2022-07-12T08:27:16.794 INFO:root:teuthology version: 1.1.1.dev381+g10062088 2022-07-12T08:27:16.798 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138 --description fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} -- /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/orig.config.yaml 2022-07-12T08:27:16.882 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-12T08:27:16.970 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138 branch: octopus-release description: fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} email: ceph-qa@ceph.io first_in_suite: false job_id: '6925138' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: octopus-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: debug ms: 1 debug osd: 20 osd objectstore: filestore osd op complaint time: 180 osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - POOL_APP_NOT_ENABLED - has not responded to cap revoke by MDS for over - MDS_CLIENT_LATE_RELEASE - responding to mclientcaps sha1: 3497be4c2159a20e7a0f23fb5e2730060a4424c0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 3497be4c2159a20e7a0f23fb5e2730060a4424c0 workunit: branch: octopus-release sha1: 297f944c54c8d8b03d4faee671f8bb06163c8262 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 - - client.2 sha1: 3497be4c2159a20e7a0f23fb5e2730060a4424c0 sleep_before_teardown: 0 suite: fs suite_branch: octopus-release suite_path: /home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 297f944c54c8d8b03d4faee671f8bb06163c8262 targets: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcA6TQiN3zR/I+ltfH7Nxg/LUprQW5CqYuCWCsUmo9hGE0XK9JhvGjPwBdpRtiHgS5zPzGCEqMTSEG1rF4IyfwCGDag12rFCVFS1/f3ZnVxIO0Pfix5Y7ZswXbE7Hht0ucOBTuPCWRgNTuYENlRPN4eM2VkqMseBkdhPdLt3pUvSZvyDF9AtcVnRvk7Apy0/gy/B6Sy12svpD8h4OZKT9IlNAMOk219Obt6ef7AgcP6UCG9NSpIczcJ9jT2X/qbfX95o5cErKVKok2Y2cTAH2vUwxiQVBMtw074iomArFc2WQUgRUwAEatg185Vn7ExcVTTNRa6CyYjdIw3Yl4VmTUV2xsfb7aFXpfP/dUc9dNYWTrZdnBfluWWtFRNQQGzpEWYYvTx5uR9Mb1T3L+HhDmPZU1+SCvnwlzjBmEBK/2NHzvfrx+1SDWGNTYypGIhWAavp7fQgW7Jz6AQQdgIMoZ3wC5YHbrfwFRZYievnzqZdEEACpRS5QCn723AtzjNTM= smithi111.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClZsT/xqX5iJXlxLK1aiwPToIUof+I7P9TZhSCfSZBmvcu1XFlnacotQnYynA3cqeKrfYgiC+JWAWzRclNsAyklBcgQ5nb6g5cpTfXXZGHpWjVYyQas7ee8J+ia9uZiWMRpoT4jQVF/iaru6uVuWe6owr25I8GKLs+dTihjj4o53huk8Sewq371Ita0J2KTXGbKCXQ6LpsG6QQzV8mKNsyHP5HTu9BjKFudZdQcNFiJHePDrl1ERrO9uJM3cll1Bv9rvA5EojiN6GnAZDEHwPUE0pb6D79dgSKk+4Syb0nPfZd3NnCgaPLF+7Sg9KjWRcqMrb49NOjPWHec9Fp1Gy6myfOBgl9NdeXmx9ZGGfVLMHqmVtiX2S34Ggg7ikximWG28CoLcGHIQeGXPIZfMnojjE7fZygvEt89UIqOgQoFVmBcD1PW09CXeSq5M54Oxh/3AokY74V0rTGxpnGuxolzE3kvkk9YnK4yyk1xU92LF6+LDrYL7uuuE0bidFN8Ec= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtVHtPFaqwQgZWMH5G5ZAhsegh+TQt0aqtK0u0xBJRy2azk1e+RgirtD3cOePJUHQ8qvZsrQpjqSD66LPMMg1QJ78nQggGT3QTXUJ7eDp9tTGvEIN4hDDhoVcSBQMLf5poh9N/DWcnbKlCZfWu8MJLxozmsLnu8u2AtwfwE2KKiu2drSCAzwuEBH24M5jOaXI2TkDso2JXarWWrH/wRTGyYPmjssAd4Ne9OgRwpOm9tUgCVds9bbt3W+XlD3X/3wlPJ8x3anRh+ySG0WIhgX5vrWY4BIM/BPBySo3z+m3fxjBJSzF0SkqNCfZCGYECNRXzayOFw+wJJRwnwvhq7oqEdzuMFqt5FDF9ZV7iGdq2fHy01MhciJr2FlsEbcVgc/LkDhUZj5iM5NJC4kxw0KFnh8NFy0xRhKrj1fAKZTo3UWc97ccZZZsKI3MrkwpuQDsE8OJpw87XVKvSTuOYzywqybchoOZmsFjUIrNMFN83JrU2etPdsgfk9GKRD4PGbH0= smithi130.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsm5mhOa0dBhV447y5mEll4JzcGa8Fq1/xsZv2bNlHY6g1vbLr9FLaBYFUqKqQaCylM/b6K5vMwfHmjNZG002uikir2nggACl8YRv5AGyhStEW6mrEBhmu/0vp/ahlNpxYiOwF1Qrm3suqIhPUP4oZmTRen0uJ9a72BbMeCebBKaPIJbvyhQg53FjSPzTkzc2zxiRBLXg31mg09lb6KUtlXc3YkW4aH7GLjoQEgovGgxNrLFfn3XvQQaWPi1cI8M42wOF6gEUfV1Ao+4+nMdOWxxtcAw1uHACTG/7hesXaLVlFX95CHcRmFdqqtwr2IkMdWwNNuj3djDzXE/jPB0aQFMRhGXIMAYV1cdtL9XSWAt9PZw6pDNO2XsIyaveuf/Dq04bD5lS5jpAm3iNXpyFxnJW7eEVAZJGzk0zRgrMQc7C9hQf07ePgAfC8j5jaUL06Ip3CKsISiBZ31rOGefGvcaGvWVXp1AxCKRrC0JlZg0YIwzFYrl0J+Y3phCHK2hU= smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnGrqG4QeVKOS3Hk30msD9taDO7J+ShxeQOBzSQB2PdOymmRpBAG3l2s2OqkHZuYpBulDWEOqtk8EkAtb7iYA2WUPVU4y5/Ksy47tYbl4ueb2NEoVqqIBe4y5L1560pSyDV9QTYpJ2affmkXieo9R2bfxxI2Rv35XAv6Los9VwOWQqdOSG3M8PDzs6zNNCTJb/GoXDy4mwS0Fi4ZWz9+xGLCu6bIjlvJaCtT9LtIbLfvlRGw85dFb0eMg0HIY7d9UFS+sa/qqoEfw//mteITTmoEoqbaLBc+xWpp03GoyYUZL0zf9/4oP8oLZ3VHCz/YMekARxIVPHpTTT5caHsa1Tv+N2E/nPE54P8Ee727QqgR7EeVMCDRgJ8QIUbPTe7c4cRoLMw++3XNbaff+m/bbLTHno+YrFxZ8Of6Pqzvh+K7IGi+OXOLlo2B/D0whq4IC2H0htewEuDPMvSA+K/3XEI70xhS/6pwLqcF+OVpkZUIaDyXjjZ9sTc2G1pJxR0kM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_misc teuthology_branch: main teuthology_sha1: 10062088f503b43eff3624326bda825b23438f9b timestamp: 2022-07-11_23:41:40 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-12T08:27:16.971 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa; will attempt to use it 2022-07-12T08:27:16.972 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks 2022-07-12T08:27:16.972 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-12T08:27:16.975 INFO:teuthology.task.internal:Checking packages... 2022-07-12T08:27:17.002 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3497be4c2159a20e7a0f23fb5e2730060a4424c0' 2022-07-12T08:27:17.002 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-12T08:27:17.003 INFO:teuthology.packaging:ref: None 2022-07-12T08:27:17.003 INFO:teuthology.packaging:tag: None 2022-07-12T08:27:17.003 INFO:teuthology.packaging:branch: octopus-release 2022-07-12T08:27:17.004 INFO:teuthology.packaging:sha1: 3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:27:17.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=octopus-release 2022-07-12T08:27:17.183 INFO:teuthology.task.internal:Found packages for ceph version 15.2.16-324-g3497be4c-1focal 2022-07-12T08:27:17.184 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-12T08:27:17.193 INFO:teuthology.task.internal:no buildpackages task found 2022-07-12T08:27:17.194 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-12T08:27:17.217 INFO:teuthology.task.internal:Saving configuration 2022-07-12T08:27:17.245 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-12T08:27:17.259 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-12T08:27:17.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-12 08:21:18.857701', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJBtifbM80wNd4DoqLeAvQmh2XF0Bf5C5qTyML36Jbrq'} 2022-07-12T08:27:17.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-12 08:21:18.855389', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIE1huyIpB9lE3RxpoGi9HJN8XlG4LCuqmiu7fx34ETvm'} 2022-07-12T08:27:17.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-12 08:21:18.860010', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINxpAG8AE2lRScgFM8DiyhJDgMWCkhZ5zYxdZpe4/d9Q'} 2022-07-12T08:27:17.369 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-12 08:21:18.861131', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN2kmHknSwVkpKzY0nKJdRY91GYDVjR9+ZjzE9Ra31ue'} 2022-07-12T08:27:17.393 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-12 08:21:18.858830', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMtIOYjp9t7ev7pgxZFcl8rLz0wbHg7XViu7V3lfR1z8'} 2022-07-12T08:27:17.394 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-12T08:27:17.413 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-07-12T08:27:17.414 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-07-12T08:27:17.414 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['client.0'] 2022-07-12T08:27:17.415 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['client.1'] 2022-07-12T08:27:17.415 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['client.2'] 2022-07-12T08:27:17.416 INFO:teuthology.run_tasks:Running task console_log... 2022-07-12T08:27:17.686 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6de1606a60>, signals=[15]) 2022-07-12T08:27:17.688 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-12T08:27:17.697 INFO:teuthology.task.internal:Opening connections... 2022-07-12T08:27:17.699 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T08:27:17.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:27:17.838 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T08:27:17.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:27:17.940 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2022-07-12T08:27:17.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:27:18.064 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2022-07-12T08:27:18.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:27:18.166 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2022-07-12T08:27:18.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:27:18.291 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-12T08:27:18.301 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2022-07-12T08:27:18.310 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2022-07-12T08:27:18.311 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2022-07-12T08:27:18.358 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Ubuntu" 2022-07-12T08:27:18.359 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-12T08:27:18.360 INFO:teuthology.orchestra.run.smithi061.stdout:ID=ubuntu 2022-07-12T08:27:18.360 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE=debian 2022-07-12T08:27:18.360 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-12T08:27:18.360 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="20.04" 2022-07-12T08:27:18.361 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-12T08:27:18.361 INFO:teuthology.orchestra.run.smithi061.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-12T08:27:18.361 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-12T08:27:18.361 INFO:teuthology.orchestra.run.smithi061.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-12T08:27:18.362 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_CODENAME=focal 2022-07-12T08:27:18.362 INFO:teuthology.orchestra.run.smithi061.stdout:UBUNTU_CODENAME=focal 2022-07-12T08:27:18.364 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2022-07-12T08:27:18.396 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2022-07-12T08:27:18.406 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2022-07-12T08:27:18.406 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2022-07-12T08:27:18.467 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="Ubuntu" 2022-07-12T08:27:18.467 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-12T08:27:18.468 INFO:teuthology.orchestra.run.smithi111.stdout:ID=ubuntu 2022-07-12T08:27:18.468 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE=debian 2022-07-12T08:27:18.469 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-12T08:27:18.469 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="20.04" 2022-07-12T08:27:18.469 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-12T08:27:18.470 INFO:teuthology.orchestra.run.smithi111.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-12T08:27:18.471 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-12T08:27:18.471 INFO:teuthology.orchestra.run.smithi111.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-12T08:27:18.472 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_CODENAME=focal 2022-07-12T08:27:18.472 INFO:teuthology.orchestra.run.smithi111.stdout:UBUNTU_CODENAME=focal 2022-07-12T08:27:18.473 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2022-07-12T08:27:18.512 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2022-07-12T08:27:18.523 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2022-07-12T08:27:18.524 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2022-07-12T08:27:18.538 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2022-07-12T08:27:18.539 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-12T08:27:18.540 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2022-07-12T08:27:18.540 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2022-07-12T08:27:18.540 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-12T08:27:18.541 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="20.04" 2022-07-12T08:27:18.541 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-12T08:27:18.541 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-12T08:27:18.542 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-12T08:27:18.542 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-12T08:27:18.543 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=focal 2022-07-12T08:27:18.543 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=focal 2022-07-12T08:27:18.544 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2022-07-12T08:27:18.570 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2022-07-12T08:27:18.578 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2022-07-12T08:27:18.579 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2022-07-12T08:27:18.590 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="Ubuntu" 2022-07-12T08:27:18.591 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-12T08:27:18.591 INFO:teuthology.orchestra.run.smithi130.stdout:ID=ubuntu 2022-07-12T08:27:18.591 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE=debian 2022-07-12T08:27:18.592 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-12T08:27:18.592 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="20.04" 2022-07-12T08:27:18.592 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-12T08:27:18.592 INFO:teuthology.orchestra.run.smithi130.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-12T08:27:18.593 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-12T08:27:18.593 INFO:teuthology.orchestra.run.smithi130.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-12T08:27:18.593 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_CODENAME=focal 2022-07-12T08:27:18.593 INFO:teuthology.orchestra.run.smithi130.stdout:UBUNTU_CODENAME=focal 2022-07-12T08:27:18.594 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2022-07-12T08:27:18.615 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2022-07-12T08:27:18.625 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2022-07-12T08:27:18.626 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2022-07-12T08:27:18.683 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Ubuntu" 2022-07-12T08:27:18.684 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-12T08:27:18.685 INFO:teuthology.orchestra.run.smithi137.stdout:ID=ubuntu 2022-07-12T08:27:18.685 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE=debian 2022-07-12T08:27:18.686 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-12T08:27:18.686 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="20.04" 2022-07-12T08:27:18.695 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-12T08:27:18.696 INFO:teuthology.orchestra.run.smithi137.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-12T08:27:18.697 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-12T08:27:18.698 INFO:teuthology.orchestra.run.smithi137.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-12T08:27:18.699 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_CODENAME=focal 2022-07-12T08:27:18.699 INFO:teuthology.orchestra.run.smithi137.stdout:UBUNTU_CODENAME=focal 2022-07-12T08:27:18.700 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2022-07-12T08:27:18.742 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-12T08:27:18.777 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-12T08:27:18.824 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-12T08:27:18.825 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2022-07-12T08:27:18.830 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2022-07-12T08:27:18.836 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2022-07-12T08:27:18.846 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2022-07-12T08:27:18.853 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2022-07-12T08:27:18.865 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-12T08:27:18.881 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-12T08:27:18.882 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2022-07-12T08:27:18.889 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2022-07-12T08:27:18.909 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2022-07-12T08:27:18.923 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2022-07-12T08:27:18.927 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2022-07-12T08:27:18.946 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-12T08:27:19.136 INFO:teuthology.run_tasks:Running task kernel... 2022-07-12T08:27:19.177 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-07-12T08:27:19.178 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-07-12T08:27:19.179 DEBUG:teuthology.orchestra.run.smithi061:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-12T08:27:19.181 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-12T08:27:19.182 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-12T08:27:19.184 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-12T08:27:19.185 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-12T08:27:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:27:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:27:19.201 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2022-07-12T08:27:19.203 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2022-07-12T08:27:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:27:19.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:27:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:27:19.213 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-07-12T08:27:19.214 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2022-07-12T08:27:19.214 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2022-07-12T08:27:19.218 INFO:teuthology.orchestra.run.smithi130.stdout:5.4.0-121-generic 2022-07-12T08:27:19.218 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.4.0-121-generic 2022-07-12T08:27:19.219 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get clean 2022-07-12T08:27:19.222 INFO:teuthology.orchestra.run.smithi121.stdout:5.4.0-121-generic 2022-07-12T08:27:19.223 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.4.0-121-generic 2022-07-12T08:27:19.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-07-12T08:27:19.253 INFO:teuthology.orchestra.run.smithi137.stdout:5.4.0-121-generic 2022-07-12T08:27:19.254 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.4.0-121-generic 2022-07-12T08:27:19.254 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2022-07-12T08:27:19.258 INFO:teuthology.orchestra.run.smithi061.stdout:5.4.0-121-generic 2022-07-12T08:27:19.258 INFO:teuthology.orchestra.run.smithi111.stdout:5.4.0-121-generic 2022-07-12T08:27:19.259 INFO:teuthology.task.kernel:Running kernel on smithi061: 5.4.0-121-generic 2022-07-12T08:27:19.259 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2022-07-12T08:27:19.260 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.4.0-121-generic 2022-07-12T08:27:19.260 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get clean 2022-07-12T08:27:19.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2022-07-12T08:27:19.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-07-12T08:27:19.395 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get update 2022-07-12T08:27:19.398 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2022-07-12T08:27:19.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2022-07-12T08:27:19.517 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-12T08:27:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:27:19.545 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-12T08:27:19.547 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-12T08:27:19.577 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:27:19.594 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:27:19.595 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-12T08:27:19.595 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-12T08:27:19.595 INFO:teuthology.orchestra.run.smithi130.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-12T08:27:19.605 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:27:19.615 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-12T08:27:19.639 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-12T08:27:19.647 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:27:19.664 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-12T08:27:19.674 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-12T08:27:19.677 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-12T08:27:19.708 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-12T08:27:19.708 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-12T08:27:19.727 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-12T08:27:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [464 kB] 2022-07-12T08:27:19.921 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [464 kB] 2022-07-12T08:27:19.955 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,593 kB] 2022-07-12T08:27:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [691 kB] 2022-07-12T08:27:19.974 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,593 kB] 2022-07-12T08:27:19.990 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [691 kB] 2022-07-12T08:27:20.007 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [271 kB] 2022-07-12T08:27:20.013 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.7 kB] 2022-07-12T08:27:20.013 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [555 kB] 2022-07-12T08:27:20.026 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-12T08:27:20.027 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,593 kB] 2022-07-12T08:27:20.046 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-12T08:27:20.047 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-12T08:27:20.066 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [691 kB] 2022-07-12T08:27:20.074 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-12T08:27:20.080 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [271 kB] 2022-07-12T08:27:20.087 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.7 kB] 2022-07-12T08:27:20.088 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [555 kB] 2022-07-12T08:27:20.095 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,964 kB] 2022-07-12T08:27:20.105 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-12T08:27:20.120 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-12T08:27:20.122 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-12T08:27:20.124 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,964 kB] 2022-07-12T08:27:20.133 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,964 kB] 2022-07-12T08:27:20.166 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [464 kB] 2022-07-12T08:27:20.174 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [354 kB] 2022-07-12T08:27:20.178 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [271 kB] 2022-07-12T08:27:20.182 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-12T08:27:20.183 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,155 kB] 2022-07-12T08:27:20.185 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.7 kB] 2022-07-12T08:27:20.186 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-12T08:27:20.187 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,964 kB] 2022-07-12T08:27:20.201 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [555 kB] 2022-07-12T08:27:20.206 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [354 kB] 2022-07-12T08:27:20.210 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-12T08:27:20.210 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,155 kB] 2022-07-12T08:27:20.212 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,964 kB] 2022-07-12T08:27:20.226 INFO:teuthology.orchestra.run.smithi130.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [165 kB] 2022-07-12T08:27:20.229 INFO:teuthology.orchestra.run.smithi130.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-12T08:27:20.236 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-12T08:27:20.240 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-12T08:27:20.247 INFO:teuthology.orchestra.run.smithi130.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-12T08:27:20.263 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [165 kB] 2022-07-12T08:27:20.264 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-12T08:27:20.290 INFO:teuthology.orchestra.run.smithi130.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,593 kB] 2022-07-12T08:27:20.300 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,593 kB] 2022-07-12T08:27:20.324 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-12T08:27:20.326 INFO:teuthology.orchestra.run.smithi130.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-12T08:27:20.327 INFO:teuthology.orchestra.run.smithi130.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-12T08:27:20.327 INFO:teuthology.orchestra.run.smithi130.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-07-12T08:27:20.328 INFO:teuthology.orchestra.run.smithi130.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2022-07-12T08:27:20.328 INFO:teuthology.orchestra.run.smithi130.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-07-12T08:27:20.329 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-12T08:27:20.330 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-12T08:27:20.331 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [354 kB] 2022-07-12T08:27:20.332 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2022-07-12T08:27:20.332 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-07-12T08:27:20.332 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-07-12T08:27:20.334 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-12T08:27:20.335 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,155 kB] 2022-07-12T08:27:20.346 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [165 kB] 2022-07-12T08:27:20.348 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-12T08:27:20.363 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-12T08:27:20.368 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [691 kB] 2022-07-12T08:27:20.387 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-12T08:27:20.393 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-12T08:27:20.393 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-07-12T08:27:20.394 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2022-07-12T08:27:20.402 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [354 kB] 2022-07-12T08:27:20.403 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-07-12T08:27:20.418 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-12T08:27:20.418 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,155 kB] 2022-07-12T08:27:20.454 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [165 kB] 2022-07-12T08:27:20.455 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-12T08:27:20.464 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-12T08:27:20.478 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-12T08:27:20.480 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-12T08:27:20.481 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2022-07-12T08:27:20.481 INFO:teuthology.orchestra.run.smithi061.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-07-12T08:27:20.482 INFO:teuthology.orchestra.run.smithi061.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-07-12T08:27:20.539 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [691 kB] 2022-07-12T08:27:20.574 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [354 kB] 2022-07-12T08:27:20.587 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15.6 kB] 2022-07-12T08:27:20.588 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,155 kB] 2022-07-12T08:27:20.624 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [165 kB] 2022-07-12T08:27:20.628 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-12T08:27:20.644 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-12T08:27:20.669 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-12T08:27:20.672 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-12T08:27:20.673 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2022-07-12T08:27:20.701 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [24.4 kB] 2022-07-12T08:27:20.701 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [592 B] 2022-07-12T08:27:20.707 INFO:teuthology.orchestra.run.smithi130.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [464 kB] 2022-07-12T08:27:20.716 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [464 kB] 2022-07-12T08:27:20.722 INFO:teuthology.orchestra.run.smithi130.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [271 kB] 2022-07-12T08:27:20.731 INFO:teuthology.orchestra.run.smithi130.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.7 kB] 2022-07-12T08:27:20.732 INFO:teuthology.orchestra.run.smithi130.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [555 kB] 2022-07-12T08:27:20.733 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [271 kB] 2022-07-12T08:27:20.740 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.7 kB] 2022-07-12T08:27:20.741 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [555 kB] 2022-07-12T08:27:20.750 INFO:teuthology.orchestra.run.smithi130.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-12T08:27:20.760 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-12T08:27:20.798 INFO:teuthology.orchestra.run.smithi130.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-12T08:27:20.802 INFO:teuthology.orchestra.run.smithi130.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-12T08:27:20.807 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-12T08:27:20.811 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-12T08:27:25.170 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 10.3 MB in 2s (5,894 kB/s) 2022-07-12T08:27:25.322 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 10.3 MB in 2s (5,961 kB/s) 2022-07-12T08:27:25.349 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 10.3 MB in 2s (5,839 kB/s) 2022-07-12T08:27:25.405 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 10.3 MB in 2s (5,818 kB/s) 2022-07-12T08:27:25.605 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 10.3 MB in 2s (5,607 kB/s) 2022-07-12T08:27:26.491 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:27:26.516 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:27:26.586 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:27:26.610 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:27:26.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:27:26.682 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:27:26.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:27:26.710 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:27:26.747 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:27:26.747 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2022-07-12T08:27:26.748 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2022-07-12T08:27:26.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:27:26.776 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:27:26.834 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:27:26.888 INFO:teuthology.orchestra.run.smithi130.stdout:The following additional packages will be installed: 2022-07-12T08:27:26.889 INFO:teuthology.orchestra.run.smithi130.stdout: linux-generic linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:26.889 INFO:teuthology.orchestra.run.smithi130.stdout: linux-headers-generic linux-image-5.4.0-122-generic 2022-07-12T08:27:26.889 INFO:teuthology.orchestra.run.smithi130.stdout: linux-modules-5.4.0-122-generic linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:26.890 INFO:teuthology.orchestra.run.smithi130.stdout:Suggested packages: 2022-07-12T08:27:26.890 INFO:teuthology.orchestra.run.smithi130.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-07-12T08:27:26.890 INFO:teuthology.orchestra.run.smithi130.stdout:Recommended packages: 2022-07-12T08:27:26.891 INFO:teuthology.orchestra.run.smithi130.stdout: thermald 2022-07-12T08:27:26.893 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-07-12T08:27:26.894 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-07-12T08:27:26.925 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:27:26.942 INFO:teuthology.orchestra.run.smithi130.stdout:The following NEW packages will be installed: 2022-07-12T08:27:26.942 INFO:teuthology.orchestra.run.smithi130.stdout: linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:26.942 INFO:teuthology.orchestra.run.smithi130.stdout: linux-image-5.4.0-122-generic linux-modules-5.4.0-122-generic 2022-07-12T08:27:26.943 INFO:teuthology.orchestra.run.smithi130.stdout: linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:26.943 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages will be upgraded: 2022-07-12T08:27:26.944 INFO:teuthology.orchestra.run.smithi130.stdout: linux-generic linux-headers-generic linux-image-generic 2022-07-12T08:27:26.950 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:27:26.952 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-07-12T08:27:26.953 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-07-12T08:27:27.009 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-07-12T08:27:27.010 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-07-12T08:27:27.011 INFO:teuthology.orchestra.run.smithi130.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:27:27.011 INFO:teuthology.orchestra.run.smithi130.stdout:Need to get 77.0 MB of archives. 2022-07-12T08:27:27.012 INFO:teuthology.orchestra.run.smithi130.stdout:After this operation, 380 MB of additional disk space will be used. 2022-07-12T08:27:27.012 INFO:teuthology.orchestra.run.smithi130.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-122-generic amd64 5.4.0-122.138 [15.0 MB] 2022-07-12T08:27:27.019 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:27:27.059 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2022-07-12T08:27:27.059 INFO:teuthology.orchestra.run.smithi111.stdout: linux-generic linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.060 INFO:teuthology.orchestra.run.smithi111.stdout: linux-headers-generic linux-image-5.4.0-122-generic 2022-07-12T08:27:27.061 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-5.4.0-122-generic linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.061 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2022-07-12T08:27:27.063 INFO:teuthology.orchestra.run.smithi111.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-07-12T08:27:27.063 INFO:teuthology.orchestra.run.smithi111.stdout:Recommended packages: 2022-07-12T08:27:27.064 INFO:teuthology.orchestra.run.smithi111.stdout: thermald 2022-07-12T08:27:27.113 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-07-12T08:27:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.114 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-generic linux-image-5.4.0-122-generic 2022-07-12T08:27:27.115 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-5.4.0-122-generic linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.116 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-07-12T08:27:27.116 INFO:teuthology.orchestra.run.smithi121.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-07-12T08:27:27.116 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2022-07-12T08:27:27.117 INFO:teuthology.orchestra.run.smithi121.stdout: thermald 2022-07-12T08:27:27.126 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2022-07-12T08:27:27.127 INFO:teuthology.orchestra.run.smithi111.stdout: linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.127 INFO:teuthology.orchestra.run.smithi111.stdout: linux-image-5.4.0-122-generic linux-modules-5.4.0-122-generic 2022-07-12T08:27:27.127 INFO:teuthology.orchestra.run.smithi111.stdout: linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.128 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages will be upgraded: 2022-07-12T08:27:27.128 INFO:teuthology.orchestra.run.smithi111.stdout: linux-generic linux-headers-generic linux-image-generic 2022-07-12T08:27:27.154 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2022-07-12T08:27:27.155 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.155 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-generic linux-image-5.4.0-122-generic 2022-07-12T08:27:27.155 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-5.4.0-122-generic linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.156 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2022-07-12T08:27:27.156 INFO:teuthology.orchestra.run.smithi137.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-07-12T08:27:27.157 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2022-07-12T08:27:27.157 INFO:teuthology.orchestra.run.smithi137.stdout: thermald 2022-07-12T08:27:27.166 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-07-12T08:27:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: linux-image-5.4.0-122-generic linux-modules-5.4.0-122-generic 2022-07-12T08:27:27.167 INFO:teuthology.orchestra.run.smithi121.stdout: linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.168 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-07-12T08:27:27.168 INFO:teuthology.orchestra.run.smithi121.stdout: linux-generic linux-headers-generic linux-image-generic 2022-07-12T08:27:27.184 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-12T08:27:27.185 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-12T08:27:27.192 INFO:teuthology.orchestra.run.smithi111.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:27:27.192 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 77.0 MB of archives. 2022-07-12T08:27:27.193 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 380 MB of additional disk space will be used. 2022-07-12T08:27:27.193 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-122-generic amd64 5.4.0-122.138 [15.0 MB] 2022-07-12T08:27:27.208 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2022-07-12T08:27:27.208 INFO:teuthology.orchestra.run.smithi137.stdout: linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.208 INFO:teuthology.orchestra.run.smithi137.stdout: linux-image-5.4.0-122-generic linux-modules-5.4.0-122-generic 2022-07-12T08:27:27.208 INFO:teuthology.orchestra.run.smithi137.stdout: linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.210 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2022-07-12T08:27:27.210 INFO:teuthology.orchestra.run.smithi137.stdout: linux-generic linux-headers-generic linux-image-generic 2022-07-12T08:27:27.290 INFO:teuthology.orchestra.run.smithi137.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:27:27.291 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 77.0 MB of archives. 2022-07-12T08:27:27.291 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 380 MB of additional disk space will be used. 2022-07-12T08:27:27.292 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-122-generic amd64 5.4.0-122.138 [15.0 MB] 2022-07-12T08:27:27.336 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2022-07-12T08:27:27.336 INFO:teuthology.orchestra.run.smithi061.stdout: linux-generic linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.337 INFO:teuthology.orchestra.run.smithi061.stdout: linux-headers-generic linux-image-5.4.0-122-generic 2022-07-12T08:27:27.337 INFO:teuthology.orchestra.run.smithi061.stdout: linux-modules-5.4.0-122-generic linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.337 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2022-07-12T08:27:27.338 INFO:teuthology.orchestra.run.smithi061.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2022-07-12T08:27:27.338 INFO:teuthology.orchestra.run.smithi061.stdout:Recommended packages: 2022-07-12T08:27:27.338 INFO:teuthology.orchestra.run.smithi061.stdout: thermald 2022-07-12T08:27:27.344 INFO:teuthology.orchestra.run.smithi121.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:27:27.344 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 77.0 MB of archives. 2022-07-12T08:27:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 380 MB of additional disk space will be used. 2022-07-12T08:27:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-122-generic amd64 5.4.0-122.138 [15.0 MB] 2022-07-12T08:27:27.388 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2022-07-12T08:27:27.388 INFO:teuthology.orchestra.run.smithi061.stdout: linux-headers-5.4.0-122 linux-headers-5.4.0-122-generic 2022-07-12T08:27:27.388 INFO:teuthology.orchestra.run.smithi061.stdout: linux-image-5.4.0-122-generic linux-modules-5.4.0-122-generic 2022-07-12T08:27:27.389 INFO:teuthology.orchestra.run.smithi061.stdout: linux-modules-extra-5.4.0-122-generic 2022-07-12T08:27:27.390 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2022-07-12T08:27:27.390 INFO:teuthology.orchestra.run.smithi061.stdout: linux-generic linux-headers-generic linux-image-generic 2022-07-12T08:27:27.447 INFO:teuthology.orchestra.run.smithi130.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-122-generic amd64 5.4.0-122.138 [10.5 MB] 2022-07-12T08:27:27.570 INFO:teuthology.orchestra.run.smithi061.stdout:3 upgraded, 5 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:27:27.570 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 77.0 MB of archives. 2022-07-12T08:27:27.570 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 380 MB of additional disk space will be used. 2022-07-12T08:27:27.571 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-122-generic amd64 5.4.0-122.138 [15.0 MB] 2022-07-12T08:27:27.710 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-122-generic amd64 5.4.0-122.138 [39.2 MB] 2022-07-12T08:27:27.832 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-122-generic amd64 5.4.0-122.138 [10.5 MB] 2022-07-12T08:27:28.121 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-122-generic amd64 5.4.0-122.138 [39.2 MB] 2022-07-12T08:27:28.371 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-122-generic amd64 5.4.0-122.138 [10.5 MB] 2022-07-12T08:27:28.503 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-122-generic amd64 5.4.0-122.138 [10.5 MB] 2022-07-12T08:27:28.700 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-122-generic amd64 5.4.0-122.138 [10.5 MB] 2022-07-12T08:27:28.771 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-122-generic amd64 5.4.0-122.138 [39.2 MB] 2022-07-12T08:27:28.774 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-122-generic amd64 5.4.0-122.138 [39.2 MB] 2022-07-12T08:27:28.967 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-122-generic amd64 5.4.0-122.138 [39.2 MB] 2022-07-12T08:27:29.348 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.122.123 [1,900 B] 2022-07-12T08:27:29.348 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.122.123 [2,564 B] 2022-07-12T08:27:29.349 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122 all 5.4.0-122.138 [11.0 MB] 2022-07-12T08:27:29.482 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.122.123 [1,900 B] 2022-07-12T08:27:29.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.122.123 [2,564 B] 2022-07-12T08:27:29.483 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122 all 5.4.0-122.138 [11.0 MB] 2022-07-12T08:27:29.652 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122-generic amd64 5.4.0-122.138 [1,385 kB] 2022-07-12T08:27:29.671 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.122.123 [2,436 B] 2022-07-12T08:27:29.703 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122-generic amd64 5.4.0-122.138 [1,385 kB] 2022-07-12T08:27:29.720 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.122.123 [2,436 B] 2022-07-12T08:27:29.997 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.122.123 [1,900 B] 2022-07-12T08:27:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.122.123 [2,564 B] 2022-07-12T08:27:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122 all 5.4.0-122.138 [11.0 MB] 2022-07-12T08:27:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.122.123 [1,900 B] 2022-07-12T08:27:30.044 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.122.123 [2,564 B] 2022-07-12T08:27:30.045 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122 all 5.4.0-122.138 [11.0 MB] 2022-07-12T08:27:30.091 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 77.0 MB in 3s (28.3 MB/s) 2022-07-12T08:27:30.146 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122-generic amd64 5.4.0-122.138 [1,385 kB] 2022-07-12T08:27:30.150 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 77.0 MB in 3s (29.8 MB/s) 2022-07-12T08:27:30.159 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.122.123 [2,436 B] 2022-07-12T08:27:30.187 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.122.123 [1,900 B] 2022-07-12T08:27:30.187 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.122.123 [2,564 B] 2022-07-12T08:27:30.188 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122 all 5.4.0-122.138 [11.0 MB] 2022-07-12T08:27:30.283 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-modules-5.4.0-122-generic. 2022-07-12T08:27:30.327 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-modules-5.4.0-122-generic. 2022-07-12T08:27:30.364 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122-generic amd64 5.4.0-122.138 [1,385 kB] 2022-07-12T08:27:30.383 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.122.123 [2,436 B] 2022-07-12T08:27:30.573 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 77.0 MB in 3s (26.2 MB/s) 2022-07-12T08:27:30.662 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-122-generic amd64 5.4.0-122.138 [1,385 kB] 2022-07-12T08:27:30.681 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.122.123 [2,436 B] 2022-07-12T08:27:30.733 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-5.4.0-122-generic. 2022-07-12T08:27:30.813 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 77.0 MB in 3s (24.0 MB/s) 2022-07-12T08:27:30.986 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-5.4.0-122-generic. 2022-07-12T08:27:31.128 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 77.0 MB in 3s (23.5 MB/s) 2022-07-12T08:27:31.282 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-modules-5.4.0-122-generic. 2022-07-12T08:27:31.613 INFO:teuthology.orchestra.run.smithi130.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-07-12T08:27:31.617 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../0-linux-modules-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:31.676 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-07-12T08:27:31.681 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../0-linux-modules-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:31.720 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:31.740 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:32.021 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-07-12T08:27:32.025 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../0-linux-modules-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:32.113 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:32.316 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-07-12T08:27:32.321 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../0-linux-modules-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:32.441 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:32.547 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102341 files and directories currently installed.) 2022-07-12T08:27:32.553 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../0-linux-modules-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:32.654 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:33.584 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-image-5.4.0-122-generic. 2022-07-12T08:27:33.598 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../1-linux-image-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:33.637 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-image-5.4.0-122-generic. 2022-07-12T08:27:33.651 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../1-linux-image-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:33.699 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:33.746 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:34.002 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-image-5.4.0-122-generic. 2022-07-12T08:27:34.011 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../1-linux-image-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:34.112 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:34.333 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-image-5.4.0-122-generic. 2022-07-12T08:27:34.350 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../1-linux-image-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:34.466 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:34.604 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-image-5.4.0-122-generic. 2022-07-12T08:27:34.617 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../1-linux-image-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:34.717 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:34.762 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-122-generic. 2022-07-12T08:27:34.777 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:34.820 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:34.832 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-122-generic. 2022-07-12T08:27:34.848 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:34.907 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:35.160 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-122-generic. 2022-07-12T08:27:35.173 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:35.218 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:35.602 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-122-generic. 2022-07-12T08:27:35.612 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:35.669 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:35.815 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-122-generic. 2022-07-12T08:27:35.828 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:35.865 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:39.593 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../3-linux-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:39.710 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:39.715 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../3-linux-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:39.768 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../3-linux-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:39.856 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:39.920 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:40.085 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:40.172 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:40.214 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-image-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:40.220 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:40.276 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-image-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:40.323 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-image-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:40.421 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../3-linux-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:40.548 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:40.558 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-headers-5.4.0-122. 2022-07-12T08:27:40.574 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../5-linux-headers-5.4.0-122_5.4.0-122.138_all.deb ... 2022-07-12T08:27:40.595 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.4.0-122. 2022-07-12T08:27:40.613 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../5-linux-headers-5.4.0-122_5.4.0-122.138_all.deb ... 2022-07-12T08:27:40.616 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:40.653 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:40.668 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-headers-5.4.0-122. 2022-07-12T08:27:40.684 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../5-linux-headers-5.4.0-122_5.4.0-122.138_all.deb ... 2022-07-12T08:27:40.720 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../3-linux-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:40.726 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:40.845 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:40.899 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:41.043 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-image-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:41.213 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:41.340 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-image-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:41.381 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-5.4.0-122. 2022-07-12T08:27:41.393 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../5-linux-headers-5.4.0-122_5.4.0-122.138_all.deb ... 2022-07-12T08:27:41.445 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:41.901 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-headers-5.4.0-122. 2022-07-12T08:27:41.916 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../5-linux-headers-5.4.0-122_5.4.0-122.138_all.deb ... 2022-07-12T08:27:41.968 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:45.850 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package linux-headers-5.4.0-122-generic. 2022-07-12T08:27:45.871 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../6-linux-headers-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:45.908 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:46.192 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package linux-headers-5.4.0-122-generic. 2022-07-12T08:27:46.193 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package linux-headers-5.4.0-122-generic. 2022-07-12T08:27:46.212 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../6-linux-headers-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:46.213 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../6-linux-headers-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:46.249 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:46.251 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:47.664 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package linux-headers-5.4.0-122-generic. 2022-07-12T08:27:47.684 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../6-linux-headers-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:47.722 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:47.953 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package linux-headers-5.4.0-122-generic. 2022-07-12T08:27:47.976 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../6-linux-headers-5.4.0-122-generic_5.4.0-122.138_amd64.deb ... 2022-07-12T08:27:48.023 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:48.558 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:48.711 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking linux-headers-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:48.948 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:49.040 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:49.069 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking linux-headers-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:49.140 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:49.270 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking linux-headers-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:49.588 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:50.045 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:50.259 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:50.385 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:50.520 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:50.546 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:50.668 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking linux-headers-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:50.696 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.122.123_amd64.deb ... 2022-07-12T08:27:50.793 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:50.824 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking linux-headers-generic (5.4.0.122.123) over (5.4.0.121.122) ... 2022-07-12T08:27:50.946 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:51.010 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:51.072 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:51.267 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:51.402 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:51.520 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:51.685 INFO:teuthology.orchestra.run.smithi111.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:27:51.685 INFO:teuthology.orchestra.run.smithi111.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:51.757 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:51.814 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-headers-generic (5.4.0.122.123) ... 2022-07-12T08:27:51.965 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:52.199 INFO:teuthology.orchestra.run.smithi130.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:27:52.199 INFO:teuthology.orchestra.run.smithi130.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:52.323 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-headers-generic (5.4.0.122.123) ... 2022-07-12T08:27:52.458 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:52.665 INFO:teuthology.orchestra.run.smithi137.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:27:52.666 INFO:teuthology.orchestra.run.smithi137.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:52.796 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-headers-generic (5.4.0.122.123) ... 2022-07-12T08:27:52.841 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:52.931 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-5.4.0-122 (5.4.0-122.138) ... 2022-07-12T08:27:52.940 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:52.993 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:53.058 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:53.120 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:53.209 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:53.459 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-image-generic (5.4.0.122.123) ... 2022-07-12T08:27:53.585 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up linux-generic (5.4.0.122.123) ... 2022-07-12T08:27:53.689 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-image-generic (5.4.0.122.123) ... 2022-07-12T08:27:53.728 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:53.812 INFO:teuthology.orchestra.run.smithi111.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-07-12T08:27:53.812 INFO:teuthology.orchestra.run.smithi111.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:53.836 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up linux-generic (5.4.0.122.123) ... 2022-07-12T08:27:53.971 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:54.037 INFO:teuthology.orchestra.run.smithi130.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-07-12T08:27:54.038 INFO:teuthology.orchestra.run.smithi130.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:54.132 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-image-generic (5.4.0.122.123) ... 2022-07-12T08:27:54.259 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up linux-generic (5.4.0.122.123) ... 2022-07-12T08:27:54.286 INFO:teuthology.orchestra.run.smithi061.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:27:54.286 INFO:teuthology.orchestra.run.smithi061.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:54.350 INFO:teuthology.orchestra.run.smithi121.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:27:54.351 INFO:teuthology.orchestra.run.smithi121.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:54.377 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:54.413 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-headers-generic (5.4.0.122.123) ... 2022-07-12T08:27:54.429 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-07-12T08:27:54.431 INFO:teuthology.orchestra.run.smithi137.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:54.477 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-headers-generic (5.4.0.122.123) ... 2022-07-12T08:27:54.539 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:54.597 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-modules-extra-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:55.755 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-image-generic (5.4.0.122.123) ... 2022-07-12T08:27:55.774 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-image-generic (5.4.0.122.123) ... 2022-07-12T08:27:55.906 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up linux-generic (5.4.0.122.123) ... 2022-07-12T08:27:56.032 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:56.041 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up linux-generic (5.4.0.122.123) ... 2022-07-12T08:27:56.097 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-07-12T08:27:56.097 INFO:teuthology.orchestra.run.smithi121.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:27:56.169 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for linux-image-5.4.0-122-generic (5.4.0-122.138) ... 2022-07-12T08:27:56.233 INFO:teuthology.orchestra.run.smithi061.stdout:/etc/kernel/postinst.d/initramfs-tools: 2022-07-12T08:27:56.233 INFO:teuthology.orchestra.run.smithi061.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:15.695 INFO:teuthology.orchestra.run.smithi111.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-07-12T08:28:15.696 INFO:teuthology.orchestra.run.smithi111.stdout:Sourcing file `/etc/default/grub' 2022-07-12T08:28:15.702 INFO:teuthology.orchestra.run.smithi111.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:15.711 INFO:teuthology.orchestra.run.smithi111.stdout:Generating grub configuration file ... 2022-07-12T08:28:15.891 INFO:teuthology.orchestra.run.smithi130.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-07-12T08:28:15.891 INFO:teuthology.orchestra.run.smithi130.stdout:Sourcing file `/etc/default/grub' 2022-07-12T08:28:15.908 INFO:teuthology.orchestra.run.smithi130.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:15.916 INFO:teuthology.orchestra.run.smithi130.stdout:Generating grub configuration file ... 2022-07-12T08:28:16.069 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:16.103 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:16.247 INFO:teuthology.orchestra.run.smithi130.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:16.271 INFO:teuthology.orchestra.run.smithi130.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:16.463 INFO:teuthology.orchestra.run.smithi111.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:16.475 INFO:teuthology.orchestra.run.smithi111.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:16.683 INFO:teuthology.orchestra.run.smithi130.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:16.694 INFO:teuthology.orchestra.run.smithi130.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:16.717 INFO:teuthology.orchestra.run.smithi137.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-07-12T08:28:16.717 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub' 2022-07-12T08:28:16.726 INFO:teuthology.orchestra.run.smithi137.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:16.739 INFO:teuthology.orchestra.run.smithi137.stdout:Generating grub configuration file ... 2022-07-12T08:28:17.118 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:17.152 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:17.618 INFO:teuthology.orchestra.run.smithi137.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:17.633 INFO:teuthology.orchestra.run.smithi137.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:18.555 INFO:teuthology.orchestra.run.smithi061.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-07-12T08:28:18.555 INFO:teuthology.orchestra.run.smithi061.stdout:Sourcing file `/etc/default/grub' 2022-07-12T08:28:18.614 INFO:teuthology.orchestra.run.smithi061.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:18.631 INFO:teuthology.orchestra.run.smithi061.stdout:Generating grub configuration file ... 2022-07-12T08:28:18.648 INFO:teuthology.orchestra.run.smithi121.stdout:/etc/kernel/postinst.d/zz-update-grub: 2022-07-12T08:28:18.648 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub' 2022-07-12T08:28:18.655 INFO:teuthology.orchestra.run.smithi121.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:18.665 INFO:teuthology.orchestra.run.smithi121.stdout:Generating grub configuration file ... 2022-07-12T08:28:19.001 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:19.030 INFO:teuthology.orchestra.run.smithi061.stdout:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:19.036 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:19.072 INFO:teuthology.orchestra.run.smithi061.stdout:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:19.094 INFO:teuthology.orchestra.run.smithi111.stdout:done 2022-07-12T08:28:19.357 INFO:teuthology.orchestra.run.smithi130.stdout:done 2022-07-12T08:28:19.440 DEBUG:teuthology.orchestra.run.smithi111:> dpkg -s linux-image-generic 2022-07-12T08:28:19.460 INFO:teuthology.orchestra.run.smithi121.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:19.472 INFO:teuthology.orchestra.run.smithi121.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:19.474 INFO:teuthology.orchestra.run.smithi111.stdout:Package: linux-image-generic 2022-07-12T08:28:19.475 INFO:teuthology.orchestra.run.smithi111.stdout:Status: install ok installed 2022-07-12T08:28:19.475 INFO:teuthology.orchestra.run.smithi111.stdout:Priority: optional 2022-07-12T08:28:19.476 INFO:teuthology.orchestra.run.smithi111.stdout:Section: kernel 2022-07-12T08:28:19.476 INFO:teuthology.orchestra.run.smithi111.stdout:Installed-Size: 19 2022-07-12T08:28:19.476 INFO:teuthology.orchestra.run.smithi111.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:19.476 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: amd64 2022-07-12T08:28:19.476 INFO:teuthology.orchestra.run.smithi111.stdout:Source: linux-meta 2022-07-12T08:28:19.477 INFO:teuthology.orchestra.run.smithi111.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:19.477 INFO:teuthology.orchestra.run.smithi111.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:19.477 INFO:teuthology.orchestra.run.smithi111.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:19.478 INFO:teuthology.orchestra.run.smithi111.stdout:Recommends: thermald 2022-07-12T08:28:19.478 INFO:teuthology.orchestra.run.smithi111.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:19.478 INFO:teuthology.orchestra.run.smithi111.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:19.478 INFO:teuthology.orchestra.run.smithi111.stdout: available. 2022-07-12T08:28:19.479 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-121-generic Expected: 5.4.0-122-generic 2022-07-12T08:28:19.479 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-07-12T08:28:19.481 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-07-12T08:28:19.481 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2022-07-12T08:28:19.482 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2022-07-12T08:28:19.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get clean 2022-07-12T08:28:19.570 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2022-07-12T08:28:19.588 INFO:teuthology.orchestra.run.smithi061.stdout:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:19.596 INFO:teuthology.orchestra.run.smithi061.stdout:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:19.708 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:28:19.716 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:28:19.737 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:28:19.795 DEBUG:teuthology.orchestra.run.smithi130:> dpkg -s linux-image-generic 2022-07-12T08:28:19.816 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:28:19.827 INFO:teuthology.orchestra.run.smithi130.stdout:Package: linux-image-generic 2022-07-12T08:28:19.827 INFO:teuthology.orchestra.run.smithi130.stdout:Status: install ok installed 2022-07-12T08:28:19.827 INFO:teuthology.orchestra.run.smithi130.stdout:Priority: optional 2022-07-12T08:28:19.828 INFO:teuthology.orchestra.run.smithi130.stdout:Section: kernel 2022-07-12T08:28:19.828 INFO:teuthology.orchestra.run.smithi130.stdout:Installed-Size: 19 2022-07-12T08:28:19.828 INFO:teuthology.orchestra.run.smithi130.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:19.828 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: amd64 2022-07-12T08:28:19.828 INFO:teuthology.orchestra.run.smithi130.stdout:Source: linux-meta 2022-07-12T08:28:19.829 INFO:teuthology.orchestra.run.smithi130.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:19.829 INFO:teuthology.orchestra.run.smithi130.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:19.829 INFO:teuthology.orchestra.run.smithi130.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:19.829 INFO:teuthology.orchestra.run.smithi130.stdout:Recommends: thermald 2022-07-12T08:28:19.830 INFO:teuthology.orchestra.run.smithi130.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:19.831 INFO:teuthology.orchestra.run.smithi130.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:19.831 INFO:teuthology.orchestra.run.smithi130.stdout: available. 2022-07-12T08:28:19.832 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-121-generic Expected: 5.4.0-122-generic 2022-07-12T08:28:19.832 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-07-12T08:28:19.833 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-07-12T08:28:19.833 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2022-07-12T08:28:19.834 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2022-07-12T08:28:19.834 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get clean 2022-07-12T08:28:19.927 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get update 2022-07-12T08:28:20.185 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:28:20.185 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:28:20.253 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:28:20.333 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:28:20.407 INFO:teuthology.orchestra.run.smithi137.stdout:done 2022-07-12T08:28:20.763 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2022-07-12T08:28:20.793 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2022-07-12T08:28:20.794 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2022-07-12T08:28:20.794 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2022-07-12T08:28:20.796 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2022-07-12T08:28:20.796 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 19 2022-07-12T08:28:20.796 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:20.796 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2022-07-12T08:28:20.797 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2022-07-12T08:28:20.797 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:20.797 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:20.797 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:20.798 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2022-07-12T08:28:20.798 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:20.798 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:20.798 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2022-07-12T08:28:20.799 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-121-generic Expected: 5.4.0-122-generic 2022-07-12T08:28:20.799 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-07-12T08:28:20.799 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-07-12T08:28:20.800 INFO:teuthology.task.kernel:Installing distro kernel on client.2... 2022-07-12T08:28:20.800 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2022-07-12T08:28:20.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get clean 2022-07-12T08:28:20.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2022-07-12T08:28:21.117 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:28:21.118 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:28:21.183 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:28:21.264 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:28:21.288 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:28:21.311 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:28:21.382 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:28:21.556 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-07-12T08:28:21.557 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-07-12T08:28:21.747 INFO:teuthology.orchestra.run.smithi111.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-12T08:28:21.748 INFO:teuthology.orchestra.run.smithi111.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:28:21.749 DEBUG:teuthology.orchestra.run.smithi111:> dpkg -s linux-image-generic 2022-07-12T08:28:21.770 INFO:teuthology.orchestra.run.smithi111.stdout:Package: linux-image-generic 2022-07-12T08:28:21.770 INFO:teuthology.orchestra.run.smithi111.stdout:Status: install ok installed 2022-07-12T08:28:21.771 INFO:teuthology.orchestra.run.smithi111.stdout:Priority: optional 2022-07-12T08:28:21.771 INFO:teuthology.orchestra.run.smithi111.stdout:Section: kernel 2022-07-12T08:28:21.771 INFO:teuthology.orchestra.run.smithi111.stdout:Installed-Size: 19 2022-07-12T08:28:21.772 INFO:teuthology.orchestra.run.smithi111.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:21.772 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: amd64 2022-07-12T08:28:21.772 INFO:teuthology.orchestra.run.smithi111.stdout:Source: linux-meta 2022-07-12T08:28:21.773 INFO:teuthology.orchestra.run.smithi111.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:21.773 INFO:teuthology.orchestra.run.smithi111.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:21.773 INFO:teuthology.orchestra.run.smithi111.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:21.773 INFO:teuthology.orchestra.run.smithi111.stdout:Recommends: thermald 2022-07-12T08:28:21.774 INFO:teuthology.orchestra.run.smithi111.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:21.774 INFO:teuthology.orchestra.run.smithi111.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:21.774 INFO:teuthology.orchestra.run.smithi111.stdout: available. 2022-07-12T08:28:21.775 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:28:21.781 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:28:21.783 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.NLGAbCecwf 2022-07-12T08:28:21.783 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NLGAbCecwf 2022-07-12T08:28:21.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.NLGAbCecwf 2022-07-12T08:28:21.804 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:28:21.870 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:28:21.889 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.NLGAbCecwf is 9KB 2022-07-12T08:28:21.906 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.NLGAbCecwf 2022-07-12T08:28:21.914 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:21.976 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:28:21.977 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-07-12T08:28:21.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:22.016 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-122-generic 2022-07-12T08:28:22.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo update-grub 2022-07-12T08:28:22.027 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2022-07-12T08:28:22.028 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2022-07-12T08:28:22.178 INFO:teuthology.orchestra.run.smithi121.stdout:done 2022-07-12T08:28:22.204 INFO:teuthology.orchestra.run.smithi130.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-12T08:28:22.205 INFO:teuthology.orchestra.run.smithi130.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:28:22.206 DEBUG:teuthology.orchestra.run.smithi130:> dpkg -s linux-image-generic 2022-07-12T08:28:22.227 INFO:teuthology.orchestra.run.smithi130.stdout:Package: linux-image-generic 2022-07-12T08:28:22.227 INFO:teuthology.orchestra.run.smithi130.stdout:Status: install ok installed 2022-07-12T08:28:22.228 INFO:teuthology.orchestra.run.smithi130.stdout:Priority: optional 2022-07-12T08:28:22.229 INFO:teuthology.orchestra.run.smithi130.stdout:Section: kernel 2022-07-12T08:28:22.229 INFO:teuthology.orchestra.run.smithi130.stdout:Installed-Size: 19 2022-07-12T08:28:22.229 INFO:teuthology.orchestra.run.smithi130.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:22.230 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: amd64 2022-07-12T08:28:22.230 INFO:teuthology.orchestra.run.smithi130.stdout:Source: linux-meta 2022-07-12T08:28:22.231 INFO:teuthology.orchestra.run.smithi130.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:22.231 INFO:teuthology.orchestra.run.smithi130.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:22.232 INFO:teuthology.orchestra.run.smithi130.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:22.232 INFO:teuthology.orchestra.run.smithi130.stdout:Recommends: thermald 2022-07-12T08:28:22.233 INFO:teuthology.orchestra.run.smithi130.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:22.234 INFO:teuthology.orchestra.run.smithi130.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:22.234 INFO:teuthology.orchestra.run.smithi130.stdout: available. 2022-07-12T08:28:22.235 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2022-07-12T08:28:22.242 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.025pT23fM2 2022-07-12T08:28:22.243 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub/grub.cfg /tmp/tmp.025pT23fM2 2022-07-12T08:28:22.259 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.025pT23fM2 2022-07-12T08:28:22.303 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.025pT23fM2 is 9KB 2022-07-12T08:28:22.367 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.025pT23fM2 2022-07-12T08:28:22.382 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:22.427 INFO:teuthology.orchestra.run.smithi061.stdout:done 2022-07-12T08:28:22.442 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:28:22.443 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-07-12T08:28:22.443 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:22.480 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-122-generic 2022-07-12T08:28:22.481 DEBUG:teuthology.orchestra.run.smithi130:> sudo update-grub 2022-07-12T08:28:22.574 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2022-07-12T08:28:22.602 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2022-07-12T08:28:22.603 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-07-12T08:28:22.603 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-07-12T08:28:22.603 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-07-12T08:28:22.604 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 19 2022-07-12T08:28:22.604 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:22.604 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-07-12T08:28:22.604 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2022-07-12T08:28:22.604 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:22.605 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:22.605 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:22.605 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-07-12T08:28:22.605 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:22.606 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-07-12T08:28:22.606 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-121-generic Expected: 5.4.0-122-generic 2022-07-12T08:28:22.607 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-07-12T08:28:22.607 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-07-12T08:28:22.607 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2022-07-12T08:28:22.608 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2022-07-12T08:28:22.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get clean 2022-07-12T08:28:22.681 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-07-12T08:28:22.726 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:28:22.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:28:22.809 DEBUG:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2022-07-12T08:28:22.816 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:28:22.837 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2022-07-12T08:28:22.838 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2022-07-12T08:28:22.838 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2022-07-12T08:28:22.839 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2022-07-12T08:28:22.839 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 19 2022-07-12T08:28:22.840 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:22.840 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2022-07-12T08:28:22.840 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2022-07-12T08:28:22.841 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:22.841 INFO:teuthology.orchestra.run.smithi061.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:22.842 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:22.842 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2022-07-12T08:28:22.843 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:22.843 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:22.843 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2022-07-12T08:28:22.844 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-121-generic Expected: 5.4.0-122-generic 2022-07-12T08:28:22.844 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2022-07-12T08:28:22.844 DEBUG:teuthology.task.kernel:src is distro, skipping download 2022-07-12T08:28:22.844 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2022-07-12T08:28:22.845 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi061.front.sepia.ceph.com, path=None, version=distro) 2022-07-12T08:28:22.845 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2022-07-12T08:28:22.908 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2022-07-12T08:28:22.938 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:28:22.939 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:28:22.990 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-07-12T08:28:22.991 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-07-12T08:28:23.007 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:28:23.048 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:28:23.048 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:28:23.057 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:28:23.082 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:28:23.091 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:28:23.174 INFO:teuthology.orchestra.run.smithi137.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-12T08:28:23.175 INFO:teuthology.orchestra.run.smithi137.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:28:23.176 DEBUG:teuthology.orchestra.run.smithi137:> dpkg -s linux-image-generic 2022-07-12T08:28:23.202 INFO:teuthology.orchestra.run.smithi137.stdout:Package: linux-image-generic 2022-07-12T08:28:23.202 INFO:teuthology.orchestra.run.smithi137.stdout:Status: install ok installed 2022-07-12T08:28:23.203 INFO:teuthology.orchestra.run.smithi137.stdout:Priority: optional 2022-07-12T08:28:23.203 INFO:teuthology.orchestra.run.smithi137.stdout:Section: kernel 2022-07-12T08:28:23.204 INFO:teuthology.orchestra.run.smithi137.stdout:Installed-Size: 19 2022-07-12T08:28:23.204 INFO:teuthology.orchestra.run.smithi137.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:23.205 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: amd64 2022-07-12T08:28:23.206 INFO:teuthology.orchestra.run.smithi137.stdout:Source: linux-meta 2022-07-12T08:28:23.206 INFO:teuthology.orchestra.run.smithi137.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:23.207 INFO:teuthology.orchestra.run.smithi137.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:23.207 INFO:teuthology.orchestra.run.smithi137.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:23.208 INFO:teuthology.orchestra.run.smithi137.stdout:Recommends: thermald 2022-07-12T08:28:23.208 INFO:teuthology.orchestra.run.smithi137.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:23.209 INFO:teuthology.orchestra.run.smithi137.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:23.209 INFO:teuthology.orchestra.run.smithi137.stdout: available. 2022-07-12T08:28:23.210 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2022-07-12T08:28:23.252 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.KtYyvWnVqx 2022-07-12T08:28:23.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub/grub.cfg /tmp/tmp.KtYyvWnVqx 2022-07-12T08:28:23.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.KtYyvWnVqx 2022-07-12T08:28:23.329 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub' 2022-07-12T08:28:23.331 INFO:teuthology.orchestra.run.smithi111.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:23.345 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2022-07-12T08:28:23.363 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.KtYyvWnVqx is 9KB 2022-07-12T08:28:23.378 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.KtYyvWnVqx 2022-07-12T08:28:23.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:23.445 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:28:23.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-07-12T08:28:23.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:23.479 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-122-generic 2022-07-12T08:28:23.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo update-grub 2022-07-12T08:28:23.595 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:23.607 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:23.873 INFO:teuthology.orchestra.run.smithi130.stderr:Sourcing file `/etc/default/grub' 2022-07-12T08:28:23.886 INFO:teuthology.orchestra.run.smithi130.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:23.907 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2022-07-12T08:28:23.961 INFO:teuthology.orchestra.run.smithi111.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:23.974 INFO:teuthology.orchestra.run.smithi111.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:24.160 INFO:teuthology.orchestra.run.smithi130.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:24.172 INFO:teuthology.orchestra.run.smithi130.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:24.517 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:28:24.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:28:24.562 INFO:teuthology.orchestra.run.smithi130.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:24.574 INFO:teuthology.orchestra.run.smithi130.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:24.607 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:28:24.683 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:28:24.708 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-12T08:28:24.759 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub' 2022-07-12T08:28:24.760 INFO:teuthology.orchestra.run.smithi137.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:24.765 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-07-12T08:28:24.766 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-07-12T08:28:24.777 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2022-07-12T08:28:24.781 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:28:24.938 INFO:teuthology.orchestra.run.smithi121.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-12T08:28:24.939 INFO:teuthology.orchestra.run.smithi121.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:28:24.942 DEBUG:teuthology.orchestra.run.smithi121:> dpkg -s linux-image-generic 2022-07-12T08:28:24.944 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-12T08:28:24.945 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-12T08:28:24.963 INFO:teuthology.orchestra.run.smithi121.stdout:Package: linux-image-generic 2022-07-12T08:28:24.963 INFO:teuthology.orchestra.run.smithi121.stdout:Status: install ok installed 2022-07-12T08:28:24.964 INFO:teuthology.orchestra.run.smithi121.stdout:Priority: optional 2022-07-12T08:28:24.964 INFO:teuthology.orchestra.run.smithi121.stdout:Section: kernel 2022-07-12T08:28:24.964 INFO:teuthology.orchestra.run.smithi121.stdout:Installed-Size: 19 2022-07-12T08:28:24.964 INFO:teuthology.orchestra.run.smithi121.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:24.965 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: amd64 2022-07-12T08:28:24.965 INFO:teuthology.orchestra.run.smithi121.stdout:Source: linux-meta 2022-07-12T08:28:24.965 INFO:teuthology.orchestra.run.smithi121.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:24.966 INFO:teuthology.orchestra.run.smithi121.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:24.966 INFO:teuthology.orchestra.run.smithi121.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:24.966 INFO:teuthology.orchestra.run.smithi121.stdout:Recommends: thermald 2022-07-12T08:28:24.967 INFO:teuthology.orchestra.run.smithi121.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:24.967 INFO:teuthology.orchestra.run.smithi121.stdout: available. 2022-07-12T08:28:24.967 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2022-07-12T08:28:24.972 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.dnBLloT6Dn 2022-07-12T08:28:24.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub/grub.cfg /tmp/tmp.dnBLloT6Dn 2022-07-12T08:28:25.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.dnBLloT6Dn 2022-07-12T08:28:25.030 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:25.041 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:25.068 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.dnBLloT6Dn is 9KB 2022-07-12T08:28:25.086 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.dnBLloT6Dn 2022-07-12T08:28:25.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:25.130 INFO:teuthology.orchestra.run.smithi061.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-12T08:28:25.131 INFO:teuthology.orchestra.run.smithi061.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:28:25.133 DEBUG:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2022-07-12T08:28:25.150 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:28:25.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-07-12T08:28:25.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:25.153 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2022-07-12T08:28:25.153 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2022-07-12T08:28:25.153 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2022-07-12T08:28:25.153 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2022-07-12T08:28:25.154 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 19 2022-07-12T08:28:25.154 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2022-07-12T08:28:25.154 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2022-07-12T08:28:25.155 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2022-07-12T08:28:25.155 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 5.4.0.122.123 2022-07-12T08:28:25.155 INFO:teuthology.orchestra.run.smithi061.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-12T08:28:25.156 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-12T08:28:25.156 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2022-07-12T08:28:25.156 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2022-07-12T08:28:25.156 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2022-07-12T08:28:25.156 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2022-07-12T08:28:25.157 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:28:25.164 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.sxScnOjusf 2022-07-12T08:28:25.165 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /boot/grub/grub.cfg /tmp/tmp.sxScnOjusf 2022-07-12T08:28:25.207 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-122-generic 2022-07-12T08:28:25.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo update-grub 2022-07-12T08:28:25.227 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.sxScnOjusf 2022-07-12T08:28:25.309 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.sxScnOjusf is 9KB 2022-07-12T08:28:25.370 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.sxScnOjusf 2022-07-12T08:28:25.378 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:25.442 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:28:25.443 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/grub.d/01_ceph_kernel 2022-07-12T08:28:25.443 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2022-07-12T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:25.480 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-122-generic 2022-07-12T08:28:25.480 DEBUG:teuthology.orchestra.run.smithi061:> sudo update-grub 2022-07-12T08:28:25.594 INFO:teuthology.orchestra.run.smithi111.stderr:done 2022-07-12T08:28:25.595 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2022-07-12T08:28:26.151 INFO:teuthology.orchestra.run.smithi130.stderr:done 2022-07-12T08:28:26.153 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2022-07-12T08:28:26.630 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub' 2022-07-12T08:28:26.631 INFO:teuthology.orchestra.run.smithi121.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:26.642 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2022-07-12T08:28:26.825 INFO:teuthology.orchestra.run.smithi061.stderr:Sourcing file `/etc/default/grub' 2022-07-12T08:28:26.826 INFO:teuthology.orchestra.run.smithi061.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2022-07-12T08:28:26.840 INFO:teuthology.orchestra.run.smithi061.stderr:Generating grub configuration file ... 2022-07-12T08:28:26.920 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:27.087 INFO:teuthology.orchestra.run.smithi137.stderr:done 2022-07-12T08:28:27.094 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2022-07-12T08:28:27.168 INFO:teuthology.orchestra.run.smithi061.stderr:Found linux image: /boot/vmlinuz-5.4.0-122-generic 2022-07-12T08:28:27.180 INFO:teuthology.orchestra.run.smithi061.stderr:Found initrd image: /boot/initrd.img-5.4.0-122-generic 2022-07-12T08:28:27.346 INFO:teuthology.orchestra.run.smithi121.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:27.351 INFO:teuthology.orchestra.run.smithi121.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:27.594 INFO:teuthology.orchestra.run.smithi061.stderr:Found linux image: /boot/vmlinuz-5.4.0-121-generic 2022-07-12T08:28:27.612 INFO:teuthology.orchestra.run.smithi061.stderr:Found initrd image: /boot/initrd.img-5.4.0-121-generic 2022-07-12T08:28:28.958 INFO:teuthology.orchestra.run.smithi121.stderr:done 2022-07-12T08:28:28.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2022-07-12T08:28:29.465 INFO:teuthology.orchestra.run.smithi061.stderr:done 2022-07-12T08:28:29.467 DEBUG:teuthology.orchestra.run.smithi061:> sudo shutdown -r now 2022-07-12T08:28:55.626 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2022-07-12T08:28:55.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-07-12T08:28:55.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:28:56.164 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2022-07-12T08:28:56.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2022-07-12T08:28:56.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:28:57.100 INFO:teuthology.task.kernel:Checking client client.2 for new kernel version... 2022-07-12T08:28:57.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-07-12T08:28:57.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:28:58.966 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2022-07-12T08:28:58.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-07-12T08:28:58.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:28:59.472 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2022-07-12T08:28:59.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2022-07-12T08:28:59.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:29:14.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-07-12T08:29:14.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2022-07-12T08:29:15.649 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2022-07-12T08:29:17.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2022-07-12T08:29:17.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2022-07-12T08:29:44.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-07-12T08:29:44.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:29:44.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2022-07-12T08:29:44.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:29:45.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-07-12T08:29:45.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:29:47.195 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2022-07-12T08:29:47.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-07-12T08:29:47.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:29:47.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2022-07-12T08:29:47.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2022-07-12T08:29:47.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:29:48.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2022-07-12T08:29:50.522 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2022-07-12T08:29:51.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2022-07-12T08:30:17.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-07-12T08:30:17.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:30:17.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2022-07-12T08:30:17.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:30:18.044 DEBUG:teuthology.orchestra.run.smithi130:> true 2022-07-12T08:30:18.700 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2022-07-12T08:30:18.700 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5.4.0-122-generic"... 2022-07-12T08:30:18.701 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2022-07-12T08:30:18.716 INFO:teuthology.orchestra.run.smithi130.stdout:5.4.0-122-generic 2022-07-12T08:30:18.717 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-122-generic vs 5.4.0-122-generic 2022-07-12T08:30:18.717 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-07-12T08:30:18.718 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-07-12T08:30:18.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-07-12T08:30:18.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:30:18.902 DEBUG:teuthology.orchestra.remote:No existing session 2022-07-12T08:30:19.050 DEBUG:teuthology.orchestra.run.smithi137:> true 2022-07-12T08:30:19.624 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-07-12T08:30:19.624 INFO:teuthology.task.kernel:Checking kernel version of client.2, want "5.4.0-122-generic"... 2022-07-12T08:30:19.625 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2022-07-12T08:30:19.636 INFO:teuthology.orchestra.run.smithi137.stdout:5.4.0-122-generic 2022-07-12T08:30:19.637 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-122-generic vs 5.4.0-122-generic 2022-07-12T08:30:19.637 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-07-12T08:30:19.637 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-07-12T08:30:19.719 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2022-07-12T08:30:19.720 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-12T08:30:19.799 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2022-07-12T08:30:19.819 DEBUG:teuthology.parallel:result is None 2022-07-12T08:30:20.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-07-12T08:30:20.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:30:20.692 INFO:teuthology.task.kernel:Enabling kdb on client.2... 2022-07-12T08:30:20.693 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-12T08:30:20.778 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2022-07-12T08:30:20.802 DEBUG:teuthology.parallel:result is None 2022-07-12T08:30:20.847 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-07-12T08:30:21.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2022-07-12T08:30:21.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:30:21.248 DEBUG:teuthology.orchestra.run.smithi061:> true 2022-07-12T08:30:21.411 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-07-12T08:30:21.412 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.4.0-122-generic"... 2022-07-12T08:30:21.412 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2022-07-12T08:30:21.419 INFO:teuthology.orchestra.run.smithi121.stdout:5.4.0-122-generic 2022-07-12T08:30:21.420 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-122-generic vs 5.4.0-122-generic 2022-07-12T08:30:21.421 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-07-12T08:30:21.421 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-07-12T08:30:21.854 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2022-07-12T08:30:21.854 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-122-generic"... 2022-07-12T08:30:21.855 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2022-07-12T08:30:21.862 INFO:teuthology.orchestra.run.smithi061.stdout:5.4.0-122-generic 2022-07-12T08:30:21.862 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-122-generic vs 5.4.0-122-generic 2022-07-12T08:30:21.863 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-07-12T08:30:21.863 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-07-12T08:30:22.423 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2022-07-12T08:30:22.424 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-12T08:30:22.514 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2022-07-12T08:30:22.536 DEBUG:teuthology.parallel:result is None 2022-07-12T08:30:22.865 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-07-12T08:30:22.866 DEBUG:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-12T08:30:22.937 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2022-07-12T08:30:22.953 DEBUG:teuthology.parallel:result is None 2022-07-12T08:30:48.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-07-12T08:30:48.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:30:49.056 DEBUG:teuthology.orchestra.run.smithi111:> true 2022-07-12T08:30:49.683 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-07-12T08:30:49.684 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-122-generic"... 2022-07-12T08:30:49.684 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2022-07-12T08:30:49.691 INFO:teuthology.orchestra.run.smithi111.stdout:5.4.0-122-generic 2022-07-12T08:30:49.692 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-122-generic vs 5.4.0-122-generic 2022-07-12T08:30:49.692 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2022-07-12T08:30:49.692 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2022-07-12T08:30:50.698 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2022-07-12T08:30:50.698 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-12T08:30:50.786 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2022-07-12T08:30:50.807 DEBUG:teuthology.parallel:result is None 2022-07-12T08:30:50.808 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-12T08:30:50.821 INFO:teuthology.task.internal:Creating test directory... 2022-07-12T08:30:50.822 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-12T08:30:50.833 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-12T08:30:50.837 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-12T08:30:50.840 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-12T08:30:50.843 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-12T08:30:50.851 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-12T08:30:50.862 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-12T08:30:50.880 INFO:teuthology.task.internal:Creating archive directory... 2022-07-12T08:30:50.881 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-12T08:30:50.885 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-12T08:30:50.887 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-12T08:30:50.890 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-12T08:30:50.893 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-12T08:30:50.923 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-12T08:30:50.934 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-12T08:30:50.936 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-12T08:30:50.945 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-12T08:30:50.949 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-12T08:30:50.956 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-12T08:30:50.959 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-12T08:30:50.964 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.971 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.975 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.980 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.983 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.985 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.992 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:50.994 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:51.008 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-12T08:30:51.012 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-12T08:30:51.020 INFO:teuthology.task.internal:Configuring sudo... 2022-07-12T08:30:51.021 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-12T08:30:51.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-12T08:30:51.031 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-12T08:30:51.052 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-12T08:30:51.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-12T08:30:51.076 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-12T08:30:51.098 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-12T08:30:51.100 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-12T08:30:51.104 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-12T08:30:51.109 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-12T08:30:51.113 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-12T08:30:51.117 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-12T08:30:51.125 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-12T08:30:51.134 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-12T08:30:51.185 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:30:51.185 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-12T08:30:51.209 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-12T08:30:51.226 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-12T08:30:51.276 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:30:51.277 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-12T08:30:51.300 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-12T08:30:51.309 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-12T08:30:51.317 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:30:51.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-12T08:30:51.338 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-12T08:30:51.346 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-12T08:30:51.353 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:30:51.353 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-12T08:30:51.373 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-12T08:30:51.381 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-12T08:30:51.439 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:30:51.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-12T08:30:51.472 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2022-07-12T08:30:51.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2022-07-12T08:30:51.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2022-07-12T08:30:51.486 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2022-07-12T08:30:51.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2022-07-12T08:30:51.569 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-12T08:30:51.588 INFO:teuthology.task.internal:Starting timer... 2022-07-12T08:30:51.588 INFO:teuthology.run_tasks:Running task pcp... 2022-07-12T08:30:51.616 INFO:teuthology.run_tasks:Running task selinux... 2022-07-12T08:30:51.640 DEBUG:teuthology.task.selinux:Excluding smithi061: OS 'ubuntu' does not support SELinux 2022-07-12T08:30:51.641 DEBUG:teuthology.task.selinux:Excluding smithi111: OS 'ubuntu' does not support SELinux 2022-07-12T08:30:51.641 DEBUG:teuthology.task.selinux:Excluding smithi121: OS 'ubuntu' does not support SELinux 2022-07-12T08:30:51.643 DEBUG:teuthology.task.selinux:Excluding smithi130: OS 'ubuntu' does not support SELinux 2022-07-12T08:30:51.643 DEBUG:teuthology.task.selinux:Excluding smithi137: OS 'ubuntu' does not support SELinux 2022-07-12T08:30:51.644 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-12T08:30:51.644 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-07-12T08:30:51.644 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-12T08:30:51.653 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-12T08:30:51.705 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-07-12T08:30:51.717 INFO:teuthology.repo_utils:Fetching main from origin 2022-07-12T08:30:51.765 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-12T08:30:51.803 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-12T08:30:51.808 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com,smithi111.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi130.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-12T08:38:01.659 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi130.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2022-07-12T08:38:01.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2022-07-12T08:38:01.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:38:01.875 DEBUG:teuthology.orchestra.run.smithi061:> true 2022-07-12T08:38:02.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-07-12T08:38:02.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:38:02.152 DEBUG:teuthology.orchestra.run.smithi111:> true 2022-07-12T08:38:02.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2022-07-12T08:38:02.276 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:38:02.428 DEBUG:teuthology.orchestra.run.smithi121:> true 2022-07-12T08:38:02.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2022-07-12T08:38:02.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:38:02.745 DEBUG:teuthology.orchestra.run.smithi130:> true 2022-07-12T08:38:02.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-07-12T08:38:02.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-12T08:38:03.044 DEBUG:teuthology.orchestra.run.smithi137:> true 2022-07-12T08:38:03.170 INFO:teuthology.run_tasks:Running task clock... 2022-07-12T08:38:03.205 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-07-12T08:38:03.206 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-12T08:38:03.213 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T08:38:03.235 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-12T08:38:03.236 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T08:38:03.240 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-12T08:38:03.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T08:38:03.246 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-12T08:38:03.247 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T08:38:03.249 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-12T08:38:03.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T08:38:03.266 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-12T08:38:03.267 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Command line: ntpd -gq 2022-07-12T08:38:03.268 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: proto: precision = 0.126 usec (-23) 2022-07-12T08:38:03.268 INFO:teuthology.orchestra.run.smithi061.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.269 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.269 INFO:teuthology.orchestra.run.smithi061.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.271 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.272 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listen and drop on 0 v6wildcard [::]:123 2022-07-12T08:38:03.272 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-12T08:38:03.272 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listen normally on 2 lo 127.0.0.1:123 2022-07-12T08:38:03.273 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listen normally on 3 enp3s0f1 172.21.15.61:123 2022-07-12T08:38:03.273 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listen normally on 4 lo [::1]:123 2022-07-12T08:38:03.273 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8b9%5]:123 2022-07-12T08:38:03.274 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:03 ntpd[16826]: Listening on routing socket on fd #22 for interface updates 2022-07-12T08:38:03.274 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-12T08:38:03.275 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Command line: ntpd -gq 2022-07-12T08:38:03.275 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: proto: precision = 0.188 usec (-22) 2022-07-12T08:38:03.275 INFO:teuthology.orchestra.run.smithi111.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.276 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.276 INFO:teuthology.orchestra.run.smithi111.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.277 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.278 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listen and drop on 0 v6wildcard [::]:123 2022-07-12T08:38:03.278 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-12T08:38:03.278 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listen normally on 2 lo 127.0.0.1:123 2022-07-12T08:38:03.278 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listen normally on 3 enp3s0f1 172.21.15.111:123 2022-07-12T08:38:03.279 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listen normally on 4 lo [::1]:123 2022-07-12T08:38:03.281 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-12T08:38:03.281 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Command line: ntpd -gq 2022-07-12T08:38:03.282 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: proto: precision = 0.068 usec (-24) 2022-07-12T08:38:03.283 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-12T08:38:03.283 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Command line: ntpd -gq 2022-07-12T08:38:03.283 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: proto: precision = 0.120 usec (-23) 2022-07-12T08:38:03.284 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc2f%5]:123 2022-07-12T08:38:03.284 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:03 ntpd[16775]: Listening on routing socket on fd #22 for interface updates 2022-07-12T08:38:03.285 INFO:teuthology.orchestra.run.smithi130.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.285 INFO:teuthology.orchestra.run.smithi121.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.286 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.286 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.287 INFO:teuthology.orchestra.run.smithi130.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.287 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.287 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.288 INFO:teuthology.orchestra.run.smithi121.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.290 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-12T08:38:03.291 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Command line: ntpd -gq 2022-07-12T08:38:03.291 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: proto: precision = 0.102 usec (-23) 2022-07-12T08:38:03.291 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listen and drop on 0 v6wildcard [::]:123 2022-07-12T08:38:03.292 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-12T08:38:03.292 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listen normally on 2 lo 127.0.0.1:123 2022-07-12T08:38:03.292 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listen normally on 3 enp3s0f1 172.21.15.130:123 2022-07-12T08:38:03.293 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listen normally on 4 lo [::1]:123 2022-07-12T08:38:03.293 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f37%5]:123 2022-07-12T08:38:03.293 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:03 ntpd[16820]: Listening on routing socket on fd #22 for interface updates 2022-07-12T08:38:03.294 INFO:teuthology.orchestra.run.smithi137.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.294 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listen and drop on 0 v6wildcard [::]:123 2022-07-12T08:38:03.296 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-12T08:38:03.296 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listen normally on 2 lo 127.0.0.1:123 2022-07-12T08:38:03.297 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listen normally on 3 enp3s0f1 172.21.15.121:123 2022-07-12T08:38:03.297 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listen normally on 4 lo [::1]:123 2022-07-12T08:38:03.298 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cba3%5]:123 2022-07-12T08:38:03.298 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:03 ntpd[16818]: Listening on routing socket on fd #22 for interface updates 2022-07-12T08:38:03.299 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-12T08:38:03.300 INFO:teuthology.orchestra.run.smithi137.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.304 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: restrict ::: KOD does nothing without LIMITED. 2022-07-12T08:38:03.305 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listen and drop on 0 v6wildcard [::]:123 2022-07-12T08:38:03.305 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-12T08:38:03.306 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listen normally on 2 lo 127.0.0.1:123 2022-07-12T08:38:03.306 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listen normally on 3 enp3s0f1 172.21.15.137:123 2022-07-12T08:38:03.307 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listen normally on 4 lo [::1]:123 2022-07-12T08:38:03.307 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8137%5]:123 2022-07-12T08:38:03.308 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:03 ntpd[16846]: Listening on routing socket on fd #22 for interface updates 2022-07-12T08:38:04.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:04 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:04 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.277 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.277 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.277 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.277 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:04.278 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:04.279 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.279 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.279 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:04.279 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:04 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:04.282 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:04 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.282 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:04 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.282 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:04 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.283 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:04 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.283 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:04 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:04.283 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:04 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:04.284 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:04 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.284 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:04 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.284 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:04 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.285 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:04 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.285 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:04 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:04.285 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:04 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:04.287 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:04 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.288 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:04 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.288 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:04 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:04.289 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:04 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:04.289 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:04 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:04.290 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:04 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:05.275 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:05 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:05.275 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:05 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:05.294 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:05 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:05.294 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:05 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:05.295 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:05 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:05.295 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:05 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:05.296 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:05 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:05.296 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:05 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.273 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:06 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:06 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:06 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.275 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:06 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.275 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:06 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.276 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:06 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.284 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.284 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.284 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.285 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.285 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.285 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.286 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.286 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.286 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.287 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.287 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.288 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.288 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.288 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:06 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.293 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:06 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.295 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:06 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.295 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:06 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.296 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:06 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.296 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:06 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.297 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:06 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.298 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.299 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.299 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.301 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:06 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.302 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.302 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.303 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.303 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.303 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.304 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.304 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.305 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:06.305 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:06.306 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:06.306 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:06.307 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:06 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:07.271 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:07 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:07 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.272 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:07 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.273 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:07 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:07 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:07.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:07 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:07.283 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:07 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.283 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:07 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.285 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:07 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.286 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:07 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.286 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:07 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.287 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:07 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:07 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:07.288 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:07 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:07.289 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:07 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.289 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:07 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.290 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:07 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:07.290 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:07 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:07.290 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:07 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:07.290 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:07 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.278 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.278 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.279 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.279 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.280 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.280 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.280 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.280 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.281 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.281 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.281 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.281 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:08 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.293 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.294 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.294 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.294 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.294 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.295 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.295 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.295 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.295 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.296 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.296 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.296 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:08 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.297 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.298 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.298 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.298 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.298 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.298 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.299 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.299 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.299 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.299 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.299 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.300 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.300 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:08 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.305 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.305 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.305 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.306 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.306 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.306 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.306 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.307 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.307 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.307 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.307 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.307 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.308 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.308 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.308 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.308 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.308 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.309 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:08 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.310 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.310 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.310 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.310 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.311 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.311 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.312 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.312 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:08.313 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:08.313 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:08.314 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:08.314 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:08 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:09.270 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:09 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:09.271 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:09 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:09.278 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:09 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:09.279 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:09 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:09.279 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:09 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:09.280 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:09 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:09.281 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:09 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:09.281 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:09 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:09.288 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:09 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:09.289 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:09 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:09.289 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:09 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:09.290 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:09 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:09.292 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:09 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:09.292 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:09 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:09 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:09.293 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:09 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:09 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:09.294 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:09 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.272 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.273 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.273 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.278 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:10 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.281 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.281 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.282 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.283 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.284 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.284 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.288 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.289 INFO:teuthology.orchestra.run.smithi111.stdout:12 Jul 08:38:10 ntpd[16775]: ntpd: time slew -0.004507 s 2022-07-12T08:38:10.290 INFO:teuthology.orchestra.run.smithi111.stdout:ntpd: time slew -0.004507s 2022-07-12T08:38:10.292 INFO:teuthology.orchestra.run.smithi111.stderr:12 Jul 08:38:10 ntpd[16775]: can't open /var/log/ntpstats/loopstats.20220712: Permission denied 2022-07-12T08:38:10.298 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.299 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.299 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.299 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.299 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.300 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.301 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.302 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.302 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.303 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.303 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.303 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.304 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.304 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.304 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.305 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.305 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.305 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.305 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.306 INFO:teuthology.orchestra.run.smithi137.stdout:12 Jul 08:38:10 ntpd[16846]: ntpd: time slew -0.001962 s 2022-07-12T08:38:10.306 INFO:teuthology.orchestra.run.smithi137.stdout:ntpd: time slew -0.001962s 2022-07-12T08:38:10.307 INFO:teuthology.orchestra.run.smithi121.stdout:12 Jul 08:38:10 ntpd[16818]: ntpd: time slew -0.006903 s 2022-07-12T08:38:10.307 INFO:teuthology.orchestra.run.smithi121.stdout:ntpd: time slew -0.006903s 2022-07-12T08:38:10.309 INFO:teuthology.orchestra.run.smithi137.stderr:12 Jul 08:38:10 ntpd[16846]: can't open /var/log/ntpstats/loopstats.20220712: Permission denied 2022-07-12T08:38:10.310 INFO:teuthology.orchestra.run.smithi121.stderr:12 Jul 08:38:10 ntpd[16818]: can't open /var/log/ntpstats/loopstats.20220712: Permission denied 2022-07-12T08:38:10.311 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.311 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.311 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.312 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.312 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.312 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.312 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.313 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.313 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.313 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.314 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.314 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.314 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.315 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.315 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:10.315 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:10.315 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:10.316 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:10.316 INFO:teuthology.orchestra.run.smithi130.stdout:12 Jul 08:38:10 ntpd[16820]: ntpd: time slew -0.008099 s 2022-07-12T08:38:10.317 INFO:teuthology.orchestra.run.smithi130.stdout:ntpd: time slew -0.008099s 2022-07-12T08:38:10.319 INFO:teuthology.orchestra.run.smithi130.stderr:12 Jul 08:38:10 ntpd[16820]: can't open /var/log/ntpstats/loopstats.20220712: Permission denied 2022-07-12T08:38:10.335 INFO:teuthology.orchestra.run.smithi130.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T08:38:10.336 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================== 2022-07-12T08:38:10.336 INFO:teuthology.orchestra.run.smithi130.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.336 INFO:teuthology.orchestra.run.smithi130.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.337 INFO:teuthology.orchestra.run.smithi130.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.337 INFO:teuthology.orchestra.run.smithi130.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.339 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T08:38:10.339 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2022-07-12T08:38:10.340 INFO:teuthology.orchestra.run.smithi111.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.340 INFO:teuthology.orchestra.run.smithi111.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.341 INFO:teuthology.orchestra.run.smithi111.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.342 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.345 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T08:38:10.346 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2022-07-12T08:38:10.346 INFO:teuthology.orchestra.run.smithi137.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.347 INFO:teuthology.orchestra.run.smithi137.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.347 INFO:teuthology.orchestra.run.smithi137.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.348 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.350 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T08:38:10.350 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-07-12T08:38:10.350 INFO:teuthology.orchestra.run.smithi121.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.351 INFO:teuthology.orchestra.run.smithi121.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.351 INFO:teuthology.orchestra.run.smithi121.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:10.351 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:11.268 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:11 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:11.268 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:11 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:11.269 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:11 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:11.269 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:11 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:11.270 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:11 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:11.270 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:11 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:12.270 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:12.270 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:12.271 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:12.271 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:12.272 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:12.272 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:12.274 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-12T08:38:12.275 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/rawstats.20220712: Permission denied 2022-07-12T08:38:12.276 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-12T08:38:12.277 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/peerstats.20220712: Permission denied 2022-07-12T08:38:12.278 INFO:teuthology.orchestra.run.smithi061.stdout:12 Jul 08:38:12 ntpd[16826]: ntpd: time slew -0.004969 s 2022-07-12T08:38:12.278 INFO:teuthology.orchestra.run.smithi061.stdout:ntpd: time slew -0.004969s 2022-07-12T08:38:12.279 INFO:teuthology.orchestra.run.smithi061.stderr:12 Jul 08:38:12 ntpd[16826]: can't open /var/log/ntpstats/loopstats.20220712: Permission denied 2022-07-12T08:38:12.328 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T08:38:12.329 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2022-07-12T08:38:12.329 INFO:teuthology.orchestra.run.smithi061.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:12.334 INFO:teuthology.orchestra.run.smithi061.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:12.335 INFO:teuthology.orchestra.run.smithi061.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:12.335 INFO:teuthology.orchestra.run.smithi061.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-12T08:38:12.336 INFO:teuthology.run_tasks:Running task install... 2022-07-12T08:38:12.350 DEBUG:teuthology.task.install:project ceph 2022-07-12T08:38:12.351 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3497be4c2159a20e7a0f23fb5e2730060a4424c0'}} 2022-07-12T08:38:12.351 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '3497be4c2159a20e7a0f23fb5e2730060a4424c0'} 2022-07-12T08:38:12.353 INFO:teuthology.task.install:Using flavor: default 2022-07-12T08:38:12.362 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-07-12T08:38:12.364 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2022-07-12T08:38:12.364 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2022-07-12T08:38:12.417 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2022-07-12T08:38:12.423 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo lsb_release -is 2022-07-12T08:38:12.426 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-key list | grep Ceph 2022-07-12T08:38:12.477 INFO:teuthology.orchestra.run.smithi111.stdout:Ubuntu 2022-07-12T08:38:12.483 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-07-12T08:38:12.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-key list | grep Ceph 2022-07-12T08:38:12.528 INFO:teuthology.orchestra.run.smithi061.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-12T08:38:12.535 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-07-12T08:38:12.539 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.540 DEBUG:teuthology.orchestra.run.smithi130:> sudo lsb_release -is 2022-07-12T08:38:12.541 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-key list | grep Ceph 2022-07-12T08:38:12.584 INFO:teuthology.orchestra.run.smithi111.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-12T08:38:12.593 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-12T08:38:12.594 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph.com (release key) 2022-07-12T08:38:12.595 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2022-07-12T08:38:12.597 INFO:teuthology.orchestra.run.smithi130.stdout:Ubuntu 2022-07-12T08:38:12.599 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2022-07-12T08:38:12.604 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-key list | grep Ceph 2022-07-12T08:38:12.641 INFO:teuthology.orchestra.run.smithi121.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-12T08:38:12.643 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-12T08:38:12.643 INFO:teuthology.orchestra.run.smithi111.stdout:uid [ unknown] Ceph.com (release key) 2022-07-12T08:38:12.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo lsb_release -is 2022-07-12T08:38:12.650 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2022-07-12T08:38:12.658 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.659 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-12T08:38:12.659 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-12T08:38:12.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:38:12.669 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2022-07-12T08:38:12.670 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-key list | grep Ceph 2022-07-12T08:38:12.709 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-12T08:38:12.710 INFO:teuthology.orchestra.run.smithi121.stdout:uid [ unknown] Ceph.com (release key) 2022-07-12T08:38:12.711 INFO:teuthology.orchestra.run.smithi111.stdout:Ubuntu 2022-07-12T08:38:12.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-07-12T08:38:12.712 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.713 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-12T08:38:12.714 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-12T08:38:12.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:38:12.718 INFO:teuthology.orchestra.run.smithi130.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-12T08:38:12.767 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-07-12T08:38:12.771 INFO:teuthology.orchestra.run.smithi130.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-12T08:38:12.772 INFO:teuthology.orchestra.run.smithi130.stdout:uid [ unknown] Ceph.com (release key) 2022-07-12T08:38:12.772 DEBUG:teuthology.orchestra.run.smithi130:> sudo lsb_release -is 2022-07-12T08:38:12.774 INFO:teuthology.orchestra.run.smithi137.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-12T08:38:12.776 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.776 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-12T08:38:12.777 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-12T08:38:12.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:38:12.833 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-12T08:38:12.834 INFO:teuthology.orchestra.run.smithi137.stdout:uid [ unknown] Ceph.com (release key) 2022-07-12T08:38:12.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2022-07-12T08:38:12.842 INFO:teuthology.orchestra.run.smithi130.stdout:Ubuntu 2022-07-12T08:38:12.843 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.843 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-12T08:38:12.844 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-12T08:38:12.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:38:12.854 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default/ 2022-07-12T08:38:12.854 INFO:teuthology.task.install.deb:Package version is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:12.906 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default/ 2022-07-12T08:38:12.906 INFO:teuthology.task.install.deb:Package version is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:12.914 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2022-07-12T08:38:12.915 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:38:12.915 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-12T08:38:12.915 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-12T08:38:12.916 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:38:12.980 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default/ 2022-07-12T08:38:12.980 INFO:teuthology.task.install.deb:Package version is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:12.989 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:38:12.989 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-12T08:38:13.020 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2022-07-12T08:38:13.034 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default/ 2022-07-12T08:38:13.035 INFO:teuthology.task.install.deb:Package version is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:13.041 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:38:13.042 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-12T08:38:13.070 DEBUG:teuthology.orchestra.run.smithi111:> sudo apt-get update 2022-07-12T08:38:13.087 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default/ 2022-07-12T08:38:13.087 INFO:teuthology.task.install.deb:Package version is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:13.104 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:38:13.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-12T08:38:13.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo apt-get update 2022-07-12T08:38:13.160 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:38:13.160 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-12T08:38:13.171 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:38:13.177 DEBUG:teuthology.orchestra.run.smithi130:> sudo apt-get update 2022-07-12T08:38:13.202 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:38:13.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-12T08:38:13.224 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal InRelease 2022-07-12T08:38:13.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo apt-get update 2022-07-12T08:38:13.241 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-12T08:38:13.264 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-12T08:38:13.274 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:38:13.276 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:38:13.279 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:38:13.282 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:38:13.285 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-12T08:38:13.302 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:38:13.322 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal InRelease 2022-07-12T08:38:13.325 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal InRelease 2022-07-12T08:38:13.325 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:38:13.325 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:38:13.328 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main i386 Packages [3,059 B] 2022-07-12T08:38:13.333 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:38:13.343 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-12T08:38:13.346 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-12T08:38:13.355 INFO:teuthology.orchestra.run.smithi130.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:38:13.359 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:38:13.365 INFO:teuthology.orchestra.run.smithi121.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-12T08:38:13.368 INFO:teuthology.orchestra.run.smithi111.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-12T08:38:13.371 INFO:teuthology.orchestra.run.smithi130.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal InRelease 2022-07-12T08:38:13.387 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:38:13.387 INFO:teuthology.orchestra.run.smithi121.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:38:13.387 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-12T08:38:13.390 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-12T08:38:13.392 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-12T08:38:13.416 INFO:teuthology.orchestra.run.smithi130.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-12T08:38:13.417 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-12T08:38:13.426 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main i386 Packages [3,059 B] 2022-07-12T08:38:13.429 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main i386 Packages [3,059 B] 2022-07-12T08:38:13.438 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main i386 Packages [3,059 B] 2022-07-12T08:38:13.439 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:38:13.463 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal InRelease 2022-07-12T08:38:13.465 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:38:13.485 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-12T08:38:13.487 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-12T08:38:13.507 INFO:teuthology.orchestra.run.smithi137.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-12T08:38:13.525 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-12T08:38:13.529 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-12T08:38:13.546 INFO:teuthology.orchestra.run.smithi111.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:38:13.570 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main i386 Packages [3,059 B] 2022-07-12T08:38:13.605 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-12T08:38:13.685 INFO:teuthology.orchestra.run.smithi137.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-12T08:38:13.839 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 26.5 kB in 1s (40.6 kB/s) 2022-07-12T08:38:13.924 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 26.5 kB in 1s (41.9 kB/s) 2022-07-12T08:38:13.925 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 26.5 kB in 1s (41.9 kB/s) 2022-07-12T08:38:13.983 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 26.5 kB in 1s (41.6 kB/s) 2022-07-12T08:38:14.074 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 26.5 kB in 1s (40.8 kB/s) 2022-07-12T08:38:14.762 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:38:14.791 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-324-g3497be4c-1focal cephadm=15.2.16-324-g3497be4c-1focal ceph-mds=15.2.16-324-g3497be4c-1focal ceph-mgr=15.2.16-324-g3497be4c-1focal ceph-common=15.2.16-324-g3497be4c-1focal ceph-fuse=15.2.16-324-g3497be4c-1focal ceph-test=15.2.16-324-g3497be4c-1focal radosgw=15.2.16-324-g3497be4c-1focal python3-rados=15.2.16-324-g3497be4c-1focal python3-rgw=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal python3-rbd=15.2.16-324-g3497be4c-1focal libcephfs2=15.2.16-324-g3497be4c-1focal libcephfs-dev=15.2.16-324-g3497be4c-1focal librados2=15.2.16-324-g3497be4c-1focal librbd1=15.2.16-324-g3497be4c-1focal rbd-fuse=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal cephfs-shell=15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:14.807 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:38:14.807 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:38:14.829 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-324-g3497be4c-1focal cephadm=15.2.16-324-g3497be4c-1focal ceph-mds=15.2.16-324-g3497be4c-1focal ceph-mgr=15.2.16-324-g3497be4c-1focal ceph-common=15.2.16-324-g3497be4c-1focal ceph-fuse=15.2.16-324-g3497be4c-1focal ceph-test=15.2.16-324-g3497be4c-1focal radosgw=15.2.16-324-g3497be4c-1focal python3-rados=15.2.16-324-g3497be4c-1focal python3-rgw=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal python3-rbd=15.2.16-324-g3497be4c-1focal libcephfs2=15.2.16-324-g3497be4c-1focal libcephfs-dev=15.2.16-324-g3497be4c-1focal librados2=15.2.16-324-g3497be4c-1focal librbd1=15.2.16-324-g3497be4c-1focal rbd-fuse=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal cephfs-shell=15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:14.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-324-g3497be4c-1focal cephadm=15.2.16-324-g3497be4c-1focal ceph-mds=15.2.16-324-g3497be4c-1focal ceph-mgr=15.2.16-324-g3497be4c-1focal ceph-common=15.2.16-324-g3497be4c-1focal ceph-fuse=15.2.16-324-g3497be4c-1focal ceph-test=15.2.16-324-g3497be4c-1focal radosgw=15.2.16-324-g3497be4c-1focal python3-rados=15.2.16-324-g3497be4c-1focal python3-rgw=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal python3-rbd=15.2.16-324-g3497be4c-1focal libcephfs2=15.2.16-324-g3497be4c-1focal libcephfs-dev=15.2.16-324-g3497be4c-1focal librados2=15.2.16-324-g3497be4c-1focal librbd1=15.2.16-324-g3497be4c-1focal rbd-fuse=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal cephfs-shell=15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:14.862 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:38:14.862 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:38:14.887 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-324-g3497be4c-1focal cephadm=15.2.16-324-g3497be4c-1focal ceph-mds=15.2.16-324-g3497be4c-1focal ceph-mgr=15.2.16-324-g3497be4c-1focal ceph-common=15.2.16-324-g3497be4c-1focal ceph-fuse=15.2.16-324-g3497be4c-1focal ceph-test=15.2.16-324-g3497be4c-1focal radosgw=15.2.16-324-g3497be4c-1focal python3-rados=15.2.16-324-g3497be4c-1focal python3-rgw=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal python3-rbd=15.2.16-324-g3497be4c-1focal libcephfs2=15.2.16-324-g3497be4c-1focal libcephfs-dev=15.2.16-324-g3497be4c-1focal librados2=15.2.16-324-g3497be4c-1focal librbd1=15.2.16-324-g3497be4c-1focal rbd-fuse=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal cephfs-shell=15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:14.916 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:38:14.917 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:38:14.960 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:38:14.961 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:38:14.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-324-g3497be4c-1focal cephadm=15.2.16-324-g3497be4c-1focal ceph-mds=15.2.16-324-g3497be4c-1focal ceph-mgr=15.2.16-324-g3497be4c-1focal ceph-common=15.2.16-324-g3497be4c-1focal ceph-fuse=15.2.16-324-g3497be4c-1focal ceph-test=15.2.16-324-g3497be4c-1focal radosgw=15.2.16-324-g3497be4c-1focal python3-rados=15.2.16-324-g3497be4c-1focal python3-rgw=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal python3-rbd=15.2.16-324-g3497be4c-1focal libcephfs2=15.2.16-324-g3497be4c-1focal libcephfs-dev=15.2.16-324-g3497be4c-1focal librados2=15.2.16-324-g3497be4c-1focal librbd1=15.2.16-324-g3497be4c-1focal rbd-fuse=15.2.16-324-g3497be4c-1focal python3-cephfs=15.2.16-324-g3497be4c-1focal cephfs-shell=15.2.16-324-g3497be4c-1focal 2022-07-12T08:38:15.055 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-12T08:38:15.055 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-12T08:38:15.060 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:38:15.094 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-07-12T08:38:15.095 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-07-12T08:38:15.095 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-07-12T08:38:15.096 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-07-12T08:38:15.116 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2022-07-12T08:38:15.117 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2022-07-12T08:38:15.204 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:38:15.212 INFO:teuthology.orchestra.run.smithi061.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:38:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:38:15.214 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2022-07-12T08:38:15.214 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.215 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.215 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-12T08:38:15.216 INFO:teuthology.orchestra.run.smithi061.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-12T08:38:15.216 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-12T08:38:15.217 INFO:teuthology.orchestra.run.smithi061.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-12T08:38:15.224 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-12T08:38:15.224 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-12T08:38:15.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-12T08:38:15.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-12T08:38:15.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-12T08:38:15.225 INFO:teuthology.orchestra.run.smithi061.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-12T08:38:15.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-12T08:38:15.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-12T08:38:15.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-12T08:38:15.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-12T08:38:15.226 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-12T08:38:15.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-12T08:38:15.227 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-12T08:38:15.227 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet 2022-07-12T08:38:15.227 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2022-07-12T08:38:15.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-12T08:38:15.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-12T08:38:15.228 INFO:teuthology.orchestra.run.smithi061.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-12T08:38:15.228 INFO:teuthology.orchestra.run.smithi061.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-12T08:38:15.229 INFO:teuthology.orchestra.run.smithi061.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-12T08:38:15.229 INFO:teuthology.orchestra.run.smithi061.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-12T08:38:15.229 INFO:teuthology.orchestra.run.smithi061.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-12T08:38:15.229 INFO:teuthology.orchestra.run.smithi061.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-12T08:38:15.230 INFO:teuthology.orchestra.run.smithi061.stdout:Recommended packages: 2022-07-12T08:38:15.230 INFO:teuthology.orchestra.run.smithi061.stdout: btrfs-tools 2022-07-12T08:38:15.235 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-07-12T08:38:15.235 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-07-12T08:38:15.240 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:38:15.241 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:38:15.242 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:38:15.242 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-07-12T08:38:15.242 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.243 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-12T08:38:15.244 INFO:teuthology.orchestra.run.smithi121.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-12T08:38:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-12T08:38:15.245 INFO:teuthology.orchestra.run.smithi121.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-12T08:38:15.260 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-12T08:38:15.261 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-12T08:38:15.261 INFO:teuthology.orchestra.run.smithi121.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-12T08:38:15.262 INFO:teuthology.orchestra.run.smithi121.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-12T08:38:15.263 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-12T08:38:15.263 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-12T08:38:15.264 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-12T08:38:15.264 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-12T08:38:15.265 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-12T08:38:15.265 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-12T08:38:15.266 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-12T08:38:15.266 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-12T08:38:15.267 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-12T08:38:15.267 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet 2022-07-12T08:38:15.268 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-07-12T08:38:15.269 INFO:teuthology.orchestra.run.smithi121.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-12T08:38:15.269 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-12T08:38:15.270 INFO:teuthology.orchestra.run.smithi121.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-12T08:38:15.271 INFO:teuthology.orchestra.run.smithi121.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-12T08:38:15.272 INFO:teuthology.orchestra.run.smithi121.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-12T08:38:15.272 INFO:teuthology.orchestra.run.smithi121.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-12T08:38:15.275 INFO:teuthology.orchestra.run.smithi121.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-12T08:38:15.277 INFO:teuthology.orchestra.run.smithi121.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-12T08:38:15.278 INFO:teuthology.orchestra.run.smithi121.stdout:Recommended packages: 2022-07-12T08:38:15.278 INFO:teuthology.orchestra.run.smithi121.stdout: btrfs-tools 2022-07-12T08:38:15.285 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:38:15.285 INFO:teuthology.orchestra.run.smithi130.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:38:15.288 INFO:teuthology.orchestra.run.smithi130.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:38:15.288 INFO:teuthology.orchestra.run.smithi130.stdout:The following additional packages will be installed: 2022-07-12T08:38:15.289 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.289 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.290 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-12T08:38:15.290 INFO:teuthology.orchestra.run.smithi130.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-12T08:38:15.291 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-12T08:38:15.291 INFO:teuthology.orchestra.run.smithi130.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-12T08:38:15.292 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-12T08:38:15.292 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-12T08:38:15.292 INFO:teuthology.orchestra.run.smithi130.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-12T08:38:15.293 INFO:teuthology.orchestra.run.smithi130.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-12T08:38:15.293 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-12T08:38:15.294 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-12T08:38:15.294 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-12T08:38:15.295 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-12T08:38:15.295 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-12T08:38:15.295 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-12T08:38:15.296 INFO:teuthology.orchestra.run.smithi130.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-12T08:38:15.296 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-12T08:38:15.297 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-12T08:38:15.297 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet 2022-07-12T08:38:15.303 INFO:teuthology.orchestra.run.smithi130.stdout:Suggested packages: 2022-07-12T08:38:15.304 INFO:teuthology.orchestra.run.smithi130.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-12T08:38:15.304 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-12T08:38:15.305 INFO:teuthology.orchestra.run.smithi130.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-12T08:38:15.316 INFO:teuthology.orchestra.run.smithi130.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-12T08:38:15.316 INFO:teuthology.orchestra.run.smithi130.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-12T08:38:15.316 INFO:teuthology.orchestra.run.smithi130.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-12T08:38:15.317 INFO:teuthology.orchestra.run.smithi130.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-12T08:38:15.317 INFO:teuthology.orchestra.run.smithi130.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-12T08:38:15.317 INFO:teuthology.orchestra.run.smithi130.stdout:Recommended packages: 2022-07-12T08:38:15.327 INFO:teuthology.orchestra.run.smithi130.stdout: btrfs-tools 2022-07-12T08:38:15.327 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:38:15.327 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:38:15.328 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:38:15.328 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2022-07-12T08:38:15.328 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.329 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.329 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-12T08:38:15.329 INFO:teuthology.orchestra.run.smithi111.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-12T08:38:15.329 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-12T08:38:15.329 INFO:teuthology.orchestra.run.smithi111.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-12T08:38:15.342 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-12T08:38:15.342 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-12T08:38:15.343 INFO:teuthology.orchestra.run.smithi111.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-12T08:38:15.343 INFO:teuthology.orchestra.run.smithi111.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-12T08:38:15.343 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-12T08:38:15.343 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-12T08:38:15.344 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-12T08:38:15.344 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-12T08:38:15.344 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-12T08:38:15.344 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-12T08:38:15.345 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-12T08:38:15.345 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-12T08:38:15.345 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-12T08:38:15.345 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet 2022-07-12T08:38:15.345 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2022-07-12T08:38:15.356 INFO:teuthology.orchestra.run.smithi111.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-12T08:38:15.357 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-12T08:38:15.357 INFO:teuthology.orchestra.run.smithi111.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-12T08:38:15.357 INFO:teuthology.orchestra.run.smithi111.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-12T08:38:15.357 INFO:teuthology.orchestra.run.smithi111.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-12T08:38:15.358 INFO:teuthology.orchestra.run.smithi111.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-12T08:38:15.358 INFO:teuthology.orchestra.run.smithi111.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-12T08:38:15.358 INFO:teuthology.orchestra.run.smithi111.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-12T08:38:15.358 INFO:teuthology.orchestra.run.smithi111.stdout:Recommended packages: 2022-07-12T08:38:15.359 INFO:teuthology.orchestra.run.smithi111.stdout: btrfs-tools 2022-07-12T08:38:15.362 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-07-12T08:38:15.363 INFO:teuthology.orchestra.run.smithi121.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-12T08:38:15.363 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.363 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.364 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-12T08:38:15.364 INFO:teuthology.orchestra.run.smithi121.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-12T08:38:15.364 INFO:teuthology.orchestra.run.smithi121.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-12T08:38:15.365 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-12T08:38:15.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-12T08:38:15.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-12T08:38:15.365 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-12T08:38:15.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-12T08:38:15.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-12T08:38:15.366 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-12T08:38:15.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-12T08:38:15.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-12T08:38:15.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-12T08:38:15.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-12T08:38:15.367 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-12T08:38:15.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-12T08:38:15.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-12T08:38:15.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-12T08:38:15.368 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-12T08:38:15.369 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-07-12T08:38:15.369 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 librbd1 2022-07-12T08:38:15.369 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2022-07-12T08:38:15.369 INFO:teuthology.orchestra.run.smithi061.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-12T08:38:15.370 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.370 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.371 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-12T08:38:15.371 INFO:teuthology.orchestra.run.smithi061.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-12T08:38:15.371 INFO:teuthology.orchestra.run.smithi061.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-12T08:38:15.371 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-12T08:38:15.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-12T08:38:15.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-12T08:38:15.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-12T08:38:15.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-12T08:38:15.372 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-12T08:38:15.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-12T08:38:15.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-12T08:38:15.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-12T08:38:15.373 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-12T08:38:15.374 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-12T08:38:15.374 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-12T08:38:15.374 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-12T08:38:15.374 INFO:teuthology.orchestra.run.smithi061.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-12T08:38:15.375 INFO:teuthology.orchestra.run.smithi061.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-12T08:38:15.375 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-12T08:38:15.375 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2022-07-12T08:38:15.375 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 librbd1 2022-07-12T08:38:15.376 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2022-07-12T08:38:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-12T08:38:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-12T08:38:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-12T08:38:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-12T08:38:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-12T08:38:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-12T08:38:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-12T08:38:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-12T08:38:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-12T08:38:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-12T08:38:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-12T08:38:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-12T08:38:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-12T08:38:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-12T08:38:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-12T08:38:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-12T08:38:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-12T08:38:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-12T08:38:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-12T08:38:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-12T08:38:15.383 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages will be upgraded: 2022-07-12T08:38:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 librbd1 2022-07-12T08:38:15.384 INFO:teuthology.orchestra.run.smithi130.stdout:The following NEW packages will be installed: 2022-07-12T08:38:15.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-12T08:38:15.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.385 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.394 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-12T08:38:15.395 INFO:teuthology.orchestra.run.smithi130.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-12T08:38:15.395 INFO:teuthology.orchestra.run.smithi130.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-12T08:38:15.395 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-12T08:38:15.396 INFO:teuthology.orchestra.run.smithi130.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-12T08:38:15.396 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-12T08:38:15.396 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-12T08:38:15.396 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-12T08:38:15.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-12T08:38:15.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-12T08:38:15.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-12T08:38:15.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-12T08:38:15.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-12T08:38:15.398 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-12T08:38:15.398 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-12T08:38:15.398 INFO:teuthology.orchestra.run.smithi130.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-12T08:38:15.398 INFO:teuthology.orchestra.run.smithi130.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-12T08:38:15.399 INFO:teuthology.orchestra.run.smithi130.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-12T08:38:15.399 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-12T08:38:15.399 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages will be upgraded: 2022-07-12T08:38:15.399 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 librbd1 2022-07-12T08:38:15.415 INFO:teuthology.orchestra.run.smithi061.stdout:2 upgraded, 101 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:38:15.416 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 153 MB of archives. 2022-07-12T08:38:15.416 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-12T08:38:15.416 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-12T08:38:15.416 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-12T08:38:15.417 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-12T08:38:15.417 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:38:15.418 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:38:15.418 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:38:15.418 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2022-07-12T08:38:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-12T08:38:15.419 INFO:teuthology.orchestra.run.smithi137.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-12T08:38:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-12T08:38:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-12T08:38:15.420 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-12T08:38:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-12T08:38:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-12T08:38:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-12T08:38:15.421 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-12T08:38:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-12T08:38:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-12T08:38:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-12T08:38:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-12T08:38:15.422 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-12T08:38:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-12T08:38:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-12T08:38:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-12T08:38:15.423 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet 2022-07-12T08:38:15.424 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2022-07-12T08:38:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-12T08:38:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-12T08:38:15.424 INFO:teuthology.orchestra.run.smithi137.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-12T08:38:15.425 INFO:teuthology.orchestra.run.smithi137.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-12T08:38:15.425 INFO:teuthology.orchestra.run.smithi137.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-12T08:38:15.425 INFO:teuthology.orchestra.run.smithi137.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-12T08:38:15.426 INFO:teuthology.orchestra.run.smithi137.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-12T08:38:15.426 INFO:teuthology.orchestra.run.smithi137.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-12T08:38:15.426 INFO:teuthology.orchestra.run.smithi137.stdout:Recommended packages: 2022-07-12T08:38:15.426 INFO:teuthology.orchestra.run.smithi137.stdout: btrfs-tools 2022-07-12T08:38:15.430 INFO:teuthology.orchestra.run.smithi111.stdout:2 upgraded, 101 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:38:15.431 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 153 MB of archives. 2022-07-12T08:38:15.431 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-12T08:38:15.432 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-12T08:38:15.432 INFO:teuthology.orchestra.run.smithi121.stdout:2 upgraded, 101 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:38:15.433 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 153 MB of archives. 2022-07-12T08:38:15.433 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-12T08:38:15.433 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-12T08:38:15.435 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-12T08:38:15.436 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-12T08:38:15.448 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-12T08:38:15.448 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-12T08:38:15.449 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-12T08:38:15.462 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-12T08:38:15.464 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-324-g3497be4c-1focal [1,698 kB] 2022-07-12T08:38:15.473 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-12T08:38:15.490 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-12T08:38:15.490 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-12T08:38:15.491 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-12T08:38:15.491 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-324-g3497be4c-1focal [1,698 kB] 2022-07-12T08:38:15.503 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-324-g3497be4c-1focal [1,698 kB] 2022-07-12T08:38:15.504 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2022-07-12T08:38:15.505 INFO:teuthology.orchestra.run.smithi137.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-12T08:38:15.505 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-12T08:38:15.505 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-12T08:38:15.505 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-12T08:38:15.506 INFO:teuthology.orchestra.run.smithi137.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-12T08:38:15.506 INFO:teuthology.orchestra.run.smithi137.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-12T08:38:15.506 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-12T08:38:15.506 INFO:teuthology.orchestra.run.smithi137.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-12T08:38:15.507 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-12T08:38:15.507 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-12T08:38:15.507 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-12T08:38:15.507 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-12T08:38:15.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-12T08:38:15.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-12T08:38:15.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-12T08:38:15.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-12T08:38:15.508 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-12T08:38:15.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-12T08:38:15.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-12T08:38:15.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-12T08:38:15.509 INFO:teuthology.orchestra.run.smithi137.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-12T08:38:15.510 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-12T08:38:15.510 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2022-07-12T08:38:15.510 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 librbd1 2022-07-12T08:38:15.511 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-12T08:38:15.511 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-12T08:38:15.512 INFO:teuthology.orchestra.run.smithi130.stdout:2 upgraded, 101 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:38:15.512 INFO:teuthology.orchestra.run.smithi130.stdout:Need to get 153 MB of archives. 2022-07-12T08:38:15.512 INFO:teuthology.orchestra.run.smithi130.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-12T08:38:15.513 INFO:teuthology.orchestra.run.smithi130.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-324-g3497be4c-1focal [1,698 kB] 2022-07-12T08:38:15.515 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-12T08:38:15.516 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-12T08:38:15.516 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-12T08:38:15.522 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-12T08:38:15.523 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-12T08:38:15.526 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-12T08:38:15.526 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-12T08:38:15.529 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-12T08:38:15.529 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-12T08:38:15.535 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-12T08:38:15.540 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-07-12T08:38:15.540 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-12T08:38:15.541 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-12T08:38:15.545 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-12T08:38:15.547 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-12T08:38:15.548 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-12T08:38:15.548 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-12T08:38:15.551 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-12T08:38:15.552 INFO:teuthology.orchestra.run.smithi130.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-12T08:38:15.554 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-12T08:38:15.554 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-12T08:38:15.559 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-12T08:38:15.568 INFO:teuthology.orchestra.run.smithi121.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-12T08:38:15.569 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-12T08:38:15.570 INFO:teuthology.orchestra.run.smithi121.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-12T08:38:15.570 INFO:teuthology.orchestra.run.smithi121.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-12T08:38:15.579 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-12T08:38:15.580 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-12T08:38:15.580 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-12T08:38:15.580 INFO:teuthology.orchestra.run.smithi061.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-12T08:38:15.580 INFO:teuthology.orchestra.run.smithi061.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-12T08:38:15.582 INFO:teuthology.orchestra.run.smithi061.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-12T08:38:15.583 INFO:teuthology.orchestra.run.smithi121.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-07-12T08:38:15.583 INFO:teuthology.orchestra.run.smithi121.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-12T08:38:15.583 INFO:teuthology.orchestra.run.smithi121.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-12T08:38:15.584 INFO:teuthology.orchestra.run.smithi111.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-12T08:38:15.584 INFO:teuthology.orchestra.run.smithi111.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-12T08:38:15.584 INFO:teuthology.orchestra.run.smithi111.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-12T08:38:15.584 INFO:teuthology.orchestra.run.smithi111.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-07-12T08:38:15.585 INFO:teuthology.orchestra.run.smithi111.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-12T08:38:15.585 INFO:teuthology.orchestra.run.smithi111.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-12T08:38:15.586 INFO:teuthology.orchestra.run.smithi121.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-12T08:38:15.586 INFO:teuthology.orchestra.run.smithi121.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-12T08:38:15.587 INFO:teuthology.orchestra.run.smithi121.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-12T08:38:15.589 INFO:teuthology.orchestra.run.smithi061.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-12T08:38:15.594 INFO:teuthology.orchestra.run.smithi111.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-12T08:38:15.595 INFO:teuthology.orchestra.run.smithi061.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-12T08:38:15.595 INFO:teuthology.orchestra.run.smithi111.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-12T08:38:15.596 INFO:teuthology.orchestra.run.smithi111.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-12T08:38:15.606 INFO:teuthology.orchestra.run.smithi121.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-12T08:38:15.606 INFO:teuthology.orchestra.run.smithi121.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-12T08:38:15.607 INFO:teuthology.orchestra.run.smithi061.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-12T08:38:15.607 INFO:teuthology.orchestra.run.smithi061.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-12T08:38:15.609 INFO:teuthology.orchestra.run.smithi121.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-12T08:38:15.609 INFO:teuthology.orchestra.run.smithi121.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-12T08:38:15.611 INFO:teuthology.orchestra.run.smithi121.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-12T08:38:15.611 INFO:teuthology.orchestra.run.smithi121.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-12T08:38:15.616 INFO:teuthology.orchestra.run.smithi121.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-12T08:38:15.617 INFO:teuthology.orchestra.run.smithi111.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-12T08:38:15.618 INFO:teuthology.orchestra.run.smithi111.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-12T08:38:15.619 INFO:teuthology.orchestra.run.smithi061.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-12T08:38:15.620 INFO:teuthology.orchestra.run.smithi111.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-12T08:38:15.621 INFO:teuthology.orchestra.run.smithi111.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-12T08:38:15.623 INFO:teuthology.orchestra.run.smithi061.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-12T08:38:15.624 INFO:teuthology.orchestra.run.smithi111.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-12T08:38:15.624 INFO:teuthology.orchestra.run.smithi111.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-12T08:38:15.626 INFO:teuthology.orchestra.run.smithi061.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-12T08:38:15.633 INFO:teuthology.orchestra.run.smithi061.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-12T08:38:15.634 INFO:teuthology.orchestra.run.smithi061.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-12T08:38:15.634 INFO:teuthology.orchestra.run.smithi061.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-12T08:38:15.634 INFO:teuthology.orchestra.run.smithi061.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-12T08:38:15.636 INFO:teuthology.orchestra.run.smithi111.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-12T08:38:15.636 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-12T08:38:15.636 INFO:teuthology.orchestra.run.smithi121.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-12T08:38:15.637 INFO:teuthology.orchestra.run.smithi121.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-12T08:38:15.637 INFO:teuthology.orchestra.run.smithi121.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-12T08:38:15.645 INFO:teuthology.orchestra.run.smithi111.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-12T08:38:15.645 INFO:teuthology.orchestra.run.smithi111.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-12T08:38:15.646 INFO:teuthology.orchestra.run.smithi111.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-12T08:38:15.646 INFO:teuthology.orchestra.run.smithi061.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-12T08:38:15.647 INFO:teuthology.orchestra.run.smithi061.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-12T08:38:15.668 INFO:teuthology.orchestra.run.smithi121.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-12T08:38:15.670 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-12T08:38:15.672 INFO:teuthology.orchestra.run.smithi121.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-12T08:38:15.675 INFO:teuthology.orchestra.run.smithi121.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-12T08:38:15.676 INFO:teuthology.orchestra.run.smithi121.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-12T08:38:15.678 INFO:teuthology.orchestra.run.smithi111.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-12T08:38:15.683 INFO:teuthology.orchestra.run.smithi111.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-12T08:38:15.684 INFO:teuthology.orchestra.run.smithi121.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-12T08:38:15.684 INFO:teuthology.orchestra.run.smithi121.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-12T08:38:15.684 INFO:teuthology.orchestra.run.smithi121.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-12T08:38:15.685 INFO:teuthology.orchestra.run.smithi111.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-12T08:38:15.685 INFO:teuthology.orchestra.run.smithi111.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-12T08:38:15.687 INFO:teuthology.orchestra.run.smithi111.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-12T08:38:15.690 INFO:teuthology.orchestra.run.smithi111.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-12T08:38:15.691 INFO:teuthology.orchestra.run.smithi137.stdout:2 upgraded, 101 newly installed, 0 to remove and 18 not upgraded. 2022-07-12T08:38:15.691 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 153 MB of archives. 2022-07-12T08:38:15.692 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-12T08:38:15.692 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-324-g3497be4c-1focal [1,698 kB] 2022-07-12T08:38:15.693 INFO:teuthology.orchestra.run.smithi111.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-12T08:38:15.694 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-12T08:38:15.710 INFO:teuthology.orchestra.run.smithi121.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-12T08:38:15.711 INFO:teuthology.orchestra.run.smithi121.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-12T08:38:15.720 INFO:teuthology.orchestra.run.smithi061.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-324-g3497be4c-1focal [3,257 kB] 2022-07-12T08:38:15.727 INFO:teuthology.orchestra.run.smithi111.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-12T08:38:15.727 INFO:teuthology.orchestra.run.smithi111.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-12T08:38:15.745 INFO:teuthology.orchestra.run.smithi111.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-324-g3497be4c-1focal [3,257 kB] 2022-07-12T08:38:15.772 INFO:teuthology.orchestra.run.smithi121.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-324-g3497be4c-1focal [3,257 kB] 2022-07-12T08:38:15.773 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-12T08:38:15.800 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-324-g3497be4c-1focal [3,257 kB] 2022-07-12T08:38:15.803 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-12T08:38:15.811 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-12T08:38:15.925 INFO:teuthology.orchestra.run.smithi061.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-12T08:38:15.936 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-12T08:38:15.944 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-12T08:38:15.944 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-12T08:38:15.959 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-12T08:38:16.018 INFO:teuthology.orchestra.run.smithi061.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-12T08:38:16.020 INFO:teuthology.orchestra.run.smithi061.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-12T08:38:16.031 INFO:teuthology.orchestra.run.smithi121.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-12T08:38:16.041 INFO:teuthology.orchestra.run.smithi061.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-12T08:38:16.047 INFO:teuthology.orchestra.run.smithi061.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-12T08:38:16.048 INFO:teuthology.orchestra.run.smithi061.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-12T08:38:16.049 INFO:teuthology.orchestra.run.smithi061.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-12T08:38:16.049 INFO:teuthology.orchestra.run.smithi061.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-12T08:38:16.051 INFO:teuthology.orchestra.run.smithi061.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-12T08:38:16.051 INFO:teuthology.orchestra.run.smithi061.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-12T08:38:16.060 INFO:teuthology.orchestra.run.smithi130.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-12T08:38:16.062 INFO:teuthology.orchestra.run.smithi121.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-12T08:38:16.065 INFO:teuthology.orchestra.run.smithi121.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-12T08:38:16.072 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-12T08:38:16.081 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-12T08:38:16.083 INFO:teuthology.orchestra.run.smithi130.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-12T08:38:16.087 INFO:teuthology.orchestra.run.smithi130.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-12T08:38:16.094 INFO:teuthology.orchestra.run.smithi061.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-324-g3497be4c-1focal [463 kB] 2022-07-12T08:38:16.095 INFO:teuthology.orchestra.run.smithi061.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-12T08:38:16.096 INFO:teuthology.orchestra.run.smithi061.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-12T08:38:16.096 INFO:teuthology.orchestra.run.smithi061.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-12T08:38:16.097 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-12T08:38:16.098 INFO:teuthology.orchestra.run.smithi121.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-12T08:38:16.098 INFO:teuthology.orchestra.run.smithi121.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-12T08:38:16.099 INFO:teuthology.orchestra.run.smithi121.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-12T08:38:16.099 INFO:teuthology.orchestra.run.smithi121.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-12T08:38:16.099 INFO:teuthology.orchestra.run.smithi121.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-12T08:38:16.103 INFO:teuthology.orchestra.run.smithi121.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-12T08:38:16.105 INFO:teuthology.orchestra.run.smithi121.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-12T08:38:16.106 INFO:teuthology.orchestra.run.smithi061.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-12T08:38:16.107 INFO:teuthology.orchestra.run.smithi061.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-12T08:38:16.110 INFO:teuthology.orchestra.run.smithi061.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-12T08:38:16.110 INFO:teuthology.orchestra.run.smithi061.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-12T08:38:16.111 INFO:teuthology.orchestra.run.smithi061.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-12T08:38:16.112 INFO:teuthology.orchestra.run.smithi130.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-12T08:38:16.114 INFO:teuthology.orchestra.run.smithi121.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-12T08:38:16.114 INFO:teuthology.orchestra.run.smithi121.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-12T08:38:16.115 INFO:teuthology.orchestra.run.smithi130.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-12T08:38:16.120 INFO:teuthology.orchestra.run.smithi130.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-12T08:38:16.121 INFO:teuthology.orchestra.run.smithi111.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-12T08:38:16.124 INFO:teuthology.orchestra.run.smithi061.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-12T08:38:16.130 INFO:teuthology.orchestra.run.smithi061.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-12T08:38:16.136 INFO:teuthology.orchestra.run.smithi130.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-07-12T08:38:16.136 INFO:teuthology.orchestra.run.smithi130.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-12T08:38:16.137 INFO:teuthology.orchestra.run.smithi130.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-12T08:38:16.137 INFO:teuthology.orchestra.run.smithi121.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-12T08:38:16.142 INFO:teuthology.orchestra.run.smithi121.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-12T08:38:16.145 INFO:teuthology.orchestra.run.smithi121.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-12T08:38:16.150 INFO:teuthology.orchestra.run.smithi061.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-324-g3497be4c-1focal [365 kB] 2022-07-12T08:38:16.151 INFO:teuthology.orchestra.run.smithi121.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-12T08:38:16.151 INFO:teuthology.orchestra.run.smithi121.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-12T08:38:16.152 INFO:teuthology.orchestra.run.smithi121.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-12T08:38:16.155 INFO:teuthology.orchestra.run.smithi121.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-12T08:38:16.155 INFO:teuthology.orchestra.run.smithi061.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-12T08:38:16.156 INFO:teuthology.orchestra.run.smithi121.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-12T08:38:16.164 INFO:teuthology.orchestra.run.smithi061.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-12T08:38:16.164 INFO:teuthology.orchestra.run.smithi061.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-12T08:38:16.165 INFO:teuthology.orchestra.run.smithi061.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-12T08:38:16.165 INFO:teuthology.orchestra.run.smithi061.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-12T08:38:16.166 INFO:teuthology.orchestra.run.smithi061.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-12T08:38:16.173 INFO:teuthology.orchestra.run.smithi121.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-12T08:38:16.175 INFO:teuthology.orchestra.run.smithi130.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-12T08:38:16.177 INFO:teuthology.orchestra.run.smithi121.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-324-g3497be4c-1focal [463 kB] 2022-07-12T08:38:16.177 INFO:teuthology.orchestra.run.smithi121.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-12T08:38:16.178 INFO:teuthology.orchestra.run.smithi130.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-12T08:38:16.179 INFO:teuthology.orchestra.run.smithi130.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-12T08:38:16.184 INFO:teuthology.orchestra.run.smithi111.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-324-g3497be4c-1focal [463 kB] 2022-07-12T08:38:16.187 INFO:teuthology.orchestra.run.smithi061.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-12T08:38:16.187 INFO:teuthology.orchestra.run.smithi061.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-12T08:38:16.189 INFO:teuthology.orchestra.run.smithi111.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-12T08:38:16.191 INFO:teuthology.orchestra.run.smithi061.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-12T08:38:16.192 INFO:teuthology.orchestra.run.smithi111.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-12T08:38:16.193 INFO:teuthology.orchestra.run.smithi061.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-12T08:38:16.203 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-12T08:38:16.208 INFO:teuthology.orchestra.run.smithi061.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-324-g3497be4c-1focal [20.8 kB] 2022-07-12T08:38:16.209 INFO:teuthology.orchestra.run.smithi061.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-324-g3497be4c-1focal [156 kB] 2022-07-12T08:38:16.210 INFO:teuthology.orchestra.run.smithi121.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-12T08:38:16.210 INFO:teuthology.orchestra.run.smithi121.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-12T08:38:16.211 INFO:teuthology.orchestra.run.smithi121.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-12T08:38:16.211 INFO:teuthology.orchestra.run.smithi121.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-12T08:38:16.219 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-12T08:38:16.223 INFO:teuthology.orchestra.run.smithi121.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-12T08:38:16.223 INFO:teuthology.orchestra.run.smithi121.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-12T08:38:16.224 INFO:teuthology.orchestra.run.smithi121.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-12T08:38:16.224 INFO:teuthology.orchestra.run.smithi121.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-12T08:38:16.225 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-12T08:38:16.227 INFO:teuthology.orchestra.run.smithi130.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-324-g3497be4c-1focal [463 kB] 2022-07-12T08:38:16.229 INFO:teuthology.orchestra.run.smithi061.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-12T08:38:16.229 INFO:teuthology.orchestra.run.smithi061.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-12T08:38:16.230 INFO:teuthology.orchestra.run.smithi061.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-12T08:38:16.233 INFO:teuthology.orchestra.run.smithi061.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-12T08:38:16.234 INFO:teuthology.orchestra.run.smithi121.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-12T08:38:16.240 INFO:teuthology.orchestra.run.smithi121.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-12T08:38:16.241 INFO:teuthology.orchestra.run.smithi121.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-324-g3497be4c-1focal [365 kB] 2022-07-12T08:38:16.243 INFO:teuthology.orchestra.run.smithi111.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-12T08:38:16.243 INFO:teuthology.orchestra.run.smithi111.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-12T08:38:16.244 INFO:teuthology.orchestra.run.smithi111.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-12T08:38:16.244 INFO:teuthology.orchestra.run.smithi111.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-12T08:38:16.245 INFO:teuthology.orchestra.run.smithi111.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-12T08:38:16.246 INFO:teuthology.orchestra.run.smithi111.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-12T08:38:16.246 INFO:teuthology.orchestra.run.smithi111.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-12T08:38:16.247 INFO:teuthology.orchestra.run.smithi111.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-324-g3497be4c-1focal [365 kB] 2022-07-12T08:38:16.249 INFO:teuthology.orchestra.run.smithi111.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-12T08:38:16.250 INFO:teuthology.orchestra.run.smithi111.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-12T08:38:16.251 INFO:teuthology.orchestra.run.smithi137.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-12T08:38:16.255 INFO:teuthology.orchestra.run.smithi130.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-12T08:38:16.257 INFO:teuthology.orchestra.run.smithi130.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-12T08:38:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-12T08:38:16.259 INFO:teuthology.orchestra.run.smithi121.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-12T08:38:16.260 INFO:teuthology.orchestra.run.smithi137.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-12T08:38:16.260 INFO:teuthology.orchestra.run.smithi137.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-12T08:38:16.261 INFO:teuthology.orchestra.run.smithi130.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-12T08:38:16.261 INFO:teuthology.orchestra.run.smithi061.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-324-g3497be4c-1focal [29.4 kB] 2022-07-12T08:38:16.262 INFO:teuthology.orchestra.run.smithi061.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-324-g3497be4c-1focal [363 kB] 2022-07-12T08:38:16.262 INFO:teuthology.orchestra.run.smithi130.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-12T08:38:16.266 INFO:teuthology.orchestra.run.smithi130.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-12T08:38:16.266 INFO:teuthology.orchestra.run.smithi130.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-12T08:38:16.268 INFO:teuthology.orchestra.run.smithi111.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-12T08:38:16.271 INFO:teuthology.orchestra.run.smithi061.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-324-g3497be4c-1focal [3,148 kB] 2022-07-12T08:38:16.272 INFO:teuthology.orchestra.run.smithi137.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2022-07-12T08:38:16.273 INFO:teuthology.orchestra.run.smithi111.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-12T08:38:16.274 INFO:teuthology.orchestra.run.smithi137.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-12T08:38:16.276 INFO:teuthology.orchestra.run.smithi111.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-12T08:38:16.276 INFO:teuthology.orchestra.run.smithi137.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-12T08:38:16.278 INFO:teuthology.orchestra.run.smithi111.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-12T08:38:16.282 INFO:teuthology.orchestra.run.smithi111.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-12T08:38:16.284 INFO:teuthology.orchestra.run.smithi111.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-12T08:38:16.285 INFO:teuthology.orchestra.run.smithi130.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-12T08:38:16.285 INFO:teuthology.orchestra.run.smithi111.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-12T08:38:16.288 INFO:teuthology.orchestra.run.smithi111.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-12T08:38:16.290 INFO:teuthology.orchestra.run.smithi111.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-12T08:38:16.292 INFO:teuthology.orchestra.run.smithi130.stdout:Get:30 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-324-g3497be4c-1focal [365 kB] 2022-07-12T08:38:16.298 INFO:teuthology.orchestra.run.smithi111.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-12T08:38:16.300 INFO:teuthology.orchestra.run.smithi121.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-324-g3497be4c-1focal [20.8 kB] 2022-07-12T08:38:16.301 INFO:teuthology.orchestra.run.smithi121.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-324-g3497be4c-1focal [156 kB] 2022-07-12T08:38:16.311 INFO:teuthology.orchestra.run.smithi111.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-12T08:38:16.312 INFO:teuthology.orchestra.run.smithi111.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-324-g3497be4c-1focal [20.8 kB] 2022-07-12T08:38:16.312 INFO:teuthology.orchestra.run.smithi111.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-12T08:38:16.312 INFO:teuthology.orchestra.run.smithi111.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-12T08:38:16.313 INFO:teuthology.orchestra.run.smithi111.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-324-g3497be4c-1focal [156 kB] 2022-07-12T08:38:16.317 INFO:teuthology.orchestra.run.smithi137.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-12T08:38:16.318 INFO:teuthology.orchestra.run.smithi111.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-12T08:38:16.319 INFO:teuthology.orchestra.run.smithi111.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-324-g3497be4c-1focal [29.4 kB] 2022-07-12T08:38:16.320 INFO:teuthology.orchestra.run.smithi111.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-324-g3497be4c-1focal [363 kB] 2022-07-12T08:38:16.321 INFO:teuthology.orchestra.run.smithi137.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-12T08:38:16.322 INFO:teuthology.orchestra.run.smithi137.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-12T08:38:16.334 INFO:teuthology.orchestra.run.smithi111.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-12T08:38:16.335 INFO:teuthology.orchestra.run.smithi111.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-12T08:38:16.338 INFO:teuthology.orchestra.run.smithi111.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-12T08:38:16.338 INFO:teuthology.orchestra.run.smithi111.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-12T08:38:16.338 INFO:teuthology.orchestra.run.smithi130.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-12T08:38:16.339 INFO:teuthology.orchestra.run.smithi130.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-12T08:38:16.339 INFO:teuthology.orchestra.run.smithi130.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-12T08:38:16.342 INFO:teuthology.orchestra.run.smithi111.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-12T08:38:16.345 INFO:teuthology.orchestra.run.smithi111.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-12T08:38:16.346 INFO:teuthology.orchestra.run.smithi121.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-324-g3497be4c-1focal [29.4 kB] 2022-07-12T08:38:16.352 INFO:teuthology.orchestra.run.smithi111.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-12T08:38:16.353 INFO:teuthology.orchestra.run.smithi121.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-324-g3497be4c-1focal [363 kB] 2022-07-12T08:38:16.354 INFO:teuthology.orchestra.run.smithi111.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-12T08:38:16.355 INFO:teuthology.orchestra.run.smithi130.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-324-g3497be4c-1focal [20.8 kB] 2022-07-12T08:38:16.359 INFO:teuthology.orchestra.run.smithi130.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-324-g3497be4c-1focal [156 kB] 2022-07-12T08:38:16.371 INFO:teuthology.orchestra.run.smithi121.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-324-g3497be4c-1focal [3,148 kB] 2022-07-12T08:38:16.382 INFO:teuthology.orchestra.run.smithi111.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-324-g3497be4c-1focal [3,148 kB] 2022-07-12T08:38:16.395 INFO:teuthology.orchestra.run.smithi137.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-12T08:38:16.398 INFO:teuthology.orchestra.run.smithi137.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-12T08:38:16.399 INFO:teuthology.orchestra.run.smithi137.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-12T08:38:16.403 INFO:teuthology.orchestra.run.smithi137.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-12T08:38:16.407 INFO:teuthology.orchestra.run.smithi137.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-12T08:38:16.408 INFO:teuthology.orchestra.run.smithi137.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-12T08:38:16.415 INFO:teuthology.orchestra.run.smithi130.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-12T08:38:16.415 INFO:teuthology.orchestra.run.smithi130.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-324-g3497be4c-1focal [29.4 kB] 2022-07-12T08:38:16.417 INFO:teuthology.orchestra.run.smithi130.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-324-g3497be4c-1focal [363 kB] 2022-07-12T08:38:16.420 INFO:teuthology.orchestra.run.smithi130.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-12T08:38:16.423 INFO:teuthology.orchestra.run.smithi130.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-12T08:38:16.424 INFO:teuthology.orchestra.run.smithi130.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-12T08:38:16.426 INFO:teuthology.orchestra.run.smithi137.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-12T08:38:16.427 INFO:teuthology.orchestra.run.smithi130.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-12T08:38:16.429 INFO:teuthology.orchestra.run.smithi130.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-324-g3497be4c-1focal [3,148 kB] 2022-07-12T08:38:16.431 INFO:teuthology.orchestra.run.smithi130.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-12T08:38:16.433 INFO:teuthology.orchestra.run.smithi130.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-12T08:38:16.476 INFO:teuthology.orchestra.run.smithi137.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-12T08:38:16.479 INFO:teuthology.orchestra.run.smithi137.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-12T08:38:16.479 INFO:teuthology.orchestra.run.smithi137.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-12T08:38:16.493 INFO:teuthology.orchestra.run.smithi130.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-12T08:38:16.495 INFO:teuthology.orchestra.run.smithi130.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-12T08:38:16.555 INFO:teuthology.orchestra.run.smithi137.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-12T08:38:16.562 INFO:teuthology.orchestra.run.smithi137.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-12T08:38:16.565 INFO:teuthology.orchestra.run.smithi137.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-12T08:38:16.566 INFO:teuthology.orchestra.run.smithi137.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-12T08:38:16.569 INFO:teuthology.orchestra.run.smithi137.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-12T08:38:16.573 INFO:teuthology.orchestra.run.smithi137.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-12T08:38:16.574 INFO:teuthology.orchestra.run.smithi137.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-12T08:38:16.635 INFO:teuthology.orchestra.run.smithi137.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-12T08:38:16.636 INFO:teuthology.orchestra.run.smithi137.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-12T08:38:16.732 INFO:teuthology.orchestra.run.smithi061.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-324-g3497be4c-1focal [116 kB] 2022-07-12T08:38:16.733 INFO:teuthology.orchestra.run.smithi061.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-324-g3497be4c-1focal [361 kB] 2022-07-12T08:38:16.768 INFO:teuthology.orchestra.run.smithi061.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-324-g3497be4c-1focal [20.1 MB] 2022-07-12T08:38:16.804 INFO:teuthology.orchestra.run.smithi121.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-324-g3497be4c-1focal [116 kB] 2022-07-12T08:38:16.816 INFO:teuthology.orchestra.run.smithi111.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-324-g3497be4c-1focal [116 kB] 2022-07-12T08:38:16.829 INFO:teuthology.orchestra.run.smithi130.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-12T08:38:16.830 INFO:teuthology.orchestra.run.smithi111.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-324-g3497be4c-1focal [361 kB] 2022-07-12T08:38:16.834 INFO:teuthology.orchestra.run.smithi121.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-324-g3497be4c-1focal [361 kB] 2022-07-12T08:38:16.866 INFO:teuthology.orchestra.run.smithi121.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-324-g3497be4c-1focal [20.1 MB] 2022-07-12T08:38:16.876 INFO:teuthology.orchestra.run.smithi111.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-324-g3497be4c-1focal [20.1 MB] 2022-07-12T08:38:16.882 INFO:teuthology.orchestra.run.smithi130.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-12T08:38:16.885 INFO:teuthology.orchestra.run.smithi130.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-12T08:38:16.906 INFO:teuthology.orchestra.run.smithi130.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-324-g3497be4c-1focal [116 kB] 2022-07-12T08:38:16.910 INFO:teuthology.orchestra.run.smithi130.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-324-g3497be4c-1focal [361 kB] 2022-07-12T08:38:16.922 INFO:teuthology.orchestra.run.smithi130.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-12T08:38:16.923 INFO:teuthology.orchestra.run.smithi130.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-12T08:38:16.947 INFO:teuthology.orchestra.run.smithi130.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-12T08:38:16.948 INFO:teuthology.orchestra.run.smithi130.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-12T08:38:16.949 INFO:teuthology.orchestra.run.smithi130.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-12T08:38:16.951 INFO:teuthology.orchestra.run.smithi130.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-12T08:38:16.952 INFO:teuthology.orchestra.run.smithi130.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-12T08:38:16.970 INFO:teuthology.orchestra.run.smithi130.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-324-g3497be4c-1focal [20.1 MB] 2022-07-12T08:38:16.971 INFO:teuthology.orchestra.run.smithi137.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-12T08:38:16.983 INFO:teuthology.orchestra.run.smithi130.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-12T08:38:16.984 INFO:teuthology.orchestra.run.smithi130.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-12T08:38:17.023 INFO:teuthology.orchestra.run.smithi137.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-12T08:38:17.024 INFO:teuthology.orchestra.run.smithi137.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-12T08:38:17.066 INFO:teuthology.orchestra.run.smithi130.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-12T08:38:17.068 INFO:teuthology.orchestra.run.smithi137.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-12T08:38:17.068 INFO:teuthology.orchestra.run.smithi137.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-12T08:38:17.069 INFO:teuthology.orchestra.run.smithi137.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-12T08:38:17.069 INFO:teuthology.orchestra.run.smithi137.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-12T08:38:17.069 INFO:teuthology.orchestra.run.smithi137.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-12T08:38:17.070 INFO:teuthology.orchestra.run.smithi137.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-12T08:38:17.070 INFO:teuthology.orchestra.run.smithi137.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-12T08:38:17.072 INFO:teuthology.orchestra.run.smithi130.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-12T08:38:17.072 INFO:teuthology.orchestra.run.smithi130.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-12T08:38:17.079 INFO:teuthology.orchestra.run.smithi130.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-12T08:38:17.080 INFO:teuthology.orchestra.run.smithi130.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-12T08:38:17.080 INFO:teuthology.orchestra.run.smithi130.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-12T08:38:17.081 INFO:teuthology.orchestra.run.smithi130.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-12T08:38:17.082 INFO:teuthology.orchestra.run.smithi130.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-12T08:38:17.129 INFO:teuthology.orchestra.run.smithi137.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-12T08:38:17.130 INFO:teuthology.orchestra.run.smithi137.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-12T08:38:17.143 INFO:teuthology.orchestra.run.smithi130.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-12T08:38:17.161 INFO:teuthology.orchestra.run.smithi130.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-12T08:38:17.183 INFO:teuthology.orchestra.run.smithi137.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-324-g3497be4c-1focal [3,257 kB] 2022-07-12T08:38:17.213 INFO:teuthology.orchestra.run.smithi137.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-12T08:38:17.218 INFO:teuthology.orchestra.run.smithi137.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-12T08:38:17.219 INFO:teuthology.orchestra.run.smithi137.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-12T08:38:17.221 INFO:teuthology.orchestra.run.smithi137.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-12T08:38:17.223 INFO:teuthology.orchestra.run.smithi137.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-12T08:38:17.224 INFO:teuthology.orchestra.run.smithi130.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-12T08:38:17.224 INFO:teuthology.orchestra.run.smithi130.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-12T08:38:17.225 INFO:teuthology.orchestra.run.smithi130.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-12T08:38:17.226 INFO:teuthology.orchestra.run.smithi130.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-12T08:38:17.227 INFO:teuthology.orchestra.run.smithi137.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-12T08:38:17.228 INFO:teuthology.orchestra.run.smithi137.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-12T08:38:17.230 INFO:teuthology.orchestra.run.smithi137.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-12T08:38:17.244 INFO:teuthology.orchestra.run.smithi130.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-12T08:38:17.244 INFO:teuthology.orchestra.run.smithi130.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-12T08:38:17.245 INFO:teuthology.orchestra.run.smithi130.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-12T08:38:17.246 INFO:teuthology.orchestra.run.smithi130.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-12T08:38:17.290 INFO:teuthology.orchestra.run.smithi137.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-12T08:38:17.296 INFO:teuthology.orchestra.run.smithi137.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-12T08:38:17.302 INFO:teuthology.orchestra.run.smithi130.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-12T08:38:17.304 INFO:teuthology.orchestra.run.smithi130.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-12T08:38:17.374 INFO:teuthology.orchestra.run.smithi137.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-12T08:38:17.375 INFO:teuthology.orchestra.run.smithi137.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-12T08:38:17.375 INFO:teuthology.orchestra.run.smithi137.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-12T08:38:17.375 INFO:teuthology.orchestra.run.smithi137.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-12T08:38:17.382 INFO:teuthology.orchestra.run.smithi130.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-12T08:38:17.385 INFO:teuthology.orchestra.run.smithi130.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-12T08:38:17.390 INFO:teuthology.orchestra.run.smithi137.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-12T08:38:17.391 INFO:teuthology.orchestra.run.smithi137.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-12T08:38:17.393 INFO:teuthology.orchestra.run.smithi137.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-12T08:38:17.393 INFO:teuthology.orchestra.run.smithi137.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-12T08:38:17.449 INFO:teuthology.orchestra.run.smithi137.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-12T08:38:17.452 INFO:teuthology.orchestra.run.smithi137.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-12T08:38:17.530 INFO:teuthology.orchestra.run.smithi137.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-12T08:38:17.532 INFO:teuthology.orchestra.run.smithi137.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-12T08:38:19.439 INFO:teuthology.orchestra.run.smithi061.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-324-g3497be4c-1focal [5,345 kB] 2022-07-12T08:38:19.678 INFO:teuthology.orchestra.run.smithi121.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-324-g3497be4c-1focal [5,345 kB] 2022-07-12T08:38:19.800 INFO:teuthology.orchestra.run.smithi137.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-324-g3497be4c-1focal [463 kB] 2022-07-12T08:38:19.801 INFO:teuthology.orchestra.run.smithi111.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-324-g3497be4c-1focal [5,345 kB] 2022-07-12T08:38:19.965 INFO:teuthology.orchestra.run.smithi130.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-324-g3497be4c-1focal [5,345 kB] 2022-07-12T08:38:20.084 INFO:teuthology.orchestra.run.smithi061.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-324-g3497be4c-1focal [163 kB] 2022-07-12T08:38:20.121 INFO:teuthology.orchestra.run.smithi061.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-324-g3497be4c-1focal [1,402 kB] 2022-07-12T08:38:20.140 INFO:teuthology.orchestra.run.smithi137.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-324-g3497be4c-1focal [365 kB] 2022-07-12T08:38:20.286 INFO:teuthology.orchestra.run.smithi121.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-324-g3497be4c-1focal [163 kB] 2022-07-12T08:38:20.289 INFO:teuthology.orchestra.run.smithi121.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-324-g3497be4c-1focal [1,402 kB] 2022-07-12T08:38:20.334 INFO:teuthology.orchestra.run.smithi061.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-324-g3497be4c-1focal [6,163 kB] 2022-07-12T08:38:20.342 INFO:teuthology.orchestra.run.smithi137.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-324-g3497be4c-1focal [20.8 kB] 2022-07-12T08:38:20.378 INFO:teuthology.orchestra.run.smithi111.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-324-g3497be4c-1focal [163 kB] 2022-07-12T08:38:20.386 INFO:teuthology.orchestra.run.smithi111.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-324-g3497be4c-1focal [1,402 kB] 2022-07-12T08:38:20.401 INFO:teuthology.orchestra.run.smithi137.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-324-g3497be4c-1focal [156 kB] 2022-07-12T08:38:20.433 INFO:teuthology.orchestra.run.smithi121.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-324-g3497be4c-1focal [6,163 kB] 2022-07-12T08:38:20.485 INFO:teuthology.orchestra.run.smithi137.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-324-g3497be4c-1focal [29.4 kB] 2022-07-12T08:38:20.486 INFO:teuthology.orchestra.run.smithi137.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-324-g3497be4c-1focal [363 kB] 2022-07-12T08:38:20.556 INFO:teuthology.orchestra.run.smithi111.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-324-g3497be4c-1focal [6,163 kB] 2022-07-12T08:38:20.732 INFO:teuthology.orchestra.run.smithi130.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-324-g3497be4c-1focal [163 kB] 2022-07-12T08:38:20.757 INFO:teuthology.orchestra.run.smithi130.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-324-g3497be4c-1focal [1,402 kB] 2022-07-12T08:38:20.770 INFO:teuthology.orchestra.run.smithi137.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-324-g3497be4c-1focal [3,148 kB] 2022-07-12T08:38:20.969 INFO:teuthology.orchestra.run.smithi130.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-324-g3497be4c-1focal [6,163 kB] 2022-07-12T08:38:21.158 INFO:teuthology.orchestra.run.smithi121.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-324-g3497be4c-1focal [24.0 MB] 2022-07-12T08:38:21.279 INFO:teuthology.orchestra.run.smithi061.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-324-g3497be4c-1focal [24.0 MB] 2022-07-12T08:38:21.321 INFO:teuthology.orchestra.run.smithi111.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-324-g3497be4c-1focal [24.0 MB] 2022-07-12T08:38:22.009 INFO:teuthology.orchestra.run.smithi130.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-324-g3497be4c-1focal [24.0 MB] 2022-07-12T08:38:22.795 INFO:teuthology.orchestra.run.smithi137.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-324-g3497be4c-1focal [116 kB] 2022-07-12T08:38:22.852 INFO:teuthology.orchestra.run.smithi137.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-324-g3497be4c-1focal [361 kB] 2022-07-12T08:38:23.036 INFO:teuthology.orchestra.run.smithi137.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-324-g3497be4c-1focal [20.1 MB] 2022-07-12T08:38:24.231 INFO:teuthology.orchestra.run.smithi121.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-324-g3497be4c-1focal [4,028 B] 2022-07-12T08:38:24.232 INFO:teuthology.orchestra.run.smithi121.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-324-g3497be4c-1focal [537 kB] 2022-07-12T08:38:24.245 INFO:teuthology.orchestra.run.smithi121.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-324-g3497be4c-1focal [2,007 kB] 2022-07-12T08:38:24.448 INFO:teuthology.orchestra.run.smithi111.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-324-g3497be4c-1focal [4,028 B] 2022-07-12T08:38:24.449 INFO:teuthology.orchestra.run.smithi111.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-324-g3497be4c-1focal [537 kB] 2022-07-12T08:38:24.450 INFO:teuthology.orchestra.run.smithi121.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-324-g3497be4c-1focal [53.2 kB] 2022-07-12T08:38:24.451 INFO:teuthology.orchestra.run.smithi121.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-324-g3497be4c-1focal [59.1 kB] 2022-07-12T08:38:24.461 INFO:teuthology.orchestra.run.smithi121.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-324-g3497be4c-1focal [3,218 kB] 2022-07-12T08:38:24.462 INFO:teuthology.orchestra.run.smithi111.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-324-g3497be4c-1focal [2,007 kB] 2022-07-12T08:38:24.713 INFO:teuthology.orchestra.run.smithi111.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-324-g3497be4c-1focal [53.2 kB] 2022-07-12T08:38:24.713 INFO:teuthology.orchestra.run.smithi111.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-324-g3497be4c-1focal [59.1 kB] 2022-07-12T08:38:24.714 INFO:teuthology.orchestra.run.smithi111.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-324-g3497be4c-1focal [3,218 kB] 2022-07-12T08:38:24.846 INFO:teuthology.orchestra.run.smithi121.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-324-g3497be4c-1focal [34.4 kB] 2022-07-12T08:38:24.847 INFO:teuthology.orchestra.run.smithi121.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-324-g3497be4c-1focal [7,891 kB] 2022-07-12T08:38:24.903 INFO:teuthology.orchestra.run.smithi061.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-324-g3497be4c-1focal [4,028 B] 2022-07-12T08:38:24.903 INFO:teuthology.orchestra.run.smithi061.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-324-g3497be4c-1focal [537 kB] 2022-07-12T08:38:24.941 INFO:teuthology.orchestra.run.smithi061.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-324-g3497be4c-1focal [2,007 kB] 2022-07-12T08:38:25.105 INFO:teuthology.orchestra.run.smithi111.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-324-g3497be4c-1focal [34.4 kB] 2022-07-12T08:38:25.106 INFO:teuthology.orchestra.run.smithi111.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-324-g3497be4c-1focal [7,891 kB] 2022-07-12T08:38:25.214 INFO:teuthology.orchestra.run.smithi061.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-324-g3497be4c-1focal [53.2 kB] 2022-07-12T08:38:25.215 INFO:teuthology.orchestra.run.smithi061.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-324-g3497be4c-1focal [59.1 kB] 2022-07-12T08:38:25.254 INFO:teuthology.orchestra.run.smithi061.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-324-g3497be4c-1focal [3,218 kB] 2022-07-12T08:38:25.754 INFO:teuthology.orchestra.run.smithi061.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-324-g3497be4c-1focal [34.4 kB] 2022-07-12T08:38:25.755 INFO:teuthology.orchestra.run.smithi061.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-324-g3497be4c-1focal [7,891 kB] 2022-07-12T08:38:25.857 INFO:teuthology.orchestra.run.smithi121.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-324-g3497be4c-1focal [4,148 B] 2022-07-12T08:38:25.857 INFO:teuthology.orchestra.run.smithi121.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-324-g3497be4c-1focal [40.2 MB] 2022-07-12T08:38:26.008 INFO:teuthology.orchestra.run.smithi130.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-324-g3497be4c-1focal [4,028 B] 2022-07-12T08:38:26.008 INFO:teuthology.orchestra.run.smithi130.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-324-g3497be4c-1focal [537 kB] 2022-07-12T08:38:26.019 INFO:teuthology.orchestra.run.smithi130.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-324-g3497be4c-1focal [2,007 kB] 2022-07-12T08:38:26.207 INFO:teuthology.orchestra.run.smithi111.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-324-g3497be4c-1focal [4,148 B] 2022-07-12T08:38:26.208 INFO:teuthology.orchestra.run.smithi111.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-324-g3497be4c-1focal [40.2 MB] 2022-07-12T08:38:26.365 INFO:teuthology.orchestra.run.smithi130.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-324-g3497be4c-1focal [53.2 kB] 2022-07-12T08:38:26.365 INFO:teuthology.orchestra.run.smithi130.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-324-g3497be4c-1focal [59.1 kB] 2022-07-12T08:38:26.366 INFO:teuthology.orchestra.run.smithi130.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-324-g3497be4c-1focal [3,218 kB] 2022-07-12T08:38:26.836 INFO:teuthology.orchestra.run.smithi061.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-324-g3497be4c-1focal [4,148 B] 2022-07-12T08:38:26.836 INFO:teuthology.orchestra.run.smithi061.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-324-g3497be4c-1focal [40.2 MB] 2022-07-12T08:38:26.853 INFO:teuthology.orchestra.run.smithi130.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-324-g3497be4c-1focal [34.4 kB] 2022-07-12T08:38:26.854 INFO:teuthology.orchestra.run.smithi130.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-324-g3497be4c-1focal [7,891 kB] 2022-07-12T08:38:28.158 INFO:teuthology.orchestra.run.smithi130.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-324-g3497be4c-1focal [4,148 B] 2022-07-12T08:38:28.159 INFO:teuthology.orchestra.run.smithi130.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-324-g3497be4c-1focal [40.2 MB] 2022-07-12T08:38:30.354 INFO:teuthology.orchestra.run.smithi137.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-324-g3497be4c-1focal [5,345 kB] 2022-07-12T08:38:31.424 INFO:teuthology.orchestra.run.smithi121.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-324-g3497be4c-1focal [15.9 kB] 2022-07-12T08:38:31.425 INFO:teuthology.orchestra.run.smithi121.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-324-g3497be4c-1focal [18.1 kB] 2022-07-12T08:38:31.433 INFO:teuthology.orchestra.run.smithi121.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-324-g3497be4c-1focal [9,667 kB] 2022-07-12T08:38:31.854 INFO:teuthology.orchestra.run.smithi111.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-324-g3497be4c-1focal [15.9 kB] 2022-07-12T08:38:31.855 INFO:teuthology.orchestra.run.smithi111.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-324-g3497be4c-1focal [18.1 kB] 2022-07-12T08:38:31.856 INFO:teuthology.orchestra.run.smithi111.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-324-g3497be4c-1focal [9,667 kB] 2022-07-12T08:38:32.002 INFO:teuthology.orchestra.run.smithi137.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-324-g3497be4c-1focal [163 kB] 2022-07-12T08:38:32.079 INFO:teuthology.orchestra.run.smithi137.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-324-g3497be4c-1focal [1,402 kB] 2022-07-12T08:38:32.283 INFO:teuthology.orchestra.run.smithi061.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-324-g3497be4c-1focal [15.9 kB] 2022-07-12T08:38:32.283 INFO:teuthology.orchestra.run.smithi061.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-324-g3497be4c-1focal [18.1 kB] 2022-07-12T08:38:32.284 INFO:teuthology.orchestra.run.smithi061.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-324-g3497be4c-1focal [9,667 kB] 2022-07-12T08:38:32.458 INFO:teuthology.orchestra.run.smithi137.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-324-g3497be4c-1focal [6,163 kB] 2022-07-12T08:38:32.498 INFO:teuthology.orchestra.run.smithi121.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-324-g3497be4c-1focal [61.4 kB] 2022-07-12T08:38:32.813 INFO:teuthology.orchestra.run.smithi111.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-324-g3497be4c-1focal [61.4 kB] 2022-07-12T08:38:32.947 INFO:teuthology.orchestra.run.smithi061.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-324-g3497be4c-1focal [61.4 kB] 2022-07-12T08:38:33.143 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 153 MB in 17s (8,920 kB/s) 2022-07-12T08:38:33.243 INFO:teuthology.orchestra.run.smithi137.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-324-g3497be4c-1focal [24.0 MB] 2022-07-12T08:38:33.356 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-12T08:38:33.356 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138766 files and directories currently installed.) 2022-07-12T08:38:33.357 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-12T08:38:33.378 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-12T08:38:33.421 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 153 MB in 17s (8,783 kB/s) 2022-07-12T08:38:33.553 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 153 MB in 18s (8,701 kB/s) 2022-07-12T08:38:33.596 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-12T08:38:33.657 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138766 files and directories currently installed.) 2022-07-12T08:38:33.657 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-12T08:38:33.687 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-12T08:38:33.694 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-12T08:38:33.719 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-12T08:38:33.737 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-12T08:38:33.742 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138766 files and directories currently installed.) 2022-07-12T08:38:33.747 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-12T08:38:33.774 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:38:33.792 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-12T08:38:34.017 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-12T08:38:34.028 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-12T08:38:34.064 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:38:34.086 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cmd2. 2022-07-12T08:38:34.093 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-12T08:38:34.097 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-12T08:38:34.117 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-12T08:38:34.156 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:38:34.173 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:38:34.348 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cmd2. 2022-07-12T08:38:34.376 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-12T08:38:34.398 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:38:34.441 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cmd2. 2022-07-12T08:38:34.453 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-12T08:38:34.457 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-12T08:38:34.486 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-12T08:38:34.533 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:38:34.537 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:38:34.778 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-12T08:38:34.792 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-12T08:38:34.876 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:38:34.891 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-12T08:38:34.906 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-12T08:38:34.925 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../004-librbd1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:34.988 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:38:35.049 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librbd1 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:35.077 INFO:teuthology.orchestra.run.smithi130.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-324-g3497be4c-1focal [15.9 kB] 2022-07-12T08:38:35.078 INFO:teuthology.orchestra.run.smithi130.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-324-g3497be4c-1focal [18.1 kB] 2022-07-12T08:38:35.079 INFO:teuthology.orchestra.run.smithi130.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-324-g3497be4c-1focal [9,667 kB] 2022-07-12T08:38:35.107 INFO:teuthology.orchestra.run.smithi137.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-324-g3497be4c-1focal [4,028 B] 2022-07-12T08:38:35.154 INFO:teuthology.orchestra.run.smithi137.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-324-g3497be4c-1focal [537 kB] 2022-07-12T08:38:35.169 INFO:teuthology.orchestra.run.smithi137.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-324-g3497be4c-1focal [2,007 kB] 2022-07-12T08:38:35.247 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../004-librbd1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:35.265 INFO:teuthology.orchestra.run.smithi137.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-324-g3497be4c-1focal [53.2 kB] 2022-07-12T08:38:35.279 INFO:teuthology.orchestra.run.smithi137.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-324-g3497be4c-1focal [59.1 kB] 2022-07-12T08:38:35.280 INFO:teuthology.orchestra.run.smithi137.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-324-g3497be4c-1focal [3,218 kB] 2022-07-12T08:38:35.306 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../004-librbd1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:35.373 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librbd1 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:35.440 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librbd1 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:35.481 INFO:teuthology.orchestra.run.smithi137.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-324-g3497be4c-1focal [34.4 kB] 2022-07-12T08:38:35.481 INFO:teuthology.orchestra.run.smithi137.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-324-g3497be4c-1focal [7,891 kB] 2022-07-12T08:38:35.572 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../005-librados2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:35.703 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librados2 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:35.752 INFO:teuthology.orchestra.run.smithi130.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-324-g3497be4c-1focal [61.4 kB] 2022-07-12T08:38:35.897 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../005-librados2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:35.976 INFO:teuthology.orchestra.run.smithi137.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-324-g3497be4c-1focal [4,148 B] 2022-07-12T08:38:35.976 INFO:teuthology.orchestra.run.smithi137.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-324-g3497be4c-1focal [40.2 MB] 2022-07-12T08:38:35.977 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../005-librados2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:36.052 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librados2 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:36.143 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librados2 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:36.358 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs2. 2022-07-12T08:38:36.374 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:36.391 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 153 MB in 20s (7,507 kB/s) 2022-07-12T08:38:36.420 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:36.527 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-12T08:38:36.585 INFO:teuthology.orchestra.run.smithi130.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138766 files and directories currently installed.) 2022-07-12T08:38:36.586 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-12T08:38:36.642 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-12T08:38:36.722 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libcephfs2. 2022-07-12T08:38:36.742 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:36.743 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rados. 2022-07-12T08:38:36.758 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../007-python3-rados_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:36.781 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:36.789 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs2. 2022-07-12T08:38:36.800 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:36.810 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:36.840 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:36.969 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-12T08:38:36.990 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-12T08:38:37.027 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:38:37.085 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rados. 2022-07-12T08:38:37.098 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../007-python3-rados_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:37.134 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rados. 2022-07-12T08:38:37.134 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.155 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../007-python3-rados_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:37.170 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-12T08:38:37.186 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:37.225 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.245 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.281 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cmd2. 2022-07-12T08:38:37.296 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-12T08:38:37.330 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:38:37.468 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-12T08:38:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:37.489 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cephfs. 2022-07-12T08:38:37.508 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:37.528 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.539 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.563 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-12T08:38:37.572 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:37.611 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.681 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-12T08:38:37.694 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-12T08:38:37.812 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:38:37.857 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cephfs. 2022-07-12T08:38:37.871 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:37.877 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-12T08:38:37.882 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:37.913 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cephfs. 2022-07-12T08:38:37.917 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.925 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:37.932 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:37.969 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:38.195 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../004-librbd1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:38.245 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-prettytable. 2022-07-12T08:38:38.259 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-12T08:38:38.260 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-12T08:38:38.261 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-12T08:38:38.279 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:38.280 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:38.297 INFO:teuthology.orchestra.run.smithi137.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-324-g3497be4c-1focal [15.9 kB] 2022-07-12T08:38:38.298 INFO:teuthology.orchestra.run.smithi137.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-324-g3497be4c-1focal [18.1 kB] 2022-07-12T08:38:38.298 INFO:teuthology.orchestra.run.smithi137.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-324-g3497be4c-1focal [9,667 kB] 2022-07-12T08:38:38.312 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-12T08:38:38.323 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:38.333 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:38.355 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librbd1 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:38.627 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-prettytable. 2022-07-12T08:38:38.645 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-prettytable. 2022-07-12T08:38:38.646 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-12T08:38:38.663 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-12T08:38:38.682 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-12T08:38:38.683 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rbd. 2022-07-12T08:38:38.708 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-12T08:38:38.709 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:38.709 INFO:teuthology.orchestra.run.smithi137.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/octopus/3497be4c2159a20e7a0f23fb5e2730060a4424c0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-324-g3497be4c-1focal [61.4 kB] 2022-07-12T08:38:38.805 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:38.867 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../005-librados2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:38.946 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rbd. 2022-07-12T08:38:38.967 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:38.980 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rbd. 2022-07-12T08:38:38.995 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:39.010 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:39.016 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librados2 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:39.038 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:39.108 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-12T08:38:39.125 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-12T08:38:39.158 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:38:39.296 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 153 MB in 23s (6,609 kB/s) 2022-07-12T08:38:39.413 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-12T08:38:39.416 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-12T08:38:39.430 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-12T08:38:39.438 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-12T08:38:39.465 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:38:39.467 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-12T08:38:39.474 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:38:39.513 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-12T08:38:39.522 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138766 files and directories currently installed.) 2022-07-12T08:38:39.532 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-12T08:38:39.532 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-12T08:38:39.571 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:38:39.605 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-12T08:38:39.704 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libcephfs2. 2022-07-12T08:38:39.727 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:39.752 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-12T08:38:39.770 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-12T08:38:39.794 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-12T08:38:39.795 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:39.818 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-12T08:38:39.819 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:38:39.849 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:38:39.945 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-12T08:38:39.958 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-12T08:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package librgw2. 2022-07-12T08:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../015-librgw2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.009 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:38:40.106 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rados. 2022-07-12T08:38:40.137 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../007-python3-rados_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.156 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.182 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librgw2. 2022-07-12T08:38:40.191 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package librgw2. 2022-07-12T08:38:40.200 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../015-librgw2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.219 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../015-librgw2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.239 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.248 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.271 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cmd2. 2022-07-12T08:38:40.297 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-12T08:38:40.319 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:38:40.500 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-12T08:38:40.520 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rgw. 2022-07-12T08:38:40.569 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.664 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-12T08:38:40.687 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-12T08:38:40.781 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:38:40.782 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rgw. 2022-07-12T08:38:40.798 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.799 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rgw. 2022-07-12T08:38:40.815 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.840 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.878 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cephfs. 2022-07-12T08:38:40.881 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.898 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-12T08:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-12T08:38:40.967 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:38:41.145 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-12T08:38:41.152 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../004-librbd1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:41.163 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-12T08:38:41.186 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-12T08:38:41.212 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:38:41.215 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-12T08:38:41.239 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:38:41.278 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librbd1 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:41.314 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-12T08:38:41.330 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:41.331 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liboath0. 2022-07-12T08:38:41.350 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-12T08:38:41.377 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:41.420 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-12T08:38:41.530 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liboath0. 2022-07-12T08:38:41.546 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-12T08:38:41.574 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liboath0. 2022-07-12T08:38:41.595 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-12T08:38:41.621 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-12T08:38:41.656 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-12T08:38:41.677 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-prettytable. 2022-07-12T08:38:41.696 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-12T08:38:41.734 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-12T08:38:41.735 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libradosstriper1. 2022-07-12T08:38:41.741 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:41.796 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:41.851 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../005-librados2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:41.947 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libradosstriper1. 2022-07-12T08:38:41.962 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:41.963 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librados2 (15.2.16-324-g3497be4c-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-12T08:38:41.965 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libradosstriper1. 2022-07-12T08:38:41.978 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rbd. 2022-07-12T08:38:41.986 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:42.002 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.003 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:42.036 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.048 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.161 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-common. 2022-07-12T08:38:42.185 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../020-ceph-common_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:42.219 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.411 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-common. 2022-07-12T08:38:42.428 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../020-ceph-common_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:42.450 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-12T08:38:42.461 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.462 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-12T08:38:42.489 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-common. 2022-07-12T08:38:42.503 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../020-ceph-common_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:42.508 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:38:42.539 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.617 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs2. 2022-07-12T08:38:42.639 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:42.680 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:42.866 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-12T08:38:42.878 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-12T08:38:42.953 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:38:43.011 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rados. 2022-07-12T08:38:43.034 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../007-python3-rados_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:43.075 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:43.294 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package librgw2. 2022-07-12T08:38:43.312 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../015-librgw2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:43.355 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:43.469 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-12T08:38:43.490 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:43.522 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:43.797 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cephfs. 2022-07-12T08:38:43.809 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:43.861 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:43.956 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rgw. 2022-07-12T08:38:43.963 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:44.006 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:44.200 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-12T08:38:44.218 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:44.261 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:44.379 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-base. 2022-07-12T08:38:44.379 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../021-ceph-base_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:44.383 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-12T08:38:44.399 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-12T08:38:44.442 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:44.474 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:38:44.616 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-base. 2022-07-12T08:38:44.631 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-prettytable. 2022-07-12T08:38:44.632 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../021-ceph-base_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:44.633 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-base. 2022-07-12T08:38:44.647 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-12T08:38:44.651 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../021-ceph-base_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:44.684 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:44.689 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-12T08:38:44.700 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:44.807 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liboath0. 2022-07-12T08:38:44.826 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-12T08:38:44.871 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-12T08:38:44.982 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rbd. 2022-07-12T08:38:44.991 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:45.025 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:45.203 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libradosstriper1. 2022-07-12T08:38:45.231 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:45.280 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:45.370 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cryptography. 2022-07-12T08:38:45.387 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-12T08:38:45.388 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-12T08:38:45.406 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-12T08:38:45.434 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:38:45.447 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:38:45.595 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cryptography. 2022-07-12T08:38:45.614 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-12T08:38:45.633 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cryptography. 2022-07-12T08:38:45.647 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-12T08:38:45.653 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:38:45.679 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:38:45.750 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-common. 2022-07-12T08:38:45.766 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../020-ceph-common_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:45.798 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-12T08:38:45.814 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-12T08:38:45.851 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-openssl. 2022-07-12T08:38:45.852 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:45.863 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-12T08:38:45.883 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:38:45.898 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-12T08:38:46.065 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-openssl. 2022-07-12T08:38:46.113 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-12T08:38:46.120 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-12T08:38:46.132 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-12T08:38:46.134 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:46.151 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-openssl. 2022-07-12T08:38:46.185 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-12T08:38:46.199 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:46.212 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-12T08:38:46.250 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package librgw2. 2022-07-12T08:38:46.265 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../015-librgw2_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:46.308 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:46.368 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-12T08:38:46.391 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:46.420 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:46.494 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-12T08:38:46.518 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:46.545 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:46.580 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-12T08:38:46.604 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-12T08:38:46.622 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:38:46.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-12T08:38:46.799 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-12T08:38:46.828 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rgw. 2022-07-12T08:38:46.848 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:46.849 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:38:46.882 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:46.950 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-12T08:38:46.959 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-12T08:38:46.965 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-12T08:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-12T08:38:47.007 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:38:47.018 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:38:47.125 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-12T08:38:47.140 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-12T08:38:47.183 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:38:47.202 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-12T08:38:47.207 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-12T08:38:47.280 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:38:47.334 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-12T08:38:47.368 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jwt. 2022-07-12T08:38:47.368 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-12T08:38:47.397 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:38:47.398 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-07-12T08:38:47.421 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:38:47.535 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jwt. 2022-07-12T08:38:47.551 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-07-12T08:38:47.596 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:38:47.649 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liboath0. 2022-07-12T08:38:47.683 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-12T08:38:47.684 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-logutils. 2022-07-12T08:38:47.684 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-12T08:38:47.724 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-12T08:38:47.725 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-12T08:38:47.757 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jwt. 2022-07-12T08:38:47.796 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-07-12T08:38:47.815 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:38:47.928 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-logutils. 2022-07-12T08:38:47.939 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-12T08:38:47.999 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-12T08:38:47.999 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-base. 2022-07-12T08:38:48.010 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../021-ceph-base_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:48.033 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libradosstriper1. 2022-07-12T08:38:48.043 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:48.058 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-12T08:38:48.060 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:48.072 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-12T08:38:48.085 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:48.126 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:38:48.314 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-12T08:38:48.328 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-12T08:38:48.378 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:38:48.450 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-mako. 2022-07-12T08:38:48.455 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-common. 2022-07-12T08:38:48.467 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-12T08:38:48.474 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../020-ceph-common_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:48.504 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:38:48.516 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:48.670 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-mako. 2022-07-12T08:38:48.687 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-12T08:38:48.734 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:38:48.787 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-logutils. 2022-07-12T08:38:48.801 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-12T08:38:48.859 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-12T08:38:48.862 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-12T08:38:48.867 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-12T08:38:48.906 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:38:48.992 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cryptography. 2022-07-12T08:38:49.002 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-12T08:38:49.068 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:38:49.115 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-12T08:38:49.116 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-12T08:38:49.158 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:38:49.169 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-12T08:38:49.178 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-12T08:38:49.221 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:38:49.478 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-openssl. 2022-07-12T08:38:49.492 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-12T08:38:49.496 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-12T08:38:49.510 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-12T08:38:49.551 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:38:49.572 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-12T08:38:49.583 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-mako. 2022-07-12T08:38:49.602 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-12T08:38:49.645 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:38:49.758 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-12T08:38:49.777 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-12T08:38:49.809 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:38:49.996 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-12T08:38:50.010 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-12T08:38:50.050 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:38:50.147 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webob. 2022-07-12T08:38:50.161 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-12T08:38:50.214 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-12T08:38:50.378 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-12T08:38:50.398 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-12T08:38:50.436 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:38:50.512 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webob. 2022-07-12T08:38:50.529 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-12T08:38:50.567 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-12T08:38:50.593 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-12T08:38:50.593 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-12T08:38:50.628 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:38:50.731 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webob. 2022-07-12T08:38:50.747 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-12T08:38:50.814 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-12T08:38:50.833 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-12T08:38:50.849 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:50.912 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-bs4. 2022-07-12T08:38:50.918 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-12T08:38:50.928 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:50.929 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-12T08:38:50.946 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-12T08:38:50.962 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-base. 2022-07-12T08:38:50.982 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../021-ceph-base_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:50.983 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:38:50.993 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-12T08:38:51.038 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:51.194 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-12T08:38:51.209 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-12T08:38:51.251 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:38:51.309 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bs4. 2022-07-12T08:38:51.324 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-12T08:38:51.333 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-12T08:38:51.333 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-12T08:38:51.362 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-waitress. 2022-07-12T08:38:51.372 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:38:51.373 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-12T08:38:51.378 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-12T08:38:51.431 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:38:51.614 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-bs4. 2022-07-12T08:38:51.622 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-12T08:38:51.662 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-12T08:38:51.706 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-12T08:38:51.730 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-12T08:38:51.760 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-waitress. 2022-07-12T08:38:51.778 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-tempita. 2022-07-12T08:38:51.779 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-12T08:38:51.788 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-12T08:38:51.816 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:38:51.817 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:38:51.830 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-12T08:38:51.934 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cryptography. 2022-07-12T08:38:51.934 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-12T08:38:51.964 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:38:52.058 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-waitress. 2022-07-12T08:38:52.074 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-12T08:38:52.121 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:38:52.146 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tempita. 2022-07-12T08:38:52.162 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-12T08:38:52.169 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jwt. 2022-07-12T08:38:52.184 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-paste. 2022-07-12T08:38:52.184 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-07-12T08:38:52.211 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-12T08:38:52.212 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-12T08:38:52.218 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:38:52.251 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:38:52.366 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-openssl. 2022-07-12T08:38:52.380 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-12T08:38:52.410 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-12T08:38:52.459 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-tempita. 2022-07-12T08:38:52.479 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-12T08:38:52.518 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-12T08:38:52.526 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-paste. 2022-07-12T08:38:52.542 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-12T08:38:52.573 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-12T08:38:52.574 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-logutils. 2022-07-12T08:38:52.589 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-12T08:38:52.591 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-12T08:38:52.594 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:38:52.634 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:38:52.655 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-12T08:38:52.713 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-12T08:38:52.730 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:38:52.775 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:52.854 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-paste. 2022-07-12T08:38:52.863 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-12T08:38:52.937 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-12T08:38:52.938 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:38:52.960 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-12T08:38:52.961 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-12T08:38:53.007 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-12T08:38:53.008 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:38:53.008 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-12T08:38:53.009 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-12T08:38:53.037 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:38:53.054 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:38:53.167 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-12T08:38:53.193 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-12T08:38:53.229 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:38:53.232 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-12T08:38:53.257 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-12T08:38:53.290 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-12T08:38:53.291 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:38:53.306 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-12T08:38:53.366 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webtest. 2022-07-12T08:38:53.382 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-12T08:38:53.383 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:38:53.393 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-mako. 2022-07-12T08:38:53.410 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-12T08:38:53.425 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-12T08:38:53.453 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:38:53.561 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-12T08:38:53.580 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-12T08:38:53.619 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:38:53.633 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-12T08:38:53.653 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-12T08:38:53.669 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webtest. 2022-07-12T08:38:53.689 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-12T08:38:53.695 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:38:53.757 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-12T08:38:53.763 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pecan. 2022-07-12T08:38:53.778 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-12T08:38:53.824 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:38:53.830 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-12T08:38:53.847 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-12T08:38:53.888 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:38:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jwt. 2022-07-12T08:38:54.019 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webtest. 2022-07-12T08:38:54.020 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2022-07-12T08:38:54.022 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-12T08:38:54.060 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:38:54.061 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-12T08:38:54.088 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pecan. 2022-07-12T08:38:54.108 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-12T08:38:54.150 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:38:54.196 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-12T08:38:54.211 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-12T08:38:54.225 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-12T08:38:54.240 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-12T08:38:54.253 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:38:54.285 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:38:54.407 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-logutils. 2022-07-12T08:38:54.426 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pecan. 2022-07-12T08:38:54.427 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-12T08:38:54.446 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-12T08:38:54.459 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-12T08:38:54.482 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:38:54.544 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-12T08:38:54.564 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-12T08:38:54.593 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-webob. 2022-07-12T08:38:54.608 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:38:54.612 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr. 2022-07-12T08:38:54.613 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-12T08:38:54.631 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:54.677 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-12T08:38:54.780 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-12T08:38:54.799 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-12T08:38:54.841 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:38:54.861 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-12T08:38:54.870 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-12T08:38:54.912 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:54.919 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:38:54.971 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr. 2022-07-12T08:38:54.994 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:55.032 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:55.047 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-12T08:38:55.063 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-12T08:38:55.111 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:38:55.190 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-mako. 2022-07-12T08:38:55.217 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-12T08:38:55.273 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:38:55.332 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr. 2022-07-12T08:38:55.353 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mon. 2022-07-12T08:38:55.355 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:55.362 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:55.396 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:55.417 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:55.451 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-bs4. 2022-07-12T08:38:55.468 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-12T08:38:55.500 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mon. 2022-07-12T08:38:55.516 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:55.541 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-12T08:38:55.562 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:55.905 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-12T08:38:55.924 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mon. 2022-07-12T08:38:55.925 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-12T08:38:55.930 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-waitress. 2022-07-12T08:38:55.937 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:55.945 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-12T08:38:55.979 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:55.992 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:38:55.994 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:38:56.279 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-12T08:38:56.298 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-12T08:38:56.328 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:38:56.409 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-osd. 2022-07-12T08:38:56.409 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-tempita. 2022-07-12T08:38:56.425 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-12T08:38:56.425 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:56.467 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:56.468 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-12T08:38:56.623 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webob. 2022-07-12T08:38:56.642 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-12T08:38:56.662 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-osd. 2022-07-12T08:38:56.677 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:56.689 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-12T08:38:56.725 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:56.800 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-paste. 2022-07-12T08:38:56.820 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-12T08:38:56.859 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:38:57.037 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-osd. 2022-07-12T08:38:57.055 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:57.058 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-12T08:38:57.069 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-12T08:38:57.096 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:57.112 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:38:57.195 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-12T08:38:57.211 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-12T08:38:57.278 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:38:57.448 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-bs4. 2022-07-12T08:38:57.469 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-12T08:38:57.503 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-12T08:38:57.603 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-12T08:38:57.622 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-12T08:38:57.662 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:38:57.865 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-waitress. 2022-07-12T08:38:57.886 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-12T08:38:57.935 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:38:57.996 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-webtest. 2022-07-12T08:38:58.008 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-12T08:38:58.047 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-12T08:38:58.318 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-tempita. 2022-07-12T08:38:58.335 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-12T08:38:58.396 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-12T08:38:58.418 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pecan. 2022-07-12T08:38:58.433 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-12T08:38:58.494 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:38:58.721 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-paste. 2022-07-12T08:38:58.738 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-12T08:38:58.779 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:38:58.881 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-12T08:38:58.896 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-12T08:38:58.908 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph. 2022-07-12T08:38:58.921 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../047-ceph_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:58.971 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:38:58.971 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.125 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-12T08:38:59.139 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-12T08:38:59.197 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:38:59.197 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph. 2022-07-12T08:38:59.210 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../047-ceph_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.252 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.274 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-fuse. 2022-07-12T08:38:59.293 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.350 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr. 2022-07-12T08:38:59.357 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.367 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.419 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.535 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-12T08:38:59.547 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-12T08:38:59.588 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:38:59.592 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph. 2022-07-12T08:38:59.602 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-fuse. 2022-07-12T08:38:59.609 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../047-ceph_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.620 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.651 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.663 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.744 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mds. 2022-07-12T08:38:59.791 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.802 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:38:59.925 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webtest. 2022-07-12T08:38:59.945 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-12T08:38:59.945 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mon. 2022-07-12T08:38:59.954 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:38:59.984 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-12T08:39:00.004 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-fuse. 2022-07-12T08:39:00.005 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:00.031 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:00.048 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mds. 2022-07-12T08:39:00.062 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:00.064 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:00.108 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:00.300 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cephadm. 2022-07-12T08:39:00.314 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../050-cephadm_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:00.354 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pecan. 2022-07-12T08:39:00.355 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:00.371 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-12T08:39:00.435 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:39:00.483 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mds. 2022-07-12T08:39:00.498 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:00.540 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:00.600 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package cephadm. 2022-07-12T08:39:00.617 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../050-cephadm_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:00.659 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:00.692 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-jinja2. 2022-07-12T08:39:00.711 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-12T08:39:00.749 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:00.831 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-12T08:39:00.849 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-12T08:39:00.890 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:39:01.030 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jinja2. 2022-07-12T08:39:01.045 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-12T08:39:01.059 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-osd. 2022-07-12T08:39:01.060 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephadm. 2022-07-12T08:39:01.078 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../050-cephadm_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:01.078 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:01.101 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-12T08:39:01.106 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:01.110 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:01.122 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:01.126 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:01.327 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr. 2022-07-12T08:39:01.350 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:01.389 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:01.464 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-12T08:39:01.475 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-jinja2. 2022-07-12T08:39:01.481 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:01.495 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-12T08:39:01.522 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-12T08:39:01.523 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:01.530 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:01.544 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-12T08:39:01.580 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:01.834 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-underscore. 2022-07-12T08:39:01.853 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-12T08:39:01.883 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-12T08:39:01.884 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-12T08:39:01.892 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:01.899 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:01.902 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-12T08:39:01.913 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mon. 2022-07-12T08:39:01.928 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:01.941 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:01.942 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:01.979 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:02.207 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjs-underscore. 2022-07-12T08:39:02.208 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-12T08:39:02.221 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-12T08:39:02.223 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-12T08:39:02.262 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:02.263 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:02.294 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-12T08:39:02.311 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-12T08:39:02.360 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:02.566 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-12T08:39:02.586 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-12T08:39:02.618 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-routes. 2022-07-12T08:39:02.631 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-12T08:39:02.632 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:02.648 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjs-underscore. 2022-07-12T08:39:02.666 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-12T08:39:02.673 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-12T08:39:02.713 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:02.987 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-routes. 2022-07-12T08:39:02.987 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-osd. 2022-07-12T08:39:03.003 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-12T08:39:03.004 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:03.010 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-12T08:39:03.026 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:03.027 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-12T08:39:03.044 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-12T08:39:03.054 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-12T08:39:03.075 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:03.086 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:03.094 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:03.382 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-12T08:39:03.397 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:03.429 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-routes. 2022-07-12T08:39:03.444 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-12T08:39:03.463 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:03.487 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-12T08:39:03.574 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph. 2022-07-12T08:39:03.594 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../047-ceph_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:03.651 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:03.863 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-12T08:39:03.875 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:03.915 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:04.002 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-fuse. 2022-07-12T08:39:04.014 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:04.066 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:04.100 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-12T08:39:04.131 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:04.198 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:04.466 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-12T08:39:04.488 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:04.524 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-decorator. 2022-07-12T08:39:04.526 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mds. 2022-07-12T08:39:04.534 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-12T08:39:04.536 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:04.554 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:04.574 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:04.598 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:04.875 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-12T08:39:04.881 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-decorator. 2022-07-12T08:39:04.895 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-12T08:39:04.896 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:04.912 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-12T08:39:04.961 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:04.962 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:04.962 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-12T08:39:04.971 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:05.131 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package cephadm. 2022-07-12T08:39:05.149 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../050-cephadm_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:05.192 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:05.272 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-12T08:39:05.294 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-12T08:39:05.306 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-decorator. 2022-07-12T08:39:05.321 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-12T08:39:05.336 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:05.359 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:05.363 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph. 2022-07-12T08:39:05.390 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../047-ceph_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:05.391 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-scipy. 2022-07-12T08:39:05.406 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-12T08:39:05.422 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:05.456 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:05.528 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-jinja2. 2022-07-12T08:39:05.547 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-12T08:39:05.604 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:05.667 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-scipy. 2022-07-12T08:39:05.680 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-12T08:39:05.700 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-12T08:39:05.717 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-12T08:39:05.718 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:05.766 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:05.800 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-fuse. 2022-07-12T08:39:05.829 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:05.896 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:06.002 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-12T08:39:06.010 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:06.067 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:06.127 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-scipy. 2022-07-12T08:39:06.155 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-12T08:39:06.181 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:06.310 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mds. 2022-07-12T08:39:06.329 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:06.370 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:06.418 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-12T08:39:06.426 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-12T08:39:06.499 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:06.797 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libjs-underscore. 2022-07-12T08:39:06.812 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-12T08:39:06.854 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:06.930 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cephadm. 2022-07-12T08:39:06.946 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../050-cephadm_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:06.988 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:07.005 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-12T08:39:07.017 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-12T08:39:07.060 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:07.148 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-12T08:39:07.165 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-12T08:39:07.215 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:07.221 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-12T08:39:07.237 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-12T08:39:07.286 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:07.326 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-jinja2. 2022-07-12T08:39:07.335 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-12T08:39:07.376 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:07.583 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-routes. 2022-07-12T08:39:07.599 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-12T08:39:07.620 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-joblib. 2022-07-12T08:39:07.639 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-12T08:39:07.684 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-12T08:39:07.685 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-12T08:39:07.708 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-12T08:39:07.733 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-12T08:39:07.736 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-12T08:39:07.760 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:07.772 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:07.849 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-joblib. 2022-07-12T08:39:07.866 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-12T08:39:07.908 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-12T08:39:08.019 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-12T08:39:08.037 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:08.077 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:08.096 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-sklearn. 2022-07-12T08:39:08.107 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-12T08:39:08.159 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:08.160 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-12T08:39:08.179 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-12T08:39:08.224 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:08.320 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn. 2022-07-12T08:39:08.343 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-12T08:39:08.344 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-joblib. 2022-07-12T08:39:08.352 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-12T08:39:08.381 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:08.394 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-12T08:39:08.525 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjs-underscore. 2022-07-12T08:39:08.545 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-12T08:39:08.579 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:08.789 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-sklearn. 2022-07-12T08:39:08.809 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-12T08:39:08.853 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-12T08:39:08.868 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:08.882 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:08.884 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-12T08:39:08.898 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-12T08:39:08.907 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:08.946 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:09.028 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-12T08:39:09.052 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:09.115 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:09.148 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-12T08:39:09.174 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:09.206 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:09.292 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-routes. 2022-07-12T08:39:09.307 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-12T08:39:09.373 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-12T08:39:09.472 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-decorator. 2022-07-12T08:39:09.497 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-12T08:39:09.534 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:09.613 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-12T08:39:09.633 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:09.687 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:09.740 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-12T08:39:09.761 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:09.796 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:09.888 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-12T08:39:09.897 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-12T08:39:09.942 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:10.042 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-cachetools. 2022-07-12T08:39:10.053 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-12T08:39:10.095 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:10.306 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-scipy. 2022-07-12T08:39:10.318 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-12T08:39:10.366 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:10.408 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cachetools. 2022-07-12T08:39:10.424 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-12T08:39:10.427 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-12T08:39:10.450 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-12T08:39:10.465 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:10.487 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:10.787 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-12T08:39:10.792 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-12T08:39:10.805 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-12T08:39:10.810 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:10.820 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-12T08:39:10.834 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-12T08:39:10.844 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:10.858 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:10.876 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:10.879 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-cachetools. 2022-07-12T08:39:10.896 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-12T08:39:10.942 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:11.159 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-12T08:39:11.177 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-12T08:39:11.199 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-12T08:39:11.206 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-rsa. 2022-07-12T08:39:11.214 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:11.218 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-12T08:39:11.223 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-12T08:39:11.249 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-decorator. 2022-07-12T08:39:11.262 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-12T08:39:11.265 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-12T08:39:11.273 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:11.311 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:11.560 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rsa. 2022-07-12T08:39:11.592 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-12T08:39:11.595 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-12T08:39:11.596 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-12T08:39:11.606 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-google-auth. 2022-07-12T08:39:11.618 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-12T08:39:11.619 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-12T08:39:11.654 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-12T08:39:11.671 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-12T08:39:11.672 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:11.672 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:11.714 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:11.812 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-12T08:39:11.836 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-12T08:39:11.887 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:11.956 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-rsa. 2022-07-12T08:39:11.960 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-google-auth. 2022-07-12T08:39:11.972 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-12T08:39:11.974 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-12T08:39:12.003 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-blinker. 2022-07-12T08:39:12.004 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-12T08:39:12.028 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:12.035 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-scipy. 2022-07-12T08:39:12.038 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-12T08:39:12.051 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-12T08:39:12.063 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:12.118 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:12.371 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-google-auth. 2022-07-12T08:39:12.372 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-12T08:39:12.373 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-blinker. 2022-07-12T08:39:12.400 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-12T08:39:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-12T08:39:12.403 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-12T08:39:12.414 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:12.429 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:12.436 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:12.482 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-joblib. 2022-07-12T08:39:12.499 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-12T08:39:12.579 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-12T08:39:12.747 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-12T08:39:12.763 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-12T08:39:12.793 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-blinker. 2022-07-12T08:39:12.825 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-12T08:39:12.835 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-12T08:39:12.836 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:12.837 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-12T08:39:12.860 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:12.861 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:12.944 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-sklearn. 2022-07-12T08:39:12.962 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-12T08:39:13.002 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:13.159 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-12T08:39:13.176 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-12T08:39:13.182 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-12T08:39:13.189 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-websocket. 2022-07-12T08:39:13.195 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-12T08:39:13.206 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-12T08:39:13.217 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:13.238 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:13.269 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:13.546 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-websocket. 2022-07-12T08:39:13.548 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-12T08:39:13.574 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-12T08:39:13.591 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-12T08:39:13.599 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-12T08:39:13.602 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-12T08:39:13.603 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:13.620 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-12T08:39:13.620 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-12T08:39:13.646 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:13.664 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:13.684 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:13.765 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-12T08:39:13.790 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:13.822 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:13.933 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-12T08:39:13.950 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-12T08:39:13.957 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-websocket. 2022-07-12T08:39:13.974 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-12T08:39:13.991 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:14.009 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:14.241 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-12T08:39:14.241 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-joblib. 2022-07-12T08:39:14.259 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-12T08:39:14.260 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:14.294 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-12T08:39:14.295 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:14.338 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-12T08:39:14.358 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-12T08:39:14.400 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:14.579 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-12T08:39:14.598 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:14.676 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:14.677 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-12T08:39:14.694 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-12T08:39:14.738 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-sklearn. 2022-07-12T08:39:14.778 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:14.779 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-12T08:39:14.792 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:14.936 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-12T08:39:14.948 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-12T08:39:14.993 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:14.998 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-12T08:39:15.007 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-cachetools. 2022-07-12T08:39:15.012 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:15.019 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-12T08:39:15.076 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:15.077 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:15.095 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-12T08:39:15.118 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:15.187 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:15.378 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-12T08:39:15.399 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-12T08:39:15.400 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-12T08:39:15.401 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:15.438 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-12T08:39:15.439 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-12T08:39:15.462 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:15.463 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:15.474 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package jq. 2022-07-12T08:39:15.481 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:15.491 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:15.539 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:15.548 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-12T08:39:15.549 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:15.588 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:15.703 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package jq. 2022-07-12T08:39:15.724 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:15.775 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:15.810 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-12T08:39:15.818 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package socat. 2022-07-12T08:39:15.828 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:15.833 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-12T08:39:15.834 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-12T08:39:15.868 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-12T08:39:15.883 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-12T08:39:15.892 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:15.894 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:16.047 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package socat. 2022-07-12T08:39:16.056 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-12T08:39:16.106 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-12T08:39:16.163 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package jq. 2022-07-12T08:39:16.176 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:16.236 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-rsa. 2022-07-12T08:39:16.248 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-12T08:39:16.249 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:16.297 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-12T08:39:16.373 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-12T08:39:16.390 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-12T08:39:16.426 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:16.503 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package socat. 2022-07-12T08:39:16.516 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-12T08:39:16.558 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-12T08:39:16.590 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-12T08:39:16.619 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-12T08:39:16.649 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:16.679 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-google-auth. 2022-07-12T08:39:16.696 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-12T08:39:16.713 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package xmlstarlet. 2022-07-12T08:39:16.734 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-12T08:39:16.735 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:16.770 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-cachetools. 2022-07-12T08:39:16.787 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:16.799 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-12T08:39:16.835 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:16.944 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package xmlstarlet. 2022-07-12T08:39:16.973 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-12T08:39:17.006 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:17.084 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-12T08:39:17.084 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-blinker. 2022-07-12T08:39:17.093 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-12T08:39:17.098 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-12T08:39:17.123 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-12T08:39:17.136 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package ceph-test. 2022-07-12T08:39:17.136 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:17.138 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:17.142 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-12T08:39:17.158 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../083-ceph-test_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:17.181 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:17.192 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:17.347 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-test. 2022-07-12T08:39:17.365 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../083-ceph-test_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:17.414 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:17.415 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package xmlstarlet. 2022-07-12T08:39:17.443 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-12T08:39:17.472 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:17.499 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-12T08:39:17.504 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-12T08:39:17.509 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-12T08:39:17.521 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-12T08:39:17.550 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:17.565 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:17.837 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-rsa. 2022-07-12T08:39:17.843 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package ceph-test. 2022-07-12T08:39:17.865 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../083-ceph-test_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:17.865 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-12T08:39:17.888 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-12T08:39:17.901 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:17.927 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-12T08:39:17.938 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-12T08:39:17.977 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:18.212 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-google-auth. 2022-07-12T08:39:18.229 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-12T08:39:18.270 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:18.304 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-websocket. 2022-07-12T08:39:18.329 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-12T08:39:18.350 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:18.613 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-blinker. 2022-07-12T08:39:18.631 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-12T08:39:18.672 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:18.714 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-12T08:39:18.728 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-12T08:39:18.755 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:19.011 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-12T08:39:19.024 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-12T08:39:19.091 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:19.329 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-12T08:39:19.343 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:19.395 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:19.446 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-12T08:39:19.461 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-12T08:39:19.502 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:19.767 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-12T08:39:19.780 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-12T08:39:19.855 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:19.856 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-websocket. 2022-07-12T08:39:19.882 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-12T08:39:19.920 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:20.286 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-12T08:39:20.286 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-12T08:39:20.287 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-12T08:39:20.296 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:20.333 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:20.333 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:20.655 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package jq. 2022-07-12T08:39:20.675 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:20.681 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-colorama. 2022-07-12T08:39:20.695 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-12T08:39:20.739 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:20.739 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:20.902 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-12T08:39:20.919 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:20.974 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:21.008 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package cephfs-shell. 2022-07-12T08:39:21.013 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package socat. 2022-07-12T08:39:21.019 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-colorama. 2022-07-12T08:39:21.029 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:21.030 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-12T08:39:21.034 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-12T08:39:21.060 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:21.074 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-12T08:39:21.077 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:21.283 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-12T08:39:21.301 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:21.350 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:21.351 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-colorama. 2022-07-12T08:39:21.351 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-12T08:39:21.351 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package cephfs-shell. 2022-07-12T08:39:21.364 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:21.365 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-12T08:39:21.365 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-12T08:39:21.396 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:21.397 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:21.420 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:21.671 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-12T08:39:21.685 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package cephfs-shell. 2022-07-12T08:39:21.685 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:21.695 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:21.724 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:21.742 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:21.762 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package nvme-cli. 2022-07-12T08:39:21.777 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-12T08:39:21.813 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-12T08:39:21.833 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:21.841 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:21.882 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:22.018 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-12T08:39:22.049 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:22.049 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-12T08:39:22.058 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-12T08:39:22.083 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:22.083 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package nvme-cli. 2022-07-12T08:39:22.108 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:22.108 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-12T08:39:22.176 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:22.185 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package jq. 2022-07-12T08:39:22.203 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-12T08:39:22.242 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-12T08:39:22.243 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:22.244 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-12T08:39:22.275 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:22.393 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package xmlstarlet. 2022-07-12T08:39:22.408 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-12T08:39:22.448 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:22.526 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package nvme-cli. 2022-07-12T08:39:22.537 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package socat. 2022-07-12T08:39:22.555 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-12T08:39:22.556 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-12T08:39:22.556 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-12T08:39:22.590 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-attr. 2022-07-12T08:39:22.591 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-12T08:39:22.597 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-12T08:39:22.598 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-12T08:39:22.605 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:22.615 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:22.648 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-12T08:39:22.811 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package ceph-test. 2022-07-12T08:39:22.829 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../083-ceph-test_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:22.859 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:22.897 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-attr. 2022-07-12T08:39:22.911 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-12T08:39:22.951 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-12T08:39:22.965 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-webencodings. 2022-07-12T08:39:22.982 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-12T08:39:23.036 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-12T08:39:23.050 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-12T08:39:23.057 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:23.090 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:23.091 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-12T08:39:23.109 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-12T08:39:23.176 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:23.305 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webencodings. 2022-07-12T08:39:23.319 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-12T08:39:23.365 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:23.371 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-html5lib. 2022-07-12T08:39:23.387 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-12T08:39:23.417 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-attr. 2022-07-12T08:39:23.430 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:23.442 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-12T08:39:23.454 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package xmlstarlet. 2022-07-12T08:39:23.470 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-12T08:39:23.482 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-12T08:39:23.539 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:23.698 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-html5lib. 2022-07-12T08:39:23.721 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-12T08:39:23.757 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:23.779 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-12T08:39:23.801 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-12T08:39:23.833 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:23.850 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-webencodings. 2022-07-12T08:39:23.885 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-12T08:39:23.901 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package ceph-test. 2022-07-12T08:39:23.902 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:23.922 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../083-ceph-test_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:23.962 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:24.115 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-12T08:39:24.136 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-12T08:39:24.189 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:24.238 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-html5lib. 2022-07-12T08:39:24.250 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-12T08:39:24.288 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:24.330 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-packaging. 2022-07-12T08:39:24.344 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-12T08:39:24.394 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-12T08:39:24.647 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-packaging. 2022-07-12T08:39:24.654 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-12T08:39:24.665 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-12T08:39:24.666 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-12T08:39:24.698 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pastescript. 2022-07-12T08:39:24.708 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:24.709 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-12T08:39:24.743 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-12T08:39:24.751 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:25.050 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastescript. 2022-07-12T08:39:25.067 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-12T08:39:25.086 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pluggy. 2022-07-12T08:39:25.099 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-12T08:39:25.108 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:25.171 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:25.247 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-packaging. 2022-07-12T08:39:25.274 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-12T08:39:25.313 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-12T08:39:25.471 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pluggy. 2022-07-12T08:39:25.491 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-12T08:39:25.492 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-psutil. 2022-07-12T08:39:25.509 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-12T08:39:25.528 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:25.550 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:25.641 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pastescript. 2022-07-12T08:39:25.656 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-12T08:39:25.698 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:25.872 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-psutil. 2022-07-12T08:39:25.896 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-12T08:39:25.926 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:26.077 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pluggy. 2022-07-12T08:39:26.093 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-12T08:39:26.143 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:26.267 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-py. 2022-07-12T08:39:26.280 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-12T08:39:26.406 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:26.416 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-py. 2022-07-12T08:39:26.436 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-12T08:39:26.447 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-colorama. 2022-07-12T08:39:26.463 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-12T08:39:26.468 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:26.505 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:26.507 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-psutil. 2022-07-12T08:39:26.521 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-12T08:39:26.575 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:26.767 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package cephfs-shell. 2022-07-12T08:39:26.785 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:26.828 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:26.832 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-12T08:39:26.862 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-12T08:39:26.889 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:26.891 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-py. 2022-07-12T08:39:26.909 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-12T08:39:26.941 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:27.063 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-12T08:39:27.085 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-12T08:39:27.086 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-12T08:39:27.103 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:27.121 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:27.144 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:27.233 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-pytest. 2022-07-12T08:39:27.250 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-12T08:39:27.262 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-12T08:39:27.280 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-12T08:39:27.296 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-12T08:39:27.321 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:27.442 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pytest. 2022-07-12T08:39:27.463 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-12T08:39:27.487 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package nvme-cli. 2022-07-12T08:39:27.499 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-12T08:39:27.508 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-12T08:39:27.586 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:27.614 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package python3-simplejson. 2022-07-12T08:39:27.635 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-12T08:39:27.640 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-pytest. 2022-07-12T08:39:27.650 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-12T08:39:27.666 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-colorama. 2022-07-12T08:39:27.670 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:27.676 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-12T08:39:27.694 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-12T08:39:27.754 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:27.864 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplejson. 2022-07-12T08:39:27.878 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-12T08:39:27.920 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:27.987 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-12T08:39:27.999 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package cephfs-shell. 2022-07-12T08:39:28.007 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-12T08:39:28.013 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-324-g3497be4c-1focal_all.deb ... 2022-07-12T08:39:28.045 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:28.068 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package python3-simplejson. 2022-07-12T08:39:28.088 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-12T08:39:28.095 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:28.126 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:28.131 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package radosgw. 2022-07-12T08:39:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../101-radosgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:28.190 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:28.335 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-12T08:39:28.350 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package radosgw. 2022-07-12T08:39:28.351 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:28.367 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../101-radosgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:28.389 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:28.402 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:28.415 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-attr. 2022-07-12T08:39:28.428 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-12T08:39:28.476 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-12T08:39:28.572 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package radosgw. 2022-07-12T08:39:28.588 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../101-radosgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:28.629 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:28.827 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-webencodings. 2022-07-12T08:39:28.827 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package nvme-cli. 2022-07-12T08:39:28.842 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-12T08:39:28.843 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-12T08:39:28.876 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:28.910 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:29.212 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-html5lib. 2022-07-12T08:39:29.232 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-12T08:39:29.306 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:29.327 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package rbd-fuse. 2022-07-12T08:39:29.335 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-12T08:39:29.363 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-12T08:39:29.363 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:29.384 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:29.390 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:29.489 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package rbd-fuse. 2022-07-12T08:39:29.504 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:29.546 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:29.649 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-12T08:39:29.670 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-12T08:39:29.687 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-12T08:39:29.711 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-attr. 2022-07-12T08:39:29.711 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:29.733 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-12T08:39:29.770 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-12T08:39:29.843 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-12T08:39:30.014 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:39:30.143 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package rbd-fuse. 2022-07-12T08:39:30.149 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:39:30.160 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-webencodings. 2022-07-12T08:39:30.174 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:30.181 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-12T08:39:30.189 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:39:30.206 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:30.210 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:30.263 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-packaging. 2022-07-12T08:39:30.270 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-12T08:39:30.274 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:30.305 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-12T08:39:30.323 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:39:30.483 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:30.509 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:30.515 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-12T08:39:30.521 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-html5lib. 2022-07-12T08:39:30.538 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-12T08:39:30.577 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:30.606 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pastescript. 2022-07-12T08:39:30.631 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-12T08:39:30.665 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:30.680 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:30.865 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:39:30.970 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-12T08:39:30.987 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-12T08:39:30.988 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:39:31.028 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:31.076 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pluggy. 2022-07-12T08:39:31.100 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-12T08:39:31.119 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user cephadm....done 2022-07-12T08:39:31.142 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:31.148 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:31.287 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user cephadm....done 2022-07-12T08:39:31.349 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:39:31.357 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:31.458 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:39:31.495 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-psutil. 2022-07-12T08:39:31.512 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-12T08:39:31.582 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-packaging. 2022-07-12T08:39:31.587 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:31.610 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-12T08:39:31.645 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-12T08:39:31.708 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:31.934 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:31.995 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-12T08:39:32.028 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user cephadm....done 2022-07-12T08:39:32.095 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-py. 2022-07-12T08:39:32.128 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pastescript. 2022-07-12T08:39:32.128 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-12T08:39:32.138 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:32.154 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:32.156 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-12T08:39:32.224 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:39:32.225 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:32.239 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-12T08:39:32.353 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-12T08:39:32.552 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-12T08:39:32.568 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-12T08:39:32.587 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:32.616 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pluggy. 2022-07-12T08:39:32.617 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:32.631 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-12T08:39:32.671 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:32.697 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-12T08:39:32.870 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:39:32.892 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-12T08:39:32.912 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-pytest. 2022-07-12T08:39:32.925 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-12T08:39:32.954 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-12T08:39:33.009 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-psutil. 2022-07-12T08:39:33.024 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-12T08:39:33.045 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:33.066 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:33.151 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:39:33.174 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:33.277 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package python3-simplejson. 2022-07-12T08:39:33.291 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-12T08:39:33.325 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:33.392 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-12T08:39:33.428 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-py. 2022-07-12T08:39:33.441 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:33.453 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-12T08:39:33.467 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:33.487 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:33.730 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package radosgw. 2022-07-12T08:39:33.740 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../101-radosgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:33.752 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:33.773 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:39:33.778 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:33.842 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:39:33.865 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-12T08:39:33.882 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-12T08:39:33.939 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:34.022 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:39:34.063 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:34.106 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:34.294 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-pytest. 2022-07-12T08:39:34.318 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:34.319 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-12T08:39:34.351 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-12T08:39:34.404 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:34.420 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:34.626 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:34.706 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:34.710 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:39:34.729 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package python3-simplejson. 2022-07-12T08:39:34.745 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-12T08:39:34.787 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:34.833 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-12T08:39:34.898 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package rbd-fuse. 2022-07-12T08:39:34.918 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:34.939 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:34.965 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:35.066 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-12T08:39:35.079 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:35.166 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:39:35.275 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package radosgw. 2022-07-12T08:39:35.285 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../101-radosgw_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:35.301 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-12T08:39:35.325 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:35.379 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:35.403 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:39:35.501 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:35.695 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:39:35.712 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:35.726 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:35.778 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:35.827 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:39:35.845 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-12T08:39:35.973 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:36.005 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:36.128 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:39:36.131 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:36.316 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:36.366 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:36.404 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package rbd-fuse. 2022-07-12T08:39:36.420 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-324-g3497be4c-1focal_amd64.deb ... 2022-07-12T08:39:36.421 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:36.465 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:36.633 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:36.686 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:36.756 INFO:teuthology.orchestra.run.smithi130.stdout:Adding system user cephadm....done 2022-07-12T08:39:36.810 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-12T08:39:36.946 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:39:37.028 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:37.041 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:37.108 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:37.141 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-12T08:39:37.264 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:37.266 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-12T08:39:37.333 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:37.335 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:37.377 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:37.417 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-12T08:39:37.538 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-12T08:39:37.612 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:37.644 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:37.669 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-12T08:39:37.690 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-12T08:39:37.714 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:37.976 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-12T08:39:37.977 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-12T08:39:38.014 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-12T08:39:38.015 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:38.273 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:39:38.284 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:39:38.319 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-12T08:39:38.320 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user cephadm....done 2022-07-12T08:39:38.407 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:39:38.407 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:39:38.424 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:39:38.500 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-12T08:39:38.638 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-12T08:39:38.691 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:39:38.692 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:39:38.719 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:38.852 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-12T08:39:38.937 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:39:38.998 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:39.064 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:39:39.103 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:39:39.104 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:39:39.156 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-12T08:39:39.267 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:39:39.296 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-12T08:39:39.387 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:39:39.426 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:39:39.437 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:39:39.562 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:39.634 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-12T08:39:39.759 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:39:39.801 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:39:39.816 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:39:39.861 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:40.037 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:39:40.038 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2022-07-12T08:39:40.159 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:40.186 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-12T08:39:40.204 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-12T08:39:40.324 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:40.324 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-12T08:39:40.329 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:40.360 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-12T08:39:40.394 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:39:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:39:40.645 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:40.677 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:40.687 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-12T08:39:40.759 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-12T08:39:40.878 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:40.880 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:41.009 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-12T08:39:41.011 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:39:41.023 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:39:41.129 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:41.156 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:41.159 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:41.189 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:41.322 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-12T08:39:41.419 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:41.502 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:41.530 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:41.531 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:39:41.537 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:41.631 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:41.632 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:41.656 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-argparse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:41.815 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:41.886 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:39:41.901 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:41.958 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:41.970 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:39:42.036 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:42.078 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-12T08:39:42.154 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:42.173 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-12T08:39:42.289 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-12T08:39:42.349 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:39:42.434 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-12T08:39:42.440 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:42.490 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:39:42.584 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:39:42.627 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:39:42.647 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-12T08:39:42.706 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:39:42.753 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-12T08:39:42.772 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-12T08:39:42.922 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-12T08:39:42.948 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:39:43.060 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-12T08:39:43.060 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-12T08:39:43.084 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-12T08:39:43.101 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:39:43.220 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:39:43.335 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:39:43.346 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:39:43.359 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-12T08:39:43.500 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:39:43.511 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-12T08:39:43.530 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:43.627 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:43.648 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:39:43.787 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:39:43.790 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-12T08:39:43.827 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-12T08:39:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-12T08:39:43.950 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:44.090 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:44.171 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:39:44.171 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-12T08:39:44.204 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:44.230 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:39:44.262 INFO:teuthology.orchestra.run.smithi111.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-12T08:39:44.368 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:39:44.440 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:39:44.472 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-12T08:39:44.556 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:44.561 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:39:44.691 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:39:44.693 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:39:44.783 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:39:44.844 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-12T08:39:44.937 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:44.999 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:39:45.041 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:45.042 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:45.179 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:45.194 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-12T08:39:45.195 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-12T08:39:45.210 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-12T08:39:45.322 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-12T08:39:45.353 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:45.374 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:45.476 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:45.524 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:45.538 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-12T08:39:45.603 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-12T08:39:45.666 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:45.696 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:45.706 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-12T08:39:45.824 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librados2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:45.922 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:45.950 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:45.956 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librados2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:45.985 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-12T08:39:46.075 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:39:46.107 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:46.174 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:46.222 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librados2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:46.340 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:46.360 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-12T08:39:46.443 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:46.568 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:46.671 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-12T08:39:46.936 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:39:47.028 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-12T08:39:47.229 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-12T08:39:47.403 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-12T08:39:47.476 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:47.522 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:39:47.536 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:47.539 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:47.599 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:47.622 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:39:47.663 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:47.733 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-12T08:39:47.764 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:47.796 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-12T08:39:47.853 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-12T08:39:47.888 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:47.892 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-12T08:39:48.014 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-12T08:39:48.111 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-12T08:39:48.168 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:39:48.183 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-12T08:39:48.207 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-12T08:39:48.343 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-12T08:39:48.434 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-12T08:39:48.435 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:39:48.468 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:48.528 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:39:48.677 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-12T08:39:48.696 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-12T08:39:48.712 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:39:48.784 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-12T08:39:48.793 INFO:teuthology.orchestra.run.smithi130.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-12T08:39:48.803 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-12T08:39:48.985 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-12T08:39:49.039 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:49.079 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:49.135 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:49.138 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-12T08:39:49.167 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:49.230 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:39:49.276 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:49.323 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:49.425 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-12T08:39:49.446 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:49.527 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:49.548 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:39:49.599 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:49.670 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:49.691 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-12T08:39:49.723 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:49.775 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:49.798 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-12T08:39:49.890 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:49.910 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:49.992 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:50.018 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:50.067 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-12T08:39:50.084 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:50.177 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-12T08:39:50.228 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:50.385 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-12T08:39:50.503 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:50.711 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-12T08:39:50.791 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librados2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:50.893 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:51.017 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-12T08:39:51.074 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librbd1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:51.150 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up librbd1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:51.212 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:51.270 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up librbd1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:51.274 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:51.334 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-12T08:39:51.337 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:51.390 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:51.398 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:51.433 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-12T08:39:51.445 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:51.521 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:51.524 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:51.629 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:51.733 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-12T08:39:51.746 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:51.867 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:51.873 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-12T08:39:51.923 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:52.003 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-12T08:39:52.024 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-12T08:39:52.110 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-12T08:39:52.198 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-12T08:39:52.219 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-12T08:39:52.297 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-12T08:39:52.306 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-12T08:39:52.372 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.392 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-12T08:39:52.528 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:52.541 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.601 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.605 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.662 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.731 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-12T08:39:52.741 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.767 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:52.885 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:52.886 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:52.970 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:52.970 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:52.996 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:52.997 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:53.014 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librados2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:53.115 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-12T08:39:53.139 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-12T08:39:53.382 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:53.447 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:53.447 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:53.492 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-12T08:39:53.507 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:53.549 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:53.568 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:53.798 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:39:53.829 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:53.835 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:53.861 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:54.085 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-12T08:39:54.152 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:39:54.177 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:39:54.235 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:39:54.436 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.508 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.546 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librgw2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.559 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.587 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:54.604 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.628 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.681 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.718 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:54.738 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.758 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.824 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-12T08:39:54.847 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.862 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.885 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-12T08:39:54.915 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:54.965 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-12T08:39:55.009 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-12T08:39:55.021 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:55.194 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-12T08:39:55.291 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:55.346 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:55.406 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-12T08:39:55.429 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:55.517 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-12T08:39:55.607 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:55.653 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:55.710 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:55.711 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:55.807 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:55.827 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-12T08:39:55.872 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:55.984 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:56.105 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:56.145 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:56.174 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:56.187 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs2 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:56.241 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:56.302 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:39:56.354 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-12T08:39:56.464 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up librbd1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:56.598 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:56.680 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libradosstriper1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:56.701 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:56.773 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-12T08:39:56.859 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:57.073 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-12T08:39:57.080 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:57.181 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:57.220 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:57.310 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-12T08:39:57.334 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:57.621 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-12T08:39:57.811 INFO:teuthology.orchestra.run.smithi111.stdout:Adding group ceph....done 2022-07-12T08:39:57.940 INFO:teuthology.orchestra.run.smithi121.stdout:Adding group ceph....done 2022-07-12T08:39:57.989 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:58.090 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:58.099 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up librbd1 (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:58.117 INFO:teuthology.orchestra.run.smithi061.stdout:Adding group ceph....done 2022-07-12T08:39:58.192 INFO:teuthology.orchestra.run.smithi111.stdout:Adding system user ceph....done 2022-07-12T08:39:58.210 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-12T08:39:58.294 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:58.295 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:58.302 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-12T08:39:58.323 INFO:teuthology.orchestra.run.smithi121.stdout:Adding system user ceph....done 2022-07-12T08:39:58.379 INFO:teuthology.orchestra.run.smithi111.stdout:Setting system user ceph properties....done 2022-07-12T08:39:58.421 INFO:teuthology.orchestra.run.smithi111.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-12T08:39:58.453 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-12T08:39:58.540 INFO:teuthology.orchestra.run.smithi121.stdout:Setting system user ceph properties....done 2022-07-12T08:39:58.558 INFO:teuthology.orchestra.run.smithi121.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-12T08:39:58.559 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user ceph....done 2022-07-12T08:39:58.559 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-12T08:39:58.703 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-12T08:39:58.727 INFO:teuthology.orchestra.run.smithi061.stdout:Setting system user ceph properties....done 2022-07-12T08:39:58.741 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:58.755 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-12T08:39:58.756 INFO:teuthology.orchestra.run.smithi061.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-12T08:39:58.864 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-12T08:39:58.877 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-12T08:39:58.902 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-12T08:39:58.909 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:39:59.025 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-12T08:39:59.186 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-12T08:39:59.210 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-12T08:39:59.237 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-12T08:39:59.319 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.471 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.497 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-modules-core (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.502 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.528 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:39:59.647 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.647 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.683 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.716 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:39:59.716 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:39:59.807 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.869 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:39:59.883 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:59.884 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-12T08:39:59.934 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:39:59.935 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:00.018 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:00.062 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:00.063 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:00.144 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:00.317 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-12T08:40:00.387 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:00.388 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libcephfs-dev (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:00.575 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-12T08:40:00.671 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:00.671 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-12T08:40:00.730 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:00.918 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-12T08:40:00.963 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.083 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-12T08:40:01.092 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.098 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-12T08:40:01.298 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-12T08:40:01.466 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.512 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.538 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-12T08:40:01.566 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.603 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:40:01.803 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:01.803 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:01.832 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.906 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rados (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:01.927 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:01.927 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:02.027 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rbd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:02.045 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:02.046 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:02.184 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up rbd-fuse (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:02.244 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:02.311 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-12T08:40:02.399 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:02.456 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:02.462 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:02.463 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:02.517 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:02.622 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:02.623 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:02.671 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:02.672 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:02.738 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-12T08:40:02.906 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.088 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.089 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.090 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-rgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.123 INFO:teuthology.orchestra.run.smithi130.stdout:Adding group ceph....done 2022-07-12T08:40:03.200 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.222 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-cephfs (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.245 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.316 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.508 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up cephfs-shell (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:03.519 INFO:teuthology.orchestra.run.smithi130.stdout:Adding system user ceph....done 2022-07-12T08:40:03.634 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-12T08:40:03.707 INFO:teuthology.orchestra.run.smithi130.stdout:Setting system user ceph properties....done 2022-07-12T08:40:03.730 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:03.730 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:03.731 INFO:teuthology.orchestra.run.smithi130.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-12T08:40:03.741 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:03.742 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:03.822 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:03.822 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:03.876 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-12T08:40:04.185 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.194 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.198 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-12T08:40:04.295 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.302 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.318 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.438 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.449 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.463 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.593 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-common (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.593 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.665 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.678 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:04.679 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:04.684 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:04.684 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:04.796 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:04.822 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:04.822 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:05.090 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:05.091 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:05.158 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.159 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.160 INFO:teuthology.orchestra.run.smithi137.stdout:Adding group ceph....done 2022-07-12T08:40:05.270 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.274 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.303 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.397 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.409 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.432 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.526 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.551 INFO:teuthology.orchestra.run.smithi137.stdout:Adding system user ceph....done 2022-07-12T08:40:05.551 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-12T08:40:05.569 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-12T08:40:05.746 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-12T08:40:05.763 INFO:teuthology.orchestra.run.smithi137.stdout:Setting system user ceph properties....done 2022-07-12T08:40:05.782 INFO:teuthology.orchestra.run.smithi137.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-12T08:40:05.816 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:05.848 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:05.882 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:05.930 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-12T08:40:06.077 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:06.187 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-12T08:40:06.251 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-12T08:40:06.644 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:06.751 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-test (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:06.849 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up radosgw (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:06.888 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:06.889 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:07.094 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:07.094 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-12T08:40:07.331 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:07.562 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:07.562 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-base (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:07.994 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:08.148 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-12T08:40:08.149 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:08.540 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:08.593 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mds (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:08.637 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:08.642 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:08.642 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:08.774 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:08.847 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:08.847 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-12T08:40:09.016 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:09.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-12T08:40:09.101 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:09.102 INFO:teuthology.orchestra.run.smithi121.stdout:Reading package lists... 2022-07-12T08:40:09.104 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:09.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-12T08:40:09.189 INFO:teuthology.orchestra.run.smithi111.stdout:Reading package lists... 2022-07-12T08:40:09.222 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:09.243 INFO:teuthology.orchestra.run.smithi061.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:09.244 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-12T08:40:09.274 INFO:teuthology.orchestra.run.smithi121.stdout:Building dependency tree... 2022-07-12T08:40:09.279 INFO:teuthology.orchestra.run.smithi121.stdout:Reading state information... 2022-07-12T08:40:09.314 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:09.333 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-12T08:40:09.365 INFO:teuthology.orchestra.run.smithi111.stdout:Building dependency tree... 2022-07-12T08:40:09.367 INFO:teuthology.orchestra.run.smithi111.stdout:Reading state information... 2022-07-12T08:40:09.377 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:09.431 INFO:teuthology.orchestra.run.smithi121.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-12T08:40:09.432 INFO:teuthology.orchestra.run.smithi121.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-12T08:40:09.432 INFO:teuthology.orchestra.run.smithi121.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-12T08:40:09.433 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:40:09.439 INFO:teuthology.orchestra.run.smithi121.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:40:09.439 INFO:teuthology.orchestra.run.smithi121.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:40:09.440 INFO:teuthology.orchestra.run.smithi121.stdout:The following additional packages will be installed: 2022-07-12T08:40:09.440 INFO:teuthology.orchestra.run.smithi121.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-07-12T08:40:09.441 INFO:teuthology.orchestra.run.smithi121.stdout:Suggested packages: 2022-07-12T08:40:09.441 INFO:teuthology.orchestra.run.smithi121.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-12T08:40:09.470 INFO:teuthology.orchestra.run.smithi121.stdout:The following NEW packages will be installed: 2022-07-12T08:40:09.472 INFO:teuthology.orchestra.run.smithi121.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-12T08:40:09.472 INFO:teuthology.orchestra.run.smithi121.stdout: libssl-dev 2022-07-12T08:40:09.473 INFO:teuthology.orchestra.run.smithi121.stdout:The following packages will be upgraded: 2022-07-12T08:40:09.473 INFO:teuthology.orchestra.run.smithi121.stdout: libssl1.1 2022-07-12T08:40:09.511 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-12T08:40:09.512 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-12T08:40:09.524 INFO:teuthology.orchestra.run.smithi111.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-12T08:40:09.524 INFO:teuthology.orchestra.run.smithi111.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-12T08:40:09.524 INFO:teuthology.orchestra.run.smithi111.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-12T08:40:09.524 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:40:09.525 INFO:teuthology.orchestra.run.smithi111.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:40:09.525 INFO:teuthology.orchestra.run.smithi111.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:40:09.526 INFO:teuthology.orchestra.run.smithi111.stdout:The following additional packages will be installed: 2022-07-12T08:40:09.526 INFO:teuthology.orchestra.run.smithi111.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-07-12T08:40:09.527 INFO:teuthology.orchestra.run.smithi111.stdout:Suggested packages: 2022-07-12T08:40:09.527 INFO:teuthology.orchestra.run.smithi111.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-12T08:40:09.532 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:09.533 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-12T08:40:09.562 INFO:teuthology.orchestra.run.smithi111.stdout:The following NEW packages will be installed: 2022-07-12T08:40:09.563 INFO:teuthology.orchestra.run.smithi111.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-12T08:40:09.563 INFO:teuthology.orchestra.run.smithi111.stdout: libssl-dev 2022-07-12T08:40:09.564 INFO:teuthology.orchestra.run.smithi111.stdout:The following packages will be upgraded: 2022-07-12T08:40:09.564 INFO:teuthology.orchestra.run.smithi111.stdout: libssl1.1 2022-07-12T08:40:09.594 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:09.594 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:09.642 INFO:teuthology.orchestra.run.smithi111.stdout:1 upgraded, 10 newly installed, 0 to remove and 17 not upgraded. 2022-07-12T08:40:09.643 INFO:teuthology.orchestra.run.smithi111.stdout:Need to get 4,234 kB of archives. 2022-07-12T08:40:09.644 INFO:teuthology.orchestra.run.smithi111.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-12T08:40:09.644 INFO:teuthology.orchestra.run.smithi111.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-12T08:40:09.653 INFO:teuthology.orchestra.run.smithi121.stdout:1 upgraded, 10 newly installed, 0 to remove and 17 not upgraded. 2022-07-12T08:40:09.654 INFO:teuthology.orchestra.run.smithi121.stdout:Need to get 4,234 kB of archives. 2022-07-12T08:40:09.654 INFO:teuthology.orchestra.run.smithi121.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-12T08:40:09.655 INFO:teuthology.orchestra.run.smithi121.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-12T08:40:09.678 INFO:teuthology.orchestra.run.smithi061.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-12T08:40:09.678 INFO:teuthology.orchestra.run.smithi061.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-12T08:40:09.679 INFO:teuthology.orchestra.run.smithi061.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-12T08:40:09.679 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:40:09.681 INFO:teuthology.orchestra.run.smithi061.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:40:09.681 INFO:teuthology.orchestra.run.smithi061.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:40:09.683 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2022-07-12T08:40:09.683 INFO:teuthology.orchestra.run.smithi061.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-07-12T08:40:09.684 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2022-07-12T08:40:09.684 INFO:teuthology.orchestra.run.smithi061.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-12T08:40:09.715 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2022-07-12T08:40:09.716 INFO:teuthology.orchestra.run.smithi061.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-12T08:40:09.717 INFO:teuthology.orchestra.run.smithi061.stdout: libssl-dev 2022-07-12T08:40:09.718 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2022-07-12T08:40:09.719 INFO:teuthology.orchestra.run.smithi061.stdout: libssl1.1 2022-07-12T08:40:09.814 INFO:teuthology.orchestra.run.smithi061.stdout:1 upgraded, 10 newly installed, 0 to remove and 17 not upgraded. 2022-07-12T08:40:09.815 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 4,234 kB of archives. 2022-07-12T08:40:09.815 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-12T08:40:09.816 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-12T08:40:09.846 INFO:teuthology.orchestra.run.smithi111.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.16 [1,321 kB] 2022-07-12T08:40:09.942 INFO:teuthology.orchestra.run.smithi111.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-12T08:40:09.963 INFO:teuthology.orchestra.run.smithi111.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-12T08:40:09.963 INFO:teuthology.orchestra.run.smithi111.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-12T08:40:09.981 INFO:teuthology.orchestra.run.smithi111.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-12T08:40:09.982 INFO:teuthology.orchestra.run.smithi111.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-12T08:40:09.983 INFO:teuthology.orchestra.run.smithi111.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-12T08:40:09.984 INFO:teuthology.orchestra.run.smithi111.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-12T08:40:09.984 INFO:teuthology.orchestra.run.smithi111.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-12T08:40:09.985 INFO:teuthology.orchestra.run.smithi111.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-12T08:40:10.015 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:10.041 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.16 [1,321 kB] 2022-07-12T08:40:10.064 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:10.128 INFO:teuthology.orchestra.run.smithi121.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.16 [1,321 kB] 2022-07-12T08:40:10.155 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-osd (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:10.175 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-12T08:40:10.201 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-12T08:40:10.201 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-12T08:40:10.215 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-12T08:40:10.216 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-12T08:40:10.216 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-12T08:40:10.217 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-12T08:40:10.217 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-12T08:40:10.226 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-12T08:40:10.278 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:10.327 INFO:teuthology.orchestra.run.smithi111.stdout:Preconfiguring packages ... 2022-07-12T08:40:10.440 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:10.469 INFO:teuthology.orchestra.run.smithi111.stdout:Fetched 4,234 kB in 0s (9,512 kB/s) 2022-07-12T08:40:10.497 INFO:teuthology.orchestra.run.smithi121.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-12T08:40:10.557 INFO:teuthology.orchestra.run.smithi061.stdout:Preconfiguring packages ... 2022-07-12T08:40:10.582 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package flex. 2022-07-12T08:40:10.592 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-12T08:40:10.627 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:10.627 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-12T08:40:10.637 INFO:teuthology.orchestra.run.smithi111.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146190 files and directories currently installed.) 2022-07-12T08:40:10.645 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:10.700 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 4,234 kB in 1s (8,104 kB/s) 2022-07-12T08:40:10.718 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-12T08:40:10.812 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package flex. 2022-07-12T08:40:10.873 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146190 files and directories currently installed.) 2022-07-12T08:40:10.878 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:10.910 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:10.919 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-12T08:40:11.055 INFO:teuthology.orchestra.run.smithi121.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-12T08:40:11.084 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-k8sevents (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:11.143 INFO:teuthology.orchestra.run.smithi121.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-12T08:40:11.173 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:11.243 INFO:teuthology.orchestra.run.smithi121.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-12T08:40:11.276 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:11.309 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mon (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:11.319 INFO:teuthology.orchestra.run.smithi121.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-12T08:40:11.400 INFO:teuthology.orchestra.run.smithi121.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-12T08:40:11.484 INFO:teuthology.orchestra.run.smithi121.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-12T08:40:11.532 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:11.533 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-12T08:40:11.533 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:11.573 INFO:teuthology.orchestra.run.smithi121.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-12T08:40:11.574 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.16) over (1.1.1f-1ubuntu2.15) ... 2022-07-12T08:40:11.648 INFO:teuthology.orchestra.run.smithi121.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-12T08:40:11.732 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.16) over (1.1.1f-1ubuntu2.15) ... 2022-07-12T08:40:12.005 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-cephadm (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:12.080 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package bison. 2022-07-12T08:40:12.080 INFO:teuthology.orchestra.run.smithi121.stdout:Preconfiguring packages ... 2022-07-12T08:40:12.089 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-12T08:40:12.124 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:12.137 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:12.211 INFO:teuthology.orchestra.run.smithi121.stdout:Fetched 4,234 kB in 2s (1,838 kB/s) 2022-07-12T08:40:12.245 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up ceph-mgr-dashboard (15.2.16-324-g3497be4c-1focal) ... 2022-07-12T08:40:12.291 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package bison. 2022-07-12T08:40:12.310 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-12T08:40:12.349 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:12.386 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package flex. 2022-07-12T08:40:12.415 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-12T08:40:12.441 INFO:teuthology.orchestra.run.smithi121.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146190 files and directories currently installed.) 2022-07-12T08:40:12.446 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:12.574 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-12T08:40:12.575 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-12T08:40:12.594 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-07-12T08:40:12.665 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:12.731 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:12.801 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-12T08:40:12.817 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-07-12T08:40:12.858 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:12.982 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package dump. 2022-07-12T08:40:12.994 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-07-12T08:40:13.066 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking dump (0.4b46-6) ... 2022-07-12T08:40:13.204 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:13.252 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package dump. 2022-07-12T08:40:13.269 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-07-12T08:40:13.314 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking dump (0.4b46-6) ... 2022-07-12T08:40:13.342 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package indent. 2022-07-12T08:40:13.355 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-07-12T08:40:13.382 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.16) over (1.1.1f-1ubuntu2.15) ... 2022-07-12T08:40:13.416 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking indent (2.2.12-1) ... 2022-07-12T08:40:13.678 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-12T08:40:13.691 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-12T08:40:13.700 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package indent. 2022-07-12T08:40:13.722 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-07-12T08:40:13.732 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:13.775 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:13.788 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking indent (2.2.12-1) ... 2022-07-12T08:40:13.858 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package bison. 2022-07-12T08:40:13.882 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-12T08:40:13.947 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:14.031 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-12T08:40:14.048 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-12T08:40:14.083 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-12T08:40:14.096 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:14.098 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-12T08:40:14.145 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:14.212 INFO:teuthology.orchestra.run.smithi130.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:14.213 DEBUG:teuthology.orchestra.run.smithi130:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-12T08:40:14.286 INFO:teuthology.orchestra.run.smithi130.stdout:Reading package lists... 2022-07-12T08:40:14.337 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-12T08:40:14.354 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-07-12T08:40:14.395 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:14.450 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-12T08:40:14.457 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-12T08:40:14.464 INFO:teuthology.orchestra.run.smithi130.stdout:Building dependency tree... 2022-07-12T08:40:14.467 INFO:teuthology.orchestra.run.smithi130.stdout:Reading state information... 2022-07-12T08:40:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-12T08:40:14.473 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:14.511 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:14.512 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:14.620 INFO:teuthology.orchestra.run.smithi130.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-12T08:40:14.621 INFO:teuthology.orchestra.run.smithi130.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-12T08:40:14.621 INFO:teuthology.orchestra.run.smithi130.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-12T08:40:14.621 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:40:14.622 INFO:teuthology.orchestra.run.smithi130.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:40:14.622 INFO:teuthology.orchestra.run.smithi130.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:40:14.622 INFO:teuthology.orchestra.run.smithi130.stdout:The following additional packages will be installed: 2022-07-12T08:40:14.627 INFO:teuthology.orchestra.run.smithi130.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-07-12T08:40:14.627 INFO:teuthology.orchestra.run.smithi130.stdout:Suggested packages: 2022-07-12T08:40:14.628 INFO:teuthology.orchestra.run.smithi130.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-12T08:40:14.657 INFO:teuthology.orchestra.run.smithi130.stdout:The following NEW packages will be installed: 2022-07-12T08:40:14.658 INFO:teuthology.orchestra.run.smithi130.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-12T08:40:14.659 INFO:teuthology.orchestra.run.smithi130.stdout: libssl-dev 2022-07-12T08:40:14.660 INFO:teuthology.orchestra.run.smithi130.stdout:The following packages will be upgraded: 2022-07-12T08:40:14.661 INFO:teuthology.orchestra.run.smithi130.stdout: libssl1.1 2022-07-12T08:40:14.728 INFO:teuthology.orchestra.run.smithi130.stdout:1 upgraded, 10 newly installed, 0 to remove and 17 not upgraded. 2022-07-12T08:40:14.728 INFO:teuthology.orchestra.run.smithi130.stdout:Need to get 4,234 kB of archives. 2022-07-12T08:40:14.728 INFO:teuthology.orchestra.run.smithi130.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-12T08:40:14.729 INFO:teuthology.orchestra.run.smithi130.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-12T08:40:14.741 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package dump. 2022-07-12T08:40:14.760 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-07-12T08:40:14.798 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking dump (0.4b46-6) ... 2022-07-12T08:40:14.806 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-12T08:40:14.828 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:14.862 INFO:teuthology.orchestra.run.smithi130.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.16 [1,321 kB] 2022-07-12T08:40:14.879 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:14.929 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-12T08:40:14.934 INFO:teuthology.orchestra.run.smithi130.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-12T08:40:14.951 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:14.951 INFO:teuthology.orchestra.run.smithi130.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-12T08:40:14.952 INFO:teuthology.orchestra.run.smithi130.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-12T08:40:14.955 INFO:teuthology.orchestra.run.smithi130.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-12T08:40:14.961 INFO:teuthology.orchestra.run.smithi130.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-12T08:40:14.963 INFO:teuthology.orchestra.run.smithi130.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-12T08:40:14.963 INFO:teuthology.orchestra.run.smithi130.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-12T08:40:14.964 INFO:teuthology.orchestra.run.smithi130.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-12T08:40:14.964 INFO:teuthology.orchestra.run.smithi130.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-12T08:40:14.988 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:15.118 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package indent. 2022-07-12T08:40:15.140 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-07-12T08:40:15.151 INFO:teuthology.orchestra.run.smithi111.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-12T08:40:15.169 INFO:teuthology.orchestra.run.smithi111.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:15.176 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking indent (2.2.12-1) ... 2022-07-12T08:40:15.211 INFO:teuthology.orchestra.run.smithi111.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:15.242 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-12T08:40:15.262 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:15.298 INFO:teuthology.orchestra.run.smithi130.stdout:Preconfiguring packages ... 2022-07-12T08:40:15.299 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:15.410 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:15.425 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-12T08:40:15.443 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-12T08:40:15.454 INFO:teuthology.orchestra.run.smithi130.stdout:Fetched 4,234 kB in 0s (13.2 MB/s) 2022-07-12T08:40:15.501 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:15.561 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package flex. 2022-07-12T08:40:15.576 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-12T08:40:15.593 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:15.626 INFO:teuthology.orchestra.run.smithi130.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146190 files and directories currently installed.) 2022-07-12T08:40:15.627 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:15.634 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:15.689 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-12T08:40:15.692 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-12T08:40:15.765 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-12T08:40:15.786 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-12T08:40:15.813 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:15.838 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:15.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-12T08:40:15.849 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:15.912 INFO:teuthology.orchestra.run.smithi137.stdout:Reading package lists... 2022-07-12T08:40:16.091 INFO:teuthology.orchestra.run.smithi137.stdout:Building dependency tree... 2022-07-12T08:40:16.091 INFO:teuthology.orchestra.run.smithi137.stdout:Reading state information... 2022-07-12T08:40:16.105 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-12T08:40:16.174 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-12T08:40:16.199 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:16.200 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:16.233 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:16.234 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:16.245 INFO:teuthology.orchestra.run.smithi137.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-12T08:40:16.246 INFO:teuthology.orchestra.run.smithi137.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-12T08:40:16.246 INFO:teuthology.orchestra.run.smithi137.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-12T08:40:16.247 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages were automatically installed and are no longer required: 2022-07-12T08:40:16.248 INFO:teuthology.orchestra.run.smithi137.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-12T08:40:16.248 INFO:teuthology.orchestra.run.smithi137.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-12T08:40:16.249 INFO:teuthology.orchestra.run.smithi137.stdout:The following additional packages will be installed: 2022-07-12T08:40:16.249 INFO:teuthology.orchestra.run.smithi137.stdout: libfl-dev libfl2 liblzo2-2 libssl1.1 2022-07-12T08:40:16.249 INFO:teuthology.orchestra.run.smithi137.stdout:Suggested packages: 2022-07-12T08:40:16.249 INFO:teuthology.orchestra.run.smithi137.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-12T08:40:16.282 INFO:teuthology.orchestra.run.smithi137.stdout:The following NEW packages will be installed: 2022-07-12T08:40:16.291 INFO:teuthology.orchestra.run.smithi137.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-12T08:40:16.292 INFO:teuthology.orchestra.run.smithi137.stdout: libssl-dev 2022-07-12T08:40:16.292 INFO:teuthology.orchestra.run.smithi137.stdout:The following packages will be upgraded: 2022-07-12T08:40:16.293 INFO:teuthology.orchestra.run.smithi137.stdout: libssl1.1 2022-07-12T08:40:16.328 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:16.330 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:16.468 INFO:teuthology.orchestra.run.smithi137.stdout:1 upgraded, 10 newly installed, 0 to remove and 17 not upgraded. 2022-07-12T08:40:16.469 INFO:teuthology.orchestra.run.smithi137.stdout:Need to get 4,234 kB of archives. 2022-07-12T08:40:16.469 INFO:teuthology.orchestra.run.smithi137.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-12T08:40:16.470 INFO:teuthology.orchestra.run.smithi137.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-12T08:40:16.510 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-12T08:40:16.529 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.16) over (1.1.1f-1ubuntu2.15) ... 2022-07-12T08:40:16.530 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:16.536 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:16.538 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:16.570 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:16.664 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:16.678 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:16.794 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:16.805 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:16.806 INFO:teuthology.orchestra.run.smithi121.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-12T08:40:16.824 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:16.868 INFO:teuthology.orchestra.run.smithi121.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:16.924 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:16.943 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up dump (0.4b46-6) ... 2022-07-12T08:40:16.944 INFO:teuthology.orchestra.run.smithi137.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.16 [1,321 kB] 2022-07-12T08:40:17.023 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:17.038 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package bison. 2022-07-12T08:40:17.038 INFO:teuthology.orchestra.run.smithi111.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-12T08:40:17.054 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-12T08:40:17.105 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:17.134 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:17.154 INFO:teuthology.orchestra.run.smithi137.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-12T08:40:17.157 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up dump (0.4b46-6) ... 2022-07-12T08:40:17.180 INFO:teuthology.orchestra.run.smithi111.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-12T08:40:17.197 INFO:teuthology.orchestra.run.smithi137.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-12T08:40:17.199 INFO:teuthology.orchestra.run.smithi137.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-12T08:40:17.207 INFO:teuthology.orchestra.run.smithi137.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-12T08:40:17.210 INFO:teuthology.orchestra.run.smithi137.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-12T08:40:17.211 INFO:teuthology.orchestra.run.smithi137.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-12T08:40:17.212 INFO:teuthology.orchestra.run.smithi137.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-12T08:40:17.213 INFO:teuthology.orchestra.run.smithi137.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-12T08:40:17.214 INFO:teuthology.orchestra.run.smithi137.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-12T08:40:17.231 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up indent (2.2.12-1) ... 2022-07-12T08:40:17.246 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-12T08:40:17.309 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:17.392 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-12T08:40:17.393 INFO:teuthology.orchestra.run.smithi111.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:17.408 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-12T08:40:17.487 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up indent (2.2.12-1) ... 2022-07-12T08:40:17.538 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:17.543 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:17.576 INFO:teuthology.orchestra.run.smithi137.stdout:Preconfiguring packages ... 2022-07-12T08:40:17.577 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-12T08:40:17.596 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-07-12T08:40:17.613 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:17.639 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:17.728 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:17.749 INFO:teuthology.orchestra.run.smithi137.stdout:Fetched 4,234 kB in 1s (4,326 kB/s) 2022-07-12T08:40:17.756 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:17.887 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package flex. 2022-07-12T08:40:17.899 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:17.941 INFO:teuthology.orchestra.run.smithi137.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 146190 files and directories currently installed.) 2022-07-12T08:40:17.947 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../00-flex_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:17.987 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package dump. 2022-07-12T08:40:17.989 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-12T08:40:18.009 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-07-12T08:40:18.045 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking dump (0.4b46-6) ... 2022-07-12T08:40:18.051 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:18.193 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:18.326 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:18.358 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package indent. 2022-07-12T08:40:18.374 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-07-12T08:40:18.406 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking indent (2.2.12-1) ... 2022-07-12T08:40:18.451 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:18.620 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:18.630 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../01-libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:18.746 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up dump (0.4b46-6) ... 2022-07-12T08:40:18.752 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-12T08:40:18.770 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-12T08:40:18.829 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.16) over (1.1.1f-1ubuntu2.15) ... 2022-07-12T08:40:18.851 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-12T08:40:18.852 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:18.913 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:18.995 INFO:teuthology.orchestra.run.smithi121.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-12T08:40:19.091 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up indent (2.2.12-1) ... 2022-07-12T08:40:19.151 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-12T08:40:19.170 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-12T08:40:19.205 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:19.234 INFO:teuthology.orchestra.run.smithi121.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:19.361 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:19.399 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package bison. 2022-07-12T08:40:19.415 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../02-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-12T08:40:19.462 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:19.616 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-12T08:40:19.617 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:19.663 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:19.688 INFO:teuthology.orchestra.run.smithi111.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-12T08:40:19.886 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-12T08:40:19.937 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-12T08:40:19.962 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-12T08:40:19.963 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2022-07-12T08:40:19.975 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:19.995 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:20.037 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:20.310 INFO:teuthology.orchestra.run.smithi130.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-12T08:40:20.326 INFO:teuthology.orchestra.run.smithi130.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:20.372 INFO:teuthology.orchestra.run.smithi130.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:20.373 INFO:teuthology.orchestra.run.smithi111.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:20.375 DEBUG:teuthology.parallel:result is None 2022-07-12T08:40:20.399 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package dump. 2022-07-12T08:40:20.418 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2022-07-12T08:40:20.459 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking dump (0.4b46-6) ... 2022-07-12T08:40:20.576 INFO:teuthology.orchestra.run.smithi061.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:20.580 DEBUG:teuthology.parallel:result is None 2022-07-12T08:40:20.787 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package indent. 2022-07-12T08:40:20.802 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../05-indent_2.2.12-1_amd64.deb ... 2022-07-12T08:40:20.848 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking indent (2.2.12-1) ... 2022-07-12T08:40:20.876 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-12T08:40:21.017 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:21.136 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-12T08:40:21.152 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-12T08:40:21.182 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:21.322 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:21.397 INFO:teuthology.orchestra.run.smithi121.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-12T08:40:21.477 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:21.499 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-12T08:40:21.513 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../07-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-12T08:40:21.554 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:21.603 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:21.767 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:21.882 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:21.903 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-12T08:40:21.915 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../08-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:21.949 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:21.983 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up dump (0.4b46-6) ... 2022-07-12T08:40:22.104 INFO:teuthology.orchestra.run.smithi130.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-12T08:40:22.144 INFO:teuthology.orchestra.run.smithi121.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:22.145 DEBUG:teuthology.parallel:result is None 2022-07-12T08:40:22.170 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:22.216 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-12T08:40:22.229 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../09-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-12T08:40:22.258 INFO:teuthology.orchestra.run.smithi130.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-12T08:40:22.268 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:22.353 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up indent (2.2.12-1) ... 2022-07-12T08:40:22.480 INFO:teuthology.orchestra.run.smithi130.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:22.554 INFO:teuthology.orchestra.run.smithi137.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-12T08:40:22.575 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing to unpack .../10-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-12T08:40:22.612 INFO:teuthology.orchestra.run.smithi137.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:22.648 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:22.785 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:23.109 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-12T08:40:23.260 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:23.627 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-12T08:40:23.754 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-12T08:40:23.910 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-12T08:40:24.036 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-12T08:40:24.197 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:24.322 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up dump (0.4b46-6) ... 2022-07-12T08:40:24.417 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-12T08:40:24.526 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-12T08:40:24.614 INFO:teuthology.orchestra.run.smithi137.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-12T08:40:24.713 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up indent (2.2.12-1) ... 2022-07-12T08:40:24.737 INFO:teuthology.orchestra.run.smithi130.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-12T08:40:24.874 INFO:teuthology.orchestra.run.smithi137.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-12T08:40:24.996 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-12T08:40:25.171 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-12T08:40:25.496 INFO:teuthology.orchestra.run.smithi130.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:25.497 DEBUG:teuthology.parallel:result is None 2022-07-12T08:40:27.109 INFO:teuthology.orchestra.run.smithi137.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-12T08:40:27.840 INFO:teuthology.orchestra.run.smithi137.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-12T08:40:27.843 DEBUG:teuthology.parallel:result is None 2022-07-12T08:40:27.844 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2022-07-12T08:40:27.909 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2022-07-12T08:40:27.916 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:40:27.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:40:28.105 DEBUG:teuthology.orchestra.run.smithi061:> dpkg-query -W -f '${Version}' ceph 2022-07-12T08:40:28.133 INFO:teuthology.orchestra.run.smithi061.stdout:15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:28.134 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:28.135 INFO:teuthology.task.install:The correct ceph version 15.2.16-324-g3497be4c-1focal is installed. 2022-07-12T08:40:28.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo lsb_release -is 2022-07-12T08:40:28.205 INFO:teuthology.orchestra.run.smithi111.stdout:Ubuntu 2022-07-12T08:40:28.218 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:40:28.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:40:28.401 DEBUG:teuthology.orchestra.run.smithi111:> dpkg-query -W -f '${Version}' ceph 2022-07-12T08:40:28.427 INFO:teuthology.orchestra.run.smithi111.stdout:15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:28.428 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:28.428 INFO:teuthology.task.install:The correct ceph version 15.2.16-324-g3497be4c-1focal is installed. 2022-07-12T08:40:28.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo lsb_release -is 2022-07-12T08:40:28.495 INFO:teuthology.orchestra.run.smithi121.stdout:Ubuntu 2022-07-12T08:40:28.501 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:40:28.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:40:28.697 DEBUG:teuthology.orchestra.run.smithi121:> dpkg-query -W -f '${Version}' ceph 2022-07-12T08:40:28.731 INFO:teuthology.orchestra.run.smithi121.stdout:15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:28.731 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:28.732 INFO:teuthology.task.install:The correct ceph version 15.2.16-324-g3497be4c-1focal is installed. 2022-07-12T08:40:28.733 DEBUG:teuthology.orchestra.run.smithi130:> sudo lsb_release -is 2022-07-12T08:40:28.790 INFO:teuthology.orchestra.run.smithi130.stdout:Ubuntu 2022-07-12T08:40:28.801 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:40:28.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:40:28.980 DEBUG:teuthology.orchestra.run.smithi130:> dpkg-query -W -f '${Version}' ceph 2022-07-12T08:40:29.037 INFO:teuthology.orchestra.run.smithi130.stdout:15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:29.037 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:29.038 INFO:teuthology.task.install:The correct ceph version 15.2.16-324-g3497be4c-1focal is installed. 2022-07-12T08:40:29.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo lsb_release -is 2022-07-12T08:40:29.095 INFO:teuthology.orchestra.run.smithi137.stdout:Ubuntu 2022-07-12T08:40:29.099 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-12T08:40:29.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=3497be4c2159a20e7a0f23fb5e2730060a4424c0 2022-07-12T08:40:29.281 DEBUG:teuthology.orchestra.run.smithi137:> dpkg-query -W -f '${Version}' ceph 2022-07-12T08:40:29.310 INFO:teuthology.orchestra.run.smithi137.stdout:15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:29.311 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-324-g3497be4c-1focal 2022-07-12T08:40:29.312 INFO:teuthology.task.install:The correct ceph version 15.2.16-324-g3497be4c-1focal is installed. 2022-07-12T08:40:29.313 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-07-12T08:40:29.314 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:29.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-12T08:40:29.334 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:29.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-12T08:40:29.358 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:40:29.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-12T08:40:29.379 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:40:29.380 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-12T08:40:29.399 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:40:29.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-12T08:40:29.420 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-07-12T08:40:29.420 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:29.421 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2022-07-12T08:40:29.440 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-12T08:40:29.512 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:29.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2022-07-12T08:40:29.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-12T08:40:29.608 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:40:29.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2022-07-12T08:40:29.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-12T08:40:29.706 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:40:29.711 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2022-07-12T08:40:29.729 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-12T08:40:29.808 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:40:29.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2022-07-12T08:40:29.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-12T08:40:29.871 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-07-12T08:40:29.871 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:29.872 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-12T08:40:29.893 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-12T08:40:29.916 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:29.916 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-12T08:40:29.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-12T08:40:29.971 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:40:29.972 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-12T08:40:30.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-12T08:40:30.055 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:40:30.055 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-12T08:40:30.084 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-12T08:40:30.147 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:40:30.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-12T08:40:30.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-12T08:40:30.211 INFO:teuthology.run_tasks:Running task ceph... 2022-07-12T08:40:30.351 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-07-12T08:40:30.352 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /var/log/ceph 2022-07-12T08:40:30.366 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2022-07-12T08:40:30.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2022-07-12T08:40:30.385 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /var/log/ceph 2022-07-12T08:40:30.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2022-07-12T08:40:30.406 INFO:tasks.ceph:Disabling ceph logrotate... 2022-07-12T08:40:30.407 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-12T08:40:30.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-12T08:40:30.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-12T08:40:30.448 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-12T08:40:30.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-12T08:40:30.468 INFO:tasks.ceph:Creating extra log directories... 2022-07-12T08:40:30.469 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-12T08:40:30.482 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-12T08:40:30.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-12T08:40:30.516 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-12T08:40:30.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-12T08:40:30.556 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-07-12T08:40:30.568 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-07-12T08:40:30.569 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-07-12T08:40:30.569 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:30.573 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-12T08:40:30.613 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-12T08:40:30.661 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:30.661 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-12T08:40:30.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-12T08:40:30.765 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:40:30.766 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-12T08:40:30.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-12T08:40:30.919 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:40:30.920 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-12T08:40:30.932 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-12T08:40:30.968 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:40:30.969 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-12T08:40:30.991 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-12T08:40:31.053 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-07-12T08:40:31.054 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-07-12T08:40:31.055 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138', 'branch': 'octopus-release', 'description': 'fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6925138', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'octopus-release'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'sha1': '3497be4c2159a20e7a0f23fb5e2730060a4424c0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3497be4c2159a20e7a0f23fb5e2730060a4424c0'}}, 'workunit': {'branch': 'octopus-release', 'sha1': '297f944c54c8d8b03d4faee671f8bb06163c8262'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'], ['client.0'], ['client.1'], ['client.2']], 'sha1': '3497be4c2159a20e7a0f23fb5e2730060a4424c0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'octopus-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '297f944c54c8d8b03d4faee671f8bb06163c8262', 'targets': {'smithi061.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcA6TQiN3zR/I+ltfH7Nxg/LUprQW5CqYuCWCsUmo9hGE0XK9JhvGjPwBdpRtiHgS5zPzGCEqMTSEG1rF4IyfwCGDag12rFCVFS1/f3ZnVxIO0Pfix5Y7ZswXbE7Hht0ucOBTuPCWRgNTuYENlRPN4eM2VkqMseBkdhPdLt3pUvSZvyDF9AtcVnRvk7Apy0/gy/B6Sy12svpD8h4OZKT9IlNAMOk219Obt6ef7AgcP6UCG9NSpIczcJ9jT2X/qbfX95o5cErKVKok2Y2cTAH2vUwxiQVBMtw074iomArFc2WQUgRUwAEatg185Vn7ExcVTTNRa6CyYjdIw3Yl4VmTUV2xsfb7aFXpfP/dUc9dNYWTrZdnBfluWWtFRNQQGzpEWYYvTx5uR9Mb1T3L+HhDmPZU1+SCvnwlzjBmEBK/2NHzvfrx+1SDWGNTYypGIhWAavp7fQgW7Jz6AQQdgIMoZ3wC5YHbrfwFRZYievnzqZdEEACpRS5QCn723AtzjNTM=', 'smithi111.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClZsT/xqX5iJXlxLK1aiwPToIUof+I7P9TZhSCfSZBmvcu1XFlnacotQnYynA3cqeKrfYgiC+JWAWzRclNsAyklBcgQ5nb6g5cpTfXXZGHpWjVYyQas7ee8J+ia9uZiWMRpoT4jQVF/iaru6uVuWe6owr25I8GKLs+dTihjj4o53huk8Sewq371Ita0J2KTXGbKCXQ6LpsG6QQzV8mKNsyHP5HTu9BjKFudZdQcNFiJHePDrl1ERrO9uJM3cll1Bv9rvA5EojiN6GnAZDEHwPUE0pb6D79dgSKk+4Syb0nPfZd3NnCgaPLF+7Sg9KjWRcqMrb49NOjPWHec9Fp1Gy6myfOBgl9NdeXmx9ZGGfVLMHqmVtiX2S34Ggg7ikximWG28CoLcGHIQeGXPIZfMnojjE7fZygvEt89UIqOgQoFVmBcD1PW09CXeSq5M54Oxh/3AokY74V0rTGxpnGuxolzE3kvkk9YnK4yyk1xU92LF6+LDrYL7uuuE0bidFN8Ec=', 'smithi121.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtVHtPFaqwQgZWMH5G5ZAhsegh+TQt0aqtK0u0xBJRy2azk1e+RgirtD3cOePJUHQ8qvZsrQpjqSD66LPMMg1QJ78nQggGT3QTXUJ7eDp9tTGvEIN4hDDhoVcSBQMLf5poh9N/DWcnbKlCZfWu8MJLxozmsLnu8u2AtwfwE2KKiu2drSCAzwuEBH24M5jOaXI2TkDso2JXarWWrH/wRTGyYPmjssAd4Ne9OgRwpOm9tUgCVds9bbt3W+XlD3X/3wlPJ8x3anRh+ySG0WIhgX5vrWY4BIM/BPBySo3z+m3fxjBJSzF0SkqNCfZCGYECNRXzayOFw+wJJRwnwvhq7oqEdzuMFqt5FDF9ZV7iGdq2fHy01MhciJr2FlsEbcVgc/LkDhUZj5iM5NJC4kxw0KFnh8NFy0xRhKrj1fAKZTo3UWc97ccZZZsKI3MrkwpuQDsE8OJpw87XVKvSTuOYzywqybchoOZmsFjUIrNMFN83JrU2etPdsgfk9GKRD4PGbH0=', 'smithi130.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsm5mhOa0dBhV447y5mEll4JzcGa8Fq1/xsZv2bNlHY6g1vbLr9FLaBYFUqKqQaCylM/b6K5vMwfHmjNZG002uikir2nggACl8YRv5AGyhStEW6mrEBhmu/0vp/ahlNpxYiOwF1Qrm3suqIhPUP4oZmTRen0uJ9a72BbMeCebBKaPIJbvyhQg53FjSPzTkzc2zxiRBLXg31mg09lb6KUtlXc3YkW4aH7GLjoQEgovGgxNrLFfn3XvQQaWPi1cI8M42wOF6gEUfV1Ao+4+nMdOWxxtcAw1uHACTG/7hesXaLVlFX95CHcRmFdqqtwr2IkMdWwNNuj3djDzXE/jPB0aQFMRhGXIMAYV1cdtL9XSWAt9PZw6pDNO2XsIyaveuf/Dq04bD5lS5jpAm3iNXpyFxnJW7eEVAZJGzk0zRgrMQc7C9hQf07ePgAfC8j5jaUL06Ip3CKsISiBZ31rOGefGvcaGvWVXp1AxCKRrC0JlZg0YIwzFYrl0J+Y3phCHK2hU=', 'smithi137.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnGrqG4QeVKOS3Hk30msD9taDO7J+ShxeQOBzSQB2PdOymmRpBAG3l2s2OqkHZuYpBulDWEOqtk8EkAtb7iYA2WUPVU4y5/Ksy47tYbl4ueb2NEoVqqIBe4y5L1560pSyDV9QTYpJ2affmkXieo9R2bfxxI2Rv35XAv6Los9VwOWQqdOSG3M8PDzs6zNNCTJb/GoXDy4mwS0Fi4ZWz9+xGLCu6bIjlvJaCtT9LtIbLfvlRGw85dFb0eMg0HIY7d9UFS+sa/qqoEfw//mteITTmoEoqbaLBc+xWpp03GoyYUZL0zf9/4oP8oLZ3VHCz/YMekARxIVPHpTTT5caHsa1Tv+N2E/nPE54P8Ee727QqgR7EeVMCDRgJ8QIUbPTe7c4cRoLMw++3XNbaff+m/bbLTHno+YrFxZ8Of6Pqzvh+K7IGi+OXOLlo2B/D0whq4IC2H0htewEuDPMvSA+K/3XEI70xhS/6pwLqcF+OVpkZUIaDyXjjZ9sTc2G1pJxR0kM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '3497be4c2159a20e7a0f23fb5e2730060a4424c0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_misc']}}], 'teuthology_branch': 'main', 'teuthology_sha1': '10062088f503b43eff3624326bda825b23438f9b', 'timestamp': '2022-07-11_23:41:40', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-07-12T08:40:31.056 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-12T08:40:31.093 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-12T08:40:31.100 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-12T08:40:31.108 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-12T08:40:31.122 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-12T08:40:31.131 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2022-07-12T08:40:31.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2022-07-12T08:40:31.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2022-07-12T08:40:31.164 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m0777 -- /var/run/ceph 2022-07-12T08:40:31.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2022-07-12T08:40:31.193 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:31.193 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/scratch_devs of=/dev/stdout 2022-07-12T08:40:31.212 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-12T08:40:31.213 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-12T08:40:31.220 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-12T08:40:31.220 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.220 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 358 Links: 1 2022-07-12T08:40:31.221 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.221 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-12 08:36:50.577277574 +0000 2022-07-12T08:40:31.221 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-12 08:36:49.849277551 +0000 2022-07-12T08:40:31.221 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-12 08:36:49.849277551 +0000 2022-07-12T08:40:31.222 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-12T08:40:31.230 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-12T08:40:31.231 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-12T08:40:31.231 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000412354 s, 1.2 MB/s 2022-07-12T08:40:31.236 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-12T08:40:31.285 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-12T08:40:31.286 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.286 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 365 Links: 1 2022-07-12T08:40:31.287 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.287 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-12 08:36:51.485277603 +0000 2022-07-12T08:40:31.287 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-12 08:36:50.805277581 +0000 2022-07-12T08:40:31.288 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-12 08:36:50.805277581 +0000 2022-07-12T08:40:31.288 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-12T08:40:31.295 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-12T08:40:31.296 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-12T08:40:31.296 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000410714 s, 1.2 MB/s 2022-07-12T08:40:31.300 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-12T08:40:31.309 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-12T08:40:31.309 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.309 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 372 Links: 1 2022-07-12T08:40:31.310 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.310 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-12 08:36:52.337277631 +0000 2022-07-12T08:40:31.310 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-12 08:36:51.697277610 +0000 2022-07-12T08:40:31.310 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-12 08:36:51.697277610 +0000 2022-07-12T08:40:31.311 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-12T08:40:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-12T08:40:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-12T08:40:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000393353 s, 1.3 MB/s 2022-07-12T08:40:31.323 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-12T08:40:31.333 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-12T08:40:31.333 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.334 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 379 Links: 1 2022-07-12T08:40:31.334 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.334 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-12 08:36:53.141277657 +0000 2022-07-12T08:40:31.335 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-12 08:36:52.573277638 +0000 2022-07-12T08:40:31.335 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-12 08:36:52.573277638 +0000 2022-07-12T08:40:31.335 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-12T08:40:31.350 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-12T08:40:31.351 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-12T08:40:31.351 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000399688 s, 1.3 MB/s 2022-07-12T08:40:31.352 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:31.353 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:31.353 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2022-07-12T08:40:31.391 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-12T08:40:31.401 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-12T08:40:31.450 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-12T08:40:31.451 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.452 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 359 Links: 1 2022-07-12T08:40:31.452 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.453 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-07-12 08:36:49.950054652 +0000 2022-07-12T08:40:31.453 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-07-12 08:36:49.946054652 +0000 2022-07-12T08:40:31.454 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-07-12 08:36:49.946054652 +0000 2022-07-12T08:40:31.454 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-07-12T08:40:31.456 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-07-12T08:40:31.456 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-07-12T08:40:31.457 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000410821 s, 1.2 MB/s 2022-07-12T08:40:31.461 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-12T08:40:31.476 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-12T08:40:31.477 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.479 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 368 Links: 1 2022-07-12T08:40:31.482 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.483 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-07-12 08:36:51.506054617 +0000 2022-07-12T08:40:31.483 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-07-12 08:36:50.874054631 +0000 2022-07-12T08:40:31.484 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-07-12 08:36:50.874054631 +0000 2022-07-12T08:40:31.484 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-07-12T08:40:31.500 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-07-12T08:40:31.500 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-07-12T08:40:31.501 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000410393 s, 1.2 MB/s 2022-07-12T08:40:31.502 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-12T08:40:31.551 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-12T08:40:31.552 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.552 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 371 Links: 1 2022-07-12T08:40:31.553 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-07-12 08:36:52.446054597 +0000 2022-07-12T08:40:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-07-12 08:36:51.714054613 +0000 2022-07-12T08:40:31.556 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-07-12 08:36:51.714054613 +0000 2022-07-12T08:40:31.556 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-07-12T08:40:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-07-12T08:40:31.560 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-07-12T08:40:31.561 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000437601 s, 1.2 MB/s 2022-07-12T08:40:31.565 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-12T08:40:31.575 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-12T08:40:31.576 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-12T08:40:31.576 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 6h/6d Inode: 380 Links: 1 2022-07-12T08:40:31.576 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-12T08:40:31.577 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2022-07-12 08:36:53.254054579 +0000 2022-07-12T08:40:31.577 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2022-07-12 08:36:52.674054592 +0000 2022-07-12T08:40:31.578 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2022-07-12 08:36:52.674054592 +0000 2022-07-12T08:40:31.578 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: - 2022-07-12T08:40:31.585 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2022-07-12T08:40:31.585 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2022-07-12T08:40:31.586 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000392524 s, 1.3 MB/s 2022-07-12T08:40:31.588 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:31.590 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-12T08:40:31.590 INFO:tasks.ceph:Generating config... 2022-07-12T08:40:31.594 INFO:tasks.ceph:[client] client mount timeout = 600 2022-07-12T08:40:31.595 INFO:tasks.ceph:[client] debug client = 20 2022-07-12T08:40:31.595 INFO:tasks.ceph:[client] debug ms = 1 2022-07-12T08:40:31.596 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2022-07-12T08:40:31.596 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2022-07-12T08:40:31.597 INFO:tasks.ceph:[mds] debug mds = 20 2022-07-12T08:40:31.598 INFO:tasks.ceph:[mds] debug ms = 1 2022-07-12T08:40:31.598 INFO:tasks.ceph:[mds] mds bal frag = True 2022-07-12T08:40:31.599 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-07-12T08:40:31.600 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-07-12T08:40:31.600 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-07-12T08:40:31.601 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-07-12T08:40:31.601 INFO:tasks.ceph:[mds] mds debug frag = True 2022-07-12T08:40:31.602 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-07-12T08:40:31.603 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-07-12T08:40:31.604 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-07-12T08:40:31.605 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-07-12T08:40:31.605 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2022-07-12T08:40:31.612 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2022-07-12T08:40:31.617 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-07-12T08:40:31.618 INFO:tasks.ceph:[mgr] debug ms = 1 2022-07-12T08:40:31.619 INFO:tasks.ceph:[mon] debug mon = 20 2022-07-12T08:40:31.619 INFO:tasks.ceph:[mon] debug ms = 1 2022-07-12T08:40:31.620 INFO:tasks.ceph:[mon] debug paxos = 20 2022-07-12T08:40:31.621 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-07-12T08:40:31.622 INFO:tasks.ceph:[osd] debug ms = 1 2022-07-12T08:40:31.623 INFO:tasks.ceph:[osd] debug osd = 20 2022-07-12T08:40:31.623 INFO:tasks.ceph:[osd] osd objectstore = filestore 2022-07-12T08:40:31.627 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-07-12T08:40:31.628 INFO:tasks.ceph:[osd] osd sloppy crc = True 2022-07-12T08:40:31.629 INFO:tasks.ceph:Setting up mon.a... 2022-07-12T08:40:31.630 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-07-12T08:40:31.683 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/ceph.keyring 2022-07-12T08:40:31.684 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-07-12T08:40:31.724 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-12T08:40:31.805 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.61'), ('mon.b', '172.21.15.111'), ('mon.c', '[v2:172.21.15.111:3301,v1:172.21.15.111:6790]')] 2022-07-12T08:40:31.806 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.61 --add b 172.21.15.111 --addv c '[v2:172.21.15.111:3301,v1:172.21.15.111:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:31.894 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:31.895 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: generated fsid 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:31.895 INFO:teuthology.orchestra.run.smithi061.stdout:epoch 0 2022-07-12T08:40:31.896 INFO:teuthology.orchestra.run.smithi061.stdout:fsid 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:31.896 INFO:teuthology.orchestra.run.smithi061.stdout:last_changed 2022-07-12T08:40:31.897863+0000 2022-07-12T08:40:31.897 INFO:teuthology.orchestra.run.smithi061.stdout:created 2022-07-12T08:40:31.897863+0000 2022-07-12T08:40:31.897 INFO:teuthology.orchestra.run.smithi061.stdout:min_mon_release 0 (unknown) 2022-07-12T08:40:31.898 INFO:teuthology.orchestra.run.smithi061.stdout:0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.a 2022-07-12T08:40:31.898 INFO:teuthology.orchestra.run.smithi061.stdout:1: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.b 2022-07-12T08:40:31.899 INFO:teuthology.orchestra.run.smithi061.stdout:2: [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] mon.c 2022-07-12T08:40:31.899 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-07-12T08:40:31.903 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a... 2022-07-12T08:40:31.947 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:40:31.962 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:40:31.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:40:31.979 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:40:31.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:40:32.002 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2022-07-12T08:40:32.003 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2022-07-12T08:40:32.003 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:40:32.004 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2022-07-12T08:40:32.004 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.005 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2022-07-12T08:40:32.005 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.006 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2022-07-12T08:40:32.007 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.007 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2022-07-12T08:40:32.007 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2022-07-12T08:40:32.008 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.008 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2022-07-12T08:40:32.009 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2022-07-12T08:40:32.009 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.010 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:40:32.010 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2022-07-12T08:40:32.011 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.011 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:40:32.011 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:40:32.012 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.012 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2022-07-12T08:40:32.012 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.013 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2022-07-12T08:40:32.013 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2022-07-12T08:40:32.014 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:40:32.014 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:40:32.014 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2022-07-12T08:40:32.015 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:40:32.015 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2022-07-12T08:40:32.015 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:40:32.016 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:40:32.016 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.016 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:40:32.017 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.017 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2022-07-12T08:40:32.018 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.019 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2022-07-12T08:40:32.019 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.019 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2022-07-12T08:40:32.020 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2022-07-12T08:40:32.020 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2022-07-12T08:40:32.021 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:40:32.021 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:32.022 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.022 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2022-07-12T08:40:32.023 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2022-07-12T08:40:32.024 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.024 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2022-07-12T08:40:32.025 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2022-07-12T08:40:32.025 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.025 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2022-07-12T08:40:32.026 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2022-07-12T08:40:32.026 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.026 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2022-07-12T08:40:32.027 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2022-07-12T08:40:32.027 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2022-07-12T08:40:32.028 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.028 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2022-07-12T08:40:32.028 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2022-07-12T08:40:32.029 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.029 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:40:32.030 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.031 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2022-07-12T08:40:32.031 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.031 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2022-07-12T08:40:32.032 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.033 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:40:32.033 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2022-07-12T08:40:32.034 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.034 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2022-07-12T08:40:32.035 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2022-07-12T08:40:32.035 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:40:32.036 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 20 2022-07-12T08:40:32.036 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = filestore 2022-07-12T08:40:32.036 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-12T08:40:32.037 INFO:teuthology.orchestra.run.smithi061.stdout: osd sloppy crc = True 2022-07-12T08:40:32.037 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.038 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2022-07-12T08:40:32.038 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:40:32.038 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2022-07-12T08:40:32.039 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-12T08:40:32.039 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-12T08:40:32.040 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.040 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.040 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2022-07-12T08:40:32.041 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.041 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2022-07-12T08:40:32.042 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:40:32.042 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-12T08:40:32.043 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2022-07-12T08:40:32.043 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-12T08:40:32.044 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2022-07-12T08:40:32.045 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2022-07-12T08:40:32.046 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.047 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2022-07-12T08:40:32.047 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2022-07-12T08:40:32.048 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.079 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.080 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:40:32.080 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:40:32.080 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:40:32.081 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.081 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:40:32.090 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:40:32.091 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:40:32.091 INFO:teuthology.orchestra.run.smithi061.stdout: mon op complaint time = 120 2022-07-12T08:40:32.092 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:40:32.092 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2022-07-12T08:40:32.093 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2022-07-12T08:40:32.093 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2022-07-12T08:40:32.093 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2022-07-12T08:40:32.094 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:40:32.094 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:40:32.095 INFO:teuthology.orchestra.run.smithi061.stdout: client mount timeout = 600 2022-07-12T08:40:32.095 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 20 2022-07-12T08:40:32.095 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:40:32.096 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.096 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.097 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2022-07-12T08:40:32.097 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.b] 2022-07-12T08:40:32.097 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.c] 2022-07-12T08:40:32.098 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-12T08:40:32.098 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-12T08:40:32.099 INFO:teuthology.orchestra.run.smithi061.stdout:[mds] 2022-07-12T08:40:32.099 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 20 2022-07-12T08:40:32.099 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:40:32.100 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal frag = True 2022-07-12T08:40:32.101 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal fragment size max = 10000 2022-07-12T08:40:32.101 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal merge size = 5 2022-07-12T08:40:32.101 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split bits = 3 2022-07-12T08:40:32.102 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split size = 100 2022-07-12T08:40:32.102 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug frag = True 2022-07-12T08:40:32.103 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug scatterstat = True 2022-07-12T08:40:32.103 INFO:teuthology.orchestra.run.smithi061.stdout: mds op complaint time = 180 2022-07-12T08:40:32.104 INFO:teuthology.orchestra.run.smithi061.stdout: mds verify scatter = True 2022-07-12T08:40:32.104 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-12T08:40:32.106 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.106 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.107 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2022-07-12T08:40:32.107 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2022-07-12T08:40:32.107 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:40:32.108 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2022-07-12T08:40:32.109 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.109 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2022-07-12T08:40:32.110 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.110 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2022-07-12T08:40:32.111 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.111 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2022-07-12T08:40:32.112 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2022-07-12T08:40:32.112 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.113 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2022-07-12T08:40:32.113 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2022-07-12T08:40:32.114 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.114 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:40:32.114 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2022-07-12T08:40:32.115 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.115 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:40:32.116 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:40:32.117 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.117 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2022-07-12T08:40:32.118 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.119 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2022-07-12T08:40:32.119 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2022-07-12T08:40:32.120 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:40:32.120 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:40:32.120 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2022-07-12T08:40:32.121 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:40:32.121 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2022-07-12T08:40:32.122 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:40:32.122 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:40:32.123 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.123 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:40:32.123 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.124 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2022-07-12T08:40:32.124 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.125 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2022-07-12T08:40:32.125 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.126 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2022-07-12T08:40:32.126 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2022-07-12T08:40:32.127 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2022-07-12T08:40:32.127 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:40:32.128 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:32.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2022-07-12T08:40:32.130 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2022-07-12T08:40:32.130 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.131 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2022-07-12T08:40:32.132 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2022-07-12T08:40:32.133 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.133 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2022-07-12T08:40:32.133 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2022-07-12T08:40:32.134 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.134 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2022-07-12T08:40:32.135 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2022-07-12T08:40:32.135 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2022-07-12T08:40:32.135 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.136 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2022-07-12T08:40:32.136 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2022-07-12T08:40:32.137 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.137 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:40:32.137 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.138 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2022-07-12T08:40:32.138 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.139 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2022-07-12T08:40:32.139 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.140 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:40:32.140 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2022-07-12T08:40:32.140 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.141 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2022-07-12T08:40:32.141 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2022-07-12T08:40:32.142 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:40:32.142 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2022-07-12T08:40:32.143 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = filestore 2022-07-12T08:40:32.143 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2022-07-12T08:40:32.143 INFO:teuthology.orchestra.run.smithi111.stdout: osd sloppy crc = True 2022-07-12T08:40:32.144 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.144 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2022-07-12T08:40:32.145 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:40:32.145 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2022-07-12T08:40:32.146 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2022-07-12T08:40:32.146 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2022-07-12T08:40:32.147 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.147 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.148 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2022-07-12T08:40:32.148 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.149 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2022-07-12T08:40:32.149 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:40:32.149 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2022-07-12T08:40:32.150 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2022-07-12T08:40:32.150 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2022-07-12T08:40:32.151 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2022-07-12T08:40:32.151 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2022-07-12T08:40:32.152 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.152 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2022-07-12T08:40:32.152 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2022-07-12T08:40:32.153 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.153 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.154 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:40:32.154 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:40:32.155 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:40:32.155 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.155 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:40:32.156 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:40:32.156 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:40:32.157 INFO:teuthology.orchestra.run.smithi111.stdout: mon op complaint time = 120 2022-07-12T08:40:32.157 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:40:32.158 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2022-07-12T08:40:32.158 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2022-07-12T08:40:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2022-07-12T08:40:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2022-07-12T08:40:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:40:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:40:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: client mount timeout = 600 2022-07-12T08:40:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2022-07-12T08:40:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:40:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.163 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2022-07-12T08:40:32.164 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2022-07-12T08:40:32.164 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2022-07-12T08:40:32.165 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2022-07-12T08:40:32.165 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2022-07-12T08:40:32.167 INFO:teuthology.orchestra.run.smithi111.stdout:[mds] 2022-07-12T08:40:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds = 20 2022-07-12T08:40:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:40:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal frag = True 2022-07-12T08:40:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal fragment size max = 10000 2022-07-12T08:40:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal merge size = 5 2022-07-12T08:40:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal split bits = 3 2022-07-12T08:40:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal split size = 100 2022-07-12T08:40:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug frag = True 2022-07-12T08:40:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug scatterstat = True 2022-07-12T08:40:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: mds op complaint time = 180 2022-07-12T08:40:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: mds verify scatter = True 2022-07-12T08:40:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2022-07-12T08:40:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.205 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2022-07-12T08:40:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2022-07-12T08:40:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:40:32.219 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2022-07-12T08:40:32.238 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2022-07-12T08:40:32.239 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2022-07-12T08:40:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.240 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2022-07-12T08:40:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2022-07-12T08:40:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2022-07-12T08:40:32.242 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2022-07-12T08:40:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:40:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2022-07-12T08:40:32.245 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:40:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:40:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.247 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2022-07-12T08:40:32.248 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2022-07-12T08:40:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2022-07-12T08:40:32.249 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:40:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:40:32.250 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2022-07-12T08:40:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:40:32.251 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2022-07-12T08:40:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:40:32.252 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:40:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.253 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:40:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.254 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2022-07-12T08:40:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.255 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2022-07-12T08:40:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.256 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2022-07-12T08:40:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2022-07-12T08:40:32.257 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2022-07-12T08:40:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:40:32.258 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:32.259 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.263 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2022-07-12T08:40:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2022-07-12T08:40:32.263 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.264 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2022-07-12T08:40:32.270 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2022-07-12T08:40:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.271 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2022-07-12T08:40:32.272 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2022-07-12T08:40:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2022-07-12T08:40:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2022-07-12T08:40:32.274 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2022-07-12T08:40:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.275 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2022-07-12T08:40:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2022-07-12T08:40:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.277 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:40:32.277 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.302 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2022-07-12T08:40:32.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.303 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2022-07-12T08:40:32.303 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.303 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:40:32.304 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2022-07-12T08:40:32.304 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.304 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2022-07-12T08:40:32.304 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2022-07-12T08:40:32.305 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:40:32.305 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2022-07-12T08:40:32.305 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = filestore 2022-07-12T08:40:32.305 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2022-07-12T08:40:32.306 INFO:teuthology.orchestra.run.smithi137.stdout: osd sloppy crc = True 2022-07-12T08:40:32.306 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.306 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2022-07-12T08:40:32.306 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:40:32.307 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2022-07-12T08:40:32.307 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2022-07-12T08:40:32.307 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2022-07-12T08:40:32.307 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.307 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.308 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2022-07-12T08:40:32.308 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.308 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2022-07-12T08:40:32.308 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:40:32.309 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2022-07-12T08:40:32.309 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2022-07-12T08:40:32.309 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2022-07-12T08:40:32.310 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2022-07-12T08:40:32.310 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2022-07-12T08:40:32.310 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.311 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2022-07-12T08:40:32.311 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2022-07-12T08:40:32.311 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.311 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.312 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:40:32.312 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:40:32.312 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:40:32.312 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.312 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:40:32.313 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:40:32.313 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:40:32.313 INFO:teuthology.orchestra.run.smithi137.stdout: mon op complaint time = 120 2022-07-12T08:40:32.313 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:40:32.314 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2022-07-12T08:40:32.314 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2022-07-12T08:40:32.314 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2022-07-12T08:40:32.315 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2022-07-12T08:40:32.315 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:40:32.315 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:40:32.315 INFO:teuthology.orchestra.run.smithi137.stdout: client mount timeout = 600 2022-07-12T08:40:32.316 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2022-07-12T08:40:32.316 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:40:32.317 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.317 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.317 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2022-07-12T08:40:32.318 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2022-07-12T08:40:32.318 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2022-07-12T08:40:32.318 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2022-07-12T08:40:32.319 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2022-07-12T08:40:32.319 INFO:teuthology.orchestra.run.smithi137.stdout:[mds] 2022-07-12T08:40:32.319 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds = 20 2022-07-12T08:40:32.320 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:40:32.320 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal frag = True 2022-07-12T08:40:32.321 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal fragment size max = 10000 2022-07-12T08:40:32.322 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal merge size = 5 2022-07-12T08:40:32.322 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal split bits = 3 2022-07-12T08:40:32.322 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal split size = 100 2022-07-12T08:40:32.322 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug frag = True 2022-07-12T08:40:32.323 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug scatterstat = True 2022-07-12T08:40:32.326 INFO:teuthology.orchestra.run.smithi137.stdout: mds op complaint time = 180 2022-07-12T08:40:32.326 INFO:teuthology.orchestra.run.smithi137.stdout: mds verify scatter = True 2022-07-12T08:40:32.328 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2022-07-12T08:40:32.329 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.330 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.331 INFO:teuthology.orchestra.run.smithi130.stdout:[global] 2022-07-12T08:40:32.332 INFO:teuthology.orchestra.run.smithi130.stdout: chdir = "" 2022-07-12T08:40:32.333 INFO:teuthology.orchestra.run.smithi130.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:40:32.333 INFO:teuthology.orchestra.run.smithi130.stdout: auth supported = cephx 2022-07-12T08:40:32.335 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.335 INFO:teuthology.orchestra.run.smithi130.stdout: filestore xattr use omap = true 2022-07-12T08:40:32.336 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.336 INFO:teuthology.orchestra.run.smithi130.stdout: mon clock drift allowed = 1.000 2022-07-12T08:40:32.337 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.337 INFO:teuthology.orchestra.run.smithi130.stdout: osd crush chooseleaf type = 0 2022-07-12T08:40:32.338 INFO:teuthology.orchestra.run.smithi130.stdout: auth debug = true 2022-07-12T08:40:32.338 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.339 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on old message = true 2022-07-12T08:40:32.339 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on bug = true 2022-07-12T08:40:32.340 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.341 INFO:teuthology.orchestra.run.smithi130.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:40:32.341 INFO:teuthology.orchestra.run.smithi130.stdout: mon pg warn max object skew = 0 2022-07-12T08:40:32.342 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.342 INFO:teuthology.orchestra.run.smithi130.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:40:32.343 INFO:teuthology.orchestra.run.smithi130.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:40:32.343 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.344 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default size = 2 2022-07-12T08:40:32.344 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.345 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow primary affinity = true 2022-07-12T08:40:32.345 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow pg remap = true 2022-07-12T08:40:32.356 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:40:32.356 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:40:32.357 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on no sortbitwise = false 2022-07-12T08:40:32.357 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:40:32.358 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on too few osds = false 2022-07-12T08:40:32.358 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:40:32.358 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:40:32.359 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.359 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:40:32.359 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.360 INFO:teuthology.orchestra.run.smithi130.stdout: osd default data pool replay window = 5 2022-07-12T08:40:32.360 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.360 INFO:teuthology.orchestra.run.smithi130.stdout: mon allow pool delete = true 2022-07-12T08:40:32.361 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.361 INFO:teuthology.orchestra.run.smithi130.stdout: mon cluster log file level = debug 2022-07-12T08:40:32.362 INFO:teuthology.orchestra.run.smithi130.stdout: debug asserts on shutdown = true 2022-07-12T08:40:32.362 INFO:teuthology.orchestra.run.smithi130.stdout: mon health detail to clog = false 2022-07-12T08:40:32.363 INFO:teuthology.orchestra.run.smithi130.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:40:32.363 INFO:teuthology.orchestra.run.smithi130.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:32.363 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.364 INFO:teuthology.orchestra.run.smithi130.stdout:[osd] 2022-07-12T08:40:32.364 INFO:teuthology.orchestra.run.smithi130.stdout: osd journal size = 100 2022-07-12T08:40:32.365 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.365 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub load threshold = 5.0 2022-07-12T08:40:32.366 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub max interval = 600 2022-07-12T08:40:32.366 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.366 INFO:teuthology.orchestra.run.smithi130.stdout: osd recover clone overlap = true 2022-07-12T08:40:32.367 INFO:teuthology.orchestra.run.smithi130.stdout: osd recovery max chunk = 1048576 2022-07-12T08:40:32.367 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.368 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug shutdown = true 2022-07-12T08:40:32.368 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug op order = true 2022-07-12T08:40:32.369 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug verify stray on activate = true 2022-07-12T08:40:32.369 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.370 INFO:teuthology.orchestra.run.smithi130.stdout: osd open classes on start = true 2022-07-12T08:40:32.370 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug pg log writeout = true 2022-07-12T08:40:32.371 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.371 INFO:teuthology.orchestra.run.smithi130.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:40:32.372 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.373 INFO:teuthology.orchestra.run.smithi130.stdout: osd map max advance = 10 2022-07-12T08:40:32.373 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.373 INFO:teuthology.orchestra.run.smithi130.stdout: journal zero on create = true 2022-07-12T08:40:32.374 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.374 INFO:teuthology.orchestra.run.smithi130.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:40:32.375 INFO:teuthology.orchestra.run.smithi130.stdout: filestore apply finisher threads = 3 2022-07-12T08:40:32.375 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.375 INFO:teuthology.orchestra.run.smithi130.stdout: bdev debug aio = true 2022-07-12T08:40:32.376 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug misdirected ops = true 2022-07-12T08:40:32.376 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:40:32.377 INFO:teuthology.orchestra.run.smithi130.stdout: debug osd = 20 2022-07-12T08:40:32.377 INFO:teuthology.orchestra.run.smithi130.stdout: osd objectstore = filestore 2022-07-12T08:40:32.378 INFO:teuthology.orchestra.run.smithi130.stdout: osd op complaint time = 180 2022-07-12T08:40:32.378 INFO:teuthology.orchestra.run.smithi130.stdout: osd sloppy crc = True 2022-07-12T08:40:32.378 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.379 INFO:teuthology.orchestra.run.smithi130.stdout:[mgr] 2022-07-12T08:40:32.379 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:40:32.380 INFO:teuthology.orchestra.run.smithi130.stdout: debug mgr = 20 2022-07-12T08:40:32.380 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2022-07-12T08:40:32.381 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2022-07-12T08:40:32.381 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.382 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.382 INFO:teuthology.orchestra.run.smithi130.stdout: mgr/telemetry/nag = false 2022-07-12T08:40:32.382 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.383 INFO:teuthology.orchestra.run.smithi130.stdout:[mon] 2022-07-12T08:40:32.383 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:40:32.384 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2022-07-12T08:40:32.384 INFO:teuthology.orchestra.run.smithi130.stdout: debug paxos = 20 2022-07-12T08:40:32.385 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2022-07-12T08:40:32.385 INFO:teuthology.orchestra.run.smithi130.stdout: mon data avail warn = 5 2022-07-12T08:40:32.394 INFO:teuthology.orchestra.run.smithi130.stdout: mon mgr mkfs grace = 240 2022-07-12T08:40:32.395 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.395 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd reporter subtree level = osd 2022-07-12T08:40:32.395 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd prime pg temp = true 2022-07-12T08:40:32.396 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.396 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.396 INFO:teuthology.orchestra.run.smithi130.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:40:32.396 INFO:teuthology.orchestra.run.smithi130.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:40:32.396 INFO:teuthology.orchestra.run.smithi130.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:40:32.397 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.397 INFO:teuthology.orchestra.run.smithi130.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:40:32.397 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:40:32.397 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:40:32.398 INFO:teuthology.orchestra.run.smithi130.stdout: mon op complaint time = 120 2022-07-12T08:40:32.398 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:40:32.398 INFO:teuthology.orchestra.run.smithi130.stdout:[client] 2022-07-12T08:40:32.399 INFO:teuthology.orchestra.run.smithi130.stdout: rgw cache enabled = true 2022-07-12T08:40:32.400 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable ops log = true 2022-07-12T08:40:32.400 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable usage log = true 2022-07-12T08:40:32.400 INFO:teuthology.orchestra.run.smithi130.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:40:32.401 INFO:teuthology.orchestra.run.smithi130.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:40:32.401 INFO:teuthology.orchestra.run.smithi130.stdout: client mount timeout = 600 2022-07-12T08:40:32.402 INFO:teuthology.orchestra.run.smithi130.stdout: debug client = 20 2022-07-12T08:40:32.403 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:40:32.404 INFO:teuthology.orchestra.run.smithi130.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.417 INFO:teuthology.orchestra.run.smithi130.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.417 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.a] 2022-07-12T08:40:32.417 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.b] 2022-07-12T08:40:32.417 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.c] 2022-07-12T08:40:32.418 INFO:teuthology.orchestra.run.smithi130.stdout:[mds.a] 2022-07-12T08:40:32.418 INFO:teuthology.orchestra.run.smithi130.stdout:[mds.b] 2022-07-12T08:40:32.418 INFO:teuthology.orchestra.run.smithi130.stdout:[mds] 2022-07-12T08:40:32.419 INFO:teuthology.orchestra.run.smithi130.stdout: debug mds = 20 2022-07-12T08:40:32.419 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:40:32.419 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal frag = True 2022-07-12T08:40:32.419 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal fragment size max = 10000 2022-07-12T08:40:32.419 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal merge size = 5 2022-07-12T08:40:32.420 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal split bits = 3 2022-07-12T08:40:32.420 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal split size = 100 2022-07-12T08:40:32.420 INFO:teuthology.orchestra.run.smithi130.stdout: mds debug frag = True 2022-07-12T08:40:32.420 INFO:teuthology.orchestra.run.smithi130.stdout: mds debug scatterstat = True 2022-07-12T08:40:32.421 INFO:teuthology.orchestra.run.smithi130.stdout: mds op complaint time = 180 2022-07-12T08:40:32.421 INFO:teuthology.orchestra.run.smithi130.stdout: mds verify scatter = True 2022-07-12T08:40:32.421 INFO:teuthology.orchestra.run.smithi130.stdout: osd op complaint time = 180 2022-07-12T08:40:32.421 INFO:teuthology.orchestra.run.smithi130.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.421 INFO:teuthology.orchestra.run.smithi130.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.422 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2022-07-12T08:40:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2022-07-12T08:40:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:40:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2022-07-12T08:40:32.423 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2022-07-12T08:40:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2022-07-12T08:40:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.424 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2022-07-12T08:40:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2022-07-12T08:40:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2022-07-12T08:40:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2022-07-12T08:40:32.425 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:40:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2022-07-12T08:40:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.426 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:40:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:40:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2022-07-12T08:40:32.427 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2022-07-12T08:40:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2022-07-12T08:40:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:40:32.428 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:40:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2022-07-12T08:40:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:40:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2022-07-12T08:40:32.429 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:40:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:40:32.430 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.431 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:40:32.431 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.431 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2022-07-12T08:40:32.431 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.432 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2022-07-12T08:40:32.432 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.432 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2022-07-12T08:40:32.432 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2022-07-12T08:40:32.432 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2022-07-12T08:40:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:40:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:40:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.433 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2022-07-12T08:40:32.433 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2022-07-12T08:40:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2022-07-12T08:40:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2022-07-12T08:40:32.434 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2022-07-12T08:40:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2022-07-12T08:40:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2022-07-12T08:40:32.435 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2022-07-12T08:40:32.436 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2022-07-12T08:40:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2022-07-12T08:40:32.438 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2022-07-12T08:40:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:40:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.439 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2022-07-12T08:40:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2022-07-12T08:40:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:40:32.440 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2022-07-12T08:40:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2022-07-12T08:40:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2022-07-12T08:40:32.441 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:40:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2022-07-12T08:40:32.442 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = filestore 2022-07-12T08:40:32.444 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-07-12T08:40:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: osd sloppy crc = True 2022-07-12T08:40:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.445 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2022-07-12T08:40:32.445 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:40:32.446 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2022-07-12T08:40:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-07-12T08:40:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-07-12T08:40:32.451 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2022-07-12T08:40:32.452 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.452 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2022-07-12T08:40:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:40:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-07-12T08:40:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2022-07-12T08:40:32.453 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-07-12T08:40:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2022-07-12T08:40:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2022-07-12T08:40:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:40:32.454 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2022-07-12T08:40:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2022-07-12T08:40:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:40:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:40:32.455 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:40:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:40:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:40:32.456 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:40:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:40:32.457 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2022-07-12T08:40:32.458 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:40:32.458 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2022-07-12T08:40:32.458 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2022-07-12T08:40:32.458 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2022-07-12T08:40:32.459 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2022-07-12T08:40:32.459 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:40:32.459 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:40:32.459 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2022-07-12T08:40:32.460 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2022-07-12T08:40:32.460 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:40:32.460 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.460 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2022-07-12T08:40:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2022-07-12T08:40:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2022-07-12T08:40:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2022-07-12T08:40:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2022-07-12T08:40:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2022-07-12T08:40:32.463 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2022-07-12T08:40:32.463 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:40:32.463 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal frag = True 2022-07-12T08:40:32.464 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal fragment size max = 10000 2022-07-12T08:40:32.464 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal merge size = 5 2022-07-12T08:40:32.464 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split bits = 3 2022-07-12T08:40:32.464 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split size = 100 2022-07-12T08:40:32.465 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2022-07-12T08:40:32.465 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2022-07-12T08:40:32.465 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2022-07-12T08:40:32.465 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2022-07-12T08:40:32.466 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-07-12T08:40:32.466 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-07-12T08:40:32.466 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-07-12T08:40:32.466 INFO:tasks.ceph:Creating admin key on mon.a... 2022-07-12T08:40:32.467 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-07-12T08:40:32.517 INFO:tasks.ceph:Copying monmap to all nodes... 2022-07-12T08:40:32.537 DEBUG:teuthology.orchestra.remote:smithi061:/etc/ceph/ceph.keyring is 206B 2022-07-12T08:40:32.642 DEBUG:teuthology.orchestra.remote:smithi061:/home/ubuntu/cephtest/ceph.monmap is 379B 2022-07-12T08:40:32.675 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T08:40:32.676 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:32.676 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-12T08:40:32.683 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-12T08:40:32.751 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-12T08:40:32.752 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:32.759 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T08:40:32.759 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:32.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-12T08:40:32.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-12T08:40:32.789 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2022-07-12T08:40:32.789 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:32.848 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2022-07-12T08:40:32.848 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:40:32.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-12T08:40:32.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-12T08:40:32.883 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2022-07-12T08:40:32.884 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:32.943 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi130.front.sepia.ceph.com 2022-07-12T08:40:32.943 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:40:32.944 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-12T08:40:32.944 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-12T08:40:32.977 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2022-07-12T08:40:32.978 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:32.983 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2022-07-12T08:40:32.984 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:40:32.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-12T08:40:32.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-12T08:40:33.020 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2022-07-12T08:40:33.021 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:33.078 INFO:tasks.ceph:Setting up mon nodes... 2022-07-12T08:40:33.079 INFO:tasks.ceph:Setting up mgr nodes... 2022-07-12T08:40:33.080 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-07-12T08:40:33.129 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-07-12T08:40:33.132 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-07-12T08:40:33.195 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-07-12T08:40:33.198 INFO:tasks.ceph:Setting up mds nodes... 2022-07-12T08:40:33.199 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-07-12T08:40:33.268 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-07-12T08:40:33.279 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-07-12T08:40:33.338 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-07-12T08:40:33.389 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-07-12T08:40:33.391 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-07-12T08:40:33.431 INFO:tasks.ceph_client:Setting up client nodes... 2022-07-12T08:40:33.432 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-07-12T08:40:33.482 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-07-12T08:40:33.498 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-07-12T08:40:33.567 INFO:teuthology.orchestra.run.smithi130.stdout:creating /etc/ceph/ceph.client.1.keyring 2022-07-12T08:40:33.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2022-07-12T08:40:33.651 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.2.keyring 2022-07-12T08:40:33.665 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-07-12T08:40:33.666 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-12T08:40:33.666 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-07-12T08:40:33.687 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:33.687 INFO:tasks.ceph:role: osd.0 2022-07-12T08:40:33.687 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T08:40:33.688 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-12T08:40:34.271 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:34.271 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:34.272 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:34.272 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-12T08:40:34.273 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:34.273 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:34.273 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:34.274 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:34.274 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:34.274 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:34.275 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-12T08:40:34.276 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-07-12T08:40:34.392 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-07-12T08:40:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:34.410 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-07-12T08:40:34.428 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:34.432 INFO:tasks.ceph:role: osd.1 2022-07-12T08:40:34.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T08:40:34.433 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-12T08:40:34.954 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:34.955 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:34.956 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:34.956 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-12T08:40:34.957 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:34.958 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:34.959 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:34.959 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:34.960 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:34.960 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:34.961 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-12T08:40:34.961 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-07-12T08:40:35.011 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-07-12T08:40:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:35.065 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:35.066 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-07-12T08:40:35.078 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:35.079 INFO:tasks.ceph:role: osd.2 2022-07-12T08:40:35.079 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T08:40:35.080 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-12T08:40:35.606 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:35.606 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:35.606 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:35.606 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-12T08:40:35.607 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:35.607 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:35.607 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:35.607 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:35.608 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:35.608 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:35.608 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-12T08:40:35.609 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-07-12T08:40:35.663 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-07-12T08:40:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:35.686 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:35.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-07-12T08:40:35.762 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:35.763 INFO:tasks.ceph:role: osd.3 2022-07-12T08:40:35.764 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T08:40:35.764 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-12T08:40:36.332 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-12T08:40:36.340 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:36.341 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:36.341 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:36.342 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:36.342 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:36.342 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:36.343 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-12T08:40:36.343 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-07-12T08:40:36.448 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-07-12T08:40:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:36.469 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:36.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:36.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.580+0000 7fc85d712d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-07-12T08:40:36.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.584+0000 7fc85d712d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-07-12T08:40:36.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.620+0000 7fc85d712d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:36.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.620+0000 7fc85d712d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:36.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.860+0000 7fc85d712d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:36.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.864+0000 7fc85d712d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:36.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:36.864+0000 7fc85d712d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:36.912 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-07-12T08:40:36.933 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:37.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.016+0000 7f41cc266d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-07-12T08:40:37.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.020+0000 7f41cc266d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-07-12T08:40:37.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.052+0000 7f41cc266d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:37.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.052+0000 7f41cc266d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:37.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.268+0000 7f41cc266d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:37.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.268+0000 7f41cc266d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:37.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.268+0000 7f41cc266d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:37.285 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-07-12T08:40:37.301 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:37.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.388+0000 7fee7307cd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-07-12T08:40:37.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.388+0000 7fee7307cd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-07-12T08:40:37.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.400+0000 7fee7307cd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:37.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.400+0000 7fee7307cd80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:37.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.536+0000 7fee7307cd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:37.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.536+0000 7fee7307cd80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:37.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.536+0000 7fee7307cd80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:37.555 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-07-12T08:40:37.616 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:37.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.656+0000 7f235c4e7d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-07-12T08:40:37.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.656+0000 7f235c4e7d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-07-12T08:40:37.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.672+0000 7f235c4e7d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:37.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.672+0000 7f235c4e7d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:37.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.816+0000 7f235c4e7d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:37.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.816+0000 7f235c4e7d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:37.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:37.816+0000 7f235c4e7d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:37.834 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-07-12T08:40:37.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-07-12T08:40:38.122 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:38.123 INFO:tasks.ceph:role: osd.4 2022-07-12T08:40:38.131 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T08:40:38.131 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-12T08:40:38.732 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2022-07-12T08:40:38.734 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:38.735 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:38.735 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:38.736 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:38.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2022-07-12T08:40:38.739 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-07-12T08:40:38.821 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-07-12T08:40:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:38.833 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:38.834 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-07-12T08:40:38.845 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:38.846 INFO:tasks.ceph:role: osd.5 2022-07-12T08:40:38.846 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T08:40:38.846 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-12T08:40:39.344 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2022-07-12T08:40:39.345 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:39.346 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:39.346 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:39.347 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:39.348 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2022-07-12T08:40:39.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-07-12T08:40:39.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-07-12T08:40:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:39.429 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:39.430 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-07-12T08:40:39.505 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:39.505 INFO:tasks.ceph:role: osd.6 2022-07-12T08:40:39.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T08:40:39.506 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-12T08:40:40.044 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:40.045 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:40.045 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:40.045 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2022-07-12T08:40:40.045 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:40.046 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:40.046 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:40.047 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:40.047 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:40.047 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:40.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2022-07-12T08:40:40.049 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-07-12T08:40:40.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-07-12T08:40:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:40.134 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:40.134 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-07-12T08:40:40.146 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-12T08:40:40.147 INFO:tasks.ceph:role: osd.7 2022-07-12T08:40:40.148 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T08:40:40.149 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-12T08:40:40.693 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-12T08:40:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-12T08:40:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-12T08:40:40.693 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 2022-07-12T08:40:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-12T08:40:40.694 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2022-07-12T08:40:40.694 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-12T08:40:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-12T08:40:40.695 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-12T08:40:40.695 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-12T08:40:40.696 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2022-07-12T08:40:40.697 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-07-12T08:40:40.748 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-07-12T08:40:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:40:40.763 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: /sbin/restorecon: command not found 2022-07-12T08:40:40.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:40.834 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:40.829+0000 7ff728371d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-07-12T08:40:40.835 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:40.829+0000 7ff728371d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-07-12T08:40:40.874 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:40.865+0000 7ff728371d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:40.875 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:40.865+0000 7ff728371d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:41.139 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.129+0000 7ff728371d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:41.139 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.129+0000 7ff728371d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:41.140 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.129+0000 7ff728371d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:41.175 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-07-12T08:40:41.208 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:41.299 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.293+0000 7f2e10919d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-07-12T08:40:41.310 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.297+0000 7f2e10919d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-07-12T08:40:41.334 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.329+0000 7f2e10919d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:41.335 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.333+0000 7f2e10919d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:41.604 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.581+0000 7f2e10919d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:41.604 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.585+0000 7f2e10919d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:41.605 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.585+0000 7f2e10919d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:41.622 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-07-12T08:40:41.704 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:41.750 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.741+0000 7f2109735d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-07-12T08:40:41.751 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.741+0000 7f2109735d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-07-12T08:40:41.801 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.757+0000 7f2109735d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:41.802 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.757+0000 7f2109735d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:41.897 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.893+0000 7f2109735d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:41.899 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.893+0000 7f2109735d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:41.899 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:41.893+0000 7f2109735d80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:41.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-07-12T08:40:42.007 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:42.093 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.085+0000 7fb9ff4eed80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-07-12T08:40:42.094 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.085+0000 7fb9ff4eed80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-07-12T08:40:42.099 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.097+0000 7fb9ff4eed80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:42.100 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.097+0000 7fb9ff4eed80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-12T08:40:42.247 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.245+0000 7fb9ff4eed80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:40:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.245+0000 7fb9ff4eed80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:42.248 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:42.245+0000 7fb9ff4eed80 -1 journal do_read_entry(4096): bad header magic 2022-07-12T08:40:42.267 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-07-12T08:40:42.284 INFO:tasks.ceph:Reading keys from all nodes... 2022-07-12T08:40:42.284 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:42.323 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.a952CHkwNJ 2022-07-12T08:40:42.342 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.a952CHkwNJ 2022-07-12T08:40:42.393 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.a952CHkwNJ 2022-07-12T08:40:42.630 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.a952CHkwNJ is 56B 2022-07-12T08:40:42.679 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.a952CHkwNJ 2022-07-12T08:40:42.692 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:42.746 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.9RvQK5IZYg 2022-07-12T08:40:42.747 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.9RvQK5IZYg 2022-07-12T08:40:42.776 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.9RvQK5IZYg 2022-07-12T08:40:42.884 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.9RvQK5IZYg is 56B 2022-07-12T08:40:42.931 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.9RvQK5IZYg 2022-07-12T08:40:42.953 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:43.003 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.oTGqMBmXTa 2022-07-12T08:40:43.004 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.oTGqMBmXTa 2022-07-12T08:40:43.024 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.oTGqMBmXTa 2022-07-12T08:40:43.115 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.oTGqMBmXTa is 56B 2022-07-12T08:40:43.205 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.oTGqMBmXTa 2022-07-12T08:40:43.220 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:43.301 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.L7x2jpdR16 2022-07-12T08:40:43.302 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.L7x2jpdR16 2022-07-12T08:40:43.358 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.L7x2jpdR16 2022-07-12T08:40:43.442 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.L7x2jpdR16 is 56B 2022-07-12T08:40:43.486 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.L7x2jpdR16 2022-07-12T08:40:43.495 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:43.555 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.ur0QXmGICd 2022-07-12T08:40:43.556 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.ur0QXmGICd 2022-07-12T08:40:43.573 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.ur0QXmGICd 2022-07-12T08:40:43.735 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.ur0QXmGICd is 56B 2022-07-12T08:40:43.811 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.ur0QXmGICd 2022-07-12T08:40:43.819 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:43.867 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.dbc7lixKQk 2022-07-12T08:40:43.868 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.dbc7lixKQk 2022-07-12T08:40:43.912 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.dbc7lixKQk 2022-07-12T08:40:43.981 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.dbc7lixKQk is 56B 2022-07-12T08:40:44.090 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.dbc7lixKQk 2022-07-12T08:40:44.107 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:44.124 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.YUQcD59cVj 2022-07-12T08:40:44.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.YUQcD59cVj 2022-07-12T08:40:44.188 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.YUQcD59cVj 2022-07-12T08:40:44.302 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.YUQcD59cVj is 56B 2022-07-12T08:40:44.414 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.YUQcD59cVj 2022-07-12T08:40:44.452 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:44.520 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.NgFpI9eFTe 2022-07-12T08:40:44.521 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.NgFpI9eFTe 2022-07-12T08:40:44.552 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.NgFpI9eFTe 2022-07-12T08:40:44.682 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.NgFpI9eFTe is 56B 2022-07-12T08:40:44.705 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.NgFpI9eFTe 2022-07-12T08:40:44.715 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:44.767 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.NGKM35jBlY 2022-07-12T08:40:44.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.NGKM35jBlY 2022-07-12T08:40:44.790 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.NGKM35jBlY 2022-07-12T08:40:44.884 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.NGKM35jBlY is 56B 2022-07-12T08:40:44.994 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.NGKM35jBlY 2022-07-12T08:40:45.019 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:45.079 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.ro66vCKXuw 2022-07-12T08:40:45.080 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.ro66vCKXuw 2022-07-12T08:40:45.113 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.ro66vCKXuw 2022-07-12T08:40:45.245 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.ro66vCKXuw is 56B 2022-07-12T08:40:45.307 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.ro66vCKXuw 2022-07-12T08:40:45.340 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:45.396 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.iUiiMsLhRt 2022-07-12T08:40:45.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.iUiiMsLhRt 2022-07-12T08:40:45.414 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.iUiiMsLhRt 2022-07-12T08:40:45.537 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.iUiiMsLhRt is 56B 2022-07-12T08:40:45.602 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.iUiiMsLhRt 2022-07-12T08:40:45.628 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:45.683 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.lnYDICXHor 2022-07-12T08:40:45.683 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.lnYDICXHor 2022-07-12T08:40:45.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.lnYDICXHor 2022-07-12T08:40:45.856 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.lnYDICXHor is 56B 2022-07-12T08:40:45.960 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.lnYDICXHor 2022-07-12T08:40:46.089 DEBUG:teuthology.orchestra.remote:smithi121:/etc/ceph/ceph.client.0.keyring is 59B 2022-07-12T08:40:46.225 DEBUG:teuthology.orchestra.remote:smithi130:/etc/ceph/ceph.client.1.keyring is 59B 2022-07-12T08:40:46.408 DEBUG:teuthology.orchestra.remote:smithi137:/etc/ceph/ceph.client.2.keyring is 59B 2022-07-12T08:40:46.537 INFO:tasks.ceph:Adding keys to all mons... 2022-07-12T08:40:46.539 DEBUG:teuthology.orchestra.run.smithi061:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-12T08:40:46.559 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-12T08:40:46.621 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.y] 2022-07-12T08:40:46.621 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg== 2022-07-12T08:40:46.627 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-12T08:40:46.628 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA== 2022-07-12T08:40:46.628 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2022-07-12T08:40:46.628 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw== 2022-07-12T08:40:46.629 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2022-07-12T08:40:46.629 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA== 2022-07-12T08:40:46.630 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2022-07-12T08:40:46.630 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ== 2022-07-12T08:40:46.631 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.3] 2022-07-12T08:40:46.631 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ== 2022-07-12T08:40:46.632 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.x] 2022-07-12T08:40:46.632 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA== 2022-07-12T08:40:46.633 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-12T08:40:46.634 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ== 2022-07-12T08:40:46.634 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.4] 2022-07-12T08:40:46.634 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg== 2022-07-12T08:40:46.635 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.5] 2022-07-12T08:40:46.635 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA== 2022-07-12T08:40:46.635 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.6] 2022-07-12T08:40:46.635 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ== 2022-07-12T08:40:46.636 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.7] 2022-07-12T08:40:46.636 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w== 2022-07-12T08:40:46.636 INFO:teuthology.orchestra.run.smithi061.stdout:[client.0] 2022-07-12T08:40:46.636 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g== 2022-07-12T08:40:46.637 INFO:teuthology.orchestra.run.smithi061.stdout:[client.1] 2022-07-12T08:40:46.637 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw== 2022-07-12T08:40:46.637 INFO:teuthology.orchestra.run.smithi061.stdout:[client.2] 2022-07-12T08:40:46.637 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A== 2022-07-12T08:40:46.638 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.y] 2022-07-12T08:40:46.639 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg== 2022-07-12T08:40:46.639 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2022-07-12T08:40:46.639 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA== 2022-07-12T08:40:46.640 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2022-07-12T08:40:46.659 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw== 2022-07-12T08:40:46.661 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2022-07-12T08:40:46.663 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA== 2022-07-12T08:40:46.663 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2022-07-12T08:40:46.664 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ== 2022-07-12T08:40:46.664 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.3] 2022-07-12T08:40:46.672 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ== 2022-07-12T08:40:46.673 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2022-07-12T08:40:46.681 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA== 2022-07-12T08:40:46.682 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2022-07-12T08:40:46.682 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ== 2022-07-12T08:40:46.683 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.4] 2022-07-12T08:40:46.683 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg== 2022-07-12T08:40:46.684 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.5] 2022-07-12T08:40:46.684 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA== 2022-07-12T08:40:46.685 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.6] 2022-07-12T08:40:46.685 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ== 2022-07-12T08:40:46.686 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.7] 2022-07-12T08:40:46.686 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w== 2022-07-12T08:40:46.687 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2022-07-12T08:40:46.687 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g== 2022-07-12T08:40:46.687 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2022-07-12T08:40:46.688 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw== 2022-07-12T08:40:46.688 INFO:teuthology.orchestra.run.smithi111.stdout:[client.2] 2022-07-12T08:40:46.689 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A== 2022-07-12T08:40:46.693 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-12T08:40:46.771 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-12T08:40:46.822 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-12T08:40:46.864 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-12T08:40:46.917 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:46.990 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.037 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.120 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.207 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.244 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.289 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-12T08:40:47.302 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-12T08:40:47.382 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-12T08:40:47.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-12T08:40:47.467 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.549 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.574 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.628 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.687 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.736 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.785 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-12T08:40:47.832 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-12T08:40:47.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-12T08:40:47.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-12T08:40:47.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-12T08:40:48.046 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-12T08:40:48.055 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-12T08:40:48.136 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-07-12T08:40:48.146 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-07-12T08:40:48.165 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-12T08:40:48.236 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-07-12T08:40:48.269 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-07-12T08:40:48.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-12T08:40:48.486 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-07-12T08:40:48.520 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-07-12T08:40:48.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-12T08:40:48.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-07-12T08:40:48.835 DEBUG:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:48.860 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-12T08:40:48.889 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-07-12T08:40:48.916 INFO:tasks.ceph.mon.a:Restarting daemon 2022-07-12T08:40:48.917 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-07-12T08:40:48.936 INFO:tasks.ceph.mon.a:Started 2022-07-12T08:40:48.937 INFO:tasks.ceph.mon.b:Restarting daemon 2022-07-12T08:40:48.938 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-07-12T08:40:48.963 INFO:tasks.ceph.mon.b:Started 2022-07-12T08:40:48.964 INFO:tasks.ceph.mon.c:Restarting daemon 2022-07-12T08:40:48.966 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-07-12T08:40:48.983 INFO:tasks.ceph.mon.c:Started 2022-07-12T08:40:48.984 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-07-12T08:40:48.984 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-07-12T08:40:48.985 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-07-12T08:40:48.995 INFO:tasks.ceph.mgr.y:Started 2022-07-12T08:40:48.996 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-07-12T08:40:48.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-07-12T08:40:49.005 INFO:tasks.ceph.mgr.x:Started 2022-07-12T08:40:49.008 INFO:tasks.ceph:Setting crush tunables to default 2022-07-12T08:40:49.009 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd crush tunables default 2022-07-12T08:40:49.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc1ca38700 1 Processor -- start 2022-07-12T08:40:49.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc1ca38700 1 -- start start 2022-07-12T08:40:49.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc1ca38700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:49.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc1ca38700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc180b6780 con 0x7fcc18066410 2022-07-12T08:40:49.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc1ca38700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc18143110 con 0x7fcc180b63a0 2022-07-12T08:40:49.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc1ca38700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180b1f30 con 0x7fcc18143af0 2022-07-12T08:40:49.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:49.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52288/0 (socket says 172.21.15.61:52288) 2022-07-12T08:40:49.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 learned_addr learned my addr 172.21.15.61:0/4251972348 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:40:49.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc16ffd700 1 --1- 172.21.15.61:0/4251972348 >> v1:172.21.15.61:6789/0 conn(0x7fcc18066410 0x7fcc1813e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40504/0 (socket says 172.21.15.61:40504) 2022-07-12T08:40:49.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-12T08:40:49.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:49.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:49.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.136+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-12T08:40:49.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.340+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:49.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.340+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-12T08:40:49.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.340+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:49.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.340+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:49.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.340+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-12T08:40:49.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.740+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:49.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.740+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-12T08:40:49.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.740+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:49.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.740+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:49.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:49.740+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2022-07-12T08:40:50.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:50.540+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:50.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:50.540+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-12T08:40:50.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:50.540+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:50.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:50.540+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:50.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:50.540+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2022-07-12T08:40:52.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 msgr2=0x7fcc180b1a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-12T08:40:52.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcc18143af0 0x7fcc180b1a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:52.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 >> v1:172.21.15.111:6789/0 conn(0x7fcc180b63a0 legacy=0x7fcc180b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:52.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 >> v1:172.21.15.61:6789/0 conn(0x7fcc18066410 legacy=0x7fcc1813e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:52.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:52.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:52.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180b1f30 con 0x7fcbf80033d0 2022-07-12T08:40:52.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80089a0 con 0x7fcbf8005e20 2022-07-12T08:40:52.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:52.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:52.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-12T08:40:52.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:52.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-07-12T08:40:52.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-12T08:40:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.140+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-12T08:40:52.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:52.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:52.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-12T08:40:52.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:52.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-07-12T08:40:52.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:52.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:52.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:52.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-12T08:40:52.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.340+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2022-07-12T08:40:52.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:52.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:52.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-12T08:40:52.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:52.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:52.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2022-07-12T08:40:52.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-07-12T08:40:52.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:52.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:52.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:52.740+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2022-07-12T08:40:53.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:53.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:53.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2022-07-12T08:40:53.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc16ffd700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2022-07-12T08:40:53.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-07-12T08:40:53.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc177fe700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-12T08:40:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-12T08:40:53.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:53.544+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2022-07-12T08:40:55.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 msgr2=0x7fcbf80082e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-12T08:40:55.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.136+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8005e20 0x7fcbf80082e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.136+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 msgr2=0x7fcbf8005830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-12T08:40:55.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf80033d0 0x7fcbf8005830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 reap_dead start 2022-07-12T08:40:55.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcbf8003820 0x7fcbf80062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:55.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf80067a0 0x7fcbf8006c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:55.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf8007260 0x7fcbf8007710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:55.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc180b1f30 con 0x7fcbf8003820 2022-07-12T08:40:55.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf80089a0 con 0x7fcbf8007260 2022-07-12T08:40:55.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc157fa700 1 -- 172.21.15.61:0/4251972348 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8007ea0 con 0x7fcbf80067a0 2022-07-12T08:40:55.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcbf8003820 0x7fcbf80062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:55.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.111:3300/0 conn(0x7fcbf8007260 0x7fcbf8007710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:55.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf80067a0 0x7fcbf8006c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:55.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf80067a0 msgr2=0x7fcbf8006c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:40:55.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf80067a0 0x7fcbf8006c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcbf8003820 msgr2=0x7fcbf80062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:40:55.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcbf8003820 0x7fcbf80062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 -- 172.21.15.61:0/4251972348 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc181433b0 con 0x7fcbf8007260 2022-07-12T08:40:55.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcbf8003820 0x7fcbf80062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:40:55.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8007260 0x7fcbf8007710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcc1000c000 tx=0x7fcc1000b610).ready entity=mon.1 client_cookie=55dbc43c46f30da8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:55.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc167fc700 1 -- 172.21.15.61:0/4251972348 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc10018070 con 0x7fcbf8007260 2022-07-12T08:40:55.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc167fc700 1 -- 172.21.15.61:0/4251972348 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc10013ac0 con 0x7fcbf8007260 2022-07-12T08:40:55.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc167fc700 1 -- 172.21.15.61:0/4251972348 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc10013e20 con 0x7fcbf8007260 2022-07-12T08:40:55.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8007260 msgr2=0x7fcbf8007710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:55.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8007260 0x7fcbf8007710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcc1000c000 tx=0x7fcc1000b610).stop 2022-07-12T08:40:55.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4251972348 shutdown_connections 2022-07-12T08:40:55.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf80067a0 0x7fcbf8006c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4251972348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf8007260 0x7fcbf8007710 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4251972348 >> v2:172.21.15.61:3300/0 conn(0x7fcbf8003820 0x7fcbf80062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4251972348 >> 172.21.15.61:0/4251972348 conn(0x7fcc18063ad0 msgr2=0x7fcc18063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4251972348 shutdown_connections 2022-07-12T08:40:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.140+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4251972348 wait complete. 2022-07-12T08:40:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 Processor -- start 2022-07-12T08:40:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 -- start start 2022-07-12T08:40:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 0x7fcc180aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf80067a0 0x7fcc180b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8007260 0x7fcc180af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc181530b0 con 0x7fcbf8003820 2022-07-12T08:40:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180afe40 con 0x7fcbf80067a0 2022-07-12T08:40:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc1ca38700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc180b0060 con 0x7fcbf8007260 2022-07-12T08:40:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 0x7fcc180aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 0x7fcc180aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52358/0 (socket says 172.21.15.61:52358) 2022-07-12T08:40:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 -- 172.21.15.61:0/4144792407 learned_addr learned my addr 172.21.15.61:0/4144792407 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:40:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc16ffd700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf80067a0 0x7fcc180b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc17fff700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8007260 0x7fcc180af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:55.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 -- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8007260 msgr2=0x7fcc180af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:55.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8007260 0x7fcc180af950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 -- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf80067a0 msgr2=0x7fcc180b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:55.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf80067a0 0x7fcc180b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 -- 172.21.15.61:0/4144792407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0800f040 con 0x7fcbf8003820 2022-07-12T08:40:55.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.144+0000 7fcc177fe700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 0x7fcc180aee40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fcc180aff80 tx=0x7fcc08009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:55.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc08018840 con 0x7fcbf8003820 2022-07-12T08:40:55.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0801d030 con 0x7fcbf8003820 2022-07-12T08:40:55.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1000b820 con 0x7fcbf8003820 2022-07-12T08:40:55.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc08018b90 con 0x7fcbf8003820 2022-07-12T08:40:55.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc18152120 con 0x7fcbf8003820 2022-07-12T08:40:55.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7fcc08013040 con 0x7fcbf8003820 2022-07-12T08:40:55.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcc08023830 con 0x7fcbf8003820 2022-07-12T08:40:55.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.148+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc180b0220 con 0x7fcbf8003820 2022-07-12T08:40:55.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.152+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7fcc180b0220 con 0x7fcbf8003820 2022-07-12T08:40:55.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.220+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcc18057460 con 0x7fcbf8003820 2022-07-12T08:40:55.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc14ff9700 1 -- 172.21.15.61:0/4144792407 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcc08023b80 con 0x7fcbf8003820 2022-07-12T08:40:55.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 msgr2=0x7fcc180aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:55.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 0x7fcc180aee40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fcc180aff80 tx=0x7fcc08009040).stop 2022-07-12T08:40:55.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 shutdown_connections 2022-07-12T08:40:55.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbf8007260 0x7fcc180af950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbf80067a0 0x7fcc180b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 --2- 172.21.15.61:0/4144792407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbf8003820 0x7fcc180aee40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:55.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 >> 172.21.15.61:0/4144792407 conn(0x7fcc18063ad0 msgr2=0x7fcc180b56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:55.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 shutdown_connections 2022-07-12T08:40:55.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:40:55.228+0000 7fcc1ca38700 1 -- 172.21.15.61:0/4144792407 wait complete. 2022-07-12T08:40:55.289 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2022-07-12T08:40:55.289 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-07-12T08:40:55.290 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:55.311 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.AmWnEW9vMQ 2022-07-12T08:40:55.312 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.AmWnEW9vMQ 2022-07-12T08:40:55.393 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.AmWnEW9vMQ 2022-07-12T08:40:55.602 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.AmWnEW9vMQ is 37B 2022-07-12T08:40:55.741 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.AmWnEW9vMQ 2022-07-12T08:40:55.754 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:55.835 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.r25CDXvmKi 2022-07-12T08:40:55.837 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.r25CDXvmKi 2022-07-12T08:40:55.875 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.r25CDXvmKi 2022-07-12T08:40:55.951 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.r25CDXvmKi is 37B 2022-07-12T08:40:56.009 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.r25CDXvmKi 2022-07-12T08:40:56.022 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:56.075 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.rsSl1jFgq9 2022-07-12T08:40:56.076 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.rsSl1jFgq9 2022-07-12T08:40:56.131 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.rsSl1jFgq9 2022-07-12T08:40:56.232 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.rsSl1jFgq9 is 37B 2022-07-12T08:40:56.255 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.rsSl1jFgq9 2022-07-12T08:40:56.289 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T08:40:56.379 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.HzUrVFiHtD 2022-07-12T08:40:56.380 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.HzUrVFiHtD 2022-07-12T08:40:56.409 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.HzUrVFiHtD 2022-07-12T08:40:56.504 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.HzUrVFiHtD is 37B 2022-07-12T08:40:56.670 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.HzUrVFiHtD 2022-07-12T08:40:56.688 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:56.708 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.wevLm1ytJg 2022-07-12T08:40:56.709 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.wevLm1ytJg 2022-07-12T08:40:56.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.wevLm1ytJg 2022-07-12T08:40:56.855 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.wevLm1ytJg is 37B 2022-07-12T08:40:56.878 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.wevLm1ytJg 2022-07-12T08:40:56.886 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:56.933 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.hwXOscv6V0 2022-07-12T08:40:56.935 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.hwXOscv6V0 2022-07-12T08:40:56.973 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.hwXOscv6V0 2022-07-12T08:40:57.117 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.hwXOscv6V0 is 37B 2022-07-12T08:40:57.212 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.hwXOscv6V0 2022-07-12T08:40:57.253 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:57.271 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.oDGRCXEoOY 2022-07-12T08:40:57.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.oDGRCXEoOY 2022-07-12T08:40:57.408 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.oDGRCXEoOY 2022-07-12T08:40:57.512 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.oDGRCXEoOY is 37B 2022-07-12T08:40:57.591 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.oDGRCXEoOY 2022-07-12T08:40:57.632 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T08:40:57.694 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.FZFxKVhGlE 2022-07-12T08:40:57.730 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.FZFxKVhGlE 2022-07-12T08:40:57.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.FZFxKVhGlE 2022-07-12T08:40:57.848 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.FZFxKVhGlE is 37B 2022-07-12T08:40:57.993 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.FZFxKVhGlE 2022-07-12T08:40:58.019 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new c5e8edcd-71d7-4bc2-84d6-819a6f4be259 0 2022-07-12T08:40:58.192 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1a021700 1 Processor -- start 2022-07-12T08:40:58.193 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1a021700 1 -- start start 2022-07-12T08:40:58.193 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1a021700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a141426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.193 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1a021700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140b5500 con 0x7f9a140b1f50 2022-07-12T08:40:58.193 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1a021700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a140b34d0 con 0x7f9a140b5ea0 2022-07-12T08:40:58.194 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1a021700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a140b3970 con 0x7f9a140b3cf0 2022-07-12T08:40:58.194 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1881e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a141426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.194 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1881e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a141426a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50598/0 (socket says 172.21.15.111:50598) 2022-07-12T08:40:58.194 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1881e700 1 -- 172.21.15.111:0/2802517529 learned_addr learned my addr 172.21.15.111:0/2802517529 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:58.195 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.169+0000 7f9a1901f700 1 --1- 172.21.15.111:0/2802517529 >> v1:172.21.15.111:6789/0 conn(0x7f9a140b5ea0 0x7f9a14141f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33748/0 (socket says 172.21.15.111:33748) 2022-07-12T08:40:58.195 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1881e700 1 -- 172.21.15.111:0/2802517529 >> v1:172.21.15.111:6789/0 conn(0x7f9a140b5ea0 legacy=0x7f9a14141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.195 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1881e700 1 -- 172.21.15.111:0/2802517529 >> v1:172.21.15.61:6789/0 conn(0x7f9a140b1f50 legacy=0x7f9a140b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.195 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f99fffff700 1 -- 172.21.15.111:0/2802517529 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633679861 0 0) 0x7f9a140b34d0 con 0x7f9a140b5ea0 2022-07-12T08:40:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1881e700 1 -- 172.21.15.111:0/2802517529 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a140b57a0 con 0x7f9a140b3cf0 2022-07-12T08:40:58.213 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1881e700 1 --2- 172.21.15.111:0/2802517529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a141426a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9a10002f70 tx=0x7f9a10006eb0).ready entity=mon.2 client_cookie=4603a35699a3c8eb server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f99fffff700 1 -- 172.21.15.111:0/2802517529 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a10009600 con 0x7f9a140b3cf0 2022-07-12T08:40:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f99fffff700 1 -- 172.21.15.111:0/2802517529 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10000e90 con 0x7f9a140b3cf0 2022-07-12T08:40:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f99fffff700 1 -- 172.21.15.111:0/2802517529 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a10009950 con 0x7f9a140b3cf0 2022-07-12T08:40:58.214 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 -- 172.21.15.111:0/2802517529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 msgr2=0x7f9a141426a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.215 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2802517529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a141426a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f9a10002f70 tx=0x7f9a10006eb0).stop 2022-07-12T08:40:58.237 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 -- 172.21.15.111:0/2802517529 shutdown_connections 2022-07-12T08:40:58.237 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2802517529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a141426a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.238 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 -- 172.21.15.111:0/2802517529 >> 172.21.15.111:0/2802517529 conn(0x7f9a14066d70 msgr2=0x7f9a14067150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:58.238 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 -- 172.21.15.111:0/2802517529 shutdown_connections 2022-07-12T08:40:58.238 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 -- 172.21.15.111:0/2802517529 wait complete. 2022-07-12T08:40:58.238 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 Processor -- start 2022-07-12T08:40:58.239 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.173+0000 7f9a1a021700 1 -- start start 2022-07-12T08:40:58.239 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a140b1f50 0x7f9a14155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.239 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a140b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.239 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 0x7f9a14112930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.239 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14112540 con 0x7f9a140b1f50 2022-07-12T08:40:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a141467c0 con 0x7f9a140b5ea0 2022-07-12T08:40:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a14146a60 con 0x7f9a140b3cf0 2022-07-12T08:40:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a19820700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 0x7f9a14112930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.240 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a140b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a19820700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 0x7f9a14112930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43848/0 (socket says 172.21.15.111:43848) 2022-07-12T08:40:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a19820700 1 -- 172.21.15.111:0/2412039780 learned_addr learned my addr 172.21.15.111:0/2412039780 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1901f700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a140b1f50 0x7f9a14155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 -- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 msgr2=0x7f9a14112930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.241 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 0x7f9a14112930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.271 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 -- 172.21.15.111:0/2412039780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a140b1f50 msgr2=0x7f9a14155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.274 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a140b1f50 0x7f9a14155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.275 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 -- 172.21.15.111:0/2412039780 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1000b1e0 con 0x7f9a140b3cf0 2022-07-12T08:40:58.275 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1901f700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a140b1f50 0x7f9a14155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:40:58.275 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a19820700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 0x7f9a14112930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:40:58.276 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1881e700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a140b7d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9a10002f70 tx=0x7f9a10015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.276 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a10021070 con 0x7f9a140b3cf0 2022-07-12T08:40:58.277 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10006b40 con 0x7f9a140b3cf0 2022-07-12T08:40:58.277 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a10011e20 con 0x7f9a140b3cf0 2022-07-12T08:40:58.282 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a14146d00 con 0x7f9a140b3cf0 2022-07-12T08:40:58.282 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.177+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a140ad500 con 0x7f9a140b3cf0 2022-07-12T08:40:58.283 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.181+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a14146e90 con 0x7f9a140b3cf0 2022-07-12T08:40:58.283 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.181+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7f9a1000f070 con 0x7f9a140b3cf0 2022-07-12T08:40:58.283 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.181+0000 7f9a06ffd700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99f0053a60 0x7f99f0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.283 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.181+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9a1006b9e0 con 0x7f9a140b3cf0 2022-07-12T08:40:58.284 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.181+0000 7f9a1901f700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99f0053a60 0x7f99f0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.284 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.181+0000 7f9a1901f700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99f0053a60 0x7f99f0055f20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9a0800b000 tx=0x7f9a080070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.284 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.185+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a100441b0 con 0x7f9a140b3cf0 2022-07-12T08:40:58.327 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.325+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", "id": 0} v 0) v1 -- 0x7f9a14057460 con 0x7f9a140b3cf0 2022-07-12T08:40:58.343 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.333+0000 7f9a06ffd700 1 -- 172.21.15.111:0/2412039780 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9a100477d0 con 0x7f9a140b3cf0 2022-07-12T08:40:58.344 INFO:teuthology.orchestra.run.smithi111.stdout:0 2022-07-12T08:40:58.345 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99f0053a60 msgr2=0x7f99f0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.345 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99f0053a60 0x7f99f0055f20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f9a0800b000 tx=0x7f9a080070e0).stop 2022-07-12T08:40:58.345 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 msgr2=0x7f9a140b7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.346 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a140b7d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f9a10002f70 tx=0x7f9a10015000).stop 2022-07-12T08:40:58.361 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 shutdown_connections 2022-07-12T08:40:58.362 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99f0053a60 0x7f99f0055f20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.362 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a140b5ea0 0x7f9a14112930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.363 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a140b3cf0 0x7f9a140b7d70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.363 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 --2- 172.21.15.111:0/2412039780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a140b1f50 0x7f9a14155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.364 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 >> 172.21.15.111:0/2412039780 conn(0x7f9a14066d70 msgr2=0x7f9a14141140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:58.364 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 shutdown_connections 2022-07-12T08:40:58.365 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.337+0000 7f9a1a021700 1 -- 172.21.15.111:0/2412039780 wait complete. 2022-07-12T08:40:58.373 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 78c8bcfb-e525-4fea-a66a-002769a0b1ee 1 2022-07-12T08:40:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d07fff700 1 Processor -- start 2022-07-12T08:40:58.494 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d07fff700 1 -- start start 2022-07-12T08:40:58.495 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 0x7f7d08143720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.495 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d07fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d080b31e0 con 0x7f7d080b2e00 2022-07-12T08:40:58.496 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d07fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d080ae520 con 0x7f7d08143c10 2022-07-12T08:40:58.496 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d07fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d080af910 con 0x7f7d08060470 2022-07-12T08:40:58.497 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.485+0000 7f7d067fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 0x7f7d08143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.497 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d067fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 0x7f7d08143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50610/0 (socket says 172.21.15.111:50610) 2022-07-12T08:40:58.498 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d067fc700 1 -- 172.21.15.111:0/1920687182 learned_addr learned my addr 172.21.15.111:0/1920687182 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:58.498 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d077fe700 1 --1- 172.21.15.111:0/1920687182 >> v1:172.21.15.111:6789/0 conn(0x7f7d08143c10 0x7f7d08128770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33766/0 (socket says 172.21.15.111:33766) 2022-07-12T08:40:58.501 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d05ffb700 1 -- 172.21.15.111:0/1920687182 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352262660 0 0) 0x7f7d080ae520 con 0x7f7d08143c10 2022-07-12T08:40:58.502 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d05ffb700 1 -- 172.21.15.111:0/1920687182 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ce8003260 con 0x7f7d08143c10 2022-07-12T08:40:58.502 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d067fc700 1 -- 172.21.15.111:0/1920687182 >> v1:172.21.15.111:6789/0 conn(0x7f7d08143c10 legacy=0x7f7d08128770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.503 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d067fc700 1 -- 172.21.15.111:0/1920687182 >> v1:172.21.15.61:6789/0 conn(0x7f7d080b2e00 legacy=0x7f7d080af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d067fc700 1 -- 172.21.15.111:0/1920687182 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d08142fe0 con 0x7f7d08060470 2022-07-12T08:40:58.504 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d067fc700 1 --2- 172.21.15.111:0/1920687182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 0x7f7d08143720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7cf0002f70 tx=0x7f7cf0006eb0).ready entity=mon.2 client_cookie=922bf199bd5805ff server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d05ffb700 1 -- 172.21.15.111:0/1920687182 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf000fa10 con 0x7f7d08060470 2022-07-12T08:40:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d05ffb700 1 -- 172.21.15.111:0/1920687182 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf0000e90 con 0x7f7d08060470 2022-07-12T08:40:58.506 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d05ffb700 1 -- 172.21.15.111:0/1920687182 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf000fd60 con 0x7f7d08060470 2022-07-12T08:40:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 -- 172.21.15.111:0/1920687182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 msgr2=0x7f7d08143720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 --2- 172.21.15.111:0/1920687182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 0x7f7d08143720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f7cf0002f70 tx=0x7f7cf0006eb0).stop 2022-07-12T08:40:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 -- 172.21.15.111:0/1920687182 shutdown_connections 2022-07-12T08:40:58.507 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 --2- 172.21.15.111:0/1920687182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d08060470 0x7f7d08143720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.508 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 -- 172.21.15.111:0/1920687182 >> 172.21.15.111:0/1920687182 conn(0x7f7d08063ab0 msgr2=0x7f7d08063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:58.508 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 -- 172.21.15.111:0/1920687182 shutdown_connections 2022-07-12T08:40:58.517 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.489+0000 7f7d07fff700 1 -- 172.21.15.111:0/1920687182 wait complete. 2022-07-12T08:40:58.518 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 Processor -- start 2022-07-12T08:40:58.527 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- start start 2022-07-12T08:40:58.528 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 0x7f7d0814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.530 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 0x7f7d08065640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.530 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d08143c10 0x7f7d08066720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.531 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0814db30 con 0x7f7d08143c10 2022-07-12T08:40:58.531 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0814ddd0 con 0x7f7d08060470 2022-07-12T08:40:58.532 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0814e070 con 0x7f7d080b2e00 2022-07-12T08:40:58.532 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d067fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 0x7f7d08065640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 0x7f7d0814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.533 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 0x7f7d0814e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43858/0 (socket says 172.21.15.111:43858) 2022-07-12T08:40:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d067fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 0x7f7d08065640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50616/0 (socket says 172.21.15.111:50616) 2022-07-12T08:40:58.534 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 -- 172.21.15.111:0/197996151 learned_addr learned my addr 172.21.15.111:0/197996151 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:58.535 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d077fe700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d08143c10 0x7f7d08066720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.536 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 -- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 msgr2=0x7f7d08065640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.536 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 0x7f7d08065640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 -- 172.21.15.111:0/197996151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d08143c10 msgr2=0x7f7d08066720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.537 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d08143c10 0x7f7d08066720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.538 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 -- 172.21.15.111:0/197996151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf000b1e0 con 0x7f7d08060470 2022-07-12T08:40:58.538 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d067fc700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 0x7f7d08065640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:40:58.539 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d06ffd700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 0x7f7d0814e510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f7d08064970 tx=0x7f7cf8007c50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.539 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf8012400 con 0x7f7d08060470 2022-07-12T08:40:58.540 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf8012560 con 0x7f7d08060470 2022-07-12T08:40:58.540 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf80128c0 con 0x7f7d08060470 2022-07-12T08:40:58.541 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0805d2f0 con 0x7f7d08060470 2022-07-12T08:40:58.541 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d08065b30 con 0x7f7d08060470 2022-07-12T08:40:58.542 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.493+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0814e230 con 0x7f7d08060470 2022-07-12T08:40:58.542 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.497+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7f7cf8019030 con 0x7f7d08060470 2022-07-12T08:40:58.543 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.497+0000 7f7cfffff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7cec0539a0 0x7f7cec055e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.554 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.497+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f7cf8069a50 con 0x7f7d08060470 2022-07-12T08:40:58.554 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.497+0000 7f7d067fc700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7cec0539a0 0x7f7cec055e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.555 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.497+0000 7f7d067fc700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7cec0539a0 0x7f7cec055e60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7cf001d000 tx=0x7f7cf0017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.555 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.501+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7cf8042110 con 0x7f7d08060470 2022-07-12T08:40:58.641 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.637+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd new", "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", "id": 1} v 0) v1 -- 0x7f7d08057460 con 0x7f7d08060470 2022-07-12T08:40:58.656 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.649+0000 7f7cfffff700 1 -- 172.21.15.111:0/197996151 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7cf8045730 con 0x7f7d08060470 2022-07-12T08:40:58.657 INFO:teuthology.orchestra.run.smithi111.stdout:1 2022-07-12T08:40:58.678 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7cec0539a0 msgr2=0x7f7cec055e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7cec0539a0 0x7f7cec055e60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f7cf001d000 tx=0x7f7cf0017000).stop 2022-07-12T08:40:58.679 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 msgr2=0x7f7d0814e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.680 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 0x7f7d0814e510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f7d08064970 tx=0x7f7cf8007c50).stop 2022-07-12T08:40:58.681 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 shutdown_connections 2022-07-12T08:40:58.681 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d08143c10 0x7f7d08066720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.682 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d080b2e00 0x7f7d08065640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.682 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7cec0539a0 0x7f7cec055e60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.683 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 --2- 172.21.15.111:0/197996151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d08060470 0x7f7d0814e510 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.683 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 >> 172.21.15.111:0/197996151 conn(0x7f7d08063ab0 msgr2=0x7f7d08144050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:58.684 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 shutdown_connections 2022-07-12T08:40:58.685 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.653+0000 7f7d07fff700 1 -- 172.21.15.111:0/197996151 wait complete. 2022-07-12T08:40:58.685 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new b765f537-14cb-46b5-be53-b78393e087ad 2 2022-07-12T08:40:58.793 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.785+0000 7fa8b61ca700 1 Processor -- start 2022-07-12T08:40:58.794 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.785+0000 7fa8b61ca700 1 -- start start 2022-07-12T08:40:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8b00ac5c0 0x7fa8b01412d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b0141890 con 0x7fa8b013ded0 2022-07-12T08:40:58.803 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b013e340 con 0x7fa8b00ac9a0 2022-07-12T08:40:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b0099e90 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b49c7700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa8b00ac9a0 0x7fa8b013d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33776/0 (socket says 172.21.15.111:33776) 2022-07-12T08:40:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b49c7700 1 -- 172.21.15.111:0/3638754593 learned_addr learned my addr 172.21.15.111:0/3638754593 (peer_addr_for_me v1:172.21.15.111:0/0) 2022-07-12T08:40:58.804 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b51c8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8b00ac5c0 0x7fa8b01412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.805 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa89ffff700 1 -- 172.21.15.111:0/3638754593 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435164486 0 0) 0x7fa8a0001110 con 0x7fa8b00ac9a0 2022-07-12T08:40:58.805 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa89ffff700 1 -- 172.21.15.111:0/3638754593 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa898003260 con 0x7fa8b00ac9a0 2022-07-12T08:40:58.809 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b51c8700 1 -- 172.21.15.111:0/3638754593 >> v1:172.21.15.111:6789/0 conn(0x7fa8b00ac9a0 legacy=0x7fa8b013d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b51c8700 1 -- 172.21.15.111:0/3638754593 >> v1:172.21.15.61:6789/0 conn(0x7fa8b013ded0 legacy=0x7fa8b00b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.810 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b51c8700 1 -- 172.21.15.111:0/3638754593 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00b2410 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.811 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b51c8700 1 --2- 172.21.15.111:0/3638754593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b01412d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa8a00012a0 tx=0x7fa8ac006df0).ready entity=mon.1 client_cookie=8e3e67450ce48665 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.811 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa89ffff700 1 -- 172.21.15.111:0/3638754593 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8ac00d5c0 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.811 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa89ffff700 1 -- 172.21.15.111:0/3638754593 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac000e90 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.818 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa89ffff700 1 -- 172.21.15.111:0/3638754593 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8ac00d910 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- 172.21.15.111:0/3638754593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 msgr2=0x7fa8b01412d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/3638754593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b01412d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa8a00012a0 tx=0x7fa8ac006df0).stop 2022-07-12T08:40:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- 172.21.15.111:0/3638754593 shutdown_connections 2022-07-12T08:40:58.819 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/3638754593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b01412d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- 172.21.15.111:0/3638754593 >> 172.21.15.111:0/3638754593 conn(0x7fa8b00b46d0 msgr2=0x7fa8b005bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- 172.21.15.111:0/3638754593 shutdown_connections 2022-07-12T08:40:58.820 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 -- 172.21.15.111:0/3638754593 wait complete. 2022-07-12T08:40:58.831 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.789+0000 7fa8b61ca700 1 Processor -- start 2022-07-12T08:40:58.831 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 -- start start 2022-07-12T08:40:58.831 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b009a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8b00ac9a0 0x7fa8b0060a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 0x7fa8b0065b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b013a120 con 0x7fa8b00ac9a0 2022-07-12T08:40:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0060f00 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.832 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0066af0 con 0x7fa8b013ded0 2022-07-12T08:40:58.836 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b009a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.836 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b59c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 0x7fa8b0065b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.837 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b009a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43872/0 (socket says 172.21.15.111:43872) 2022-07-12T08:40:58.837 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b59c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 0x7fa8b0065b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50634/0 (socket says 172.21.15.111:50634) 2022-07-12T08:40:58.837 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 -- 172.21.15.111:0/1309966553 learned_addr learned my addr 172.21.15.111:0/1309966553 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:58.837 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b49c7700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8b00ac9a0 0x7fa8b0060a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.838 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 -- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 msgr2=0x7fa8b0065b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.838 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 0x7fa8b0065b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.838 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 -- 172.21.15.111:0/1309966553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8b00ac9a0 msgr2=0x7fa8b0060a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.838 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8b00ac9a0 0x7fa8b0060a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.839 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 -- 172.21.15.111:0/1309966553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac00b180 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.839 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b59c9700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 0x7fa8b0065b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:40:58.839 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b51c8700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b009a750 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa8a00012a0 tx=0x7fa8ac014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.839 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8ac013da0 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac002f70 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b00675d0 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8ac01e5f0 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.793+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b0098c90 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.840 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.797+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b0139f80 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.797+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7fa8ac01c070 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.797+0000 7fa89e7fc700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8900539a0 0x7fa890055e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.797+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1360+0+0 (secure 0 0 0) 0x7fa8ac06b610 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.797+0000 7fa8b49c7700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8900539a0 0x7fa890055e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.797+0000 7fa8b49c7700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8900539a0 0x7fa890055e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa8a000d000 tx=0x7fa8a0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:58.842 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.805+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8b0139f80 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.945 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.941+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", "id": 2} v 0) v1 -- 0x7fa8b0139f80 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.953 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.949+0000 7fa89e7fc700 1 -- 172.21.15.111:0/1309966553 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa8b0139f80 con 0x7fa8b00ac5c0 2022-07-12T08:40:58.954 INFO:teuthology.orchestra.run.smithi111.stdout:2 2022-07-12T08:40:58.956 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8900539a0 msgr2=0x7fa890055e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.957 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8900539a0 0x7fa890055e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa8a000d000 tx=0x7fa8a0009040).stop 2022-07-12T08:40:58.957 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 msgr2=0x7fa8b009a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:58.957 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b009a750 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa8a00012a0 tx=0x7fa8ac014000).stop 2022-07-12T08:40:58.958 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 shutdown_connections 2022-07-12T08:40:58.958 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8b013ded0 0x7fa8b0065b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.958 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8b00ac9a0 0x7fa8b0060a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.959 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8900539a0 0x7fa890055e60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.959 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 --2- 172.21.15.111:0/1309966553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8b00ac5c0 0x7fa8b009a750 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:58.960 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 >> 172.21.15.111:0/1309966553 conn(0x7fa8b00b46d0 msgr2=0x7fa8b00b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:58.960 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 shutdown_connections 2022-07-12T08:40:58.961 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:58.953+0000 7fa8b61ca700 1 -- 172.21.15.111:0/1309966553 wait complete. 2022-07-12T08:40:58.968 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 0f043fbb-3135-49f3-9b49-a6ce7e7347b4 3 2022-07-12T08:40:59.076 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 Processor -- start 2022-07-12T08:40:59.076 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 -- start start 2022-07-12T08:40:59.079 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1fc0b9f00 0x7fa1fc13fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.080 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b3e50 0x7fa1fc140090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.080 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1fc1437b0 con 0x7fa1fc0b00a0 2022-07-12T08:40:59.080 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1fc0ba2e0 con 0x7fa1fc0b9f00 2022-07-12T08:40:59.081 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa201b9f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc05d0b0 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.081 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa1faffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b3e50 0x7fa1fc140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.081 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa1fb7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1fc0b9f00 0x7fa1fc13fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.081 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa1faffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b3e50 0x7fa1fc140090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50640/0 (socket says 172.21.15.111:50640) 2022-07-12T08:40:59.090 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.073+0000 7fa1faffd700 1 -- 172.21.15.111:0/3829778086 learned_addr learned my addr 172.21.15.111:0/3829778086 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:59.091 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fb7fe700 1 -- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b3e50 msgr2=0x7fa1fc140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:40:59.091 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fb7fe700 1 --2- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b3e50 0x7fa1fc140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.092 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fb7fe700 1 -- 172.21.15.111:0/3829778086 >> v1:172.21.15.111:6789/0 conn(0x7fa1fc0b00a0 legacy=0x7fa1fc146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.092 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fa7fc700 1 -- 172.21.15.111:0/3829778086 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2946176829 0 0) 0x7fa1fc1437b0 con 0x7fa1fc0b00a0 2022-07-12T08:40:59.092 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fb7fe700 1 -- 172.21.15.111:0/3829778086 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1fc1439e0 con 0x7fa1fc0b9f00 2022-07-12T08:40:59.093 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fb7fe700 1 --2- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b9f00 0x7fa1fc13fba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa1ec0032f0 tx=0x7fa1ec0091d0).ready entity=mon.1 client_cookie=4c7bb2c0e83bc3d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.093 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fa7fc700 1 -- 172.21.15.111:0/3829778086 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ec00b6c0 con 0x7fa1fc0b9f00 2022-07-12T08:40:59.093 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fa7fc700 1 -- 172.21.15.111:0/3829778086 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ec00b820 con 0x7fa1fc0b9f00 2022-07-12T08:40:59.094 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fa7fc700 1 -- 172.21.15.111:0/3829778086 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ec00b990 con 0x7fa1fc0b9f00 2022-07-12T08:40:59.094 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b9f00 msgr2=0x7fa1fc13fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.094 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b9f00 0x7fa1fc13fba0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fa1ec0032f0 tx=0x7fa1ec0091d0).stop 2022-07-12T08:40:59.094 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- 172.21.15.111:0/3829778086 shutdown_connections 2022-07-12T08:40:59.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b3e50 0x7fa1fc140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3829778086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b9f00 0x7fa1fc13fba0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- 172.21.15.111:0/3829778086 >> 172.21.15.111:0/3829778086 conn(0x7fa1fc0679d0 msgr2=0x7fa1fc0b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:59.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- 172.21.15.111:0/3829778086 shutdown_connections 2022-07-12T08:40:59.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- 172.21.15.111:0/3829778086 wait complete. 2022-07-12T08:40:59.151 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 Processor -- start 2022-07-12T08:40:59.151 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- start start 2022-07-12T08:40:59.151 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1fc0b00a0 0x7fa1fc13cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 0x7fa1fc0ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b9f00 0x7fa1fc0aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.152 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0ae860 con 0x7fa1fc0b00a0 2022-07-12T08:40:59.153 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc1551a0 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.153 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa201b9f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1fc0aea90 con 0x7fa1fc0b9f00 2022-07-12T08:40:59.153 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 0x7fa1fc0ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b9f00 0x7fa1fc0aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 0x7fa1fc0ad780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43888/0 (socket says 172.21.15.111:43888) 2022-07-12T08:40:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1faffd700 1 -- 172.21.15.111:0/3259499048 learned_addr learned my addr 172.21.15.111:0/3259499048 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:59.154 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.077+0000 7fa1fb7fe700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1fc0b00a0 0x7fa1fc13cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1faffd700 1 -- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b9f00 msgr2=0x7fa1fc0aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1faffd700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b9f00 0x7fa1fc0aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1faffd700 1 -- 172.21.15.111:0/3259499048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1fc0b00a0 msgr2=0x7fa1fc13cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1faffd700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1fc0b00a0 0x7fa1fc13cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1faffd700 1 -- 172.21.15.111:0/3259499048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec009470 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.158 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1fb7fe700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1fc0b00a0 0x7fa1fc13cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:40:59.159 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1faffd700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 0x7fa1fc0ad780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa1fc0acd50 tx=0x7fa1e4008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.162 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1e4013610 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.162 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e4013770 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.162 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1fc152d70 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.162 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1e4013ad0 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.163 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1fc148ef0 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.163 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc0aec50 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.163 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7fa1e4013c30 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.164 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1f8ff9700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1e0053910 0x7fa1e0055dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.165 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1499+0+0 (secure 0 0 0) 0x7fa1e406ac10 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.165 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1fb7fe700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1e0053910 0x7fa1e0055dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.165 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.081+0000 7fa1fb7fe700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1e0053910 0x7fa1e0055dd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa1ec019000 tx=0x7fa1ec016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.166 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.089+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1e4042df0 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.334 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.229+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", "id": 3} v 0) v1 -- 0x7fa1fc057460 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.335 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.237+0000 7fa1f8ff9700 1 -- 172.21.15.111:0/3259499048 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa1e4046410 con 0x7fa1fc0b3e50 2022-07-12T08:40:59.336 INFO:teuthology.orchestra.run.smithi111.stdout:3 2022-07-12T08:40:59.349 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1e0053910 msgr2=0x7fa1e0055dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.350 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1e0053910 0x7fa1e0055dd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa1ec019000 tx=0x7fa1ec016000).stop 2022-07-12T08:40:59.350 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 msgr2=0x7fa1fc0ad780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.394 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 0x7fa1fc0ad780 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fa1fc0acd50 tx=0x7fa1e4008040).stop 2022-07-12T08:40:59.394 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 shutdown_connections 2022-07-12T08:40:59.395 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1fc0b9f00 0x7fa1fc0aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.395 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1e0053910 0x7fa1e0055dd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.395 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1fc0b3e50 0x7fa1fc0ad780 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.396 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 --2- 172.21.15.111:0/3259499048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1fc0b00a0 0x7fa1fc13cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.396 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 >> 172.21.15.111:0/3259499048 conn(0x7fa1fc0679d0 msgr2=0x7fa1fc0b9490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:59.396 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 shutdown_connections 2022-07-12T08:40:59.397 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.241+0000 7fa201b9f700 1 -- 172.21.15.111:0/3259499048 wait complete. 2022-07-12T08:40:59.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new b2005cdb-ea05-443e-8816-aaa7dfabfd84 4 2022-07-12T08:40:59.613 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 Processor -- start 2022-07-12T08:40:59.613 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- start start 2022-07-12T08:40:59.614 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa30c14aba0 0x7fa30c152840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.614 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30c05d350 con 0x7fa30c060870 2022-07-12T08:40:59.615 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30c05d0b0 con 0x7fa30c060c50 2022-07-12T08:40:59.615 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30c09afe0 con 0x7fa30c14aba0 2022-07-12T08:40:59.615 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3037fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa30c060c50 0x7fa30c147570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33806/0 (socket says 172.21.15.111:33806) 2022-07-12T08:40:59.616 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa310cc4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa30c14aba0 0x7fa30c152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.617 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3037fe700 1 -- 172.21.15.111:0/1212999710 learned_addr learned my addr 172.21.15.111:0/1212999710 (peer_addr_for_me v1:172.21.15.111:0/0) 2022-07-12T08:40:59.617 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa30b7fe700 1 -- 172.21.15.111:0/1212999710 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472381377 0 0) 0x7fa30c05d0b0 con 0x7fa30c060c50 2022-07-12T08:40:59.617 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa30b7fe700 1 -- 172.21.15.111:0/1212999710 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec003260 con 0x7fa30c060c50 2022-07-12T08:40:59.641 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa30b7fe700 1 -- 172.21.15.111:0/1212999710 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142392368 0 0) 0x7fa2ec003260 con 0x7fa30c060c50 2022-07-12T08:40:59.641 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa310cc4700 1 -- 172.21.15.111:0/1212999710 >> v1:172.21.15.111:6789/0 conn(0x7fa30c060c50 legacy=0x7fa30c147570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.642 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa310cc4700 1 -- 172.21.15.111:0/1212999710 >> v1:172.21.15.61:6789/0 conn(0x7fa30c060870 legacy=0x7fa30c14a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.642 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa310cc4700 1 -- 172.21.15.111:0/1212999710 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c0b0ec0 con 0x7fa30c14aba0 2022-07-12T08:40:59.643 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa310cc4700 1 --2- 172.21.15.111:0/1212999710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c152840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fa2fc001160 tx=0x7fa3040091d0).ready entity=mon.1 client_cookie=47f07f69dcb92162 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.643 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa30b7fe700 1 -- 172.21.15.111:0/1212999710 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f4001660 con 0x7fa30c14aba0 2022-07-12T08:40:59.643 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa30b7fe700 1 -- 172.21.15.111:0/1212999710 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30400b500 con 0x7fa30c14aba0 2022-07-12T08:40:59.644 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa30b7fe700 1 -- 172.21.15.111:0/1212999710 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa30400b670 con 0x7fa30c14aba0 2022-07-12T08:40:59.644 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- 172.21.15.111:0/1212999710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 msgr2=0x7fa30c152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 --2- 172.21.15.111:0/1212999710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c152840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fa2fc001160 tx=0x7fa3040091d0).stop 2022-07-12T08:40:59.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- 172.21.15.111:0/1212999710 shutdown_connections 2022-07-12T08:40:59.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 --2- 172.21.15.111:0/1212999710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c152840 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.658 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- 172.21.15.111:0/1212999710 >> 172.21.15.111:0/1212999710 conn(0x7fa30c05bd10 msgr2=0x7fa30c05c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:59.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.565+0000 7fa3114c5700 1 -- 172.21.15.111:0/1212999710 shutdown_connections 2022-07-12T08:40:59.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 -- 172.21.15.111:0/1212999710 wait complete. 2022-07-12T08:40:59.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 Processor -- start 2022-07-12T08:40:59.660 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 -- start start 2022-07-12T08:40:59.660 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 0x7fa30c063000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa30c060c50 0x7fa30c0ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c0ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c063630 con 0x7fa30c060c50 2022-07-12T08:40:59.664 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c0638d0 con 0x7fa30c14aba0 2022-07-12T08:40:59.664 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c066a90 con 0x7fa30c060870 2022-07-12T08:40:59.664 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 0x7fa30c063000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.665 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa310cc4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c0ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.665 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 0x7fa30c063000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50658/0 (socket says 172.21.15.111:50658) 2022-07-12T08:40:59.666 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa310cc4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c0ae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43904/0 (socket says 172.21.15.111:43904) 2022-07-12T08:40:59.666 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 -- 172.21.15.111:0/350898697 learned_addr learned my addr 172.21.15.111:0/350898697 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:59.666 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3037fe700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa30c060c50 0x7fa30c0ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 -- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 msgr2=0x7fa30c0ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.667 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c0ae580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.668 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 -- 172.21.15.111:0/350898697 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa30c060c50 msgr2=0x7fa30c0ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.669 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa30c060c50 0x7fa30c0ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.669 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 -- 172.21.15.111:0/350898697 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304009470 con 0x7fa30c060870 2022-07-12T08:40:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa310cc4700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c0ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:40:59.670 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa30bfff700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 0x7fa30c063000 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa2fc001220 tx=0x7fa2fc007c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.671 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2fc00ce20 con 0x7fa30c060870 2022-07-12T08:40:59.672 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc00f470 con 0x7fa30c060870 2022-07-12T08:40:59.672 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30c0643b0 con 0x7fa30c060870 2022-07-12T08:40:59.673 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.569+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2fc00f7d0 con 0x7fa30c060870 2022-07-12T08:40:59.673 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30c066f30 con 0x7fa30c060870 2022-07-12T08:40:59.674 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30c066c50 con 0x7fa30c060870 2022-07-12T08:40:59.674 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7fa2fc019030 con 0x7fa30c060870 2022-07-12T08:40:59.675 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa309ffb700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2f00539a0 0x7fa2f0055e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.675 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1614+0+0 (secure 0 0 0) 0x7fa2fc069ff0 con 0x7fa30c060870 2022-07-12T08:40:59.676 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa3037fe700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2f00539a0 0x7fa2f0055e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.676 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.573+0000 7fa3037fe700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2f00539a0 0x7fa2f0055e60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa2f400d000 tx=0x7fa2f4000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.676 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.577+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa2fc042210 con 0x7fa30c060870 2022-07-12T08:40:59.718 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.709+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", "id": 4} v 0) v1 -- 0x7fa30c057460 con 0x7fa30c060870 2022-07-12T08:40:59.724 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.721+0000 7fa309ffb700 1 -- 172.21.15.111:0/350898697 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa2fc017020 con 0x7fa30c060870 2022-07-12T08:40:59.725 INFO:teuthology.orchestra.run.smithi111.stdout:4 2022-07-12T08:40:59.729 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.721+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2f00539a0 msgr2=0x7fa2f0055e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.730 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.721+0000 7fa3114c5700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2f00539a0 0x7fa2f0055e60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa2f400d000 tx=0x7fa2f4000bc0).stop 2022-07-12T08:40:59.731 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.721+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 msgr2=0x7fa30c063000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.731 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.721+0000 7fa3114c5700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 0x7fa30c063000 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa2fc001220 tx=0x7fa2fc007c00).stop 2022-07-12T08:40:59.731 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 shutdown_connections 2022-07-12T08:40:59.732 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2f00539a0 0x7fa2f0055e60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.732 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa30c14aba0 0x7fa30c0ae580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.733 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa30c060c50 0x7fa30c0ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.734 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 --2- 172.21.15.111:0/350898697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa30c060870 0x7fa30c063000 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.746 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 >> 172.21.15.111:0/350898697 conn(0x7fa30c05bd10 msgr2=0x7fa30c098d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:59.746 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 shutdown_connections 2022-07-12T08:40:59.747 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.725+0000 7fa3114c5700 1 -- 172.21.15.111:0/350898697 wait complete. 2022-07-12T08:40:59.749 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 9b0164a4-42b8-409a-86fb-bafc85e28b29 5 2022-07-12T08:40:59.894 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 Processor -- start 2022-07-12T08:40:59.894 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 -- start start 2022-07-12T08:40:59.945 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 0x7ff8fc0b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.955 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8fc1467a0 0x7ff8fc141e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:40:59.955 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc05ccc0 con 0x7ff8fc05c8e0 2022-07-12T08:40:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc0b1640 con 0x7ff8fc1467a0 2022-07-12T08:40:59.956 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff900e4a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc0adec0 con 0x7ff8fc05f3c0 2022-07-12T08:40:59.957 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff8faffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 0x7ff8fc0b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.957 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff8faffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 0x7ff8fc0b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50668/0 (socket says 172.21.15.111:50668) 2022-07-12T08:40:59.958 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.889+0000 7ff8faffd700 1 -- 172.21.15.111:0/1198911697 learned_addr learned my addr 172.21.15.111:0/1198911697 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:40:59.958 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8fbfff700 1 --2- 172.21.15.111:0/1198911697 >> v2:172.21.15.111:3300/0 conn(0x7ff8fc1467a0 0x7ff8fc141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:40:59.959 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8faffd700 1 -- 172.21.15.111:0/1198911697 >> v2:172.21.15.111:3300/0 conn(0x7ff8fc1467a0 msgr2=0x7ff8fc141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:40:59.959 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8faffd700 1 --2- 172.21.15.111:0/1198911697 >> v2:172.21.15.111:3300/0 conn(0x7ff8fc1467a0 0x7ff8fc141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.960 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8faffd700 1 -- 172.21.15.111:0/1198911697 >> v1:172.21.15.111:6789/0 conn(0x7ff8fc05c8e0 legacy=0x7ff8fc146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.960 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8faffd700 1 -- 172.21.15.111:0/1198911697 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0ae160 con 0x7ff8fc05f3c0 2022-07-12T08:40:59.961 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/1198911697 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048639990 0 0) 0x7ff8fc05ccc0 con 0x7ff8fc05c8e0 2022-07-12T08:40:59.961 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8faffd700 1 --2- 172.21.15.111:0/1198911697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 0x7ff8fc0b1150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff8e40032f0 tx=0x7ff8e4009290).ready entity=mon.2 client_cookie=dcc1bd00f3bb9838 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:40:59.961 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/1198911697 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e400b8a0 con 0x7ff8fc05f3c0 2022-07-12T08:40:59.980 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/1198911697 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e400ba00 con 0x7ff8fc05f3c0 2022-07-12T08:40:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff8fa7fc700 1 -- 172.21.15.111:0/1198911697 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e400bb70 con 0x7ff8fc05f3c0 2022-07-12T08:40:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 -- 172.21.15.111:0/1198911697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 msgr2=0x7ff8fc0b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:40:59.995 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 --2- 172.21.15.111:0/1198911697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 0x7ff8fc0b1150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7ff8e40032f0 tx=0x7ff8e4009290).stop 2022-07-12T08:40:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 -- 172.21.15.111:0/1198911697 shutdown_connections 2022-07-12T08:40:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 --2- 172.21.15.111:0/1198911697 >> v2:172.21.15.111:3300/0 conn(0x7ff8fc1467a0 0x7ff8fc141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 --2- 172.21.15.111:0/1198911697 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05f3c0 0x7ff8fc0b1150 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:40:59.997 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 -- 172.21.15.111:0/1198911697 >> 172.21.15.111:0/1198911697 conn(0x7ff8fc063a00 msgr2=0x7ff8fc063e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:40:59.999 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 -- 172.21.15.111:0/1198911697 shutdown_connections 2022-07-12T08:41:00.005 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 -- 172.21.15.111:0/1198911697 wait complete. 2022-07-12T08:41:00.014 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 Processor -- start 2022-07-12T08:41:00.015 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.893+0000 7ff900e4a700 1 -- start start 2022-07-12T08:41:00.015 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05c8e0 0x7ff8fc0b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.016 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc05f3c0 0x7ff8fc151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.016 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8fc1467a0 0x7ff8fc1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.016 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc115510 con 0x7ff8fc1467a0 2022-07-12T08:41:00.017 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc1157b0 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.017 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc0b4970 con 0x7ff8fc05c8e0 2022-07-12T08:41:00.047 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8fb7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05c8e0 0x7ff8fc0b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc05f3c0 0x7ff8fc151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.085 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8fb7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05c8e0 0x7ff8fc0b6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50672/0 (socket says 172.21.15.111:50672) 2022-07-12T08:41:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8fb7fe700 1 -- 172.21.15.111:0/3878015741 learned_addr learned my addr 172.21.15.111:0/3878015741 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:41:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8fbfff700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8fc1467a0 0x7ff8fc1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 -- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05c8e0 msgr2=0x7ff8fc0b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.087 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05c8e0 0x7ff8fc0b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 -- 172.21.15.111:0/3878015741 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8fc1467a0 msgr2=0x7ff8fc1523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.088 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8fc1467a0 0x7ff8fc1523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 -- 172.21.15.111:0/3878015741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e4009530 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.089 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8faffd700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc05f3c0 0x7ff8fc151eb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff8e40032f0 tx=0x7ff8e4016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.090 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e400b8a0 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.091 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4021630 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.092 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc0b52f0 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.093 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8e4021990 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.093 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.897+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc0b5030 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.901+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc0b4b30 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.095 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.901+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7ff8e4027020 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.096 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.901+0000 7ff8f8ff9700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8e00539a0 0x7ff8e0055e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.096 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.901+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1729+0+0 (secure 0 0 0) 0x7ff8e406c220 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.901+0000 7ff8fb7fe700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8e00539a0 0x7ff8e0055e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.097 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.901+0000 7ff8fb7fe700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8e00539a0 0x7ff8e0055e60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff8ec00d000 tx=0x7ff8ec007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.099 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:40:59.905+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8e40443c0 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.103 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.037+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff8e400f070 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.104 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.045+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", "id": 5} v 0) v1 -- 0x7ff8fc057460 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.104 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.053+0000 7ff8f8ff9700 1 -- 172.21.15.111:0/3878015741 <== mon.1 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff8e4015070 con 0x7ff8fc05f3c0 2022-07-12T08:41:00.105 INFO:teuthology.orchestra.run.smithi111.stdout:5 2022-07-12T08:41:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8e00539a0 msgr2=0x7ff8e0055e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.109 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8e00539a0 0x7ff8e0055e60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff8ec00d000 tx=0x7ff8ec007b50).stop 2022-07-12T08:41:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc05f3c0 msgr2=0x7ff8fc151eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.110 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc05f3c0 0x7ff8fc151eb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7ff8e40032f0 tx=0x7ff8e4016000).stop 2022-07-12T08:41:00.111 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 shutdown_connections 2022-07-12T08:41:00.116 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8fc1467a0 0x7ff8fc1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.117 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8e00539a0 0x7ff8e0055e60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.117 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8fc05f3c0 0x7ff8fc151eb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.118 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 --2- 172.21.15.111:0/3878015741 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8fc05c8e0 0x7ff8fc0b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.119 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 >> 172.21.15.111:0/3878015741 conn(0x7ff8fc063a00 msgr2=0x7ff8fc145fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 shutdown_connections 2022-07-12T08:41:00.120 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.057+0000 7ff900e4a700 1 -- 172.21.15.111:0/3878015741 wait complete. 2022-07-12T08:41:00.121 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 1718afd4-28d5-4c9e-bbb8-f8b29600a5fa 6 2022-07-12T08:41:00.237 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 Processor -- start 2022-07-12T08:41:00.237 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 -- start start 2022-07-12T08:41:00.309 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdde4064f90 0x7fdde40aff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.325 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdde4142f00 0x7fdde40b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.328 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde40b2550 con 0x7fdde40b1210 2022-07-12T08:41:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40b08e0 con 0x7fdde4064f90 2022-07-12T08:41:00.338 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddec4e3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde4065370 con 0x7fdde4142f00 2022-07-12T08:41:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fdde9a7e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdde4142f00 0x7fdde40b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.339 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdde4142f00 0x7fdde40b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43924/0 (socket says 172.21.15.111:43924) 2022-07-12T08:41:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.233+0000 7fddeaa80700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fdde40b1210 0x7fdde4066650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33836/0 (socket says 172.21.15.111:33836) 2022-07-12T08:41:00.340 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 -- 172.21.15.111:0/3912446753 learned_addr learned my addr 172.21.15.111:0/3912446753 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:41:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddea27f700 1 --2- 172.21.15.111:0/3912446753 >> v2:172.21.15.61:3300/0 conn(0x7fdde4064f90 0x7fdde40aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 -- 172.21.15.111:0/3912446753 >> v2:172.21.15.61:3300/0 conn(0x7fdde4064f90 msgr2=0x7fdde40aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:00.341 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 --2- 172.21.15.111:0/3912446753 >> v2:172.21.15.61:3300/0 conn(0x7fdde4064f90 0x7fdde40aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.367 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 -- 172.21.15.111:0/3912446753 >> v1:172.21.15.111:6789/0 conn(0x7fdde40b1210 legacy=0x7fdde4066650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde927d700 1 -- 172.21.15.111:0/3912446753 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176933228 0 0) 0x7fdde40b2550 con 0x7fdde40b1210 2022-07-12T08:41:00.368 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 -- 172.21.15.111:0/3912446753 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde40b2010 con 0x7fdde4142f00 2022-07-12T08:41:00.369 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde9a7e700 1 --2- 172.21.15.111:0/3912446753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b03f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fddd4002f70 tx=0x7fddd4006eb0).ready entity=mon.1 client_cookie=f5182c3c37a38c88 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.371 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde927d700 1 -- 172.21.15.111:0/3912446753 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd400d5c0 con 0x7fdde4142f00 2022-07-12T08:41:00.371 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde927d700 1 -- 172.21.15.111:0/3912446753 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4000e90 con 0x7fdde4142f00 2022-07-12T08:41:00.372 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fdde927d700 1 -- 172.21.15.111:0/3912446753 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd400d910 con 0x7fdde4142f00 2022-07-12T08:41:00.373 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 -- 172.21.15.111:0/3912446753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 msgr2=0x7fdde40b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.373 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 --2- 172.21.15.111:0/3912446753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b03f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fddd4002f70 tx=0x7fddd4006eb0).stop 2022-07-12T08:41:00.374 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 -- 172.21.15.111:0/3912446753 shutdown_connections 2022-07-12T08:41:00.374 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 --2- 172.21.15.111:0/3912446753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b03f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.375 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 --2- 172.21.15.111:0/3912446753 >> v2:172.21.15.61:3300/0 conn(0x7fdde4064f90 0x7fdde40aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.375 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 -- 172.21.15.111:0/3912446753 >> 172.21.15.111:0/3912446753 conn(0x7fdde4063c40 msgr2=0x7fdde4064020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:00.376 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 -- 172.21.15.111:0/3912446753 shutdown_connections 2022-07-12T08:41:00.376 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 -- 172.21.15.111:0/3912446753 wait complete. 2022-07-12T08:41:00.392 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 Processor -- start 2022-07-12T08:41:00.404 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.237+0000 7fddec4e3700 1 -- start start 2022-07-12T08:41:00.404 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdde4064f90 0x7fdde4155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.405 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 0x7fdde4110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.405 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde41104f0 con 0x7fdde4064f90 2022-07-12T08:41:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4110710 con 0x7fdde4142f00 2022-07-12T08:41:00.406 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde41109b0 con 0x7fdde40b1210 2022-07-12T08:41:00.407 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 0x7fdde4110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.425 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddeaa80700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.428 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 0x7fdde4110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50686/0 (socket says 172.21.15.111:50686) 2022-07-12T08:41:00.429 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 -- 172.21.15.111:0/653291846 learned_addr learned my addr 172.21.15.111:0/653291846 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:41:00.429 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddea27f700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdde4064f90 0x7fdde4155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 -- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 msgr2=0x7fdde40b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.430 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 -- 172.21.15.111:0/653291846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdde4064f90 msgr2=0x7fdde4155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.431 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdde4064f90 0x7fdde4155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.432 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 -- 172.21.15.111:0/653291846 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd400b1e0 con 0x7fdde40b1210 2022-07-12T08:41:00.432 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fdde9a7e700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 0x7fdde4110000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fdde4110630 tx=0x7fddd4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.432 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd40116e0 con 0x7fdde40b1210 2022-07-12T08:41:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4011840 con 0x7fdde40b1210 2022-07-12T08:41:00.433 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4111ed0 con 0x7fdde40b1210 2022-07-12T08:41:00.446 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd4011ba0 con 0x7fdde40b1210 2022-07-12T08:41:00.446 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde40b4700 con 0x7fdde40b1210 2022-07-12T08:41:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde410fbb0 con 0x7fdde40b1210 2022-07-12T08:41:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fddd4025030 con 0x7fdde40b1210 2022-07-12T08:41:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.241+0000 7fddd37fe700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fddc4053a40 0x7fddc4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.447 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.245+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1844+0+0 (secure 0 0 0) 0x7fddd406c010 con 0x7fdde40b1210 2022-07-12T08:41:00.448 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.245+0000 7fddea27f700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fddc4053a40 0x7fddc4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.449 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.245+0000 7fddea27f700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fddc4053a40 0x7fddc4055f00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fdde000b000 tx=0x7fdde0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.449 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.249+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdde410fbb0 con 0x7fdde40b1210 2022-07-12T08:41:00.450 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.385+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", "id": 6} v 0) v1 -- 0x7fdde410fbb0 con 0x7fdde40b1210 2022-07-12T08:41:00.451 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.397+0000 7fddd37fe700 1 -- 172.21.15.111:0/653291846 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdde410fbb0 con 0x7fdde40b1210 2022-07-12T08:41:00.451 INFO:teuthology.orchestra.run.smithi111.stdout:6 2022-07-12T08:41:00.476 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fddc4053a40 msgr2=0x7fddc4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.476 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fddc4053a40 0x7fddc4055f00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fdde000b000 tx=0x7fdde0008040).stop 2022-07-12T08:41:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 msgr2=0x7fdde4110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 0x7fdde4110000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fdde4110630 tx=0x7fddd4015000).stop 2022-07-12T08:41:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 shutdown_connections 2022-07-12T08:41:00.477 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fddc4053a40 0x7fddc4055f00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdde4142f00 0x7fdde40b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdde40b1210 0x7fdde4110000 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 --2- 172.21.15.111:0/653291846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdde4064f90 0x7fdde4155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.478 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 >> 172.21.15.111:0/653291846 conn(0x7fdde4063c40 msgr2=0x7fdde413fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:00.479 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 shutdown_connections 2022-07-12T08:41:00.479 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.401+0000 7fddec4e3700 1 -- 172.21.15.111:0/653291846 wait complete. 2022-07-12T08:41:00.486 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd new 87944907-c10c-4187-abd4-fd919e7520be 7 2022-07-12T08:41:00.643 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa068620700 1 Processor -- start 2022-07-12T08:41:00.643 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa068620700 1 -- start start 2022-07-12T08:41:00.644 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa068620700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa0600b9f00 0x7fa0601466f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.644 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa068620700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0601462c0 con 0x7fa0600b5380 2022-07-12T08:41:00.644 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa068620700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0600b5c10 con 0x7fa06005bd10 2022-07-12T08:41:00.644 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa068620700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0600ba2e0 con 0x7fa0600b9f00 2022-07-12T08:41:00.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0663bc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa0600b9f00 0x7fa0601466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0663bc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa0600b9f00 0x7fa0601466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43936/0 (socket says 172.21.15.111:43936) 2022-07-12T08:41:00.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0663bc700 1 -- 172.21.15.111:0/3058299669 learned_addr learned my addr 172.21.15.111:0/3058299669 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:41:00.645 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa066bbd700 1 --1- 172.21.15.111:0/3058299669 >> v1:172.21.15.111:6789/0 conn(0x7fa06005bd10 0x7fa06005c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.111:33848/0 (socket says 172.21.15.111:33848) 2022-07-12T08:41:00.646 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0663bc700 1 -- 172.21.15.111:0/3058299669 >> v1:172.21.15.111:6789/0 conn(0x7fa06005bd10 legacy=0x7fa06005c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.646 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0663bc700 1 -- 172.21.15.111:0/3058299669 >> v1:172.21.15.61:6789/0 conn(0x7fa0600b5380 legacy=0x7fa060146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.646 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0653ba700 1 -- 172.21.15.111:0/3058299669 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097483550 0 0) 0x7fa0600b5c10 con 0x7fa06005bd10 2022-07-12T08:41:00.647 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.625+0000 7fa0663bc700 1 -- 172.21.15.111:0/3058299669 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0600b0c70 con 0x7fa0600b9f00 2022-07-12T08:41:00.647 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa0663bc700 1 --2- 172.21.15.111:0/3058299669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601466f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa05c002f70 tx=0x7fa05c006eb0).ready entity=mon.1 client_cookie=af06cc9c92adfb76 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.647 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa0653ba700 1 -- 172.21.15.111:0/3058299669 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa05c00d660 con 0x7fa0600b9f00 2022-07-12T08:41:00.647 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa0653ba700 1 -- 172.21.15.111:0/3058299669 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c000e90 con 0x7fa0600b9f00 2022-07-12T08:41:00.648 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa0653ba700 1 -- 172.21.15.111:0/3058299669 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa05c00d9b0 con 0x7fa0600b9f00 2022-07-12T08:41:00.651 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 -- 172.21.15.111:0/3058299669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 msgr2=0x7fa0601466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.653 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 --2- 172.21.15.111:0/3058299669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601466f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa05c002f70 tx=0x7fa05c006eb0).stop 2022-07-12T08:41:00.653 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 -- 172.21.15.111:0/3058299669 shutdown_connections 2022-07-12T08:41:00.653 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 --2- 172.21.15.111:0/3058299669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601466f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.654 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 -- 172.21.15.111:0/3058299669 >> 172.21.15.111:0/3058299669 conn(0x7fa0600679d0 msgr2=0x7fa0600b94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:00.654 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 -- 172.21.15.111:0/3058299669 shutdown_connections 2022-07-12T08:41:00.655 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 -- 172.21.15.111:0/3058299669 wait complete. 2022-07-12T08:41:00.655 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 Processor -- start 2022-07-12T08:41:00.655 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.629+0000 7fa068620700 1 -- start start 2022-07-12T08:41:00.656 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa06005bd10 0x7fa0601479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.656 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 0x7fa060149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.656 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600ad570 con 0x7fa06005bd10 2022-07-12T08:41:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa060146c40 con 0x7fa0600b9f00 2022-07-12T08:41:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa060146ee0 con 0x7fa0600b5380 2022-07-12T08:41:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa065bbb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 0x7fa060149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.657 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa065bbb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 0x7fa060149f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.111:50702/0 (socket says 172.21.15.111:50702) 2022-07-12T08:41:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa065bbb700 1 -- 172.21.15.111:0/2545270557 learned_addr learned my addr 172.21.15.111:0/2545270557 (peer_addr_for_me v2:172.21.15.111:0/0) 2022-07-12T08:41:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa0663bc700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa06005bd10 0x7fa0601479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.658 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 -- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 msgr2=0x7fa060149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 0x7fa060149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 -- 172.21.15.111:0/2545270557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa06005bd10 msgr2=0x7fa0601479d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa06005bd10 0x7fa0601479d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.659 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 -- 172.21.15.111:0/2545270557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c00b1e0 con 0x7fa0600b9f00 2022-07-12T08:41:00.660 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa065bbb700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 0x7fa060149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:00.660 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa066bbd700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601551e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa058001360 tx=0x7fa058007d90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.660 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa058005bb0 con 0x7fa0600b9f00 2022-07-12T08:41:00.660 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058005d10 con 0x7fa0600b9f00 2022-07-12T08:41:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa060146550 con 0x7fa0600b9f00 2022-07-12T08:41:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0580072c0 con 0x7fa0600b9f00 2022-07-12T08:41:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0600ad9d0 con 0x7fa0600b9f00 2022-07-12T08:41:00.661 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa060146aa0 con 0x7fa0600b9f00 2022-07-12T08:41:00.663 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.633+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa05801a060 con 0x7fa0600b9f00 2022-07-12T08:41:00.663 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.637+0000 7fa04f7fe700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa040053ad0 0x7fa040055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:00.663 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.637+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1959+0+0 (secure 0 0 0) 0x7fa05806a930 con 0x7fa0600b9f00 2022-07-12T08:41:00.664 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.637+0000 7fa0663bc700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa040053ad0 0x7fa040055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:00.664 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.637+0000 7fa0663bc700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa040053ad0 0x7fa040055f90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa05c01a000 tx=0x7fa05c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:00.665 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.641+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa060146aa0 con 0x7fa0600b9f00 2022-07-12T08:41:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.777+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd new", "uuid": "87944907-c10c-4187-abd4-fd919e7520be", "id": 7} v 0) v1 -- 0x7fa060146aa0 con 0x7fa0600b9f00 2022-07-12T08:41:00.805 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.789+0000 7fa04f7fe700 1 -- 172.21.15.111:0/2545270557 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87944907-c10c-4187-abd4-fd919e7520be", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa060146aa0 con 0x7fa0600b9f00 2022-07-12T08:41:00.838 INFO:teuthology.orchestra.run.smithi111.stdout:7 2022-07-12T08:41:00.840 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa040053ad0 msgr2=0x7fa040055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa040053ad0 0x7fa040055f90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa05c01a000 tx=0x7fa05c015000).stop 2022-07-12T08:41:00.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 msgr2=0x7fa0601551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:00.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601551e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa058001360 tx=0x7fa058007d90).stop 2022-07-12T08:41:00.841 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 shutdown_connections 2022-07-12T08:41:00.848 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa040053ad0 0x7fa040055f90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.849 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0600b9f00 0x7fa0601551e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa0600b5380 0x7fa060149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.850 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 --2- 172.21.15.111:0/2545270557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa06005bd10 0x7fa0601479d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:00.851 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 >> 172.21.15.111:0/2545270557 conn(0x7fa0600679d0 msgr2=0x7fa0601411c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:00.851 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 shutdown_connections 2022-07-12T08:41:00.851 INFO:teuthology.orchestra.run.smithi111.stderr:2022-07-12T08:41:00.793+0000 7fa068620700 1 -- 172.21.15.111:0/2545270557 wait complete. 2022-07-12T08:41:00.852 INFO:tasks.ceph.osd.0:Restarting daemon 2022-07-12T08:41:00.853 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-07-12T08:41:00.857 INFO:tasks.ceph.osd.0:Started 2022-07-12T08:41:00.860 INFO:tasks.ceph.osd.1:Restarting daemon 2022-07-12T08:41:00.861 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-07-12T08:41:00.906 INFO:tasks.ceph.osd.1:Started 2022-07-12T08:41:00.907 INFO:tasks.ceph.osd.2:Restarting daemon 2022-07-12T08:41:00.908 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-07-12T08:41:00.944 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:00.928+0000 7fd9c1495d80 -1 Falling back to public interface 2022-07-12T08:41:00.955 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:00.948+0000 7fd9c1495d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:00.966 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:00.948+0000 7fd9c1495d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:00.966 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:00.948+0000 7fd9c1495d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:00.976 INFO:tasks.ceph.osd.2:Started 2022-07-12T08:41:00.977 INFO:tasks.ceph.osd.3:Restarting daemon 2022-07-12T08:41:01.016 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-07-12T08:41:01.018 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:00.956+0000 7fd9c1495d80 -1 osd.0 0 log_to_monitors {default=true} 2022-07-12T08:41:01.042 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:00.964+0000 7fdfa9e87d80 -1 Falling back to public interface 2022-07-12T08:41:01.056 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:00.980+0000 7fdfa9e87d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.057 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:00.980+0000 7fdfa9e87d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.057 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:00.980+0000 7fdfa9e87d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.057 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:00.984+0000 7fdfa9e87d80 -1 osd.1 0 log_to_monitors {default=true} 2022-07-12T08:41:01.083 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:01.020+0000 7f5a08f92d80 -1 Falling back to public interface 2022-07-12T08:41:01.084 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:01.044+0000 7f5a08f92d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.084 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:01.044+0000 7f5a08f92d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.085 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:01.044+0000 7f5a08f92d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.085 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:01.052+0000 7f5a08f92d80 -1 osd.2 0 log_to_monitors {default=true} 2022-07-12T08:41:01.094 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:01.095 INFO:tasks.ceph.osd.3:Started 2022-07-12T08:41:01.096 INFO:tasks.ceph.osd.4:Restarting daemon 2022-07-12T08:41:01.096 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-07-12T08:41:01.113 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:01.114 INFO:tasks.ceph.osd.4:Started 2022-07-12T08:41:01.114 INFO:tasks.ceph.osd.5:Restarting daemon 2022-07-12T08:41:01.115 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-07-12T08:41:01.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:01.121 INFO:tasks.ceph.osd.5:Started 2022-07-12T08:41:01.121 INFO:tasks.ceph.osd.6:Restarting daemon 2022-07-12T08:41:01.121 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-07-12T08:41:01.123 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:01.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:01.168 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:01.164+0000 7f7598b2fd80 -1 Falling back to public interface 2022-07-12T08:41:01.177 INFO:tasks.ceph.osd.6:Started 2022-07-12T08:41:01.178 INFO:tasks.ceph.osd.7:Restarting daemon 2022-07-12T08:41:01.179 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-07-12T08:41:01.186 INFO:tasks.ceph.osd.7:Started 2022-07-12T08:41:01.187 INFO:tasks.ceph:Waiting for OSDs to come up 2022-07-12T08:41:01.189 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:01.185+0000 7f5d495c3d80 -1 Falling back to public interface 2022-07-12T08:41:01.189 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:01.188+0000 7f7598b2fd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.190 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:01.188+0000 7f7598b2fd80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.191 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:01.188+0000 7f7598b2fd80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.214 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:01.197+0000 7f2bd4913d80 -1 Falling back to public interface 2022-07-12T08:41:01.216 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:01.200+0000 7f7598b2fd80 -1 osd.3 0 log_to_monitors {default=true} 2022-07-12T08:41:01.217 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:01.209+0000 7f5d495c3d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.223 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:01.209+0000 7f5d495c3d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.224 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:01.209+0000 7f5d495c3d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.226 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:01.213+0000 7f2bd4913d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.227 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:01.213+0000 7f2bd4913d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.227 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:01.213+0000 7f2bd4913d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.228 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:01.217+0000 7f5d495c3d80 -1 osd.4 0 log_to_monitors {default=true} 2022-07-12T08:41:01.239 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:01.217+0000 7f2bd4913d80 -1 osd.5 0 log_to_monitors {default=true} 2022-07-12T08:41:01.242 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:01.229+0000 7fdb60f53d80 -1 Falling back to public interface 2022-07-12T08:41:01.254 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:01.245+0000 7f954f28cd80 -1 Falling back to public interface 2022-07-12T08:41:01.257 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:01.249+0000 7fdb60f53d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.258 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:01.249+0000 7fdb60f53d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.267 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:01.249+0000 7fdb60f53d80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.269 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:01.253+0000 7fdb60f53d80 -1 osd.6 0 log_to_monitors {default=true} 2022-07-12T08:41:01.278 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:01.261+0000 7f954f28cd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-12T08:41:01.279 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:01.261+0000 7f954f28cd80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.279 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:01.261+0000 7f954f28cd80 -1 journal do_read_entry(8192): bad header magic 2022-07-12T08:41:01.280 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:01.269+0000 7f954f28cd80 -1 osd.7 0 log_to_monitors {default=true} 2022-07-12T08:41:02.016 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:02.125 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.120+0000 7f5be95ec700 1 Processor -- start 2022-07-12T08:41:02.127 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.120+0000 7f5be95ec700 1 -- start start 2022-07-12T08:41:02.128 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.120+0000 7f5be95ec700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be41498c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:02.128 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.120+0000 7f5be95ec700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be41426a0 con 0x7f5be4143080 2022-07-12T08:41:02.129 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.120+0000 7f5be95ec700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be4142940 con 0x7f5be4060290 2022-07-12T08:41:02.129 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.120+0000 7f5be95ec700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be413fac0 con 0x7f5be413f630 2022-07-12T08:41:02.130 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5be4143080 0x7f5be413ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40634/0 (socket says 172.21.15.61:40634) 2022-07-12T08:41:02.131 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be37fe700 1 -- 172.21.15.61:0/4087701829 learned_addr learned my addr 172.21.15.61:0/4087701829 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:02.131 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be8deb700 1 --2- 172.21.15.61:0/4087701829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be41498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:02.131 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be2ffd700 1 -- 172.21.15.61:0/4087701829 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537993199 0 0) 0x7f5be41426a0 con 0x7f5be4143080 2022-07-12T08:41:02.132 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be2ffd700 1 -- 172.21.15.61:0/4087701829 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0003260 con 0x7f5be4143080 2022-07-12T08:41:02.161 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be2ffd700 1 -- 172.21.15.61:0/4087701829 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997635589 0 0) 0x7f5bd0003260 con 0x7f5be4143080 2022-07-12T08:41:02.161 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be8deb700 1 -- 172.21.15.61:0/4087701829 >> v1:172.21.15.111:6789/0 conn(0x7f5be4060290 legacy=0x7f5be4060670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.162 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be8deb700 1 -- 172.21.15.61:0/4087701829 >> v1:172.21.15.61:6789/0 conn(0x7f5be4143080 legacy=0x7f5be413ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.162 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be8deb700 1 -- 172.21.15.61:0/4087701829 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be4142be0 con 0x7f5be413f630 2022-07-12T08:41:02.163 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be8deb700 1 --2- 172.21.15.61:0/4087701829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be41498c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5bd4001b80 tx=0x7f5bdc006df0).ready entity=mon.2 client_cookie=814c5fd875218c9b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:02.163 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be2ffd700 1 -- 172.21.15.61:0/4087701829 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bd40019e0 con 0x7f5be413f630 2022-07-12T08:41:02.164 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be2ffd700 1 -- 172.21.15.61:0/4087701829 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc000e90 con 0x7f5be413f630 2022-07-12T08:41:02.164 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be2ffd700 1 -- 172.21.15.61:0/4087701829 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bdc00d4a0 con 0x7f5be413f630 2022-07-12T08:41:02.165 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 -- 172.21.15.61:0/4087701829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 msgr2=0x7f5be41498c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.165 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 --2- 172.21.15.61:0/4087701829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be41498c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5bd4001b80 tx=0x7f5bdc006df0).stop 2022-07-12T08:41:02.166 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 -- 172.21.15.61:0/4087701829 shutdown_connections 2022-07-12T08:41:02.166 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 --2- 172.21.15.61:0/4087701829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be41498c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.167 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 -- 172.21.15.61:0/4087701829 >> 172.21.15.61:0/4087701829 conn(0x7f5be40b5700 msgr2=0x7f5be40b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:02.167 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 -- 172.21.15.61:0/4087701829 shutdown_connections 2022-07-12T08:41:02.167 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.124+0000 7f5be95ec700 1 -- 172.21.15.61:0/4087701829 wait complete. 2022-07-12T08:41:02.168 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 Processor -- start 2022-07-12T08:41:02.168 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 -- start start 2022-07-12T08:41:02.169 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 0x7f5be413c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:02.169 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be40667c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:02.170 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5be4143080 0x7f5be40642e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:02.170 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40655f0 con 0x7f5be4060290 2022-07-12T08:41:02.170 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4138320 con 0x7f5be4143080 2022-07-12T08:41:02.171 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be4138040 con 0x7f5be413f630 2022-07-12T08:41:02.171 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 0x7f5be413c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:02.172 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 0x7f5be413c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52424/0 (socket says 172.21.15.61:52424) 2022-07-12T08:41:02.172 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 -- 172.21.15.61:0/439572274 learned_addr learned my addr 172.21.15.61:0/439572274 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:02.173 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be37fe700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be40667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:02.173 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be8deb700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5be4143080 0x7f5be40642e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:02.173 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 -- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 msgr2=0x7f5be40667c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.190 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be40667c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.190 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 -- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5be4143080 msgr2=0x7f5be40642e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.191 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5be4143080 0x7f5be40642e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.191 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 -- 172.21.15.61:0/439572274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc00b180 con 0x7f5be4060290 2022-07-12T08:41:02.191 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be37fe700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be40667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:02.192 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be8deb700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5be4143080 0x7f5be40642e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:02.192 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be3fff700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 0x7f5be413c150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f5be4062700 tx=0x7f5bd8007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:02.193 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bd8012400 con 0x7f5be4060290 2022-07-12T08:41:02.193 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd8012560 con 0x7f5be4060290 2022-07-12T08:41:02.194 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be4066330 con 0x7f5be4060290 2022-07-12T08:41:02.194 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bd80128c0 con 0x7f5be4060290 2022-07-12T08:41:02.194 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.128+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be41390c0 con 0x7f5be4060290 2022-07-12T08:41:02.207 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.132+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be4066e50 con 0x7f5be4060290 2022-07-12T08:41:02.207 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.132+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5bd8019030 con 0x7f5be4060290 2022-07-12T08:41:02.208 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.132+0000 7f5be17fa700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5bc40539b0 0x7f5bc4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:02.208 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.132+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2312+0+0 (secure 0 0 0) 0x7f5bd806a2d0 con 0x7f5be4060290 2022-07-12T08:41:02.209 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.132+0000 7f5be37fe700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5bc40539b0 0x7f5bc4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:02.209 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.132+0000 7f5be37fe700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5bc40539b0 0x7f5bc4055e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f5bd400d000 tx=0x7f5bd4000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:02.217 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.140+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5bd8042080 con 0x7f5be4060290 2022-07-12T08:41:02.270 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.268+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5be4057460 con 0x7f5be4060290 2022-07-12T08:41:02.270 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.268+0000 7f5be17fa700 1 -- 172.21.15.61:0/439572274 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8274 (secure 0 0 0) 0x7f5bd80456a0 con 0x7f5be4060290 2022-07-12T08:41:02.271 INFO:teuthology.misc.health.smithi061.stdout: 2022-07-12T08:41:02.272 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":11,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:01.790052+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"5","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:02.288 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5bc40539b0 msgr2=0x7f5bc4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.288 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5bc40539b0 0x7f5bc4055e70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f5bd400d000 tx=0x7f5bd4000f40).stop 2022-07-12T08:41:02.289 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 msgr2=0x7f5be413c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:02.289 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 0x7f5be413c150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f5be4062700 tx=0x7f5bd8007c50).stop 2022-07-12T08:41:02.289 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 shutdown_connections 2022-07-12T08:41:02.290 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5bc40539b0 0x7f5bc4055e70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.290 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5be4143080 0x7f5be40642e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.291 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5be413f630 0x7f5be40667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.291 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 --2- 172.21.15.61:0/439572274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5be4060290 0x7f5be413c150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:02.291 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 >> 172.21.15.61:0/439572274 conn(0x7f5be40b5700 msgr2=0x7f5be40b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:02.292 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 shutdown_connections 2022-07-12T08:41:02.292 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:02.272+0000 7f5be95ec700 1 -- 172.21.15.61:0/439572274 wait complete. 2022-07-12T08:41:02.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-07-12T08:41:02.803 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:02.801+0000 7f9539351700 -1 osd.7 0 waiting for initial osdmap 2022-07-12T08:41:02.805 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:02.804+0000 7f59f3057700 -1 osd.2 0 waiting for initial osdmap 2022-07-12T08:41:02.805 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:02.801+0000 7f2bbe9d8700 -1 osd.5 0 waiting for initial osdmap 2022-07-12T08:41:02.806 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:02.808+0000 7fdf93f4c700 -1 osd.1 0 waiting for initial osdmap 2022-07-12T08:41:02.807 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:02.801+0000 7f5d33688700 -1 osd.4 0 waiting for initial osdmap 2022-07-12T08:41:02.807 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:02.808+0000 7fd9ab55a700 -1 osd.0 0 waiting for initial osdmap 2022-07-12T08:41:02.810 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:02.808+0000 7f7582bf4700 -1 osd.3 0 waiting for initial osdmap 2022-07-12T08:41:02.811 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-12T08:41:02.812+0000 7fdfa33cd700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.812 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-12T08:41:02.808+0000 7f5a024d8700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.812 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-12T08:41:02.812+0000 7fd9ba9db700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.813 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:02.805+0000 7fdb4b018700 -1 osd.6 0 waiting for initial osdmap 2022-07-12T08:41:02.814 INFO:tasks.ceph.osd.4.smithi111.stderr:2022-07-12T08:41:02.805+0000 7f5d42b09700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.815 INFO:tasks.ceph.osd.6.smithi111.stderr:2022-07-12T08:41:02.809+0000 7fdb5a499700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.818 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-12T08:41:02.812+0000 7f7592075700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.827 INFO:tasks.ceph.osd.5.smithi111.stderr:2022-07-12T08:41:02.825+0000 7f2bcde59700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:02.842 INFO:tasks.ceph.osd.7.smithi111.stderr:2022-07-12T08:41:02.829+0000 7f95487d2700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-12T08:41:09.138 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:09.437 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 Processor -- start 2022-07-12T08:41:09.437 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 -- start start 2022-07-12T08:41:09.449 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f49c40b61d0 0x7f49c40af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.449 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49c40afbc0 0x7f49c4142f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.449 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40ada80 0x7f49c4143400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.450 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c4062c90 con 0x7f49c40b61d0 2022-07-12T08:41:09.450 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c40ac9d0 con 0x7f49c40afbc0 2022-07-12T08:41:09.451 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.396+0000 7f49c3fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c40ac500 con 0x7f49c40ada80 2022-07-12T08:41:09.451 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f49c40b61d0 0x7f49c40af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.452 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f49c40b61d0 0x7f49c40af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52620/0 (socket says 172.21.15.61:52620) 2022-07-12T08:41:09.452 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 -- 172.21.15.61:0/2999323243 learned_addr learned my addr 172.21.15.61:0/2999323243 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:09.453 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c37fe700 1 --2- 172.21.15.61:0/2999323243 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40ada80 0x7f49c4143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.462 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c27fc700 1 --2- 172.21.15.61:0/2999323243 >> v2:172.21.15.111:3300/0 conn(0x7f49c40afbc0 0x7f49c4142f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.462 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 -- 172.21.15.61:0/2999323243 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40ada80 msgr2=0x7f49c4143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:09.463 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 --2- 172.21.15.61:0/2999323243 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40ada80 0x7f49c4143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.463 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 -- 172.21.15.61:0/2999323243 >> v2:172.21.15.111:3300/0 conn(0x7f49c40afbc0 msgr2=0x7f49c4142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:09.464 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 --2- 172.21.15.61:0/2999323243 >> v2:172.21.15.111:3300/0 conn(0x7f49c40afbc0 0x7f49c4142f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.464 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 -- 172.21.15.61:0/2999323243 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c40b65b0 con 0x7f49c40b61d0 2022-07-12T08:41:09.464 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c27fc700 1 --2- 172.21.15.61:0/2999323243 >> v2:172.21.15.111:3300/0 conn(0x7f49c40afbc0 0x7f49c4142f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:09.465 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c2ffd700 1 --2- 172.21.15.61:0/2999323243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40b61d0 0x7f49c40af6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f49c4062dd0 tx=0x7f49b4006eb0).ready entity=mon.0 client_cookie=4f6f84a3871d59d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:09.465 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c1ffb700 1 -- 172.21.15.61:0/2999323243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b400d570 con 0x7f49c40b61d0 2022-07-12T08:41:09.467 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c1ffb700 1 -- 172.21.15.61:0/2999323243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b4000e90 con 0x7f49c40b61d0 2022-07-12T08:41:09.471 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c1ffb700 1 -- 172.21.15.61:0/2999323243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b400d900 con 0x7f49c40b61d0 2022-07-12T08:41:09.472 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 -- 172.21.15.61:0/2999323243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40b61d0 msgr2=0x7f49c40af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.472 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2999323243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40b61d0 0x7f49c40af6d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f49c4062dd0 tx=0x7f49b4006eb0).stop 2022-07-12T08:41:09.472 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 -- 172.21.15.61:0/2999323243 shutdown_connections 2022-07-12T08:41:09.473 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2999323243 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40ada80 0x7f49c4143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.473 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2999323243 >> v2:172.21.15.111:3300/0 conn(0x7f49c40afbc0 0x7f49c4142f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.474 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2999323243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40b61d0 0x7f49c40af6d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.475 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 -- 172.21.15.61:0/2999323243 >> 172.21.15.61:0/2999323243 conn(0x7f49c40639f0 msgr2=0x7f49c40b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:09.476 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 -- 172.21.15.61:0/2999323243 shutdown_connections 2022-07-12T08:41:09.476 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 -- 172.21.15.61:0/2999323243 wait complete. 2022-07-12T08:41:09.476 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 Processor -- start 2022-07-12T08:41:09.477 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.400+0000 7f49c3fff700 1 -- start start 2022-07-12T08:41:09.478 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c40ada80 0x7f49c4149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.478 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 0x7f49c4155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.478 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40b61d0 0x7f49c4067300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.479 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c40677f0 con 0x7f49c40afbc0 2022-07-12T08:41:09.479 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c4067a10 con 0x7f49c40ada80 2022-07-12T08:41:09.479 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c4067cb0 con 0x7f49c40b61d0 2022-07-12T08:41:09.479 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 0x7f49c4155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.480 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 0x7f49c4155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52628/0 (socket says 172.21.15.61:52628) 2022-07-12T08:41:09.480 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 -- 172.21.15.61:0/2276304983 learned_addr learned my addr 172.21.15.61:0/2276304983 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:09.480 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c2ffd700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c40ada80 0x7f49c4149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.481 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c37fe700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40b61d0 0x7f49c4067300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.481 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 -- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40b61d0 msgr2=0x7f49c4067300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.481 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40b61d0 0x7f49c4067300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.481 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 -- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c40ada80 msgr2=0x7f49c4149ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.482 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c40ada80 0x7f49c4149ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.482 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 -- 172.21.15.61:0/2276304983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b400b1e0 con 0x7f49c40afbc0 2022-07-12T08:41:09.482 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c37fe700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40b61d0 0x7f49c4067300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:09.483 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c27fc700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 0x7f49c4155160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f49b800a000 tx=0x7f49b8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:09.483 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b8011780 con 0x7f49c40afbc0 2022-07-12T08:41:09.484 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b80118e0 con 0x7f49c40afbc0 2022-07-12T08:41:09.484 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c4146030 con 0x7f49c40afbc0 2022-07-12T08:41:09.485 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49b8011c40 con 0x7f49c40afbc0 2022-07-12T08:41:09.485 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c4144630 con 0x7f49c40afbc0 2022-07-12T08:41:09.486 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.404+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c4057460 con 0x7f49c40afbc0 2022-07-12T08:41:09.486 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.408+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f49b801c460 con 0x7f49c40afbc0 2022-07-12T08:41:09.487 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.408+0000 7f49b3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49a0053af0 0x7f49a0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.487 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.408+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f49b806bd20 con 0x7f49c40afbc0 2022-07-12T08:41:09.488 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.408+0000 7f49c2ffd700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49a0053af0 0x7f49a0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.488 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.408+0000 7f49c2ffd700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49a0053af0 0x7f49a0055fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f49b401b000 tx=0x7f49b4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:09.488 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.412+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f49b80430b0 con 0x7f49c40afbc0 2022-07-12T08:41:09.548 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.536+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49c405fcb0 con 0x7f49c40afbc0 2022-07-12T08:41:09.572 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49b3fff700 1 -- 172.21.15.61:0/2276304983 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11853 (secure 0 0 0) 0x7f49b80466d0 con 0x7f49c40afbc0 2022-07-12T08:41:09.576 INFO:teuthology.misc.health.smithi061.stdout: 2022-07-12T08:41:09.577 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":15,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:05.820174+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:09.579 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49a0053af0 msgr2=0x7f49a0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.580 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49a0053af0 0x7f49a0055fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f49b401b000 tx=0x7f49b4015000).stop 2022-07-12T08:41:09.580 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 msgr2=0x7f49c4155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.581 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 0x7f49c4155160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f49b800a000 tx=0x7f49b8008040).stop 2022-07-12T08:41:09.581 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 shutdown_connections 2022-07-12T08:41:09.582 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f49c40b61d0 0x7f49c4067300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.582 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49c40afbc0 0x7f49c4155160 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.582 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49a0053af0 0x7f49a0055fb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.583 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 --2- 172.21.15.61:0/2276304983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49c40ada80 0x7f49c4149ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.583 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 >> 172.21.15.61:0/2276304983 conn(0x7f49c40639f0 msgr2=0x7f49c413eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:09.584 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 shutdown_connections 2022-07-12T08:41:09.584 INFO:teuthology.misc.health.smithi061.stderr:2022-07-12T08:41:09.540+0000 7f49c3fff700 1 -- 172.21.15.61:0/2276304983 wait complete. 2022-07-12T08:41:09.585 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-07-12T08:41:09.586 INFO:tasks.ceph:Creating RBD pool 2022-07-12T08:41:09.587 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd pool create rbd 8 2022-07-12T08:41:09.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 Processor -- start 2022-07-12T08:41:09.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 -- start start 2022-07-12T08:41:09.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f68e80b0120 0x7f68e8144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68e814aad0 0x7f68e814af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e80b0500 con 0x7f68e80ab730 2022-07-12T08:41:09.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e80b1040 con 0x7f68e814aad0 2022-07-12T08:41:09.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68f0548700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e8144d50 con 0x7f68e80b0120 2022-07-12T08:41:09.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68edae3700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f68e80ab730 0x7f68e8144640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40854/0 (socket says 172.21.15.61:40854) 2022-07-12T08:41:09.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68edae3700 1 -- 172.21.15.61:0/1788335917 learned_addr learned my addr 172.21.15.61:0/1788335917 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:09.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68eeae5700 1 --2- 172.21.15.61:0/1788335917 >> v2:172.21.15.61:3300/0 conn(0x7f68e814aad0 0x7f68e814af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68ee2e4700 1 --2- 172.21.15.61:0/1788335917 >> v2:172.21.15.111:3300/0 conn(0x7f68e80b0120 0x7f68e8144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68ed2e2700 1 -- 172.21.15.61:0/1788335917 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245749296 0 0) 0x7f68e80b0500 con 0x7f68e80ab730 2022-07-12T08:41:09.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68ed2e2700 1 -- 172.21.15.61:0/1788335917 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d0003260 con 0x7f68e80ab730 2022-07-12T08:41:09.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68eeae5700 1 -- 172.21.15.61:0/1788335917 >> v2:172.21.15.111:3300/0 conn(0x7f68e80b0120 msgr2=0x7f68e8144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:09.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68eeae5700 1 --2- 172.21.15.61:0/1788335917 >> v2:172.21.15.111:3300/0 conn(0x7f68e80b0120 0x7f68e8144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68eeae5700 1 -- 172.21.15.61:0/1788335917 >> v1:172.21.15.61:6789/0 conn(0x7f68e80ab730 legacy=0x7f68e8144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68eeae5700 1 -- 172.21.15.61:0/1788335917 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e805d0b0 con 0x7f68e814aad0 2022-07-12T08:41:09.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68eeae5700 1 --2- 172.21.15.61:0/1788335917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e814aad0 0x7f68e814af10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f68e00032f0 tx=0x7f68e00091d0).ready entity=mon.0 client_cookie=aa29e8fe225a0f28 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:09.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68ed2e2700 1 -- 172.21.15.61:0/1788335917 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68e000b600 con 0x7f68e814aad0 2022-07-12T08:41:09.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.776+0000 7f68ed2e2700 1 -- 172.21.15.61:0/1788335917 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e000b760 con 0x7f68e814aad0 2022-07-12T08:41:09.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68ed2e2700 1 -- 172.21.15.61:0/1788335917 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68e000b8d0 con 0x7f68e814aad0 2022-07-12T08:41:09.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 -- 172.21.15.61:0/1788335917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e814aad0 msgr2=0x7f68e814af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 --2- 172.21.15.61:0/1788335917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e814aad0 0x7f68e814af10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f68e00032f0 tx=0x7f68e00091d0).stop 2022-07-12T08:41:09.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 -- 172.21.15.61:0/1788335917 shutdown_connections 2022-07-12T08:41:09.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 --2- 172.21.15.61:0/1788335917 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e814aad0 0x7f68e814af10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 --2- 172.21.15.61:0/1788335917 >> v2:172.21.15.111:3300/0 conn(0x7f68e80b0120 0x7f68e8144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 -- 172.21.15.61:0/1788335917 >> 172.21.15.61:0/1788335917 conn(0x7f68e80b5290 msgr2=0x7f68e80b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:09.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 -- 172.21.15.61:0/1788335917 shutdown_connections 2022-07-12T08:41:09.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 -- 172.21.15.61:0/1788335917 wait complete. 2022-07-12T08:41:09.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 Processor -- start 2022-07-12T08:41:09.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.780+0000 7f68f0548700 1 -- start start 2022-07-12T08:41:09.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f68e80ab730 0x7f68e809f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 0x7f68e80ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e814aad0 0x7f68e80acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e80af400 con 0x7f68e80b0120 2022-07-12T08:41:09.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e809e1a0 con 0x7f68e814aad0 2022-07-12T08:41:09.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e809e440 con 0x7f68e80ab730 2022-07-12T08:41:09.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 0x7f68e80ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 0x7f68e80ac6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52642/0 (socket says 172.21.15.61:52642) 2022-07-12T08:41:09.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 -- 172.21.15.61:0/2676891103 learned_addr learned my addr 172.21.15.61:0/2676891103 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:09.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68ee2e4700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f68e80ab730 0x7f68e809f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68eeae5700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e814aad0 0x7f68e80acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 -- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f68e80ab730 msgr2=0x7f68e809f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f68e80ab730 0x7f68e809f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 -- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e814aad0 msgr2=0x7f68e80acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:09.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e814aad0 0x7f68e80acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:09.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 -- 172.21.15.61:0/2676891103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0009470 con 0x7f68e80b0120 2022-07-12T08:41:09.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68edae3700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 0x7f68e80ac6d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f68e4001110 tx=0x7f68e4007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:09.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68e40072e0 con 0x7f68e80b0120 2022-07-12T08:41:09.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e4007440 con 0x7f68e80b0120 2022-07-12T08:41:09.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e80aef60 con 0x7f68e80b0120 2022-07-12T08:41:09.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68e4005800 con 0x7f68e80b0120 2022-07-12T08:41:09.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e80a1cc0 con 0x7f68e80b0120 2022-07-12T08:41:09.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.784+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e809ee90 con 0x7f68e80b0120 2022-07-12T08:41:09.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.788+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f68e4010070 con 0x7f68e80b0120 2022-07-12T08:41:09.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.788+0000 7f68d77fe700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68c8053a40 0x7f68c8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:09.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.788+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f68e406b0f0 con 0x7f68e80b0120 2022-07-12T08:41:09.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.788+0000 7f68ee2e4700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68c8053a40 0x7f68c8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:09.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.788+0000 7f68ee2e4700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68c8053a40 0x7f68c8055f00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f68d800b000 tx=0x7f68d8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:09.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.796+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68e809ee90 con 0x7f68e80b0120 2022-07-12T08:41:09.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:09.920+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f68e809ee90 con 0x7f68e80b0120 2022-07-12T08:41:10.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.048+0000 7f68d77fe700 1 -- 172.21.15.61:0/2676891103 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7f68e809ee90 con 0x7f68e80b0120 2022-07-12T08:41:10.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68c8053a40 msgr2=0x7f68c8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68c8053a40 0x7f68c8055f00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f68d800b000 tx=0x7f68d8008040).stop 2022-07-12T08:41:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 msgr2=0x7f68e80ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 0x7f68e80ac6d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f68e4001110 tx=0x7f68e4007ad0).stop 2022-07-12T08:41:10.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 shutdown_connections 2022-07-12T08:41:10.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68c8053a40 0x7f68c8055f00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68e814aad0 0x7f68e80acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68e80b0120 0x7f68e80ac6d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 --2- 172.21.15.61:0/2676891103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f68e80ab730 0x7f68e809f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 >> 172.21.15.61:0/2676891103 conn(0x7f68e80b5290 msgr2=0x7f68e8142f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:10.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 shutdown_connections 2022-07-12T08:41:10.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.056+0000 7f68f0548700 1 -- 172.21.15.61:0/2676891103 wait complete. 2022-07-12T08:41:10.088 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'rbd' created 2022-07-12T08:41:10.089 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-07-12T08:41:10.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb30c90c700 1 Processor -- start 2022-07-12T08:41:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb30c90c700 1 -- start start 2022-07-12T08:41:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb30c90c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb30805d060 0x7fb308145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb30c90c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3080afca0 con 0x7fb308067860 2022-07-12T08:41:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb30c90c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30805d440 con 0x7fb3080b3c60 2022-07-12T08:41:10.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb30c90c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3080aff00 con 0x7fb30805d060 2022-07-12T08:41:10.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb307fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb308067860 0x7fb308146760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40868/0 (socket says 172.21.15.61:40868) 2022-07-12T08:41:10.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb307fff700 1 -- 172.21.15.61:0/991571318 learned_addr learned my addr 172.21.15.61:0/991571318 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:10.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.212+0000 7fb3077fe700 1 --2- 172.21.15.61:0/991571318 >> v2:172.21.15.111:3300/0 conn(0x7fb30805d060 0x7fb308145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:10.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3067fc700 1 -- 172.21.15.61:0/991571318 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518210189 0 0) 0x7fb3080afca0 con 0x7fb308067860 2022-07-12T08:41:10.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3067fc700 1 -- 172.21.15.61:0/991571318 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e8003260 con 0x7fb308067860 2022-07-12T08:41:10.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3077fe700 1 -- 172.21.15.61:0/991571318 >> v1:172.21.15.111:6789/0 conn(0x7fb3080b3c60 legacy=0x7fb308145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3077fe700 1 -- 172.21.15.61:0/991571318 >> v1:172.21.15.61:6789/0 conn(0x7fb308067860 legacy=0x7fb308146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3077fe700 1 -- 172.21.15.61:0/991571318 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080b4200 con 0x7fb30805d060 2022-07-12T08:41:10.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3077fe700 1 --2- 172.21.15.61:0/991571318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb30805d060 0x7fb308145680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb2f80032f0 tx=0x7fb2f80091d0).ready entity=mon.1 client_cookie=85d2ab8bfcb08f48 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:10.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3067fc700 1 -- 172.21.15.61:0/991571318 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f800b6c0 con 0x7fb30805d060 2022-07-12T08:41:10.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3067fc700 1 -- 172.21.15.61:0/991571318 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f800b820 con 0x7fb30805d060 2022-07-12T08:41:10.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb3067fc700 1 -- 172.21.15.61:0/991571318 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f800b990 con 0x7fb30805d060 2022-07-12T08:41:10.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 -- 172.21.15.61:0/991571318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb30805d060 msgr2=0x7fb308145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 --2- 172.21.15.61:0/991571318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb30805d060 0x7fb308145680 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb2f80032f0 tx=0x7fb2f80091d0).stop 2022-07-12T08:41:10.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 -- 172.21.15.61:0/991571318 shutdown_connections 2022-07-12T08:41:10.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 --2- 172.21.15.61:0/991571318 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb30805d060 0x7fb308145680 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 -- 172.21.15.61:0/991571318 >> 172.21.15.61:0/991571318 conn(0x7fb3080b67c0 msgr2=0x7fb308066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:10.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 -- 172.21.15.61:0/991571318 shutdown_connections 2022-07-12T08:41:10.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.216+0000 7fb30c90c700 1 -- 172.21.15.61:0/991571318 wait complete. 2022-07-12T08:41:10.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 Processor -- start 2022-07-12T08:41:10.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 -- start start 2022-07-12T08:41:10.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 0x7fb308146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb308067860 0x7fb30809ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3080b3c60 0x7fb308143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb308147950 con 0x7fb30805d060 2022-07-12T08:41:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080b29a0 con 0x7fb308067860 2022-07-12T08:41:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080b1940 con 0x7fb3080b3c60 2022-07-12T08:41:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 0x7fb308146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 0x7fb308146a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52654/0 (socket says 172.21.15.61:52654) 2022-07-12T08:41:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 -- 172.21.15.61:0/3137770894 learned_addr learned my addr 172.21.15.61:0/3137770894 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb306ffd700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb308067860 0x7fb30809ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb307fff700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3080b3c60 0x7fb308143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 -- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3080b3c60 msgr2=0x7fb308143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3080b3c60 0x7fb308143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 -- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb308067860 msgr2=0x7fb30809ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:10.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb308067860 0x7fb30809ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:10.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 -- 172.21.15.61:0/3137770894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8009470 con 0x7fb30805d060 2022-07-12T08:41:10.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb307fff700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3080b3c60 0x7fb308143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:10.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb3077fe700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 0x7fb308146a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb2f80032f0 tx=0x7fb2f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f800b6c0 con 0x7fb30805d060 2022-07-12T08:41:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8021670 con 0x7fb30805d060 2022-07-12T08:41:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3080b2490 con 0x7fb30805d060 2022-07-12T08:41:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f80219d0 con 0x7fb30805d060 2022-07-12T08:41:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.220+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3080b3430 con 0x7fb30805d060 2022-07-12T08:41:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.224+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb308147ac0 con 0x7fb30805d060 2022-07-12T08:41:10.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.224+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb2f8028070 con 0x7fb30805d060 2022-07-12T08:41:10.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.224+0000 7fb304ff9700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ec053940 0x7fb2ec055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.224+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fb2f806ce20 con 0x7fb30805d060 2022-07-12T08:41:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.224+0000 7fb306ffd700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ec053940 0x7fb2ec055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.224+0000 7fb306ffd700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ec053940 0x7fb2ec055e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb2f000d000 tx=0x7fb2f0000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.228+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb2f8043fb0 con 0x7fb30805d060 2022-07-12T08:41:10.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:10.356+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb308057460 con 0x7fb30805d060 2022-07-12T08:41:11.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.060+0000 7fb304ff9700 1 -- 172.21.15.61:0/3137770894 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7fb2f8015070 con 0x7fb30805d060 2022-07-12T08:41:11.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ec053940 msgr2=0x7fb2ec055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ec053940 0x7fb2ec055e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fb2f000d000 tx=0x7fb2f0000bc0).stop 2022-07-12T08:41:11.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 msgr2=0x7fb308146a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 0x7fb308146a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fb2f80032f0 tx=0x7fb2f8016000).stop 2022-07-12T08:41:11.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 shutdown_connections 2022-07-12T08:41:11.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3080b3c60 0x7fb308143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ec053940 0x7fb2ec055e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb308067860 0x7fb30809ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 --2- 172.21.15.61:0/3137770894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb30805d060 0x7fb308146a00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 >> 172.21.15.61:0/3137770894 conn(0x7fb3080b67c0 msgr2=0x7fb3080af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:11.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 shutdown_connections 2022-07-12T08:41:11.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.064+0000 7fb30c90c700 1 -- 172.21.15.61:0/3137770894 wait complete. 2022-07-12T08:41:11.093 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rbd' on pool 'rbd' 2022-07-12T08:41:11.094 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-07-12T08:41:11.095 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:41:11.096 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:41:11.131 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:41:11.132 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:41:11.132 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:41:11.170 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:41:11.187 INFO:tasks.ceph:Setting up CephFS filesystem... 2022-07-12T08:41:11.187 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:11.204 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:41:11.242 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:41:11.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 Processor -- start 2022-07-12T08:41:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 -- start start 2022-07-12T08:41:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48180b6ac0 0x7f48180b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48180b11c0 0x7f481814a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481805f550 con 0x7f4818064e40 2022-07-12T08:41:11.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48180b1740 con 0x7f48180b6ac0 2022-07-12T08:41:11.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481ee47700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180b6ea0 con 0x7f48180b11c0 2022-07-12T08:41:11.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481de45700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48180b6ac0 0x7f48180b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481de45700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48180b6ac0 0x7f48180b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52676/0 (socket says 172.21.15.61:52676) 2022-07-12T08:41:11.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481de45700 1 -- 172.21.15.61:0/2271236726 learned_addr learned my addr 172.21.15.61:0/2271236726 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:11.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481d644700 1 --2- 172.21.15.61:0/2271236726 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48180b11c0 0x7f481814a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.308+0000 7f481de45700 1 -- 172.21.15.61:0/2271236726 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48180b11c0 msgr2=0x7f481814a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:11.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481de45700 1 --2- 172.21.15.61:0/2271236726 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48180b11c0 0x7f481814a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481de45700 1 -- 172.21.15.61:0/2271236726 >> v1:172.21.15.61:6789/0 conn(0x7f4818064e40 legacy=0x7f48180652f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481de45700 1 -- 172.21.15.61:0/2271236726 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481805f7b0 con 0x7f48180b6ac0 2022-07-12T08:41:11.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481de45700 1 --2- 172.21.15.61:0/2271236726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f48180b0cb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f48100032f0 tx=0x7f4810009290).ready entity=mon.0 client_cookie=3187d987d5962f6b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ce43700 1 -- 172.21.15.61:0/2271236726 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481000b8a0 con 0x7f48180b6ac0 2022-07-12T08:41:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ce43700 1 -- 172.21.15.61:0/2271236726 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481000ba00 con 0x7f48180b6ac0 2022-07-12T08:41:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ce43700 1 -- 172.21.15.61:0/2271236726 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481000bb70 con 0x7f48180b6ac0 2022-07-12T08:41:11.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 -- 172.21.15.61:0/2271236726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 msgr2=0x7f48180b0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 --2- 172.21.15.61:0/2271236726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f48180b0cb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f48100032f0 tx=0x7f4810009290).stop 2022-07-12T08:41:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 -- 172.21.15.61:0/2271236726 shutdown_connections 2022-07-12T08:41:11.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 --2- 172.21.15.61:0/2271236726 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48180b11c0 0x7f481814a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 --2- 172.21.15.61:0/2271236726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f48180b0cb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 -- 172.21.15.61:0/2271236726 >> 172.21.15.61:0/2271236726 conn(0x7f4818064570 msgr2=0x7f48180b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:11.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 -- 172.21.15.61:0/2271236726 shutdown_connections 2022-07-12T08:41:11.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 -- 172.21.15.61:0/2271236726 wait complete. 2022-07-12T08:41:11.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 Processor -- start 2022-07-12T08:41:11.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.312+0000 7f481ee47700 1 -- start start 2022-07-12T08:41:11.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4818064e40 0x7f48180b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48180b11c0 0x7f4818159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f481814c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48180ad4f0 con 0x7f48180b6ac0 2022-07-12T08:41:11.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481815a290 con 0x7f48180b11c0 2022-07-12T08:41:11.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48180ad750 con 0x7f4818064e40 2022-07-12T08:41:11.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f481814c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f481814c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52686/0 (socket says 172.21.15.61:52686) 2022-07-12T08:41:11.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 -- 172.21.15.61:0/1185299077 learned_addr learned my addr 172.21.15.61:0/1185299077 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:11.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481de45700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4818064e40 0x7f48180b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481d644700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48180b11c0 0x7f4818159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 -- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4818064e40 msgr2=0x7f48180b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4818064e40 0x7f48180b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 -- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48180b11c0 msgr2=0x7f4818159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48180b11c0 0x7f4818159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 -- 172.21.15.61:0/1185299077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4810009530 con 0x7f48180b6ac0 2022-07-12T08:41:11.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481e646700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f481814c6f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4818064d20 tx=0x7f4814007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4814013450 con 0x7f48180b6ac0 2022-07-12T08:41:11.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48140135b0 con 0x7f48180b6ac0 2022-07-12T08:41:11.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48180ad9f0 con 0x7f48180b6ac0 2022-07-12T08:41:11.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4814013910 con 0x7f48180b6ac0 2022-07-12T08:41:11.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48180ae4f0 con 0x7f48180b6ac0 2022-07-12T08:41:11.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4818057460 con 0x7f48180b6ac0 2022-07-12T08:41:11.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4814010070 con 0x7f48180b6ac0 2022-07-12T08:41:11.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.316+0000 7f480effd700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4800053ad0 0x7f4800055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.320+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f481406b1e0 con 0x7f48180b6ac0 2022-07-12T08:41:11.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.320+0000 7f481de45700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4800053ad0 0x7f4800055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.320+0000 7f481de45700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4800053ad0 0x7f4800055f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f481001b000 tx=0x7f4810016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.324+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4814042470 con 0x7f48180b6ac0 2022-07-12T08:41:11.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.448+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4818068350 con 0x7f48180b6ac0 2022-07-12T08:41:11.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.448+0000 7f480effd700 1 -- 172.21.15.61:0/1185299077 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f4814018020 con 0x7f48180b6ac0 2022-07-12T08:41:11.458 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:11.459 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:11.048257+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:11.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4800053ad0 msgr2=0x7f4800055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4800053ad0 0x7f4800055f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f481001b000 tx=0x7f4810016000).stop 2022-07-12T08:41:11.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 msgr2=0x7f481814c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f481814c6f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f4818064d20 tx=0x7f4814007bc0).stop 2022-07-12T08:41:11.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 shutdown_connections 2022-07-12T08:41:11.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48180b6ac0 0x7f481814c6f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4800053ad0 0x7f4800055f90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48180b11c0 0x7f4818159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 --2- 172.21.15.61:0/1185299077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4818064e40 0x7f48180b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 >> 172.21.15.61:0/1185299077 conn(0x7f4818064570 msgr2=0x7f481805fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 shutdown_connections 2022-07-12T08:41:11.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.452+0000 7f481ee47700 1 -- 172.21.15.61:0/1185299077 wait complete. 2022-07-12T08:41:11.491 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:41:11.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 Processor -- start 2022-07-12T08:41:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 -- start start 2022-07-12T08:41:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d34060a70 0x7f2d3409a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2d3409ae20 0x7f2d34142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340ab010 con 0x7f2d340abe90 2022-07-12T08:41:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d34097aa0 con 0x7f2d3409ae20 2022-07-12T08:41:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d38b00700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34143ea0 con 0x7f2d34060a70 2022-07-12T08:41:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d337fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d34060a70 0x7f2d3409a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d337fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d34060a70 0x7f2d3409a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:34886/0 (socket says 172.21.15.61:34886) 2022-07-12T08:41:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d337fe700 1 -- 172.21.15.61:0/967959767 learned_addr learned my addr 172.21.15.61:0/967959767 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d32ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2d3409ae20 0x7f2d34142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d32ffd700 1 -- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d34060a70 msgr2=0x7f2d3409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d32ffd700 1 --2- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d34060a70 0x7f2d3409a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d32ffd700 1 -- 172.21.15.61:0/967959767 >> v1:172.21.15.111:6789/0 conn(0x7f2d340abe90 legacy=0x7f2d341431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d32ffd700 1 -- 172.21.15.61:0/967959767 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d341438d0 con 0x7f2d3409ae20 2022-07-12T08:41:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d32ffd700 1 --2- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d34142cd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f2d28002f70 tx=0x7f2d28006df0).ready entity=mon.1 client_cookie=4cae2075c4f2d7b7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d327fc700 1 -- 172.21.15.61:0/967959767 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d2800d570 con 0x7f2d3409ae20 2022-07-12T08:41:11.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d327fc700 1 -- 172.21.15.61:0/967959767 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d28000e90 con 0x7f2d3409ae20 2022-07-12T08:41:11.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.612+0000 7f2d327fc700 1 -- 172.21.15.61:0/967959767 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d2800d8c0 con 0x7f2d3409ae20 2022-07-12T08:41:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 msgr2=0x7f2d34142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 --2- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d34142cd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f2d28002f70 tx=0x7f2d28006df0).stop 2022-07-12T08:41:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- 172.21.15.61:0/967959767 shutdown_connections 2022-07-12T08:41:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 --2- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d34142cd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 --2- 172.21.15.61:0/967959767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d34060a70 0x7f2d3409a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- 172.21.15.61:0/967959767 >> 172.21.15.61:0/967959767 conn(0x7f2d3405bd10 msgr2=0x7f2d3405c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:11.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- 172.21.15.61:0/967959767 shutdown_connections 2022-07-12T08:41:11.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- 172.21.15.61:0/967959767 wait complete. 2022-07-12T08:41:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 Processor -- start 2022-07-12T08:41:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- start start 2022-07-12T08:41:11.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 0x7f2d340af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d341488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d340abe90 0x7f2d341527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34145520 con 0x7f2d34060a70 2022-07-12T08:41:11.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340afcf0 con 0x7f2d3409ae20 2022-07-12T08:41:11.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d38b00700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34152080 con 0x7f2d340abe90 2022-07-12T08:41:11.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.616+0000 7f2d337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 0x7f2d340af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 0x7f2d340af800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52696/0 (socket says 172.21.15.61:52696) 2022-07-12T08:41:11.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 -- 172.21.15.61:0/2372225648 learned_addr learned my addr 172.21.15.61:0/2372225648 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:11.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d33fff700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d340abe90 0x7f2d341527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d32ffd700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d341488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 -- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d340abe90 msgr2=0x7f2d341527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d340abe90 0x7f2d341527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 -- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 msgr2=0x7f2d341488b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d341488b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 -- 172.21.15.61:0/2372225648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2800b180 con 0x7f2d34060a70 2022-07-12T08:41:11.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d33fff700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d340abe90 0x7f2d341527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:11.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d337fe700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 0x7f2d340af800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2d2400c000 tx=0x7f2d24008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d24013610 con 0x7f2d34060a70 2022-07-12T08:41:11.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24013770 con 0x7f2d34060a70 2022-07-12T08:41:11.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d34152320 con 0x7f2d34060a70 2022-07-12T08:41:11.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d24013ad0 con 0x7f2d34060a70 2022-07-12T08:41:11.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d340ac510 con 0x7f2d34060a70 2022-07-12T08:41:11.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d34151ed0 con 0x7f2d34060a70 2022-07-12T08:41:11.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2d24023070 con 0x7f2d34060a70 2022-07-12T08:41:11.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d30ff9700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2d14053a40 0x7f2d14055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.620+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f2d2406be60 con 0x7f2d34060a70 2022-07-12T08:41:11.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.624+0000 7f2d32ffd700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2d14053a40 0x7f2d14055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.624+0000 7f2d32ffd700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2d14053a40 0x7f2d14055f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2d2801a000 tx=0x7f2d28014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.628+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d24042fe0 con 0x7f2d34060a70 2022-07-12T08:41:11.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.752+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2d34057460 con 0x7f2d34060a70 2022-07-12T08:41:11.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.752+0000 7f2d30ff9700 1 -- 172.21.15.61:0/2372225648 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2d24046600 con 0x7f2d34060a70 2022-07-12T08:41:11.794 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2d14053a40 msgr2=0x7f2d14055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2d14053a40 0x7f2d14055f00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f2d2801a000 tx=0x7f2d28014000).stop 2022-07-12T08:41:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 msgr2=0x7f2d340af800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 0x7f2d340af800 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2d2400c000 tx=0x7f2d24008040).stop 2022-07-12T08:41:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 shutdown_connections 2022-07-12T08:41:11.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2d340abe90 0x7f2d341527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2d14053a40 0x7f2d14055f00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d3409ae20 0x7f2d341488b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 --2- 172.21.15.61:0/2372225648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d34060a70 0x7f2d340af800 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 >> 172.21.15.61:0/2372225648 conn(0x7f2d3405bd10 msgr2=0x7f2d340ab440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:11.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 shutdown_connections 2022-07-12T08:41:11.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.756+0000 7f2d38b00700 1 -- 172.21.15.61:0/2372225648 wait complete. 2022-07-12T08:41:11.814 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:11.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 Processor -- start 2022-07-12T08:41:11.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- start start 2022-07-12T08:41:11.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fad3c0b0280 0x7fad3c14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c05c3e0 con 0x7fad3c0aa830 2022-07-12T08:41:11.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3c0b5d00 con 0x7fad3c0b5920 2022-07-12T08:41:11.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3c05c610 con 0x7fad3c0b0280 2022-07-12T08:41:11.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad40804700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fad3c0b0280 0x7fad3c14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad41806700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fad3c0aa830 0x7fad3c0aac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40926/0 (socket says 172.21.15.61:40926) 2022-07-12T08:41:11.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad41806700 1 -- 172.21.15.61:0/1837975567 learned_addr learned my addr 172.21.15.61:0/1837975567 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:11.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad40804700 1 -- 172.21.15.61:0/1837975567 >> v1:172.21.15.111:6789/0 conn(0x7fad3c0b5920 legacy=0x7fad3c14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad40804700 1 -- 172.21.15.61:0/1837975567 >> v1:172.21.15.61:6789/0 conn(0x7fad3c0aa830 legacy=0x7fad3c0aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad40804700 1 -- 172.21.15.61:0/1837975567 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c05c8b0 con 0x7fad3c0b0280 2022-07-12T08:41:11.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad40804700 1 --2- 172.21.15.61:0/1837975567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b0280 0x7fad3c14b0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fad2c002f70 tx=0x7fad2c006df0).ready entity=mon.0 client_cookie=f55d37c2207e31e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:11.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad33fff700 1 -- 172.21.15.61:0/1837975567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad2c00d610 con 0x7fad3c0b0280 2022-07-12T08:41:11.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad33fff700 1 -- 172.21.15.61:0/1837975567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c000e90 con 0x7fad3c0b0280 2022-07-12T08:41:11.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad33fff700 1 -- 172.21.15.61:0/1837975567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad2c00d960 con 0x7fad3c0b0280 2022-07-12T08:41:11.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- 172.21.15.61:0/1837975567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b0280 msgr2=0x7fad3c14b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:11.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 --2- 172.21.15.61:0/1837975567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b0280 0x7fad3c14b0f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fad2c002f70 tx=0x7fad2c006df0).stop 2022-07-12T08:41:11.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- 172.21.15.61:0/1837975567 shutdown_connections 2022-07-12T08:41:11.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 --2- 172.21.15.61:0/1837975567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b0280 0x7fad3c14b0f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:11.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- 172.21.15.61:0/1837975567 >> 172.21.15.61:0/1837975567 conn(0x7fad3c063340 msgr2=0x7fad3c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:11.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- 172.21.15.61:0/1837975567 shutdown_connections 2022-07-12T08:41:11.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.936+0000 7fad42007700 1 -- 172.21.15.61:0/1837975567 wait complete. 2022-07-12T08:41:11.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 Processor -- start 2022-07-12T08:41:11.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 -- start start 2022-07-12T08:41:11.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad3c0aa830 0x7fad3c063bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad3c0b0280 0x7fad3c147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 0x7fad3c147510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:11.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0acc80 con 0x7fad3c0b5920 2022-07-12T08:41:11.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0acf20 con 0x7fad3c0b0280 2022-07-12T08:41:11.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c0ac430 con 0x7fad3c0aa830 2022-07-12T08:41:11.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 0x7fad3c147510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:11.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 0x7fad3c147510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52714/0 (socket says 172.21.15.61:52714) 2022-07-12T08:41:12.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 -- 172.21.15.61:0/4200482198 learned_addr learned my addr 172.21.15.61:0/4200482198 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:12.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41005700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad3c0aa830 0x7fad3c063bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad40804700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad3c0b0280 0x7fad3c147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 -- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad3c0aa830 msgr2=0x7fad3c063bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad3c0aa830 0x7fad3c063bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 -- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad3c0b0280 msgr2=0x7fad3c147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad3c0b0280 0x7fad3c147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 -- 172.21.15.61:0/4200482198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c00b180 con 0x7fad3c0b5920 2022-07-12T08:41:12.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad41806700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 0x7fad3c147510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fad3c0b5f30 tx=0x7fad34007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad34005ba0 con 0x7fad3c0b5920 2022-07-12T08:41:12.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad34005d00 con 0x7fad3c0b5920 2022-07-12T08:41:12.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c147a00 con 0x7fad3c0b5920 2022-07-12T08:41:12.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad340072c0 con 0x7fad3c0b5920 2022-07-12T08:41:12.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.940+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c0adb50 con 0x7fad3c0b5920 2022-07-12T08:41:12.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.944+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c0add40 con 0x7fad3c0b5920 2022-07-12T08:41:12.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.944+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fad34010070 con 0x7fad3c0b5920 2022-07-12T08:41:12.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.944+0000 7fad327fc700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad24053ad0 0x7fad24055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.944+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fad3406b4a0 con 0x7fad3c0b5920 2022-07-12T08:41:12.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.944+0000 7fad41005700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad24053ad0 0x7fad24055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.944+0000 7fad41005700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad24053ad0 0x7fad24055f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fad3800d000 tx=0x7fad38003040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:11.952+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad34042690 con 0x7fad3c0b5920 2022-07-12T08:41:12.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fad3c0ac5f0 con 0x7fad3c0b5920 2022-07-12T08:41:12.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad327fc700 1 -- 172.21.15.61:0/4200482198 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7fad34045cb0 con 0x7fad3c0b5920 2022-07-12T08:41:12.083 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:12.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad24053ad0 msgr2=0x7fad24055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad24053ad0 0x7fad24055f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fad3800d000 tx=0x7fad38003040).stop 2022-07-12T08:41:12.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 msgr2=0x7fad3c147510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 0x7fad3c147510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fad3c0b5f30 tx=0x7fad34007f40).stop 2022-07-12T08:41:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 shutdown_connections 2022-07-12T08:41:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad3c0b5920 0x7fad3c147510 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad24053ad0 0x7fad24055f90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad3c0b0280 0x7fad3c147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 --2- 172.21.15.61:0/4200482198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad3c0aa830 0x7fad3c063bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 >> 172.21.15.61:0/4200482198 conn(0x7fad3c063340 msgr2=0x7fad3c0b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:12.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 shutdown_connections 2022-07-12T08:41:12.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.084+0000 7fad42007700 1 -- 172.21.15.61:0/4200482198 wait complete. 2022-07-12T08:41:12.097 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:41:12.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.204+0000 7fdcc62d7700 1 Processor -- start 2022-07-12T08:41:12.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.204+0000 7fdcc62d7700 1 -- start start 2022-07-12T08:41:12.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc0146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdcc00b4720 0x7fdcc0146570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcc0146a60 0x7fdcc005bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00b6b50 con 0x7fdcc0146a60 2022-07-12T08:41:12.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc00b4f10 con 0x7fdcc00b4720 2022-07-12T08:41:12.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b51b0 con 0x7fdcc00b6770 2022-07-12T08:41:12.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcc0146a60 0x7fdcc005bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcc0146a60 0x7fdcc005bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52722/0 (socket says 172.21.15.61:52722) 2022-07-12T08:41:12.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2835630538 learned_addr learned my addr 172.21.15.61:0/2835630538 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:12.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc52d5700 1 --2- 172.21.15.61:0/2835630538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc0146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc4ad4700 1 --2- 172.21.15.61:0/2835630538 >> v2:172.21.15.111:3300/0 conn(0x7fdcc00b4720 0x7fdcc0146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2835630538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 msgr2=0x7fdcc0146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:12.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 --2- 172.21.15.61:0/2835630538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc0146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2835630538 >> v2:172.21.15.111:3300/0 conn(0x7fdcc00b4720 msgr2=0x7fdcc0146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:12.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 --2- 172.21.15.61:0/2835630538 >> v2:172.21.15.111:3300/0 conn(0x7fdcc00b4720 0x7fdcc0146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2835630538 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00b5450 con 0x7fdcc0146a60 2022-07-12T08:41:12.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc4ad4700 1 --2- 172.21.15.61:0/2835630538 >> v2:172.21.15.111:3300/0 conn(0x7fdcc00b4720 0x7fdcc0146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:12.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc5ad6700 1 --2- 172.21.15.61:0/2835630538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc005bd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fdcb80032f0 tx=0x7fdcb8009290).ready entity=mon.0 client_cookie=5a7847b8b379c9e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcb7fff700 1 -- 172.21.15.61:0/2835630538 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb800b870 con 0x7fdcc0146a60 2022-07-12T08:41:12.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcb7fff700 1 -- 172.21.15.61:0/2835630538 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb800b9d0 con 0x7fdcc0146a60 2022-07-12T08:41:12.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcb7fff700 1 -- 172.21.15.61:0/2835630538 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb800bb40 con 0x7fdcc0146a60 2022-07-12T08:41:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2835630538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 msgr2=0x7fdcc005bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2835630538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc005bd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fdcb80032f0 tx=0x7fdcb8009290).stop 2022-07-12T08:41:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2835630538 shutdown_connections 2022-07-12T08:41:12.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2835630538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc005bd10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2835630538 >> v2:172.21.15.111:3300/0 conn(0x7fdcc00b4720 0x7fdcc0146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2835630538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc0146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2835630538 >> 172.21.15.61:0/2835630538 conn(0x7fdcc00b9d50 msgr2=0x7fdcc00b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:12.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2835630538 shutdown_connections 2022-07-12T08:41:12.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.208+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2835630538 wait complete. 2022-07-12T08:41:12.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.212+0000 7fdcc62d7700 1 Processor -- start 2022-07-12T08:41:12.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.212+0000 7fdcc62d7700 1 -- start start 2022-07-12T08:41:12.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc00b4720 0x7fdcc0145220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc00b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc0155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc009cd30 con 0x7fdcc0146a60 2022-07-12T08:41:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b0a60 con 0x7fdcc00b4720 2022-07-12T08:41:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00b0c80 con 0x7fdcc00b6770 2022-07-12T08:41:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc0155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52728/0 (socket says 172.21.15.61:52728) 2022-07-12T08:41:12.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2152483337 learned_addr learned my addr 172.21.15.61:0/2152483337 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:12.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc4ad4700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc00b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc52d5700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc00b4720 0x7fdcc0145220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 msgr2=0x7fdcc00b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc00b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc00b4720 msgr2=0x7fdcc0145220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc00b4720 0x7fdcc0145220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 -- 172.21.15.61:0/2152483337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8009530 con 0x7fdcc0146a60 2022-07-12T08:41:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc52d5700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc00b4720 0x7fdcc0145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc5ad6700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc0155270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fdcb80032f0 tx=0x7fdcb8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb800b870 con 0x7fdcc0146a60 2022-07-12T08:41:12.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb8020470 con 0x7fdcc0146a60 2022-07-12T08:41:12.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc00b0f20 con 0x7fdcc0146a60 2022-07-12T08:41:12.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb80207d0 con 0x7fdcc0146a60 2022-07-12T08:41:12.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc00b1800 con 0x7fdcc0146a60 2022-07-12T08:41:12.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc00698d0 con 0x7fdcc0146a60 2022-07-12T08:41:12.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdcb800f070 con 0x7fdcc0146a60 2022-07-12T08:41:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.216+0000 7fdcb67fc700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdca80539b0 0x7fdca8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.220+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdcb806c780 con 0x7fdcc0146a60 2022-07-12T08:41:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.220+0000 7fdcc52d5700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdca80539b0 0x7fdca8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.220+0000 7fdcc52d5700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdca80539b0 0x7fdca8055e70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fdcbc00b000 tx=0x7fdcbc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.224+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdcc00698d0 con 0x7fdcc0146a60 2022-07-12T08:41:12.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.352+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdcc00698d0 con 0x7fdcc0146a60 2022-07-12T08:41:12.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.352+0000 7fdcb67fc700 1 -- 172.21.15.61:0/2152483337 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7fdcc00698d0 con 0x7fdcc0146a60 2022-07-12T08:41:12.355 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:12.356 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:41:12.356 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:12.357 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:41:12.357 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:41:12.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:12.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:12.358 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:41:12.376 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:41:12.376 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:12.376 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:41:12.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdca80539b0 msgr2=0x7fdca8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdca80539b0 0x7fdca8055e70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fdcbc00b000 tx=0x7fdcbc008040).stop 2022-07-12T08:41:12.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 msgr2=0x7fdcc0155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc0155270 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fdcb80032f0 tx=0x7fdcb8016000).stop 2022-07-12T08:41:12.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 shutdown_connections 2022-07-12T08:41:12.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcc0146a60 0x7fdcc0155270 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdcc00b6770 0x7fdcc00b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdca80539b0 0x7fdca8055e70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 --2- 172.21.15.61:0/2152483337 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdcc00b4720 0x7fdcc0145220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 >> 172.21.15.61:0/2152483337 conn(0x7fdcc00b9d50 msgr2=0x7fdcc005c600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:12.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 shutdown_connections 2022-07-12T08:41:12.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.356+0000 7fdcc62d7700 1 -- 172.21.15.61:0/2152483337 wait complete. 2022-07-12T08:41:12.451 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:41:12.452 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:41:12.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 Processor -- start 2022-07-12T08:41:12.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 -- start start 2022-07-12T08:41:12.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08f4060690 0x7f08f40b2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f413fd20 con 0x7f08f40b3860 2022-07-12T08:41:12.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f4098c60 con 0x7f08f4060690 2022-07-12T08:41:12.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08fb53e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f40b3c40 con 0x7f08f40b2920 2022-07-12T08:41:12.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08f4060690 0x7f08f40b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f92da700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f08f40b3860 0x7f08f40b1d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40950/0 (socket says 172.21.15.61:40950) 2022-07-12T08:41:12.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08f4060690 0x7f08f40b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52734/0 (socket says 172.21.15.61:52734) 2022-07-12T08:41:12.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f92da700 1 -- 172.21.15.61:0/1465833325 learned_addr learned my addr 172.21.15.61:0/1465833325 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:12.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f9adb700 1 --2- 172.21.15.61:0/1465833325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 -- 172.21.15.61:0/1465833325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 msgr2=0x7f08f413ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:12.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 --2- 172.21.15.61:0/1465833325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f413ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 -- 172.21.15.61:0/1465833325 >> v1:172.21.15.61:6789/0 conn(0x7f08f40b3860 legacy=0x7f08f40b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 -- 172.21.15.61:0/1465833325 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4098280 con 0x7f08f4060690 2022-07-12T08:41:12.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08f8ad9700 1 --2- 172.21.15.61:0/1465833325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f4060690 0x7f08f40b2430 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f08e4002f70 tx=0x7f08e4006df0).ready entity=mon.0 client_cookie=d97da252a864d493 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08ebfff700 1 -- 172.21.15.61:0/1465833325 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e400d570 con 0x7f08f4060690 2022-07-12T08:41:12.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08ebfff700 1 -- 172.21.15.61:0/1465833325 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e4000e90 con 0x7f08f4060690 2022-07-12T08:41:12.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.588+0000 7f08ebfff700 1 -- 172.21.15.61:0/1465833325 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e400d8c0 con 0x7f08f4060690 2022-07-12T08:41:12.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- 172.21.15.61:0/1465833325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f4060690 msgr2=0x7f08f40b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 --2- 172.21.15.61:0/1465833325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f4060690 0x7f08f40b2430 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f08e4002f70 tx=0x7f08e4006df0).stop 2022-07-12T08:41:12.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- 172.21.15.61:0/1465833325 shutdown_connections 2022-07-12T08:41:12.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 --2- 172.21.15.61:0/1465833325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 --2- 172.21.15.61:0/1465833325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f4060690 0x7f08f40b2430 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- 172.21.15.61:0/1465833325 >> 172.21.15.61:0/1465833325 conn(0x7f08f40b6e40 msgr2=0x7f08f40b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:12.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- 172.21.15.61:0/1465833325 shutdown_connections 2022-07-12T08:41:12.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- 172.21.15.61:0/1465833325 wait complete. 2022-07-12T08:41:12.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 Processor -- start 2022-07-12T08:41:12.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- start start 2022-07-12T08:41:12.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4060690 0x7f08f40ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f409ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 0x7f08f40afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f409bea0 con 0x7f08f40b3860 2022-07-12T08:41:12.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f409c140 con 0x7f08f4060690 2022-07-12T08:41:12.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08fb53e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f40b0150 con 0x7f08f40b2920 2022-07-12T08:41:12.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 0x7f08f40afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 0x7f08f40afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52740/0 (socket says 172.21.15.61:52740) 2022-07-12T08:41:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 -- 172.21.15.61:0/2764262261 learned_addr learned my addr 172.21.15.61:0/2764262261 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f8ad9700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f409ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f92da700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4060690 0x7f08f40ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 -- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 msgr2=0x7f08f409ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f409ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 -- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4060690 msgr2=0x7f08f40ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4060690 0x7f08f40ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f9adb700 1 -- 172.21.15.61:0/2764262261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e400b180 con 0x7f08f40b3860 2022-07-12T08:41:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.592+0000 7f08f8ad9700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f409ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08f9adb700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 0x7f08f40afc60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f08ec00c000 tx=0x7f08ec0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ec00c850 con 0x7f08f40b3860 2022-07-12T08:41:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec00c9b0 con 0x7f08f40b3860 2022-07-12T08:41:12.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ec00cd10 con 0x7f08f40b3860 2022-07-12T08:41:12.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f409a0f0 con 0x7f08f40b3860 2022-07-12T08:41:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f409b9a0 con 0x7f08f40b3860 2022-07-12T08:41:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f409c300 con 0x7f08f40b3860 2022-07-12T08:41:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f08ec022070 con 0x7f08f40b3860 2022-07-12T08:41:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08ea7fc700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08dc053940 0x7f08dc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:12.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f08ec06c230 con 0x7f08f40b3860 2022-07-12T08:41:12.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.596+0000 7f08f92da700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08dc053940 0x7f08dc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:12.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.600+0000 7f08f92da700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08dc053940 0x7f08dc055e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f08f000b000 tx=0x7f08f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:12.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.600+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08ec0434c0 con 0x7f08f40b3860 2022-07-12T08:41:12.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:12.732+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f08f4057460 con 0x7f08f40b3860 2022-07-12T08:41:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.240+0000 7f08ea7fc700 1 -- 172.21.15.61:0/2764262261 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7f08ec0160a0 con 0x7f08f40b3860 2022-07-12T08:41:13.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.244+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08dc053940 msgr2=0x7f08dc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:13.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.244+0000 7f08fb53e700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08dc053940 0x7f08dc055e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f08f000b000 tx=0x7f08f0009040).stop 2022-07-12T08:41:13.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.244+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 msgr2=0x7f08f40afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:13.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.244+0000 7f08fb53e700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 0x7f08f40afc60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f08ec00c000 tx=0x7f08ec0091d0).stop 2022-07-12T08:41:13.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 shutdown_connections 2022-07-12T08:41:13.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08f40b3860 0x7f08f40afc60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08f40b2920 0x7f08f409ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08dc053940 0x7f08dc055e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 --2- 172.21.15.61:0/2764262261 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08f4060690 0x7f08f40ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 >> 172.21.15.61:0/2764262261 conn(0x7f08f40b6e40 msgr2=0x7f08f405c930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:13.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 shutdown_connections 2022-07-12T08:41:13.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.248+0000 7f08fb53e700 1 -- 172.21.15.61:0/2764262261 wait complete. 2022-07-12T08:41:13.296 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:41:13.297 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:41:13.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 Processor -- start 2022-07-12T08:41:13.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- start start 2022-07-12T08:41:13.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb4940b05f0 0x7fb4940b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:13.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4940b3730 0x7fb494142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:13.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4940b3560 con 0x7fb4940b5480 2022-07-12T08:41:13.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4940b5b80 con 0x7fb4940b3730 2022-07-12T08:41:13.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4940b5e20 con 0x7fb4940b05f0 2022-07-12T08:41:13.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4940b3730 0x7fb494142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:13.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4940b3730 0x7fb494142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52748/0 (socket says 172.21.15.61:52748) 2022-07-12T08:41:13.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 -- 172.21.15.61:0/2893197990 learned_addr learned my addr 172.21.15.61:0/2893197990 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:13.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb491d9b700 1 --2- 172.21.15.61:0/2893197990 >> v2:172.21.15.111:3300/0 conn(0x7fb4940b05f0 0x7fb4940b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:13.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 -- 172.21.15.61:0/2893197990 >> v2:172.21.15.111:3300/0 conn(0x7fb4940b05f0 msgr2=0x7fb4940b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:13.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 --2- 172.21.15.61:0/2893197990 >> v2:172.21.15.111:3300/0 conn(0x7fb4940b05f0 0x7fb4940b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 -- 172.21.15.61:0/2893197990 >> v1:172.21.15.111:6789/0 conn(0x7fb4940b5480 legacy=0x7fb4940afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:13.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 -- 172.21.15.61:0/2893197990 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4940b5860 con 0x7fb4940b3730 2022-07-12T08:41:13.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb492d9d700 1 --2- 172.21.15.61:0/2893197990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b3730 0x7fb494142f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb48c0032f0 tx=0x7fb48c009290).ready entity=mon.0 client_cookie=7b57d41f18f1365f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:13.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb49159a700 1 -- 172.21.15.61:0/2893197990 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb48c00b870 con 0x7fb4940b3730 2022-07-12T08:41:13.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb49159a700 1 -- 172.21.15.61:0/2893197990 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c00b9d0 con 0x7fb4940b3730 2022-07-12T08:41:13.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb49159a700 1 -- 172.21.15.61:0/2893197990 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb48c00bb40 con 0x7fb4940b3730 2022-07-12T08:41:13.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- 172.21.15.61:0/2893197990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b3730 msgr2=0x7fb494142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:13.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 --2- 172.21.15.61:0/2893197990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b3730 0x7fb494142f90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb48c0032f0 tx=0x7fb48c009290).stop 2022-07-12T08:41:13.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- 172.21.15.61:0/2893197990 shutdown_connections 2022-07-12T08:41:13.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 --2- 172.21.15.61:0/2893197990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b3730 0x7fb494142f90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 --2- 172.21.15.61:0/2893197990 >> v2:172.21.15.111:3300/0 conn(0x7fb4940b05f0 0x7fb4940b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- 172.21.15.61:0/2893197990 >> 172.21.15.61:0/2893197990 conn(0x7fb4940b96b0 msgr2=0x7fb4940b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:13.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.428+0000 7fb498b7b700 1 -- 172.21.15.61:0/2893197990 shutdown_connections 2022-07-12T08:41:13.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 -- 172.21.15.61:0/2893197990 wait complete. 2022-07-12T08:41:13.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 Processor -- start 2022-07-12T08:41:13.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 -- start start 2022-07-12T08:41:13.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 0x7fb4940afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:13.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb4940b3730 0x7fb4941422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:13.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4940b5480 0x7fb4940acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:13.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940af7f0 con 0x7fb4940b05f0 2022-07-12T08:41:13.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4940ad790 con 0x7fb4940b5480 2022-07-12T08:41:13.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb498b7b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb494141360 con 0x7fb4940b3730 2022-07-12T08:41:13.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb49259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 0x7fb4940afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:13.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb49259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 0x7fb4940afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52750/0 (socket says 172.21.15.61:52750) 2022-07-12T08:41:13.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb49259c700 1 -- 172.21.15.61:0/683516919 learned_addr learned my addr 172.21.15.61:0/683516919 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:13.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb491d9b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb4940b3730 0x7fb4941422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:13.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb492d9d700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4940b5480 0x7fb4940acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:13.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb49259c700 1 -- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb4940b3730 msgr2=0x7fb4941422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:13.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.432+0000 7fb49259c700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb4940b3730 0x7fb4941422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb49259c700 1 -- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4940b5480 msgr2=0x7fb4940acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:13.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb49259c700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4940b5480 0x7fb4940acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:13.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb49259c700 1 -- 172.21.15.61:0/683516919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c009530 con 0x7fb4940b05f0 2022-07-12T08:41:13.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb49259c700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 0x7fb4940afc50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fb494066150 tx=0x7fb484007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:13.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb484012400 con 0x7fb4940b05f0 2022-07-12T08:41:13.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb484012560 con 0x7fb4940b05f0 2022-07-12T08:41:13.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4940ad4f0 con 0x7fb4940b05f0 2022-07-12T08:41:13.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4840128c0 con 0x7fb4940b05f0 2022-07-12T08:41:13.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4940ad050 con 0x7fb4940b05f0 2022-07-12T08:41:13.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4940ad360 con 0x7fb4940b05f0 2022-07-12T08:41:13.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb484019030 con 0x7fb4940b05f0 2022-07-12T08:41:13.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb4837fe700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb470053ad0 0x7fb470055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:13.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fb48406b0f0 con 0x7fb4940b05f0 2022-07-12T08:41:13.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.436+0000 7fb491d9b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb470053ad0 0x7fb470055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:13.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.440+0000 7fb491d9b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb470053ad0 0x7fb470055f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb48800b000 tx=0x7fb488006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:13.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.444+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb484042110 con 0x7fb4940b05f0 2022-07-12T08:41:13.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:13.564+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fb494057460 con 0x7fb4940b05f0 2022-07-12T08:41:14.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.252+0000 7fb4837fe700 1 -- 172.21.15.61:0/683516919 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v20) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb484045730 con 0x7fb4940b05f0 2022-07-12T08:41:14.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb470053ad0 msgr2=0x7fb470055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb470053ad0 0x7fb470055f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fb48800b000 tx=0x7fb488006df0).stop 2022-07-12T08:41:14.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 msgr2=0x7fb4940afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 0x7fb4940afc50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fb494066150 tx=0x7fb484007c50).stop 2022-07-12T08:41:14.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 shutdown_connections 2022-07-12T08:41:14.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb470053ad0 0x7fb470055f90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4940b5480 0x7fb4940acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb4940b3730 0x7fb4941422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 --2- 172.21.15.61:0/683516919 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4940b05f0 0x7fb4940afc50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 >> 172.21.15.61:0/683516919 conn(0x7fb4940b96b0 msgr2=0x7fb494148620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:14.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 shutdown_connections 2022-07-12T08:41:14.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.256+0000 7fb498b7b700 1 -- 172.21.15.61:0/683516919 wait complete. 2022-07-12T08:41:14.272 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:41:14.274 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:41:14.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.388+0000 7f542a0da700 1 Processor -- start 2022-07-12T08:41:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.388+0000 7f542a0da700 1 -- start start 2022-07-12T08:41:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f542a0da700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5424060560 0x7f54240b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f542a0da700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f542405bd10 0x7f54240af530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f542a0da700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54240ae110 con 0x7f54240b2640 2022-07-12T08:41:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f542a0da700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f542405c430 con 0x7f5424060560 2022-07-12T08:41:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f542a0da700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5424142e80 con 0x7f542405bd10 2022-07-12T08:41:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54237fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f54240b2640 0x7f542405f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40976/0 (socket says 172.21.15.61:40976) 2022-07-12T08:41:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54237fe700 1 -- 172.21.15.61:0/2820390147 learned_addr learned my addr 172.21.15.61:0/2820390147 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5422ffd700 1 --2- 172.21.15.61:0/2820390147 >> v2:172.21.15.111:3300/0 conn(0x7f5424060560 0x7f54240b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5423fff700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f542405bd10 0x7f54240af530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54227fc700 1 -- 172.21.15.61:0/2820390147 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062095842 0 0) 0x7f54240ae110 con 0x7f54240b2640 2022-07-12T08:41:14.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54227fc700 1 -- 172.21.15.61:0/2820390147 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5410003260 con 0x7f54240b2640 2022-07-12T08:41:14.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54227fc700 1 -- 172.21.15.61:0/2820390147 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859563659 0 0) 0x7f5410003260 con 0x7f54240b2640 2022-07-12T08:41:14.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5422ffd700 1 -- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f542405bd10 msgr2=0x7f54240af530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:14.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5422ffd700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f542405bd10 0x7f54240af530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5422ffd700 1 -- 172.21.15.61:0/2820390147 >> v1:172.21.15.61:6789/0 conn(0x7f54240b2640 legacy=0x7f542405f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5422ffd700 1 -- 172.21.15.61:0/2820390147 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542405f850 con 0x7f5424060560 2022-07-12T08:41:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5423fff700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f542405bd10 0x7f54240af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f5422ffd700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5424060560 0x7f54240b0d60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f5414001800 tx=0x7f5418006df0).ready entity=mon.1 client_cookie=2c3e08a0885ca61 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54227fc700 1 -- 172.21.15.61:0/2820390147 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5414001660 con 0x7f5424060560 2022-07-12T08:41:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54227fc700 1 -- 172.21.15.61:0/2820390147 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5418000e90 con 0x7f5424060560 2022-07-12T08:41:14.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.392+0000 7f54227fc700 1 -- 172.21.15.61:0/2820390147 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5418012460 con 0x7f5424060560 2022-07-12T08:41:14.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5424060560 msgr2=0x7f54240b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5424060560 0x7f54240b0d60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f5414001800 tx=0x7f5418006df0).stop 2022-07-12T08:41:14.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- 172.21.15.61:0/2820390147 shutdown_connections 2022-07-12T08:41:14.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f542405bd10 0x7f54240af530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 --2- 172.21.15.61:0/2820390147 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5424060560 0x7f54240b0d60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- 172.21.15.61:0/2820390147 >> 172.21.15.61:0/2820390147 conn(0x7f54240633a0 msgr2=0x7f54240637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- 172.21.15.61:0/2820390147 shutdown_connections 2022-07-12T08:41:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- 172.21.15.61:0/2820390147 wait complete. 2022-07-12T08:41:14.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 Processor -- start 2022-07-12T08:41:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- start start 2022-07-12T08:41:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f542405bd10 0x7f54240b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5424060560 0x7f54240657d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 0x7f5424065cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54240667e0 con 0x7f54240b2640 2022-07-12T08:41:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5424066a80 con 0x7f542405bd10 2022-07-12T08:41:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f542a0da700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5424066d20 con 0x7f5424060560 2022-07-12T08:41:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.396+0000 7f5423fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 0x7f5424065cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 0x7f5424065cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52768/0 (socket says 172.21.15.61:52768) 2022-07-12T08:41:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 -- 172.21.15.61:0/1015364981 learned_addr learned my addr 172.21.15.61:0/1015364981 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5422ffd700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5424060560 0x7f54240657d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f54237fe700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f542405bd10 0x7f54240b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 -- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5424060560 msgr2=0x7f54240657d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5424060560 0x7f54240657d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 -- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f542405bd10 msgr2=0x7f54240b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f542405bd10 0x7f54240b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 -- 172.21.15.61:0/1015364981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c009470 con 0x7f54240b2640 2022-07-12T08:41:14.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5423fff700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 0x7f5424065cc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f541c00c000 tx=0x7f541c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:14.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f541c00c850 con 0x7f54240b2640 2022-07-12T08:41:14.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541c00c9b0 con 0x7f54240b2640 2022-07-12T08:41:14.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541800b180 con 0x7f54240b2640 2022-07-12T08:41:14.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f541c00cd10 con 0x7f54240b2640 2022-07-12T08:41:14.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54240649c0 con 0x7f54240b2640 2022-07-12T08:41:14.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5424066ee0 con 0x7f54240b2640 2022-07-12T08:41:14.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.400+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f541c022070 con 0x7f54240b2640 2022-07-12T08:41:14.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.404+0000 7f5420ff9700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5404053a40 0x7f5404055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.404+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f541c06c5a0 con 0x7f54240b2640 2022-07-12T08:41:14.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.404+0000 7f54237fe700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5404053a40 0x7f5404055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.404+0000 7f54237fe700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5404053a40 0x7f5404055f00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f541400d000 tx=0x7f5414001340).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:14.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.412+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f541c043570 con 0x7f54240b2640 2022-07-12T08:41:14.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.576+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f5424057460 con 0x7f54240b2640 2022-07-12T08:41:14.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.588+0000 7f5420ff9700 1 -- 172.21.15.61:0/1015364981 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 187+0+0 (secure 0 0 0) 0x7f541c046b90 con 0x7f54240b2640 2022-07-12T08:41:14.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5404053a40 msgr2=0x7f5404055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5404053a40 0x7f5404055f00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f541400d000 tx=0x7f5414001340).stop 2022-07-12T08:41:14.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 msgr2=0x7f5424065cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 0x7f5424065cc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f541c00c000 tx=0x7f541c0091d0).stop 2022-07-12T08:41:14.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 shutdown_connections 2022-07-12T08:41:14.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54240b2640 0x7f5424065cc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5424060560 0x7f54240657d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5404053a40 0x7f5404055f00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 --2- 172.21.15.61:0/1015364981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f542405bd10 0x7f54240b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 >> 172.21.15.61:0/1015364981 conn(0x7f54240633a0 msgr2=0x7f542405f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:14.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 shutdown_connections 2022-07-12T08:41:14.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.592+0000 7f542a0da700 1 -- 172.21.15.61:0/1015364981 wait complete. 2022-07-12T08:41:14.604 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 3 and data pool 4 2022-07-12T08:41:14.619 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:14.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.736+0000 7f5a74070700 1 Processor -- start 2022-07-12T08:41:14.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.736+0000 7f5a74070700 1 -- start start 2022-07-12T08:41:14.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a6c143720 0x7f5a6c143b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c061030 con 0x7f5a6c1442b0 2022-07-12T08:41:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c0b1d20 con 0x7f5a6c143720 2022-07-12T08:41:14.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c0612d0 con 0x7f5a6c060b00 2022-07-12T08:41:14.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7260d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a71e0c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5a6c143720 0x7f5a6c143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7160b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f5a6c1442b0 0x7f5a6c0603f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:54592/0 (socket says 172.21.15.61:54592) 2022-07-12T08:41:14.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7160b700 1 -- 172.21.15.61:0/2676682124 learned_addr learned my addr 172.21.15.61:0/2676682124 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:14.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a70e0a700 1 -- 172.21.15.61:0/2676682124 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015171537 0 0) 0x7f5a6c061030 con 0x7f5a6c1442b0 2022-07-12T08:41:14.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a70e0a700 1 -- 172.21.15.61:0/2676682124 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a54003260 con 0x7f5a6c1442b0 2022-07-12T08:41:14.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7260d700 1 -- 172.21.15.61:0/2676682124 >> v2:172.21.15.111:3300/0 conn(0x7f5a6c143720 msgr2=0x7f5a6c143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:14.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7260d700 1 --2- 172.21.15.61:0/2676682124 >> v2:172.21.15.111:3300/0 conn(0x7f5a6c143720 0x7f5a6c143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7260d700 1 -- 172.21.15.61:0/2676682124 >> v1:172.21.15.111:6789/0 conn(0x7f5a6c1442b0 legacy=0x7f5a6c0603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7260d700 1 -- 172.21.15.61:0/2676682124 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c140f40 con 0x7f5a6c060b00 2022-07-12T08:41:14.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a7260d700 1 --2- 172.21.15.61:0/2676682124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0981e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5a640032f0 tx=0x7f5a640091d0).ready entity=mon.2 client_cookie=f6bf52b8baa663b7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:14.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a70e0a700 1 -- 172.21.15.61:0/2676682124 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6400b600 con 0x7f5a6c060b00 2022-07-12T08:41:14.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a70e0a700 1 -- 172.21.15.61:0/2676682124 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6400b760 con 0x7f5a6c060b00 2022-07-12T08:41:14.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a70e0a700 1 -- 172.21.15.61:0/2676682124 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6400b8d0 con 0x7f5a6c060b00 2022-07-12T08:41:14.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- 172.21.15.61:0/2676682124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 msgr2=0x7f5a6c0981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 --2- 172.21.15.61:0/2676682124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0981e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5a640032f0 tx=0x7f5a640091d0).stop 2022-07-12T08:41:14.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- 172.21.15.61:0/2676682124 shutdown_connections 2022-07-12T08:41:14.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 --2- 172.21.15.61:0/2676682124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0981e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 --2- 172.21.15.61:0/2676682124 >> v2:172.21.15.111:3300/0 conn(0x7f5a6c143720 0x7f5a6c143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- 172.21.15.61:0/2676682124 >> 172.21.15.61:0/2676682124 conn(0x7f5a6c064720 msgr2=0x7f5a6c0b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- 172.21.15.61:0/2676682124 shutdown_connections 2022-07-12T08:41:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.740+0000 7f5a74070700 1 -- 172.21.15.61:0/2676682124 wait complete. 2022-07-12T08:41:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 Processor -- start 2022-07-12T08:41:14.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 -- start start 2022-07-12T08:41:14.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0676a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c143720 0x7f5a6c152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 0x7f5a6c152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c0650c0 con 0x7f5a6c1442b0 2022-07-12T08:41:14.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c065360 con 0x7f5a6c143720 2022-07-12T08:41:14.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a74070700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1523f0 con 0x7f5a6c060b00 2022-07-12T08:41:14.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a7260d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 0x7f5a6c152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a7260d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 0x7f5a6c152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52782/0 (socket says 172.21.15.61:52782) 2022-07-12T08:41:14.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a7260d700 1 -- 172.21.15.61:0/1661245594 learned_addr learned my addr 172.21.15.61:0/1661245594 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:14.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a7160b700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c143720 0x7f5a6c152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.744+0000 7f5a71e0c700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7260d700 1 -- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 msgr2=0x7f5a6c0676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7260d700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0676a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7260d700 1 -- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c143720 msgr2=0x7f5a6c152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7260d700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c143720 0x7f5a6c152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7260d700 1 -- 172.21.15.61:0/1661245594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64009470 con 0x7f5a6c1442b0 2022-07-12T08:41:14.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7160b700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c143720 0x7f5a6c152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:14.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a7260d700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 0x7f5a6c152d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f5a640032f0 tx=0x7f5a64016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:14.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6400b8d0 con 0x7f5a6c1442b0 2022-07-12T08:41:14.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a64021630 con 0x7f5a6c1442b0 2022-07-12T08:41:14.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c153270 con 0x7f5a6c1442b0 2022-07-12T08:41:14.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a6400b600 con 0x7f5a6c1442b0 2022-07-12T08:41:14.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c066ea0 con 0x7f5a6c1442b0 2022-07-12T08:41:14.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c064f30 con 0x7f5a6c1442b0 2022-07-12T08:41:14.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5a64027020 con 0x7f5a6c1442b0 2022-07-12T08:41:14.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a5affd700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5a4c0539b0 0x7f5a4c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:14.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.748+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a6406d590 con 0x7f5a6c1442b0 2022-07-12T08:41:14.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.752+0000 7f5a71e0c700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5a4c0539b0 0x7f5a4c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:14.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.752+0000 7f5a71e0c700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5a4c0539b0 0x7f5a4c055e70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5a6800b000 tx=0x7f5a68006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:14.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.760+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a640443e0 con 0x7f5a6c1442b0 2022-07-12T08:41:14.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.884+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a6c057460 con 0x7f5a6c1442b0 2022-07-12T08:41:14.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a5affd700 1 -- 172.21.15.61:0/1661245594 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f5a64015090 con 0x7f5a6c1442b0 2022-07-12T08:41:14.892 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:14.892 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":21,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:14.580081+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:14.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5a4c0539b0 msgr2=0x7f5a4c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5a4c0539b0 0x7f5a4c055e70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5a6800b000 tx=0x7f5a68006df0).stop 2022-07-12T08:41:14.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 msgr2=0x7f5a6c152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:14.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 0x7f5a6c152d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f5a640032f0 tx=0x7f5a64016000).stop 2022-07-12T08:41:14.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 shutdown_connections 2022-07-12T08:41:14.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a6c1442b0 0x7f5a6c152d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5a4c0539b0 0x7f5a4c055e70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c143720 0x7f5a6c152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 --2- 172.21.15.61:0/1661245594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5a6c060b00 0x7f5a6c0676a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:14.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 >> 172.21.15.61:0/1661245594 conn(0x7f5a6c064720 msgr2=0x7f5a6c13ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:14.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 shutdown_connections 2022-07-12T08:41:14.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:14.888+0000 7f5a74070700 1 -- 172.21.15.61:0/1661245594 wait complete. 2022-07-12T08:41:14.900 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:15.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 Processor -- start 2022-07-12T08:41:15.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 -- start start 2022-07-12T08:41:15.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbc4c0af0b0 0x7fbc4c05bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc4c060b80 0x7fbc4c05c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c05cb80 con 0x7fbc4c060b80 2022-07-12T08:41:15.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c0b02e0 con 0x7fbc4c0af0b0 2022-07-12T08:41:15.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc509f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0af490 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc4c060b80 0x7fbc4c05c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc4c060b80 0x7fbc4c05c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52788/0 (socket says 172.21.15.61:52788) 2022-07-12T08:41:15.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 -- 172.21.15.61:0/2781566171 learned_addr learned my addr 172.21.15.61:0/2781566171 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:15.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4bfff700 1 --2- 172.21.15.61:0/2781566171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4b7fe700 1 --2- 172.21.15.61:0/2781566171 >> v2:172.21.15.111:3300/0 conn(0x7fbc4c0af0b0 0x7fbc4c05bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 -- 172.21.15.61:0/2781566171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c05c6f0 msgr2=0x7fbc4c13f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:15.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 --2- 172.21.15.61:0/2781566171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 -- 172.21.15.61:0/2781566171 >> v2:172.21.15.111:3300/0 conn(0x7fbc4c0af0b0 msgr2=0x7fbc4c05bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:15.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 --2- 172.21.15.61:0/2781566171 >> v2:172.21.15.111:3300/0 conn(0x7fbc4c0af0b0 0x7fbc4c05bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 -- 172.21.15.61:0/2781566171 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c05d240 con 0x7fbc4c060b80 2022-07-12T08:41:15.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4b7fe700 1 --2- 172.21.15.61:0/2781566171 >> v2:172.21.15.111:3300/0 conn(0x7fbc4c0af0b0 0x7fbc4c05bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:15.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4affd700 1 --2- 172.21.15.61:0/2781566171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c060b80 0x7fbc4c05c200 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbc3c0032f0 tx=0x7fbc3c009290).ready entity=mon.0 client_cookie=ba476aff6468789f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4a7fc700 1 -- 172.21.15.61:0/2781566171 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc3c00b870 con 0x7fbc4c060b80 2022-07-12T08:41:15.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4a7fc700 1 -- 172.21.15.61:0/2781566171 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c00b9d0 con 0x7fbc4c060b80 2022-07-12T08:41:15.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.036+0000 7fbc4a7fc700 1 -- 172.21.15.61:0/2781566171 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc3c00bb40 con 0x7fbc4c060b80 2022-07-12T08:41:15.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/2781566171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c060b80 msgr2=0x7fbc4c05c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- 172.21.15.61:0/2781566171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c060b80 0x7fbc4c05c200 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fbc3c0032f0 tx=0x7fbc3c009290).stop 2022-07-12T08:41:15.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/2781566171 shutdown_connections 2022-07-12T08:41:15.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- 172.21.15.61:0/2781566171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- 172.21.15.61:0/2781566171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c060b80 0x7fbc4c05c200 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- 172.21.15.61:0/2781566171 >> v2:172.21.15.111:3300/0 conn(0x7fbc4c0af0b0 0x7fbc4c05bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/2781566171 >> 172.21.15.61:0/2781566171 conn(0x7fbc4c0b5780 msgr2=0x7fbc4c0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:15.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/2781566171 shutdown_connections 2022-07-12T08:41:15.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/2781566171 wait complete. 2022-07-12T08:41:15.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 Processor -- start 2022-07-12T08:41:15.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- start start 2022-07-12T08:41:15.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc4c060b80 0x7fbc4c138fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c0af0b0 0x7fbc4c1394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0ad740 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0ad200 con 0x7fbc4c060b80 2022-07-12T08:41:15.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c0ad4a0 con 0x7fbc4c0af0b0 2022-07-12T08:41:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52792/0 (socket says 172.21.15.61:52792) 2022-07-12T08:41:15.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 -- 172.21.15.61:0/1097750821 learned_addr learned my addr 172.21.15.61:0/1097750821 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:15.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4bfff700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c0af0b0 0x7fbc4c1394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4affd700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc4c060b80 0x7fbc4c138fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 -- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c0af0b0 msgr2=0x7fbc4c1394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c0af0b0 0x7fbc4c1394d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 -- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc4c060b80 msgr2=0x7fbc4c138fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc4c060b80 0x7fbc4c138fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 -- 172.21.15.61:0/1097750821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c009530 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4affd700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc4c060b80 0x7fbc4c138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:15.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc4b7fe700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbc40006940 tx=0x7fbc40008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc40011780 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc400118e0 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c13fd60 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc40011c40 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.040+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c13ebe0 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.044+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c061240 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.044+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbc40012420 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.044+0000 7fbc48ff9700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc2c0539b0 0x7fbc2c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.044+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc4006bd60 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.044+0000 7fbc4affd700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc2c0539b0 0x7fbc2c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.044+0000 7fbc4affd700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc2c0539b0 0x7fbc2c055e70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fbc3c009000 tx=0x7fbc3c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.052+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc40042c60 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.176+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc4c065fd0 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.176+0000 7fbc48ff9700 1 -- 172.21.15.61:0/1097750821 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16099 (secure 0 0 0) 0x7fbc40046280 con 0x7fbc4c05c6f0 2022-07-12T08:41:15.178 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:15.178 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":21,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:14.580081+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:15.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc2c0539b0 msgr2=0x7fbc2c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc2c0539b0 0x7fbc2c055e70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fbc3c009000 tx=0x7fbc3c016000).stop 2022-07-12T08:41:15.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 msgr2=0x7fbc4c13f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f870 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbc40006940 tx=0x7fbc40008040).stop 2022-07-12T08:41:15.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 shutdown_connections 2022-07-12T08:41:15.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc4c0af0b0 0x7fbc4c1394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc2c0539b0 0x7fbc2c055e70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc4c060b80 0x7fbc4c138fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 --2- 172.21.15.61:0/1097750821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc4c05c6f0 0x7fbc4c13f870 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 >> 172.21.15.61:0/1097750821 conn(0x7fbc4c0b5780 msgr2=0x7fbc4c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:15.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 shutdown_connections 2022-07-12T08:41:15.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.180+0000 7fbc509f8700 1 -- 172.21.15.61:0/1097750821 wait complete. 2022-07-12T08:41:15.203 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:41:15.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.336+0000 7f993a37d700 1 Processor -- start 2022-07-12T08:41:15.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.336+0000 7f993a37d700 1 -- start start 2022-07-12T08:41:15.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f99340b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340b3120 con 0x7f99340b9db0 2022-07-12T08:41:15.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340b33c0 con 0x7f993405c810 2022-07-12T08:41:15.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993405bd10 con 0x7f99340b3ea0 2022-07-12T08:41:15.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9933fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f99340b9db0 0x7f993405c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41018/0 (socket says 172.21.15.61:41018) 2022-07-12T08:41:15.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9933fff700 1 -- 172.21.15.61:0/2678966573 learned_addr learned my addr 172.21.15.61:0/2678966573 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:15.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f99337fe700 1 --2- 172.21.15.61:0/2678966573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f99340b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9932ffd700 1 -- 172.21.15.61:0/2678966573 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891660884 0 0) 0x7f99340b3120 con 0x7f99340b9db0 2022-07-12T08:41:15.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9932ffd700 1 -- 172.21.15.61:0/2678966573 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920003260 con 0x7f99340b9db0 2022-07-12T08:41:15.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9932ffd700 1 -- 172.21.15.61:0/2678966573 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047972924 0 0) 0x7f99340b33c0 con 0x7f993405c810 2022-07-12T08:41:15.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f99337fe700 1 -- 172.21.15.61:0/2678966573 >> v1:172.21.15.111:6789/0 conn(0x7f993405c810 legacy=0x7f99340b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f99337fe700 1 -- 172.21.15.61:0/2678966573 >> v1:172.21.15.61:6789/0 conn(0x7f99340b9db0 legacy=0x7f993405c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f99337fe700 1 -- 172.21.15.61:0/2678966573 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993405cc60 con 0x7f99340b3ea0 2022-07-12T08:41:15.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f99337fe700 1 --2- 172.21.15.61:0/2678966573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f99340b2a40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9924002f70 tx=0x7f9924006df0).ready entity=mon.2 client_cookie=a563b94756a59a5c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9932ffd700 1 -- 172.21.15.61:0/2678966573 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f992400d570 con 0x7f99340b3ea0 2022-07-12T08:41:15.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9932ffd700 1 -- 172.21.15.61:0/2678966573 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924000e90 con 0x7f99340b3ea0 2022-07-12T08:41:15.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f9932ffd700 1 -- 172.21.15.61:0/2678966573 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f992400d8c0 con 0x7f99340b3ea0 2022-07-12T08:41:15.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- 172.21.15.61:0/2678966573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 msgr2=0x7f99340b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 --2- 172.21.15.61:0/2678966573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f99340b2a40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f9924002f70 tx=0x7f9924006df0).stop 2022-07-12T08:41:15.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- 172.21.15.61:0/2678966573 shutdown_connections 2022-07-12T08:41:15.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 --2- 172.21.15.61:0/2678966573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f99340b2a40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- 172.21.15.61:0/2678966573 >> 172.21.15.61:0/2678966573 conn(0x7f9934067960 msgr2=0x7f99340b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:15.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- 172.21.15.61:0/2678966573 shutdown_connections 2022-07-12T08:41:15.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.340+0000 7f993a37d700 1 -- 172.21.15.61:0/2678966573 wait complete. 2022-07-12T08:41:15.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 Processor -- start 2022-07-12T08:41:15.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 -- start start 2022-07-12T08:41:15.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 0x7f99341138f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f9934113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99340b9db0 0x7f9934155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934114400 con 0x7f993405c810 2022-07-12T08:41:15.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99340b01c0 con 0x7f99340b9db0 2022-07-12T08:41:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9934116320 con 0x7f99340b3ea0 2022-07-12T08:41:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 0x7f99341138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 0x7f99341138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52806/0 (socket says 172.21.15.61:52806) 2022-07-12T08:41:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 -- 172.21.15.61:0/4197511203 learned_addr learned my addr 172.21.15.61:0/4197511203 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f99337fe700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f9934113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993891a700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99340b9db0 0x7f9934155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 -- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 msgr2=0x7f9934113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f9934113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 -- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99340b9db0 msgr2=0x7f9934155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99340b9db0 0x7f9934155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 -- 172.21.15.61:0/4197511203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992400b180 con 0x7f993405c810 2022-07-12T08:41:15.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f99337fe700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f9934113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:15.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993891a700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99340b9db0 0x7f9934155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:15.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f9933fff700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 0x7f99341138f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9928001110 tx=0x7f9928007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99280070d0 con 0x7f993405c810 2022-07-12T08:41:15.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9928007230 con 0x7f993405c810 2022-07-12T08:41:15.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9928005610 con 0x7f993405c810 2022-07-12T08:41:15.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99340af1a0 con 0x7f993405c810 2022-07-12T08:41:15.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.344+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99340afde0 con 0x7f993405c810 2022-07-12T08:41:15.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.348+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99341133f0 con 0x7f993405c810 2022-07-12T08:41:15.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.348+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9928010070 con 0x7f993405c810 2022-07-12T08:41:15.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.348+0000 7f99317fa700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99140539b0 0x7f9914055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.348+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f992806b4b0 con 0x7f993405c810 2022-07-12T08:41:15.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.348+0000 7f99337fe700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99140539b0 0x7f9914055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.348+0000 7f99337fe700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99140539b0 0x7f9914055e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9924015000 tx=0x7f9924014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.356+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99341133f0 con 0x7f993405c810 2022-07-12T08:41:15.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.528+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f99341133f0 con 0x7f993405c810 2022-07-12T08:41:15.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.596+0000 7f99317fa700 1 -- 172.21.15.61:0/4197511203 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7f99341133f0 con 0x7f993405c810 2022-07-12T08:41:15.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99140539b0 msgr2=0x7f9914055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99140539b0 0x7f9914055e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9924015000 tx=0x7f9924014000).stop 2022-07-12T08:41:15.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 msgr2=0x7f99341138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 0x7f99341138f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9928001110 tx=0x7f9928007aa0).stop 2022-07-12T08:41:15.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 shutdown_connections 2022-07-12T08:41:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99140539b0 0x7f9914055e70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99340b9db0 0x7f9934155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99340b3ea0 0x7f9934113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 --2- 172.21.15.61:0/4197511203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f993405c810 0x7f99341138f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 >> 172.21.15.61:0/4197511203 conn(0x7f9934067960 msgr2=0x7f99340b8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:15.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 shutdown_connections 2022-07-12T08:41:15.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.600+0000 7f993a37d700 1 -- 172.21.15.61:0/4197511203 wait complete. 2022-07-12T08:41:15.635 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:15.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 Processor -- start 2022-07-12T08:41:15.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 -- start start 2022-07-12T08:41:15.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72d80b01f0 0x7f72d80ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d814a8f0 0x7f72d81443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d80ad2f0 con 0x7f72d805c050 2022-07-12T08:41:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d80b0a40 con 0x7f72d80b01f0 2022-07-12T08:41:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dec81700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d805c430 con 0x7f72d814a8f0 2022-07-12T08:41:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dca1d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f72d805c050 0x7f72d805cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41032/0 (socket says 172.21.15.61:41032) 2022-07-12T08:41:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dca1d700 1 -- 172.21.15.61:0/323746975 learned_addr learned my addr 172.21.15.61:0/323746975 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72d7fff700 1 --2- 172.21.15.61:0/323746975 >> v2:172.21.15.111:3300/0 conn(0x7f72d80b01f0 0x7f72d80ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.752+0000 7f72dd21e700 1 --2- 172.21.15.61:0/323746975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d814a8f0 0x7f72d81443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d77fe700 1 -- 172.21.15.61:0/323746975 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733173110 0 0) 0x7f72d80ad2f0 con 0x7f72d805c050 2022-07-12T08:41:15.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d77fe700 1 -- 172.21.15.61:0/323746975 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c4003260 con 0x7f72d805c050 2022-07-12T08:41:15.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d77fe700 1 -- 172.21.15.61:0/323746975 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1323361279 0 0) 0x7f72c4003260 con 0x7f72d805c050 2022-07-12T08:41:15.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dd21e700 1 -- 172.21.15.61:0/323746975 >> v2:172.21.15.111:3300/0 conn(0x7f72d80b01f0 msgr2=0x7f72d80ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:15.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dd21e700 1 --2- 172.21.15.61:0/323746975 >> v2:172.21.15.111:3300/0 conn(0x7f72d80b01f0 0x7f72d80ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dd21e700 1 -- 172.21.15.61:0/323746975 >> v1:172.21.15.61:6789/0 conn(0x7f72d805c050 legacy=0x7f72d805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dd21e700 1 -- 172.21.15.61:0/323746975 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d80b2420 con 0x7f72d814a8f0 2022-07-12T08:41:15.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d7fff700 1 --2- 172.21.15.61:0/323746975 >> v2:172.21.15.111:3300/0 conn(0x7f72d80b01f0 0x7f72d80ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:15.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dd21e700 1 --2- 172.21.15.61:0/323746975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d814a8f0 0x7f72d81443c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f72cc001800 tx=0x7f72c8006df0).ready entity=mon.2 client_cookie=b28a0ba23fde48cd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d77fe700 1 -- 172.21.15.61:0/323746975 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72cc001660 con 0x7f72d814a8f0 2022-07-12T08:41:15.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d77fe700 1 -- 172.21.15.61:0/323746975 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c8000e90 con 0x7f72d814a8f0 2022-07-12T08:41:15.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72d77fe700 1 -- 172.21.15.61:0/323746975 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72c8012460 con 0x7f72d814a8f0 2022-07-12T08:41:15.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 -- 172.21.15.61:0/323746975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d814a8f0 msgr2=0x7f72d81443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 --2- 172.21.15.61:0/323746975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d814a8f0 0x7f72d81443c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f72cc001800 tx=0x7f72c8006df0).stop 2022-07-12T08:41:15.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 -- 172.21.15.61:0/323746975 shutdown_connections 2022-07-12T08:41:15.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 --2- 172.21.15.61:0/323746975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d814a8f0 0x7f72d81443c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 --2- 172.21.15.61:0/323746975 >> v2:172.21.15.111:3300/0 conn(0x7f72d80b01f0 0x7f72d80ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 -- 172.21.15.61:0/323746975 >> 172.21.15.61:0/323746975 conn(0x7f72d8062c00 msgr2=0x7f72d8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:15.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 -- 172.21.15.61:0/323746975 shutdown_connections 2022-07-12T08:41:15.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 -- 172.21.15.61:0/323746975 wait complete. 2022-07-12T08:41:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 Processor -- start 2022-07-12T08:41:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.756+0000 7f72dec81700 1 -- start start 2022-07-12T08:41:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d805c050 0x7f72d8145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 0x7f72d809fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d814a8f0 0x7f72d80ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80a0130 con 0x7f72d80b01f0 2022-07-12T08:41:15.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80ae540 con 0x7f72d814a8f0 2022-07-12T08:41:15.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80a0330 con 0x7f72d805c050 2022-07-12T08:41:15.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 0x7f72d809fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 0x7f72d809fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52824/0 (socket says 172.21.15.61:52824) 2022-07-12T08:41:15.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 -- 172.21.15.61:0/1811041090 learned_addr learned my addr 172.21.15.61:0/1811041090 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:15.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dca1d700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d805c050 0x7f72d8145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dd21e700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d814a8f0 0x7f72d80ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 -- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d805c050 msgr2=0x7f72d8145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d805c050 0x7f72d8145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 -- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d814a8f0 msgr2=0x7f72d80ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d814a8f0 0x7f72d80ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 -- 172.21.15.61:0/1811041090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0009470 con 0x7f72d80b01f0 2022-07-12T08:41:15.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d7fff700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 0x7f72d809fc40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f72d000c000 tx=0x7f72d00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d000c850 con 0x7f72d80b01f0 2022-07-12T08:41:15.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d000c9b0 con 0x7f72d80b01f0 2022-07-12T08:41:15.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72d000cd10 con 0x7f72d80b01f0 2022-07-12T08:41:15.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c800b180 con 0x7f72d80b01f0 2022-07-12T08:41:15.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d809eaf0 con 0x7f72d80b01f0 2022-07-12T08:41:15.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d80adda0 con 0x7f72d80b01f0 2022-07-12T08:41:15.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.760+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f72d001b400 con 0x7f72d80b01f0 2022-07-12T08:41:15.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.764+0000 7f72d5ffb700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72b8053a40 0x7f72b8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:15.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.764+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f72d006c910 con 0x7f72d80b01f0 2022-07-12T08:41:15.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.764+0000 7f72dca1d700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72b8053a40 0x7f72b8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:15.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.764+0000 7f72dca1d700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72b8053a40 0x7f72b8055f00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f72cc00d000 tx=0x7f72cc0013c0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.768+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72d80adda0 con 0x7f72d80b01f0 2022-07-12T08:41:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.924+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72d80adda0 con 0x7f72d80b01f0 2022-07-12T08:41:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72d5ffb700 1 -- 172.21.15.61:0/1811041090 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+2137 (secure 0 0 0) 0x7f72d80adda0 con 0x7f72d80b01f0 2022-07-12T08:41:15.928 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:15.928 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:15.584363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.111:6835/1570404457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1570404457},{"type":"v1","addr":"172.21.15.111:6835","nonce":1570404457}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:15.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72b8053a40 msgr2=0x7f72b8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72b8053a40 0x7f72b8055f00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f72cc00d000 tx=0x7f72cc0013c0).stop 2022-07-12T08:41:15.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 msgr2=0x7f72d809fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:15.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 0x7f72d809fc40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f72d000c000 tx=0x7f72d00091d0).stop 2022-07-12T08:41:15.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 shutdown_connections 2022-07-12T08:41:15.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72b8053a40 0x7f72b8055f00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72d814a8f0 0x7f72d80ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72d80b01f0 0x7f72d809fc40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 --2- 172.21.15.61:0/1811041090 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72d805c050 0x7f72d8145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:15.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 >> 172.21.15.61:0/1811041090 conn(0x7f72d8062c00 msgr2=0x7f72d80b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:15.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 shutdown_connections 2022-07-12T08:41:15.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:15.928+0000 7f72dec81700 1 -- 172.21.15.61:0/1811041090 wait complete. 2022-07-12T08:41:15.945 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2022-07-12T08:41:15.947 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.068+0000 7fd592555700 1 Processor -- start 2022-07-12T08:41:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.068+0000 7fd592555700 1 -- start start 2022-07-12T08:41:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd58c05c740 0x7fd58c098070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c0b1210 0x7fd58c142640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd58c05f340 0x7fd58c142b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c0aaa00 con 0x7fd58c05f340 2022-07-12T08:41:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58c09b020 con 0x7fd58c05c740 2022-07-12T08:41:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c05d220 con 0x7fd58c0b1210 2022-07-12T08:41:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd58c05f340 0x7fd58c142b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd58c05f340 0x7fd58c142b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52832/0 (socket says 172.21.15.61:52832) 2022-07-12T08:41:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 -- 172.21.15.61:0/367135782 learned_addr learned my addr 172.21.15.61:0/367135782 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd590d52700 1 --2- 172.21.15.61:0/367135782 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c0b1210 0x7fd58c142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591553700 1 --2- 172.21.15.61:0/367135782 >> v2:172.21.15.111:3300/0 conn(0x7fd58c05c740 0x7fd58c098070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 -- 172.21.15.61:0/367135782 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c0b1210 msgr2=0x7fd58c142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 --2- 172.21.15.61:0/367135782 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c0b1210 0x7fd58c142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 -- 172.21.15.61:0/367135782 >> v2:172.21.15.111:3300/0 conn(0x7fd58c05c740 msgr2=0x7fd58c098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 --2- 172.21.15.61:0/367135782 >> v2:172.21.15.111:3300/0 conn(0x7fd58c05c740 0x7fd58c098070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 -- 172.21.15.61:0/367135782 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c140200 con 0x7fd58c05f340 2022-07-12T08:41:16.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd591d54700 1 --2- 172.21.15.61:0/367135782 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c05f340 0x7fd58c142b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fd58c0aadc0 tx=0x7fd584009290).ready entity=mon.0 client_cookie=9cc58b8967841e99 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd57bfff700 1 -- 172.21.15.61:0/367135782 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58400b830 con 0x7fd58c05f340 2022-07-12T08:41:16.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd57bfff700 1 -- 172.21.15.61:0/367135782 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58400b990 con 0x7fd58c05f340 2022-07-12T08:41:16.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd57bfff700 1 -- 172.21.15.61:0/367135782 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58400bb00 con 0x7fd58c05f340 2022-07-12T08:41:16.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- 172.21.15.61:0/367135782 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c05f340 msgr2=0x7fd58c142b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- 172.21.15.61:0/367135782 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c05f340 0x7fd58c142b30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fd58c0aadc0 tx=0x7fd584009290).stop 2022-07-12T08:41:16.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- 172.21.15.61:0/367135782 shutdown_connections 2022-07-12T08:41:16.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- 172.21.15.61:0/367135782 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c0b1210 0x7fd58c142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- 172.21.15.61:0/367135782 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c05f340 0x7fd58c142b30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 --2- 172.21.15.61:0/367135782 >> v2:172.21.15.111:3300/0 conn(0x7fd58c05c740 0x7fd58c098070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- 172.21.15.61:0/367135782 >> 172.21.15.61:0/367135782 conn(0x7fd58c0b1950 msgr2=0x7fd58c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:16.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- 172.21.15.61:0/367135782 shutdown_connections 2022-07-12T08:41:16.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- 172.21.15.61:0/367135782 wait complete. 2022-07-12T08:41:16.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 Processor -- start 2022-07-12T08:41:16.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.072+0000 7fd592555700 1 -- start start 2022-07-12T08:41:16.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd58c05c740 0x7fd58c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c05f340 0x7fd58c0ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 0x7fd58c0ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c140010 con 0x7fd58c0b1210 2022-07-12T08:41:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c142e30 con 0x7fd58c05c740 2022-07-12T08:41:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c1430d0 con 0x7fd58c05f340 2022-07-12T08:41:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 0x7fd58c0ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 0x7fd58c0ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52838/0 (socket says 172.21.15.61:52838) 2022-07-12T08:41:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 -- 172.21.15.61:0/3249906517 learned_addr learned my addr 172.21.15.61:0/3249906517 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd590d52700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c05f340 0x7fd58c0ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591553700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd58c05c740 0x7fd58c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 -- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c05f340 msgr2=0x7fd58c0ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c05f340 0x7fd58c0ac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 -- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd58c05c740 msgr2=0x7fd58c13f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd58c05c740 0x7fd58c13f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 -- 172.21.15.61:0/3249906517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd584009530 con 0x7fd58c0b1210 2022-07-12T08:41:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd591d54700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 0x7fd58c0ace90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd58c0aadc0 tx=0x7fd584016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58400b830 con 0x7fd58c0b1210 2022-07-12T08:41:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd584021670 con 0x7fd58c0b1210 2022-07-12T08:41:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c143370 con 0x7fd58c0b1210 2022-07-12T08:41:16.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd5840219d0 con 0x7fd58c0b1210 2022-07-12T08:41:16.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c0ac3c0 con 0x7fd58c0b1210 2022-07-12T08:41:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58c0903b0 con 0x7fd58c0b1210 2022-07-12T08:41:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.076+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd58400f070 con 0x7fd58c0b1210 2022-07-12T08:41:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.080+0000 7fd57a7fc700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd56c0539b0 0x7fd56c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.080+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd58406cff0 con 0x7fd58c0b1210 2022-07-12T08:41:16.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.080+0000 7fd591553700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd56c0539b0 0x7fd56c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.080+0000 7fd591553700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd56c0539b0 0x7fd56c055e70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd58800c000 tx=0x7fd588008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.084+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd58c0903b0 con 0x7fd58c0b1210 2022-07-12T08:41:16.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.212+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd58c0903b0 con 0x7fd58c0b1210 2022-07-12T08:41:16.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd57a7fc700 1 -- 172.21.15.61:0/3249906517 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fd58c0903b0 con 0x7fd58c0b1210 2022-07-12T08:41:16.220 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:16.220 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:16.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd56c0539b0 msgr2=0x7fd56c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd56c0539b0 0x7fd56c055e70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd58800c000 tx=0x7fd588008040).stop 2022-07-12T08:41:16.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 msgr2=0x7fd58c0ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 0x7fd58c0ace90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fd58c0aadc0 tx=0x7fd584016000).stop 2022-07-12T08:41:16.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 shutdown_connections 2022-07-12T08:41:16.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd58c0b1210 0x7fd58c0ace90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd58c05f340 0x7fd58c0ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd56c0539b0 0x7fd56c055e70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 --2- 172.21.15.61:0/3249906517 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd58c05c740 0x7fd58c13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 >> 172.21.15.61:0/3249906517 conn(0x7fd58c0b1950 msgr2=0x7fd58c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:16.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 shutdown_connections 2022-07-12T08:41:16.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.216+0000 7fd592555700 1 -- 172.21.15.61:0/3249906517 wait complete. 2022-07-12T08:41:16.228 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:16.229 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-07-12T08:41:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d549c700 1 Processor -- start 2022-07-12T08:41:16.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d549c700 1 -- start start 2022-07-12T08:41:16.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d549c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00af280 0x7f05d0060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d549c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d00aecb0 con 0x7f05d00b3f60 2022-07-12T08:41:16.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d549c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d01435c0 con 0x7f05d009bce0 2022-07-12T08:41:16.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d549c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d00afd70 con 0x7f05d00af280 2022-07-12T08:41:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d4c9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f05d00b3f60 0x7f05d00b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41064/0 (socket says 172.21.15.61:41064) 2022-07-12T08:41:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05d4c9b700 1 -- 172.21.15.61:0/2042885690 learned_addr learned my addr 172.21.15.61:0/2042885690 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05cffff700 1 --2- 172.21.15.61:0/2042885690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00af280 0x7f05d0060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05ceffd700 1 -- 172.21.15.61:0/2042885690 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158284883 0 0) 0x7f05d00aecb0 con 0x7f05d00b3f60 2022-07-12T08:41:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05ceffd700 1 -- 172.21.15.61:0/2042885690 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05bc003260 con 0x7f05d00b3f60 2022-07-12T08:41:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05cffff700 1 -- 172.21.15.61:0/2042885690 >> v1:172.21.15.111:6789/0 conn(0x7f05d009bce0 legacy=0x7f05d00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05cffff700 1 -- 172.21.15.61:0/2042885690 >> v1:172.21.15.61:6789/0 conn(0x7f05d00b3f60 legacy=0x7f05d00b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05cffff700 1 -- 172.21.15.61:0/2042885690 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d0143860 con 0x7f05d00af280 2022-07-12T08:41:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05cffff700 1 --2- 172.21.15.61:0/2042885690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00af280 0x7f05d0060680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f05c00032f0 tx=0x7f05c00091a0).ready entity=mon.2 client_cookie=67d9230ea1b64075 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05ceffd700 1 -- 172.21.15.61:0/2042885690 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c000b6c0 con 0x7f05d00af280 2022-07-12T08:41:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05ceffd700 1 -- 172.21.15.61:0/2042885690 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c000b820 con 0x7f05d00af280 2022-07-12T08:41:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.332+0000 7f05ceffd700 1 -- 172.21.15.61:0/2042885690 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c000b990 con 0x7f05d00af280 2022-07-12T08:41:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- 172.21.15.61:0/2042885690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00af280 msgr2=0x7f05d0060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 --2- 172.21.15.61:0/2042885690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00af280 0x7f05d0060680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f05c00032f0 tx=0x7f05c00091a0).stop 2022-07-12T08:41:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- 172.21.15.61:0/2042885690 shutdown_connections 2022-07-12T08:41:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 --2- 172.21.15.61:0/2042885690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00af280 0x7f05d0060680 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- 172.21.15.61:0/2042885690 >> 172.21.15.61:0/2042885690 conn(0x7f05d00b92c0 msgr2=0x7f05d00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- 172.21.15.61:0/2042885690 shutdown_connections 2022-07-12T08:41:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- 172.21.15.61:0/2042885690 wait complete. 2022-07-12T08:41:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 Processor -- start 2022-07-12T08:41:16.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- start start 2022-07-12T08:41:16.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 0x7f05d0156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d00af280 0x7f05d00b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00b3f60 0x7f05d01485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d009e150 con 0x7f05d009bce0 2022-07-12T08:41:16.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d009e380 con 0x7f05d00af280 2022-07-12T08:41:16.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d009e620 con 0x7f05d00b3f60 2022-07-12T08:41:16.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 0x7f05d0156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 0x7f05d0156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52848/0 (socket says 172.21.15.61:52848) 2022-07-12T08:41:16.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 -- 172.21.15.61:0/458594521 learned_addr learned my addr 172.21.15.61:0/458594521 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:16.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d4c9b700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00b3f60 0x7f05d01485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cf7fe700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d00af280 0x7f05d00b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 -- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00b3f60 msgr2=0x7f05d01485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00b3f60 0x7f05d01485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 -- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d00af280 msgr2=0x7f05d00b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d00af280 0x7f05d00b3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 -- 172.21.15.61:0/458594521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c0009420 con 0x7f05d009bce0 2022-07-12T08:41:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d4c9b700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00b3f60 0x7f05d01485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cf7fe700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d00af280 0x7f05d00b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cffff700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 0x7f05d0156190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f05c00032f0 tx=0x7f05c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c000b6c0 con 0x7f05d009bce0 2022-07-12T08:41:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c0005e70 con 0x7f05d009bce0 2022-07-12T08:41:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d009ed40 con 0x7f05d009bce0 2022-07-12T08:41:16.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.336+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c0006220 con 0x7f05d009bce0 2022-07-12T08:41:16.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d0151ff0 con 0x7f05d009bce0 2022-07-12T08:41:16.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d009e7e0 con 0x7f05d009bce0 2022-07-12T08:41:16.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f05c000f070 con 0x7f05d009bce0 2022-07-12T08:41:16.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05cd7fa700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05b0053af0 0x7f05b0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f05c006d330 con 0x7f05d009bce0 2022-07-12T08:41:16.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05cf7fe700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05b0053af0 0x7f05b0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.340+0000 7f05cf7fe700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05b0053af0 0x7f05b0055fb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f05c400b000 tx=0x7f05c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.344+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05c0044290 con 0x7f05d009bce0 2022-07-12T08:41:16.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.468+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05d006b2b0 con 0x7f05d009bce0 2022-07-12T08:41:16.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05cd7fa700 1 -- 172.21.15.61:0/458594521 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f05c0015020 con 0x7f05d009bce0 2022-07-12T08:41:16.475 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:16.485 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:16.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05b0053af0 msgr2=0x7f05b0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05b0053af0 0x7f05b0055fb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f05c400b000 tx=0x7f05c4009040).stop 2022-07-12T08:41:16.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 msgr2=0x7f05d0156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 0x7f05d0156190 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f05c00032f0 tx=0x7f05c0016000).stop 2022-07-12T08:41:16.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 shutdown_connections 2022-07-12T08:41:16.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05d00b3f60 0x7f05d01485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05b0053af0 0x7f05b0055fb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05d00af280 0x7f05d00b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 --2- 172.21.15.61:0/458594521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05d009bce0 0x7f05d0156190 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 >> 172.21.15.61:0/458594521 conn(0x7f05d00b92c0 msgr2=0x7f05d00af660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:16.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 shutdown_connections 2022-07-12T08:41:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.472+0000 7f05d549c700 1 -- 172.21.15.61:0/458594521 wait complete. 2022-07-12T08:41:16.521 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:16.523 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:16.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.648+0000 7ff551ac1700 1 Processor -- start 2022-07-12T08:41:16.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.648+0000 7ff551ac1700 1 -- start start 2022-07-12T08:41:16.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff54c0b2c70 0x7ff54c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c0b1910 con 0x7ff54c140140 2022-07-12T08:41:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c066350 con 0x7ff54c060430 2022-07-12T08:41:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c140d00 con 0x7ff54c0b2c70 2022-07-12T08:41:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5512c0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff54c140140 0x7ff54c13ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41078/0 (socket says 172.21.15.61:41078) 2022-07-12T08:41:16.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5512c0700 1 -- 172.21.15.61:0/653286625 learned_addr learned my addr 172.21.15.61:0/653286625 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:16.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff550abf700 1 --2- 172.21.15.61:0/653286625 >> v2:172.21.15.111:3300/0 conn(0x7ff54c0b2c70 0x7ff54c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5437fe700 1 -- 172.21.15.61:0/653286625 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654473203 0 0) 0x7ff54c0b1910 con 0x7ff54c140140 2022-07-12T08:41:16.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5437fe700 1 -- 172.21.15.61:0/653286625 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538003260 con 0x7ff54c140140 2022-07-12T08:41:16.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff550abf700 1 -- 172.21.15.61:0/653286625 >> v1:172.21.15.111:6789/0 conn(0x7ff54c060430 legacy=0x7ff54c13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff550abf700 1 -- 172.21.15.61:0/653286625 >> v1:172.21.15.61:6789/0 conn(0x7ff54c140140 legacy=0x7ff54c13ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff550abf700 1 -- 172.21.15.61:0/653286625 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0665f0 con 0x7ff54c0b2c70 2022-07-12T08:41:16.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff550abf700 1 --2- 172.21.15.61:0/653286625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0b2c70 0x7ff54c0b1d40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff548002f70 tx=0x7ff548006df0).ready entity=mon.1 client_cookie=3647d6cb73aef44d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5437fe700 1 -- 172.21.15.61:0/653286625 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54800d610 con 0x7ff54c0b2c70 2022-07-12T08:41:16.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5437fe700 1 -- 172.21.15.61:0/653286625 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff548000e90 con 0x7ff54c0b2c70 2022-07-12T08:41:16.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff5437fe700 1 -- 172.21.15.61:0/653286625 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54800d960 con 0x7ff54c0b2c70 2022-07-12T08:41:16.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- 172.21.15.61:0/653286625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0b2c70 msgr2=0x7ff54c0b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 --2- 172.21.15.61:0/653286625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0b2c70 0x7ff54c0b1d40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7ff548002f70 tx=0x7ff548006df0).stop 2022-07-12T08:41:16.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- 172.21.15.61:0/653286625 shutdown_connections 2022-07-12T08:41:16.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 --2- 172.21.15.61:0/653286625 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c0b2c70 0x7ff54c0b1d40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- 172.21.15.61:0/653286625 >> 172.21.15.61:0/653286625 conn(0x7ff54c063ad0 msgr2=0x7ff54c063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:16.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- 172.21.15.61:0/653286625 shutdown_connections 2022-07-12T08:41:16.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.652+0000 7ff551ac1700 1 -- 172.21.15.61:0/653286625 wait complete. 2022-07-12T08:41:16.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 Processor -- start 2022-07-12T08:41:16.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 -- start start 2022-07-12T08:41:16.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 0x7ff54c13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff54c0b2c70 0x7ff54c13bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c140140 0x7ff54c13e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c13f2d0 con 0x7ff54c060430 2022-07-12T08:41:16.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0ae780 con 0x7ff54c140140 2022-07-12T08:41:16.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff551ac1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0af0c0 con 0x7ff54c0b2c70 2022-07-12T08:41:16.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff550abf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 0x7ff54c13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff550abf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 0x7ff54c13cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52864/0 (socket says 172.21.15.61:52864) 2022-07-12T08:41:16.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff550abf700 1 -- 172.21.15.61:0/553235178 learned_addr learned my addr 172.21.15.61:0/553235178 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:16.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff543fff700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff54c0b2c70 0x7ff54c13bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.656+0000 7ff5512c0700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c140140 0x7ff54c13e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff550abf700 1 -- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff54c0b2c70 msgr2=0x7ff54c13bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff550abf700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff54c0b2c70 0x7ff54c13bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff550abf700 1 -- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c140140 msgr2=0x7ff54c13e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff550abf700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c140140 0x7ff54c13e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff550abf700 1 -- 172.21.15.61:0/553235178 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54800b180 con 0x7ff54c060430 2022-07-12T08:41:16.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff550abf700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 0x7ff54c13cca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff54c13e1c0 tx=0x7ff548014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff548016070 con 0x7ff54c060430 2022-07-12T08:41:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54800de90 con 0x7ff54c060430 2022-07-12T08:41:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff548013d60 con 0x7ff54c060430 2022-07-12T08:41:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c0aec50 con 0x7ff54c060430 2022-07-12T08:41:16.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c13e810 con 0x7ff54c060430 2022-07-12T08:41:16.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff548025030 con 0x7ff54c060430 2022-07-12T08:41:16.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c0b0380 con 0x7ff54c060430 2022-07-12T08:41:16.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff541ffb700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff52c053920 0x7ff52c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:16.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.660+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff54806cdf0 con 0x7ff54c060430 2022-07-12T08:41:16.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.664+0000 7ff543fff700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff52c053920 0x7ff52c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:16.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.664+0000 7ff543fff700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff52c053920 0x7ff52c055de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff53c00d000 tx=0x7ff53c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:16.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.668+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff54c0b0380 con 0x7ff54c060430 2022-07-12T08:41:16.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.792+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff54c0b0380 con 0x7ff54c060430 2022-07-12T08:41:16.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.792+0000 7ff541ffb700 1 -- 172.21.15.61:0/553235178 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff54c0b0380 con 0x7ff54c060430 2022-07-12T08:41:16.828 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:16.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff52c053920 msgr2=0x7ff52c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff52c053920 0x7ff52c055de0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7ff53c00d000 tx=0x7ff53c000bc0).stop 2022-07-12T08:41:16.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 msgr2=0x7ff54c13cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:16.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 0x7ff54c13cca0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7ff54c13e1c0 tx=0x7ff548014000).stop 2022-07-12T08:41:16.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 shutdown_connections 2022-07-12T08:41:16.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff52c053920 0x7ff52c055de0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff54c140140 0x7ff54c13e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff54c0b2c70 0x7ff54c13bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 --2- 172.21.15.61:0/553235178 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54c060430 0x7ff54c13cca0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 >> 172.21.15.61:0/553235178 conn(0x7ff54c063ad0 msgr2=0x7ff54c065920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 shutdown_connections 2022-07-12T08:41:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:16.796+0000 7ff551ac1700 1 -- 172.21.15.61:0/553235178 wait complete. 2022-07-12T08:41:16.891 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:17.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 Processor -- start 2022-07-12T08:41:17.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- start start 2022-07-12T08:41:17.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea6c0b0120 0x7fea6c144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c144640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c0b0500 con 0x7fea6c14aad0 2022-07-12T08:41:17.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c0b1040 con 0x7fea6c0b0120 2022-07-12T08:41:17.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c05d0b0 con 0x7fea6c0ab730 2022-07-12T08:41:17.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6ad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fea6c14aad0 0x7fea6c14af60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41090/0 (socket says 172.21.15.61:41090) 2022-07-12T08:41:17.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6ad9d700 1 -- 172.21.15.61:0/2071635232 learned_addr learned my addr 172.21.15.61:0/2071635232 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:17.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea6c0b0120 0x7fea6c144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea69d9b700 1 --2- 172.21.15.61:0/2071635232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6959a700 1 -- 172.21.15.61:0/2071635232 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651112100 0 0) 0x7fea6c0b0500 con 0x7fea6c14aad0 2022-07-12T08:41:17.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6959a700 1 -- 172.21.15.61:0/2071635232 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea50003260 con 0x7fea6c14aad0 2022-07-12T08:41:17.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6a59c700 1 -- 172.21.15.61:0/2071635232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 msgr2=0x7fea6c144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6a59c700 1 --2- 172.21.15.61:0/2071635232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6959a700 1 -- 172.21.15.61:0/2071635232 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973252395 0 0) 0x7fea50003260 con 0x7fea6c14aad0 2022-07-12T08:41:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6a59c700 1 -- 172.21.15.61:0/2071635232 >> v1:172.21.15.61:6789/0 conn(0x7fea6c14aad0 legacy=0x7fea6c14af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6a59c700 1 -- 172.21.15.61:0/2071635232 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c05d350 con 0x7fea6c0b0120 2022-07-12T08:41:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6a59c700 1 --2- 172.21.15.61:0/2071635232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c0b0120 0x7fea6c144150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fea64001800 tx=0x7fea60006df0).ready entity=mon.0 client_cookie=6be917cef89185c5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6959a700 1 -- 172.21.15.61:0/2071635232 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea64001660 con 0x7fea6c0b0120 2022-07-12T08:41:17.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6959a700 1 -- 172.21.15.61:0/2071635232 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea60000e90 con 0x7fea6c0b0120 2022-07-12T08:41:17.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea6959a700 1 -- 172.21.15.61:0/2071635232 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea60012460 con 0x7fea6c0b0120 2022-07-12T08:41:17.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- 172.21.15.61:0/2071635232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c0b0120 msgr2=0x7fea6c144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 --2- 172.21.15.61:0/2071635232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c0b0120 0x7fea6c144150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fea64001800 tx=0x7fea60006df0).stop 2022-07-12T08:41:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- 172.21.15.61:0/2071635232 shutdown_connections 2022-07-12T08:41:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 --2- 172.21.15.61:0/2071635232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 --2- 172.21.15.61:0/2071635232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c0b0120 0x7fea6c144150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- 172.21.15.61:0/2071635232 >> 172.21.15.61:0/2071635232 conn(0x7fea6c0b5290 msgr2=0x7fea6c0b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- 172.21.15.61:0/2071635232 shutdown_connections 2022-07-12T08:41:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.008+0000 7fea70fab700 1 -- 172.21.15.61:0/2071635232 wait complete. 2022-07-12T08:41:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 Processor -- start 2022-07-12T08:41:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 -- start start 2022-07-12T08:41:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c0b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c0b0120 0x7fea6c0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 0x7fea6c159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c064850 con 0x7fea6c14aad0 2022-07-12T08:41:17.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c064ab0 con 0x7fea6c0b0120 2022-07-12T08:41:17.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c159970 con 0x7fea6c0ab730 2022-07-12T08:41:17.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 0x7fea6c159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 0x7fea6c159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52880/0 (socket says 172.21.15.61:52880) 2022-07-12T08:41:17.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 -- 172.21.15.61:0/2354792309 learned_addr learned my addr 172.21.15.61:0/2354792309 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea69d9b700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c0b0120 0x7fea6c0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6a59c700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c0b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 -- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 msgr2=0x7fea6c0b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c0b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 -- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c0b0120 msgr2=0x7fea6c0ad9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c0b0120 0x7fea6c0ad9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 -- 172.21.15.61:0/2354792309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6000b180 con 0x7fea6c14aad0 2022-07-12T08:41:17.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6a59c700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c0b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:17.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea6ad9d700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 0x7fea6c159e10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fea640018c0 tx=0x7fea64001de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea64007170 con 0x7fea6c14aad0 2022-07-12T08:41:17.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea640072d0 con 0x7fea6c14aad0 2022-07-12T08:41:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c0adec0 con 0x7fea6c14aad0 2022-07-12T08:41:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea64004710 con 0x7fea6c14aad0 2022-07-12T08:41:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.012+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c0ac8f0 con 0x7fea6c14aad0 2022-07-12T08:41:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.016+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c0ad810 con 0x7fea6c14aad0 2022-07-12T08:41:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.016+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fea6401b070 con 0x7fea6c14aad0 2022-07-12T08:41:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.016+0000 7fea5b7fe700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea48053ad0 0x7fea48055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.016+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fea6406b4a0 con 0x7fea6c14aad0 2022-07-12T08:41:17.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.016+0000 7fea6a59c700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea48053ad0 0x7fea48055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.016+0000 7fea6a59c700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea48053ad0 0x7fea48055f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fea60018000 tx=0x7fea60014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.020+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea6c0ad810 con 0x7fea6c14aad0 2022-07-12T08:41:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.148+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fea6c0ad810 con 0x7fea6c14aad0 2022-07-12T08:41:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.148+0000 7fea5b7fe700 1 -- 172.21.15.61:0/2354792309 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fea6c0ad810 con 0x7fea6c14aad0 2022-07-12T08:41:17.148 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:17.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea48053ad0 msgr2=0x7fea48055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea48053ad0 0x7fea48055f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fea60018000 tx=0x7fea60014000).stop 2022-07-12T08:41:17.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 msgr2=0x7fea6c159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 0x7fea6c159e10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fea640018c0 tx=0x7fea64001de0).stop 2022-07-12T08:41:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 shutdown_connections 2022-07-12T08:41:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6c14aad0 0x7fea6c159e10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea48053ad0 0x7fea48055f90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6c0b0120 0x7fea6c0ad9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 --2- 172.21.15.61:0/2354792309 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea6c0ab730 0x7fea6c0b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 >> 172.21.15.61:0/2354792309 conn(0x7fea6c0b5290 msgr2=0x7fea6c0aad20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 shutdown_connections 2022-07-12T08:41:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.152+0000 7fea70fab700 1 -- 172.21.15.61:0/2354792309 wait complete. 2022-07-12T08:41:17.161 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:17.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 Processor -- start 2022-07-12T08:41:17.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 -- start start 2022-07-12T08:41:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22a40af4b0 0x7f22a40af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a4142120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f22a4142610 0x7f22a4146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a4143610 con 0x7f22a40af4b0 2022-07-12T08:41:17.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a4143840 con 0x7f22a4142610 2022-07-12T08:41:17.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22ab116700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a40afef0 con 0x7f22a4060b50 2022-07-12T08:41:17.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22aa114700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22a40af4b0 0x7f22a40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22aa114700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22a40af4b0 0x7f22a40af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52884/0 (socket says 172.21.15.61:52884) 2022-07-12T08:41:17.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22aa114700 1 -- 172.21.15.61:0/4077383465 learned_addr learned my addr 172.21.15.61:0/4077383465 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22a9913700 1 --2- 172.21.15.61:0/4077383465 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a4142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.276+0000 7f22aa915700 1 --2- 172.21.15.61:0/4077383465 >> v2:172.21.15.111:3300/0 conn(0x7f22a4142610 0x7f22a4146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22aa114700 1 -- 172.21.15.61:0/4077383465 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 msgr2=0x7f22a4142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:17.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22aa114700 1 --2- 172.21.15.61:0/4077383465 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a4142120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22aa114700 1 -- 172.21.15.61:0/4077383465 >> v2:172.21.15.111:3300/0 conn(0x7f22a4142610 msgr2=0x7f22a4146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:17.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22aa114700 1 --2- 172.21.15.61:0/4077383465 >> v2:172.21.15.111:3300/0 conn(0x7f22a4142610 0x7f22a4146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22aa114700 1 -- 172.21.15.61:0/4077383465 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40b3c60 con 0x7f22a40af4b0 2022-07-12T08:41:17.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22aa114700 1 --2- 172.21.15.61:0/4077383465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a40af890 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2298002f70 tx=0x7f2298006eb0).ready entity=mon.0 client_cookie=36cb0a08bcd43d1b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22a9112700 1 -- 172.21.15.61:0/4077383465 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f229800d570 con 0x7f22a40af4b0 2022-07-12T08:41:17.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22a9112700 1 -- 172.21.15.61:0/4077383465 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2298000e90 con 0x7f22a40af4b0 2022-07-12T08:41:17.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22a9112700 1 -- 172.21.15.61:0/4077383465 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f229800d900 con 0x7f22a40af4b0 2022-07-12T08:41:17.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 -- 172.21.15.61:0/4077383465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 msgr2=0x7f22a40af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 --2- 172.21.15.61:0/4077383465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a40af890 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f2298002f70 tx=0x7f2298006eb0).stop 2022-07-12T08:41:17.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 -- 172.21.15.61:0/4077383465 shutdown_connections 2022-07-12T08:41:17.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 --2- 172.21.15.61:0/4077383465 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a4142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 --2- 172.21.15.61:0/4077383465 >> v2:172.21.15.111:3300/0 conn(0x7f22a4142610 0x7f22a4146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 --2- 172.21.15.61:0/4077383465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a40af890 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 -- 172.21.15.61:0/4077383465 >> 172.21.15.61:0/4077383465 conn(0x7f22a40b92c0 msgr2=0x7f22a40b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 -- 172.21.15.61:0/4077383465 shutdown_connections 2022-07-12T08:41:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 -- 172.21.15.61:0/4077383465 wait complete. 2022-07-12T08:41:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 Processor -- start 2022-07-12T08:41:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.280+0000 7f22ab116700 1 -- start start 2022-07-12T08:41:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a40981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a409d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22a4142610 0x7f22a4159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4159570 con 0x7f22a40af4b0 2022-07-12T08:41:17.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4159810 con 0x7f22a4142610 2022-07-12T08:41:17.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4159ab0 con 0x7f22a4060b50 2022-07-12T08:41:17.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a409d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a409d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52892/0 (socket says 172.21.15.61:52892) 2022-07-12T08:41:17.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 -- 172.21.15.61:0/1587384296 learned_addr learned my addr 172.21.15.61:0/1587384296 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22aa114700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a40981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22aa915700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22a4142610 0x7f22a4159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 -- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 msgr2=0x7f22a40981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a40981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 -- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22a4142610 msgr2=0x7f22a4159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22a4142610 0x7f22a4159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 -- 172.21.15.61:0/1587384296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229800b1e0 con 0x7f22a40af4b0 2022-07-12T08:41:17.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a9913700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a409d9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f229c006e00 tx=0x7f229c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f229c00ee60 con 0x7f22a40af4b0 2022-07-12T08:41:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229c00c850 con 0x7f22a40af4b0 2022-07-12T08:41:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a40992a0 con 0x7f22a40af4b0 2022-07-12T08:41:17.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f229c00cbb0 con 0x7f22a40af4b0 2022-07-12T08:41:17.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a4155ed0 con 0x7f22a40af4b0 2022-07-12T08:41:17.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a4159c70 con 0x7f22a40af4b0 2022-07-12T08:41:17.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.284+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f229c022070 con 0x7f22a40af4b0 2022-07-12T08:41:17.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.288+0000 7f22a37fe700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2284053a40 0x7f2284055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.288+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f229c06c550 con 0x7f22a40af4b0 2022-07-12T08:41:17.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.288+0000 7f22aa114700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2284053a40 0x7f2284055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.288+0000 7f22aa114700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2284053a40 0x7f2284055f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f229801c000 tx=0x7f2298015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.292+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f229c0434b0 con 0x7f22a40af4b0 2022-07-12T08:41:17.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.416+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f22a4057460 con 0x7f22a40af4b0 2022-07-12T08:41:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22a37fe700 1 -- 172.21.15.61:0/1587384296 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f229c016020 con 0x7f22a40af4b0 2022-07-12T08:41:17.427 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:17.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2284053a40 msgr2=0x7f2284055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2284053a40 0x7f2284055f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f229801c000 tx=0x7f2298015000).stop 2022-07-12T08:41:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 msgr2=0x7f22a409d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a409d9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f229c006e00 tx=0x7f229c0091d0).stop 2022-07-12T08:41:17.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 shutdown_connections 2022-07-12T08:41:17.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2284053a40 0x7f2284055f00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22a4142610 0x7f22a4159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22a40af4b0 0x7f22a409d9b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 --2- 172.21.15.61:0/1587384296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22a4060b50 0x7f22a40981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 >> 172.21.15.61:0/1587384296 conn(0x7f22a40b92c0 msgr2=0x7f22a41482b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 shutdown_connections 2022-07-12T08:41:17.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.420+0000 7f22ab116700 1 -- 172.21.15.61:0/1587384296 wait complete. 2022-07-12T08:41:17.454 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:17.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 Processor -- start 2022-07-12T08:41:17.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 -- start start 2022-07-12T08:41:17.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f92800b7eb0 0x7f9280140d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f92800ad120 0x7f928005c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f928005c6a0 0x7f9280066ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92800b8290 con 0x7f92800b7eb0 2022-07-12T08:41:17.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92800acb60 con 0x7f92800ad120 2022-07-12T08:41:17.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f92868bb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92800ace00 con 0x7f928005c6a0 2022-07-12T08:41:17.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f92800b7eb0 0x7f9280140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f92800b7eb0 0x7f9280140d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52898/0 (socket says 172.21.15.61:52898) 2022-07-12T08:41:17.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 -- 172.21.15.61:0/2866136031 learned_addr learned my addr 172.21.15.61:0/2866136031 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927f7fe700 1 --2- 172.21.15.61:0/2866136031 >> v2:172.21.15.111:3300/0 conn(0x7f92800ad120 0x7f928005c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f9284e58700 1 --2- 172.21.15.61:0/2866136031 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f928005c6a0 0x7f9280066ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 -- 172.21.15.61:0/2866136031 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f928005c6a0 msgr2=0x7f9280066ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 --2- 172.21.15.61:0/2866136031 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f928005c6a0 0x7f9280066ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 -- 172.21.15.61:0/2866136031 >> v2:172.21.15.111:3300/0 conn(0x7f92800ad120 msgr2=0x7f928005c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:17.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 --2- 172.21.15.61:0/2866136031 >> v2:172.21.15.111:3300/0 conn(0x7f92800ad120 0x7f928005c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 -- 172.21.15.61:0/2866136031 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928005bd10 con 0x7f92800b7eb0 2022-07-12T08:41:17.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927ffff700 1 --2- 172.21.15.61:0/2866136031 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800b7eb0 0x7f9280140d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9270002f70 tx=0x7f9270006e50).ready entity=mon.0 client_cookie=2211b6c369a5439c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927effd700 1 -- 172.21.15.61:0/2866136031 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f927000d570 con 0x7f92800b7eb0 2022-07-12T08:41:17.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927effd700 1 -- 172.21.15.61:0/2866136031 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9270000e90 con 0x7f92800b7eb0 2022-07-12T08:41:17.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.572+0000 7f927effd700 1 -- 172.21.15.61:0/2866136031 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f927000d900 con 0x7f92800b7eb0 2022-07-12T08:41:17.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 -- 172.21.15.61:0/2866136031 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800b7eb0 msgr2=0x7f9280140d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 --2- 172.21.15.61:0/2866136031 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800b7eb0 0x7f9280140d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9270002f70 tx=0x7f9270006e50).stop 2022-07-12T08:41:17.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 -- 172.21.15.61:0/2866136031 shutdown_connections 2022-07-12T08:41:17.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 --2- 172.21.15.61:0/2866136031 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f928005c6a0 0x7f9280066ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 --2- 172.21.15.61:0/2866136031 >> v2:172.21.15.111:3300/0 conn(0x7f92800ad120 0x7f928005c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 --2- 172.21.15.61:0/2866136031 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800b7eb0 0x7f9280140d40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 -- 172.21.15.61:0/2866136031 >> 172.21.15.61:0/2866136031 conn(0x7f9280065850 msgr2=0x7f92800b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 -- 172.21.15.61:0/2866136031 shutdown_connections 2022-07-12T08:41:17.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 -- 172.21.15.61:0/2866136031 wait complete. 2022-07-12T08:41:17.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 Processor -- start 2022-07-12T08:41:17.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.576+0000 7f92868bb700 1 -- start start 2022-07-12T08:41:17.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f928005c6a0 0x7f92800b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 0x7f9280142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f92800b7eb0 0x7f9280110a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92800aee80 con 0x7f92800ad120 2022-07-12T08:41:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92800ae940 con 0x7f928005c6a0 2022-07-12T08:41:17.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92800aebe0 con 0x7f92800b7eb0 2022-07-12T08:41:17.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 0x7f9280142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 0x7f9280142590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52906/0 (socket says 172.21.15.61:52906) 2022-07-12T08:41:17.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 -- 172.21.15.61:0/2439815137 learned_addr learned my addr 172.21.15.61:0/2439815137 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927ffff700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f928005c6a0 0x7f92800b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f9284e58700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f92800b7eb0 0x7f9280110a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 -- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f92800b7eb0 msgr2=0x7f9280110a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f92800b7eb0 0x7f9280110a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 -- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f928005c6a0 msgr2=0x7f92800b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f928005c6a0 0x7f92800b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 -- 172.21.15.61:0/2439815137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927000b1e0 con 0x7f92800ad120 2022-07-12T08:41:17.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f9284e58700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f92800b7eb0 0x7f9280110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:17.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927f7fe700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 0x7f9280142590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f92740030b0 tx=0x7f9274008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9274011780 con 0x7f92800ad120 2022-07-12T08:41:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92740118e0 con 0x7f92800ad120 2022-07-12T08:41:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9280143450 con 0x7f92800ad120 2022-07-12T08:41:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9274011c40 con 0x7f92800ad120 2022-07-12T08:41:17.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92800af460 con 0x7f92800ad120 2022-07-12T08:41:17.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.580+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9280057460 con 0x7f92800ad120 2022-07-12T08:41:17.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.584+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9274012420 con 0x7f92800ad120 2022-07-12T08:41:17.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.584+0000 7f927d7fa700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9260053af0 0x7f9260055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.584+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f927406be20 con 0x7f92800ad120 2022-07-12T08:41:17.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.584+0000 7f927ffff700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9260053af0 0x7f9260055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.584+0000 7f927ffff700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9260053af0 0x7f9260055fb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f927001c000 tx=0x7f9270015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.592+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9274042b90 con 0x7f92800ad120 2022-07-12T08:41:17.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.720+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92800447a0 con 0x7f92800ad120 2022-07-12T08:41:17.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.720+0000 7f927d7fa700 1 -- 172.21.15.61:0/2439815137 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f92740461b0 con 0x7f92800ad120 2022-07-12T08:41:17.726 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:17.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9260053af0 msgr2=0x7f9260055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9260053af0 0x7f9260055fb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f927001c000 tx=0x7f9270015000).stop 2022-07-12T08:41:17.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 msgr2=0x7f9280142590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 0x7f9280142590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f92740030b0 tx=0x7f9274008040).stop 2022-07-12T08:41:17.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 shutdown_connections 2022-07-12T08:41:17.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f92800b7eb0 0x7f9280110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92800ad120 0x7f9280142590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9260053af0 0x7f9260055fb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 --2- 172.21.15.61:0/2439815137 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f928005c6a0 0x7f92800b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 >> 172.21.15.61:0/2439815137 conn(0x7f9280065850 msgr2=0x7f92800b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 shutdown_connections 2022-07-12T08:41:17.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.724+0000 7f92868bb700 1 -- 172.21.15.61:0/2439815137 wait complete. 2022-07-12T08:41:17.741 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-07-12T08:41:17.742 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-07-12T08:41:17.742 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-07-12T08:41:17.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 Processor -- start 2022-07-12T08:41:17.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 -- start start 2022-07-12T08:41:17.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4c2c0ab290 0x7f4c2c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c0afa60 0x7f4c2c0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0abc70 con 0x7f4c2c05f140 2022-07-12T08:41:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c05f580 con 0x7f4c2c0ab290 2022-07-12T08:41:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c314a0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c05c720 con 0x7f4c2c0afa60 2022-07-12T08:41:17.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4c2c0ab290 0x7f4c2c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4c2c0ab290 0x7f4c2c0ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52912/0 (socket says 172.21.15.61:52912) 2022-07-12T08:41:17.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 -- 172.21.15.61:0/3838956868 learned_addr learned my addr 172.21.15.61:0/3838956868 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c30c9f700 1 --2- 172.21.15.61:0/3838956868 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c0afa60 0x7f4c2c0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 -- 172.21.15.61:0/3838956868 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c0afa60 msgr2=0x7f4c2c0aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:17.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 --2- 172.21.15.61:0/3838956868 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c0afa60 0x7f4c2c0aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 -- 172.21.15.61:0/3838956868 >> v1:172.21.15.111:6789/0 conn(0x7f4c2c05f140 legacy=0x7f4c2c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 -- 172.21.15.61:0/3838956868 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c05c9c0 con 0x7f4c2c0ab290 2022-07-12T08:41:17.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.848+0000 7f4c2bfff700 1 --2- 172.21.15.61:0/3838956868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c0ab670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f4c240032f0 tx=0x7f4c24009290).ready entity=mon.0 client_cookie=2a907652ba4f1a4e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c2affd700 1 -- 172.21.15.61:0/3838956868 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c2400b8a0 con 0x7f4c2c0ab290 2022-07-12T08:41:17.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c2affd700 1 -- 172.21.15.61:0/3838956868 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2400ba00 con 0x7f4c2c0ab290 2022-07-12T08:41:17.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c2affd700 1 -- 172.21.15.61:0/3838956868 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c2400bb70 con 0x7f4c2c0ab290 2022-07-12T08:41:17.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 -- 172.21.15.61:0/3838956868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 msgr2=0x7f4c2c0ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 --2- 172.21.15.61:0/3838956868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c0ab670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f4c240032f0 tx=0x7f4c24009290).stop 2022-07-12T08:41:17.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 -- 172.21.15.61:0/3838956868 shutdown_connections 2022-07-12T08:41:17.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 --2- 172.21.15.61:0/3838956868 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c0afa60 0x7f4c2c0aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 --2- 172.21.15.61:0/3838956868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c0ab670 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 -- 172.21.15.61:0/3838956868 >> 172.21.15.61:0/3838956868 conn(0x7f4c2c0b1950 msgr2=0x7f4c2c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:17.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 -- 172.21.15.61:0/3838956868 shutdown_connections 2022-07-12T08:41:17.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 -- 172.21.15.61:0/3838956868 wait complete. 2022-07-12T08:41:17.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 Processor -- start 2022-07-12T08:41:17.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.852+0000 7f4c314a0700 1 -- start start 2022-07-12T08:41:17.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c05f140 0x7f4c2c1472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c066720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c0afa60 0x7f4c2c159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c065260 con 0x7f4c2c0ab290 2022-07-12T08:41:17.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0af110 con 0x7f4c2c0afa60 2022-07-12T08:41:17.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c065fe0 con 0x7f4c2c05f140 2022-07-12T08:41:17.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c066720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c066720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52920/0 (socket says 172.21.15.61:52920) 2022-07-12T08:41:17.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 -- 172.21.15.61:0/1396062484 learned_addr learned my addr 172.21.15.61:0/1396062484 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:17.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c30c9f700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c0afa60 0x7f4c2c159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2bfff700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c05f140 0x7f4c2c1472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 -- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c05f140 msgr2=0x7f4c2c1472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c05f140 0x7f4c2c1472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 -- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c0afa60 msgr2=0x7f4c2c159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c0afa60 0x7f4c2c159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 -- 172.21.15.61:0/1396062484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24009530 con 0x7f4c2c0ab290 2022-07-12T08:41:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c2b7fe700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c066720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f4c2c0615d0 tx=0x7f4c20007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c20012400 con 0x7f4c2c0ab290 2022-07-12T08:41:17.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c20012560 con 0x7f4c2c0ab290 2022-07-12T08:41:17.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c066280 con 0x7f4c2c0ab290 2022-07-12T08:41:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c200128c0 con 0x7f4c2c0ab290 2022-07-12T08:41:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.856+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c0aec70 con 0x7f4c2c0ab290 2022-07-12T08:41:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.860+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c0aef80 con 0x7f4c2c0ab290 2022-07-12T08:41:17.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.860+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4c20019030 con 0x7f4c2c0ab290 2022-07-12T08:41:17.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.860+0000 7f4c297fa700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4c0c053a40 0x7f4c0c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.860+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4c2006b1b0 con 0x7f4c2c0ab290 2022-07-12T08:41:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.860+0000 7f4c2bfff700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4c0c053a40 0x7f4c0c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.860+0000 7f4c2bfff700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4c0c053a40 0x7f4c0c055f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4c2401b000 tx=0x7f4c24016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:17.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:17.868+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c200420b0 con 0x7f4c2c0ab290 2022-07-12T08:41:18.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.024+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4c2c066410 con 0x7f4c2c0ab290 2022-07-12T08:41:18.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.028+0000 7f4c297fa700 1 -- 172.21.15.61:0/1396062484 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+131710 (secure 0 0 0) 0x7f4c200456d0 con 0x7f4c2c0ab290 2022-07-12T08:41:18.041 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:18.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4c0c053a40 msgr2=0x7f4c0c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4c0c053a40 0x7f4c0c055f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4c2401b000 tx=0x7f4c24016000).stop 2022-07-12T08:41:18.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 msgr2=0x7f4c2c066720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c066720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f4c2c0615d0 tx=0x7f4c20007c50).stop 2022-07-12T08:41:18.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 shutdown_connections 2022-07-12T08:41:18.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4c0c053a40 0x7f4c0c055f00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4c2c0afa60 0x7f4c2c159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c2c0ab290 0x7f4c2c066720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 --2- 172.21.15.61:0/1396062484 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4c2c05f140 0x7f4c2c1472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 >> 172.21.15.61:0/1396062484 conn(0x7f4c2c0b1950 msgr2=0x7f4c2c05bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 shutdown_connections 2022-07-12T08:41:18.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.032+0000 7f4c314a0700 1 -- 172.21.15.61:0/1396062484 wait complete. 2022-07-12T08:41:18.052 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":4,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":21499},{"type":"v1","addr":"172.21.15.111:6801","nonce":21499}]},"active_addr":"172.21.15.111:6801/21499","active_change":"2022-07-12T08:40:57.073777+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[{"gid":4099,"name":"y","mgr_features":4540138292840890367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1681575181}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3596702918}]}]}} 2022-07-12T08:41:18.059 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-07-12T08:41:18.059 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-07-12T08:41:18.061 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 Processor -- start 2022-07-12T08:41:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 -- start start 2022-07-12T08:41:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdfbc0b3630 0x7fdfbc05bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdfbc14abf0 0x7fdfbc146290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfbc148110 con 0x7fdfbc14a810 2022-07-12T08:41:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfbc066bc0 con 0x7fdfbc14abf0 2022-07-12T08:41:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbb59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfbc0b3080 con 0x7fdfbc0b3630 2022-07-12T08:41:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdfbc14abf0 0x7fdfbc146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfba59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdfbc14a810 0x7fdfbc147a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41144/0 (socket says 172.21.15.61:41144) 2022-07-12T08:41:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfba59c700 1 -- 172.21.15.61:0/111272656 learned_addr learned my addr 172.21.15.61:0/111272656 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfb9d9b700 1 --2- 172.21.15.61:0/111272656 >> v2:172.21.15.111:3300/0 conn(0x7fdfbc0b3630 0x7fdfbc05bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbad9d700 1 -- 172.21.15.61:0/111272656 >> v2:172.21.15.111:3300/0 conn(0x7fdfbc0b3630 msgr2=0x7fdfbc05bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbad9d700 1 --2- 172.21.15.61:0/111272656 >> v2:172.21.15.111:3300/0 conn(0x7fdfbc0b3630 0x7fdfbc05bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbad9d700 1 -- 172.21.15.61:0/111272656 >> v1:172.21.15.61:6789/0 conn(0x7fdfbc14a810 legacy=0x7fdfbc147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbad9d700 1 -- 172.21.15.61:0/111272656 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc14b140 con 0x7fdfbc14abf0 2022-07-12T08:41:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfbad9d700 1 --2- 172.21.15.61:0/111272656 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc146290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdfb4002f70 tx=0x7fdfb4006df0).ready entity=mon.0 client_cookie=8d3ca7845d439638 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfb959a700 1 -- 172.21.15.61:0/111272656 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb400d570 con 0x7fdfbc14abf0 2022-07-12T08:41:18.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfb959a700 1 -- 172.21.15.61:0/111272656 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb4000e90 con 0x7fdfbc14abf0 2022-07-12T08:41:18.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.172+0000 7fdfb959a700 1 -- 172.21.15.61:0/111272656 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb400d8c0 con 0x7fdfbc14abf0 2022-07-12T08:41:18.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- 172.21.15.61:0/111272656 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14abf0 msgr2=0x7fdfbc146290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/111272656 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc146290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdfb4002f70 tx=0x7fdfb4006df0).stop 2022-07-12T08:41:18.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- 172.21.15.61:0/111272656 shutdown_connections 2022-07-12T08:41:18.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/111272656 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc146290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/111272656 >> v2:172.21.15.111:3300/0 conn(0x7fdfbc0b3630 0x7fdfbc05bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- 172.21.15.61:0/111272656 >> 172.21.15.61:0/111272656 conn(0x7fdfbc0b70b0 msgr2=0x7fdfbc0b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- 172.21.15.61:0/111272656 shutdown_connections 2022-07-12T08:41:18.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- 172.21.15.61:0/111272656 wait complete. 2022-07-12T08:41:18.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 Processor -- start 2022-07-12T08:41:18.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- start start 2022-07-12T08:41:18.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdfbc0b3630 0x7fdfbc0b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 0x7fdfbc0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc0b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc09c120 con 0x7fdfbc14a810 2022-07-12T08:41:18.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc09ac40 con 0x7fdfbc14abf0 2022-07-12T08:41:18.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbb59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc0ae4f0 con 0x7fdfbc0b3630 2022-07-12T08:41:18.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfb9d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 0x7fdfbc0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfb9d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 0x7fdfbc0aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52934/0 (socket says 172.21.15.61:52934) 2022-07-12T08:41:18.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfb9d9b700 1 -- 172.21.15.61:0/1191306183 learned_addr learned my addr 172.21.15.61:0/1191306183 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:18.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfbad9d700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc0b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.176+0000 7fdfba59c700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdfbc0b3630 0x7fdfbc0b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb9d9b700 1 -- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdfbc0b3630 msgr2=0x7fdfbc0b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb9d9b700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdfbc0b3630 0x7fdfbc0b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb9d9b700 1 -- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfbc14abf0 msgr2=0x7fdfbc0b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb9d9b700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc0b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb9d9b700 1 -- 172.21.15.61:0/1191306183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb400b180 con 0x7fdfbc14a810 2022-07-12T08:41:18.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfba59c700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdfbc0b3630 0x7fdfbc0b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:18.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb9d9b700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 0x7fdfbc0aebf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fdfa400c000 tx=0x7fdfa4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfa4010780 con 0x7fdfbc14a810 2022-07-12T08:41:18.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa40108e0 con 0x7fdfbc14a810 2022-07-12T08:41:18.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfa4010c40 con 0x7fdfbc14a810 2022-07-12T08:41:18.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfbc09ae70 con 0x7fdfbc14a810 2022-07-12T08:41:18.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfbc156f00 con 0x7fdfbc14a810 2022-07-12T08:41:18.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfbc09a9e0 con 0x7fdfbc14a810 2022-07-12T08:41:18.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdfa401c460 con 0x7fdfbc14a810 2022-07-12T08:41:18.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb37fe700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfa0053920 0x7fdfa0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdfa406c2f0 con 0x7fdfbc14a810 2022-07-12T08:41:18.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.180+0000 7fdfba59c700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfa0053920 0x7fdfa0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.184+0000 7fdfba59c700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfa0053920 0x7fdfa0055de0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdfac00b000 tx=0x7fdfac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.188+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdfa4043150 con 0x7fdfbc14a810 2022-07-12T08:41:18.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.312+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfbc057460 con 0x7fdfbc14a810 2022-07-12T08:41:18.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.312+0000 7fdfb37fe700 1 -- 172.21.15.61:0/1191306183 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fdfa4046770 con 0x7fdfbc14a810 2022-07-12T08:41:18.321 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:18.322 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:18.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfa0053920 msgr2=0x7fdfa0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfa0053920 0x7fdfa0055de0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fdfac00b000 tx=0x7fdfac009040).stop 2022-07-12T08:41:18.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 msgr2=0x7fdfbc0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 0x7fdfbc0aebf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fdfa400c000 tx=0x7fdfa4006df0).stop 2022-07-12T08:41:18.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 shutdown_connections 2022-07-12T08:41:18.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfa0053920 0x7fdfa0055de0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfbc14abf0 0x7fdfbc0b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdfbc14a810 0x7fdfbc0aebf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 --2- 172.21.15.61:0/1191306183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdfbc0b3630 0x7fdfbc0b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 >> 172.21.15.61:0/1191306183 conn(0x7fdfbc0b70b0 msgr2=0x7fdfbc05cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 shutdown_connections 2022-07-12T08:41:18.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.316+0000 7fdfbb59e700 1 -- 172.21.15.61:0/1191306183 wait complete. 2022-07-12T08:41:18.337 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-07-12T08:41:18.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:18.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 Processor -- start 2022-07-12T08:41:18.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 -- start start 2022-07-12T08:41:18.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405caf0 0x7fd9040b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd90405d040 0x7fd9040b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040607e0 con 0x7fd9040b6ac0 2022-07-12T08:41:18.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd904060340 con 0x7fd90405d040 2022-07-12T08:41:18.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90aea5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040b6ea0 con 0x7fd90405caf0 2022-07-12T08:41:18.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90a6a4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd90405d040 0x7fd9040b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90a6a4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd90405d040 0x7fd9040b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52942/0 (socket says 172.21.15.61:52942) 2022-07-12T08:41:18.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.464+0000 7fd90a6a4700 1 -- 172.21.15.61:0/2715069000 learned_addr learned my addr 172.21.15.61:0/2715069000 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:18.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd9096a2700 1 --2- 172.21.15.61:0/2715069000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405caf0 0x7fd9040b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90a6a4700 1 -- 172.21.15.61:0/2715069000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405caf0 msgr2=0x7fd9040b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:18.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90a6a4700 1 --2- 172.21.15.61:0/2715069000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405caf0 0x7fd9040b1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90a6a4700 1 -- 172.21.15.61:0/2715069000 >> v1:172.21.15.111:6789/0 conn(0x7fd9040b6ac0 legacy=0x7fd904143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90a6a4700 1 -- 172.21.15.61:0/2715069000 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904060540 con 0x7fd90405d040 2022-07-12T08:41:18.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90a6a4700 1 --2- 172.21.15.61:0/2715069000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405d040 0x7fd9040b2360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd9040b1d50 tx=0x7fd900006eb0).ready entity=mon.0 client_cookie=5b2aee4a525a6526 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd908ea1700 1 -- 172.21.15.61:0/2715069000 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90000d5c0 con 0x7fd90405d040 2022-07-12T08:41:18.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd908ea1700 1 -- 172.21.15.61:0/2715069000 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd900000e90 con 0x7fd90405d040 2022-07-12T08:41:18.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd908ea1700 1 -- 172.21.15.61:0/2715069000 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd90000d950 con 0x7fd90405d040 2022-07-12T08:41:18.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 -- 172.21.15.61:0/2715069000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405d040 msgr2=0x7fd9040b2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 --2- 172.21.15.61:0/2715069000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405d040 0x7fd9040b2360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd9040b1d50 tx=0x7fd900006eb0).stop 2022-07-12T08:41:18.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 -- 172.21.15.61:0/2715069000 shutdown_connections 2022-07-12T08:41:18.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 --2- 172.21.15.61:0/2715069000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405d040 0x7fd9040b2360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 --2- 172.21.15.61:0/2715069000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405caf0 0x7fd9040b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 -- 172.21.15.61:0/2715069000 >> 172.21.15.61:0/2715069000 conn(0x7fd904064570 msgr2=0x7fd9040b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 -- 172.21.15.61:0/2715069000 shutdown_connections 2022-07-12T08:41:18.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 -- 172.21.15.61:0/2715069000 wait complete. 2022-07-12T08:41:18.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 Processor -- start 2022-07-12T08:41:18.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.468+0000 7fd90aea5700 1 -- start start 2022-07-12T08:41:18.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 0x7fd904143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405d040 0x7fd90413a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9040b6ac0 0x7fd90413ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90413b370 con 0x7fd90405caf0 2022-07-12T08:41:18.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90413fb50 con 0x7fd9040b6ac0 2022-07-12T08:41:18.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd904141a20 con 0x7fd90405d040 2022-07-12T08:41:18.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 0x7fd904143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 0x7fd904143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52946/0 (socket says 172.21.15.61:52946) 2022-07-12T08:41:18.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 -- 172.21.15.61:0/33363364 learned_addr learned my addr 172.21.15.61:0/33363364 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:18.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd9096a2700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405d040 0x7fd90413a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90a6a4700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9040b6ac0 0x7fd90413ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 -- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405d040 msgr2=0x7fd90413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405d040 0x7fd90413a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 -- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9040b6ac0 msgr2=0x7fd90413ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9040b6ac0 0x7fd90413ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 -- 172.21.15.61:0/33363364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90000b1e0 con 0x7fd90405caf0 2022-07-12T08:41:18.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90a6a4700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9040b6ac0 0x7fd90413ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:18.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd909ea3700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 0x7fd904143b00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd90413fc90 tx=0x7fd8fc001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8fc00e4a0 con 0x7fd90405caf0 2022-07-12T08:41:18.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc00e600 con 0x7fd90405caf0 2022-07-12T08:41:18.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd904153a20 con 0x7fd90405caf0 2022-07-12T08:41:18.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8fc00e960 con 0x7fd90405caf0 2022-07-12T08:41:18.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90414f8c0 con 0x7fd90405caf0 2022-07-12T08:41:18.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90414fa80 con 0x7fd90405caf0 2022-07-12T08:41:18.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.472+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd8fc019030 con 0x7fd90405caf0 2022-07-12T08:41:18.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.476+0000 7fd8f2ffd700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8e4053920 0x7fd8e4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.476+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8fc06b4e0 con 0x7fd90405caf0 2022-07-12T08:41:18.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.476+0000 7fd9096a2700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8e4053920 0x7fd8e4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.476+0000 7fd9096a2700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8e4053920 0x7fd8e4055de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd8f400c000 tx=0x7fd8f4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.480+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8fc042440 con 0x7fd90405caf0 2022-07-12T08:41:18.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.608+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd904057460 con 0x7fd90405caf0 2022-07-12T08:41:18.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.608+0000 7fd8f2ffd700 1 -- 172.21.15.61:0/33363364 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fd8fc045a60 con 0x7fd90405caf0 2022-07-12T08:41:18.609 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:18.609 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:18.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.608+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8e4053920 msgr2=0x7fd8e4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.608+0000 7fd90aea5700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8e4053920 0x7fd8e4055de0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd8f400c000 tx=0x7fd8f4008040).stop 2022-07-12T08:41:18.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.608+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 msgr2=0x7fd904143b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.608+0000 7fd90aea5700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 0x7fd904143b00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd90413fc90 tx=0x7fd8fc001400).stop 2022-07-12T08:41:18.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 shutdown_connections 2022-07-12T08:41:18.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8e4053920 0x7fd8e4055de0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9040b6ac0 0x7fd90413ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd90405d040 0x7fd90413a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 --2- 172.21.15.61:0/33363364 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd90405caf0 0x7fd904143b00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 >> 172.21.15.61:0/33363364 conn(0x7fd904064570 msgr2=0x7fd904067550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 shutdown_connections 2022-07-12T08:41:18.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.612+0000 7fd90aea5700 1 -- 172.21.15.61:0/33363364 wait complete. 2022-07-12T08:41:18.658 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-07-12T08:41:18.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 Processor -- start 2022-07-12T08:41:18.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- start start 2022-07-12T08:41:18.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb2540ab290 0x7fb2540ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2540afa60 0x7fb2540aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2540abc70 con 0x7fb25405f140 2022-07-12T08:41:18.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25405f580 con 0x7fb2540ab290 2022-07-12T08:41:18.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25405c720 con 0x7fb2540afa60 2022-07-12T08:41:18.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb2540ab290 0x7fb2540ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb2540ab290 0x7fb2540ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52954/0 (socket says 172.21.15.61:52954) 2022-07-12T08:41:18.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 -- 172.21.15.61:0/1873722346 learned_addr learned my addr 172.21.15.61:0/1873722346 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:18.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259d31700 1 --2- 172.21.15.61:0/1873722346 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2540afa60 0x7fb2540aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 -- 172.21.15.61:0/1873722346 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2540afa60 msgr2=0x7fb2540aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:18.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 --2- 172.21.15.61:0/1873722346 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2540afa60 0x7fb2540aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 -- 172.21.15.61:0/1873722346 >> v1:172.21.15.111:6789/0 conn(0x7fb25405f140 legacy=0x7fb25405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:18.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 -- 172.21.15.61:0/1873722346 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25405c9c0 con 0x7fb2540ab290 2022-07-12T08:41:18.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb259530700 1 --2- 172.21.15.61:0/1873722346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540ab290 0x7fb2540ab670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb2540615d0 tx=0x7fb24c009290).ready entity=mon.0 client_cookie=69e670d9885da03e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb24bfff700 1 -- 172.21.15.61:0/1873722346 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb24c00b860 con 0x7fb2540ab290 2022-07-12T08:41:18.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb24bfff700 1 -- 172.21.15.61:0/1873722346 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c00b9c0 con 0x7fb2540ab290 2022-07-12T08:41:18.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb24bfff700 1 -- 172.21.15.61:0/1873722346 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb24c00bb70 con 0x7fb2540ab290 2022-07-12T08:41:18.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- 172.21.15.61:0/1873722346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540ab290 msgr2=0x7fb2540ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 --2- 172.21.15.61:0/1873722346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540ab290 0x7fb2540ab670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fb2540615d0 tx=0x7fb24c009290).stop 2022-07-12T08:41:18.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- 172.21.15.61:0/1873722346 shutdown_connections 2022-07-12T08:41:18.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 --2- 172.21.15.61:0/1873722346 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2540afa60 0x7fb2540aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 --2- 172.21.15.61:0/1873722346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540ab290 0x7fb2540ab670 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- 172.21.15.61:0/1873722346 >> 172.21.15.61:0/1873722346 conn(0x7fb2540b1950 msgr2=0x7fb2540b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- 172.21.15.61:0/1873722346 shutdown_connections 2022-07-12T08:41:18.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.784+0000 7fb25a532700 1 -- 172.21.15.61:0/1873722346 wait complete. 2022-07-12T08:41:18.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 Processor -- start 2022-07-12T08:41:18.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 -- start start 2022-07-12T08:41:18.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb25405f140 0x7fb254147130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540ab290 0x7fb254067030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540afa60 0x7fb254159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2540679e0 con 0x7fb2540afa60 2022-07-12T08:41:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb254067520 con 0x7fb2540ab290 2022-07-12T08:41:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb254067740 con 0x7fb25405f140 2022-07-12T08:41:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540afa60 0x7fb254159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540afa60 0x7fb254159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52964/0 (socket says 172.21.15.61:52964) 2022-07-12T08:41:18.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 -- 172.21.15.61:0/486400805 learned_addr learned my addr 172.21.15.61:0/486400805 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:18.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259530700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb25405f140 0x7fb254147130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb258d2f700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540ab290 0x7fb254067030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 -- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb25405f140 msgr2=0x7fb254147130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb25405f140 0x7fb254147130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 -- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540ab290 msgr2=0x7fb254067030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540ab290 0x7fb254067030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 -- 172.21.15.61:0/486400805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c009530 con 0x7fb2540afa60 2022-07-12T08:41:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb258d2f700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2540ab290 0x7fb254067030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb259d31700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540afa60 0x7fb254159420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb254065210 tx=0x7fb250008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb250011780 con 0x7fb2540afa60 2022-07-12T08:41:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2500118e0 con 0x7fb2540afa60 2022-07-12T08:41:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb254066050 con 0x7fb2540afa60 2022-07-12T08:41:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb250011c40 con 0x7fb2540afa60 2022-07-12T08:41:18.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.788+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2540adfe0 con 0x7fb2540afa60 2022-07-12T08:41:18.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb254065e80 con 0x7fb2540afa60 2022-07-12T08:41:18.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb250012420 con 0x7fb2540afa60 2022-07-12T08:41:18.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb23c0539b0 0x7fb23c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb25006bd90 con 0x7fb2540afa60 2022-07-12T08:41:18.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fb23c05ac90 0x7fb23c05d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:18.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 --> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb23c05d770 con 0x7fb23c05ac90 2022-07-12T08:41:18.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb250042bf0 con 0x7fb2540afa60 2022-07-12T08:41:18.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb259530700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb23c0539b0 0x7fb23c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb258d2f700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fb23c05ac90 0x7fb23c05d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:18.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb259530700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb23c0539b0 0x7fb23c055e70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb24c019000 tx=0x7fb24c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb258d2f700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fb23c05ac90 0x7fb23c05d0f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:18.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.792+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== osd.0 v2:172.21.15.61:6800/21668 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fb23c05d770 con 0x7fb23c05ac90 2022-07-12T08:41:18.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.808+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 --> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb25405bfd0 con 0x7fb23c05ac90 2022-07-12T08:41:18.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb24a7fc700 1 -- 172.21.15.61:0/486400805 <== osd.0 v2:172.21.15.61:6800/21668 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb25405bfd0 con 0x7fb23c05ac90 2022-07-12T08:41:18.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fb23c05ac90 msgr2=0x7fb23c05d0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fb23c05ac90 0x7fb23c05d0f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:18.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb23c0539b0 msgr2=0x7fb23c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb23c0539b0 0x7fb23c055e70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fb24c019000 tx=0x7fb24c016000).stop 2022-07-12T08:41:18.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540afa60 msgr2=0x7fb254159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:18.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 --2- 172.21.15.61:0/486400805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2540afa60 0x7fb254159420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fb254065210 tx=0x7fb250008040).stop 2022-07-12T08:41:18.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb259d31700 1 -- 172.21.15.61:0/486400805 reap_dead start 2022-07-12T08:41:18.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 shutdown_connections 2022-07-12T08:41:18.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 >> 172.21.15.61:0/486400805 conn(0x7fb2540b1950 msgr2=0x7fb2540b33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:18.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 shutdown_connections 2022-07-12T08:41:18.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:18.812+0000 7fb25a532700 1 -- 172.21.15.61:0/486400805 wait complete. 2022-07-12T08:41:18.927 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:18.927 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-07-12T08:41:19.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.048+0000 7f60f359e700 1 Processor -- start 2022-07-12T08:41:19.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.048+0000 7f60f359e700 1 -- start start 2022-07-12T08:41:19.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60f4140160 0x7f60f413ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4140680 con 0x7f60f405f480 2022-07-12T08:41:19.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4140cd0 con 0x7f60f405f860 2022-07-12T08:41:19.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60f4140f70 con 0x7f60f4140160 2022-07-12T08:41:19.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f60f405f480 0x7f60f413c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41188/0 (socket says 172.21.15.61:41188) 2022-07-12T08:41:19.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f259c700 1 -- 172.21.15.61:0/1404460072 learned_addr learned my addr 172.21.15.61:0/1404460072 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:19.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60e9d9b700 1 --2- 172.21.15.61:0/1404460072 >> v2:172.21.15.111:3300/0 conn(0x7f60f4140160 0x7f60f413ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547271581 0 0) 0x7f60f4140680 con 0x7f60f405f480 2022-07-12T08:41:19.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d4003260 con 0x7f60f405f480 2022-07-12T08:41:19.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792699585 0 0) 0x7f60f4140cd0 con 0x7f60f405f860 2022-07-12T08:41:19.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4140680 con 0x7f60f405f860 2022-07-12T08:41:19.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60e9d9b700 1 -- 172.21.15.61:0/1404460072 >> v1:172.21.15.111:6789/0 conn(0x7f60f405f860 legacy=0x7f60f4141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60e9d9b700 1 -- 172.21.15.61:0/1404460072 >> v1:172.21.15.61:6789/0 conn(0x7f60f405f480 legacy=0x7f60f413c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60e9d9b700 1 -- 172.21.15.61:0/1404460072 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f405fca0 con 0x7f60f4140160 2022-07-12T08:41:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60e9d9b700 1 --2- 172.21.15.61:0/1404460072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f4140160 0x7f60f413ce20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f60dc002f70 tx=0x7f60dc006df0).ready entity=mon.1 client_cookie=442b3714d1aeb5f2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60dc00d540 con 0x7f60f4140160 2022-07-12T08:41:19.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc000e90 con 0x7f60f4140160 2022-07-12T08:41:19.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f1d9b700 1 -- 172.21.15.61:0/1404460072 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60dc00d890 con 0x7f60f4140160 2022-07-12T08:41:19.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- 172.21.15.61:0/1404460072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f4140160 msgr2=0x7f60f413ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 --2- 172.21.15.61:0/1404460072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f4140160 0x7f60f413ce20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f60dc002f70 tx=0x7f60dc006df0).stop 2022-07-12T08:41:19.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- 172.21.15.61:0/1404460072 shutdown_connections 2022-07-12T08:41:19.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 --2- 172.21.15.61:0/1404460072 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f4140160 0x7f60f413ce20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- 172.21.15.61:0/1404460072 >> 172.21.15.61:0/1404460072 conn(0x7f60f4063a60 msgr2=0x7f60f4063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- 172.21.15.61:0/1404460072 shutdown_connections 2022-07-12T08:41:19.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.052+0000 7f60f359e700 1 -- 172.21.15.61:0/1404460072 wait complete. 2022-07-12T08:41:19.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 Processor -- start 2022-07-12T08:41:19.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 -- start start 2022-07-12T08:41:19.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60f405f480 0x7f60f40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f405f860 0x7f60f40671b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60f4140160 0x7f60f4151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4143430 con 0x7f60f405f480 2022-07-12T08:41:19.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f40b1c20 con 0x7f60f405f860 2022-07-12T08:41:19.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f4143190 con 0x7f60f4140160 2022-07-12T08:41:19.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60f405f480 0x7f60f40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60f405f480 0x7f60f40b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52978/0 (socket says 172.21.15.61:52978) 2022-07-12T08:41:19.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 -- 172.21.15.61:0/4084430191 learned_addr learned my addr 172.21.15.61:0/4084430191 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:19.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60e9d9b700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f405f860 0x7f60f40671b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f2d9d700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60f4140160 0x7f60f4151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 -- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60f4140160 msgr2=0x7f60f4151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60f4140160 0x7f60f4151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 -- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f405f860 msgr2=0x7f60f40671b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60f405f860 0x7f60f40671b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 -- 172.21.15.61:0/4084430191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc00b180 con 0x7f60f405f480 2022-07-12T08:41:19.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f259c700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60f405f480 0x7f60f40b0e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f60e40011d0 tx=0x7f60e4007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60e40070d0 con 0x7f60f405f480 2022-07-12T08:41:19.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e4007230 con 0x7f60f405f480 2022-07-12T08:41:19.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f4066d10 con 0x7f60f405f480 2022-07-12T08:41:19.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60e4005610 con 0x7f60f405f480 2022-07-12T08:41:19.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.056+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f40b2750 con 0x7f60f405f480 2022-07-12T08:41:19.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60f4066ea0 con 0x7f60f405f480 2022-07-12T08:41:19.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f60e4010070 con 0x7f60f405f480 2022-07-12T08:41:19.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60ebfff700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f60d8053ad0 0x7f60d8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f60e406b4c0 con 0x7f60f405f480 2022-07-12T08:41:19.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60ebfff700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f60d805ae30 0x7f60d805d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 --> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60d805d870 con 0x7f60d805ae30 2022-07-12T08:41:19.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60e406db50 con 0x7f60f405f480 2022-07-12T08:41:19.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60e9d9b700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f60d8053ad0 0x7f60d8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60f2d9d700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f60d805ae30 0x7f60d805d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60e9d9b700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f60d8053ad0 0x7f60d8055f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f60dc018000 tx=0x7f60dc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.060+0000 7f60f2d9d700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f60d805ae30 0x7f60d805d290 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.064+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== osd.1 v2:172.21.15.61:6808/21713 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f60d805d870 con 0x7f60d805ae30 2022-07-12T08:41:19.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 --> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60f4057460 con 0x7f60d805ae30 2022-07-12T08:41:19.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60ebfff700 1 -- 172.21.15.61:0/4084430191 <== osd.1 v2:172.21.15.61:6808/21713 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f60f4057460 con 0x7f60d805ae30 2022-07-12T08:41:19.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f60d805ae30 msgr2=0x7f60d805d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f60d805ae30 0x7f60d805d290 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f60d8053ad0 msgr2=0x7f60d8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f60d8053ad0 0x7f60d8055f90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f60dc018000 tx=0x7f60dc014000).stop 2022-07-12T08:41:19.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60f405f480 msgr2=0x7f60f40b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f359e700 1 --2- 172.21.15.61:0/4084430191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60f405f480 0x7f60f40b0e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f60e40011d0 tx=0x7f60e4007b50).stop 2022-07-12T08:41:19.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.080+0000 7f60f2d9d700 1 -- 172.21.15.61:0/4084430191 reap_dead start 2022-07-12T08:41:19.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.084+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 shutdown_connections 2022-07-12T08:41:19.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.084+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 >> 172.21.15.61:0/4084430191 conn(0x7f60f4063a60 msgr2=0x7f60f40b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.084+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 shutdown_connections 2022-07-12T08:41:19.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.084+0000 7f60f359e700 1 -- 172.21.15.61:0/4084430191 wait complete. 2022-07-12T08:41:19.144 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:19.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-07-12T08:41:19.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 Processor -- start 2022-07-12T08:41:19.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 -- start start 2022-07-12T08:41:19.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe34009bce0 0x7fe3400af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe3400b3f60 0x7fe3400b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3400aecb0 con 0x7fe3400af280 2022-07-12T08:41:19.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe340143b00 con 0x7fe3400b3f60 2022-07-12T08:41:19.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3474fe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3401435c0 con 0x7fe34009bce0 2022-07-12T08:41:19.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe3400b3f60 0x7fe3400b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe3400b3f60 0x7fe3400b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52990/0 (socket says 172.21.15.61:52990) 2022-07-12T08:41:19.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 -- 172.21.15.61:0/3847467220 learned_addr learned my addr 172.21.15.61:0/3847467220 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:19.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe3464fc700 1 --1- 172.21.15.61:0/3847467220 >> v1:172.21.15.61:6789/0 conn(0x7fe3400af280 0x7fe340060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41206/0 (socket says 172.21.15.61:41206) 2022-07-12T08:41:19.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe345cfb700 1 --2- 172.21.15.61:0/3847467220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe34009bce0 0x7fe3400af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 -- 172.21.15.61:0/3847467220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe34009bce0 msgr2=0x7fe3400af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:19.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 --2- 172.21.15.61:0/3847467220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe34009bce0 0x7fe3400af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 -- 172.21.15.61:0/3847467220 >> v1:172.21.15.61:6789/0 conn(0x7fe3400af280 legacy=0x7fe340060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 -- 172.21.15.61:0/3847467220 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe340143860 con 0x7fe3400b3f60 2022-07-12T08:41:19.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.276+0000 7fe346cfd700 1 --2- 172.21.15.61:0/3847467220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3400b3f60 0x7fe3400b4810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fe3380032f0 tx=0x7fe338009290).ready entity=mon.0 client_cookie=f082b38be7ac534f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3454fa700 1 -- 172.21.15.61:0/3847467220 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33800b8a0 con 0x7fe3400b3f60 2022-07-12T08:41:19.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3454fa700 1 -- 172.21.15.61:0/3847467220 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33800ba00 con 0x7fe3400b3f60 2022-07-12T08:41:19.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3454fa700 1 -- 172.21.15.61:0/3847467220 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33800bb70 con 0x7fe3400b3f60 2022-07-12T08:41:19.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- 172.21.15.61:0/3847467220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3400b3f60 msgr2=0x7fe3400b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 --2- 172.21.15.61:0/3847467220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3400b3f60 0x7fe3400b4810 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fe3380032f0 tx=0x7fe338009290).stop 2022-07-12T08:41:19.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- 172.21.15.61:0/3847467220 shutdown_connections 2022-07-12T08:41:19.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 --2- 172.21.15.61:0/3847467220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3400b3f60 0x7fe3400b4810 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 --2- 172.21.15.61:0/3847467220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe34009bce0 0x7fe3400af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- 172.21.15.61:0/3847467220 >> 172.21.15.61:0/3847467220 conn(0x7fe3400b92c0 msgr2=0x7fe3400b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- 172.21.15.61:0/3847467220 shutdown_connections 2022-07-12T08:41:19.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- 172.21.15.61:0/3847467220 wait complete. 2022-07-12T08:41:19.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 Processor -- start 2022-07-12T08:41:19.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- start start 2022-07-12T08:41:19.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe34009bce0 0x7fe34009c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3400af280 0x7fe340142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe3400b3f60 0x7fe3400b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340152890 con 0x7fe34009bce0 2022-07-12T08:41:19.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340146c90 con 0x7fe3400af280 2022-07-12T08:41:19.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.280+0000 7fe3474fe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe340152ac0 con 0x7fe3400b3f60 2022-07-12T08:41:19.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe34009bce0 0x7fe34009c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe34009bce0 0x7fe34009c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52992/0 (socket says 172.21.15.61:52992) 2022-07-12T08:41:19.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 -- 172.21.15.61:0/292300074 learned_addr learned my addr 172.21.15.61:0/292300074 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:19.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe346cfd700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe3400b3f60 0x7fe3400b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe345cfb700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3400af280 0x7fe340142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 -- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe3400b3f60 msgr2=0x7fe3400b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe3400b3f60 0x7fe3400b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 -- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3400af280 msgr2=0x7fe340142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3400af280 0x7fe340142240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 -- 172.21.15.61:0/292300074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe338009530 con 0x7fe34009bce0 2022-07-12T08:41:19.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe346cfd700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe3400b3f60 0x7fe3400b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:19.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe345cfb700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe3400af280 0x7fe340142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:19.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3464fc700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe34009bce0 0x7fe34009c350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe340060440 tx=0x7fe33c007b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33c00ce00 con 0x7fe34009bce0 2022-07-12T08:41:19.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c0072e0 con 0x7fe34009bce0 2022-07-12T08:41:19.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe340145ad0 con 0x7fe34009bce0 2022-07-12T08:41:19.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe33c0056d0 con 0x7fe34009bce0 2022-07-12T08:41:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3400b0850 con 0x7fe34009bce0 2022-07-12T08:41:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe340146b10 con 0x7fe34009bce0 2022-07-12T08:41:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe33c01a060 con 0x7fe34009bce0 2022-07-12T08:41:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe32f7fe700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe320053a40 0x7fe320055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.284+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe33c06bd50 con 0x7fe34009bce0 2022-07-12T08:41:19.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe32f7fe700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7fe32005ada0 0x7fe32005d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 --> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe32005d7e0 con 0x7fe32005ada0 2022-07-12T08:41:19.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe33c06c130 con 0x7fe34009bce0 2022-07-12T08:41:19.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe345cfb700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe320053a40 0x7fe320055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe346cfd700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7fe32005ada0 0x7fe32005d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe345cfb700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe320053a40 0x7fe320055f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe33000b000 tx=0x7fe330008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe346cfd700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7fe32005ada0 0x7fe32005d200 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.288+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== osd.2 v2:172.21.15.61:6816/21871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fe32005d7e0 con 0x7fe32005ada0 2022-07-12T08:41:19.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 --> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe3400b3490 con 0x7fe32005ada0 2022-07-12T08:41:19.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe32f7fe700 1 -- 172.21.15.61:0/292300074 <== osd.2 v2:172.21.15.61:6816/21871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe3400b3490 con 0x7fe32005ada0 2022-07-12T08:41:19.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7fe32005ada0 msgr2=0x7fe32005d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7fe32005ada0 0x7fe32005d200 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe320053a40 msgr2=0x7fe320055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe320053a40 0x7fe320055f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fe33000b000 tx=0x7fe330008040).stop 2022-07-12T08:41:19.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe34009bce0 msgr2=0x7fe34009c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 --2- 172.21.15.61:0/292300074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe34009bce0 0x7fe34009c350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fe340060440 tx=0x7fe33c007b30).stop 2022-07-12T08:41:19.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe346cfd700 1 -- 172.21.15.61:0/292300074 reap_dead start 2022-07-12T08:41:19.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 shutdown_connections 2022-07-12T08:41:19.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 >> 172.21.15.61:0/292300074 conn(0x7fe3400b92c0 msgr2=0x7fe34005bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 shutdown_connections 2022-07-12T08:41:19.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.304+0000 7fe3474fe700 1 -- 172.21.15.61:0/292300074 wait complete. 2022-07-12T08:41:19.354 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:19.356 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-07-12T08:41:19.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 Processor -- start 2022-07-12T08:41:19.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 -- start start 2022-07-12T08:41:19.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b6770 0x7fb214146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb21405cdd0 0x7fb214141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2140b6b50 con 0x7fb2140b4720 2022-07-12T08:41:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2140b4f10 con 0x7fb21405cdd0 2022-07-12T08:41:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21b08d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140b51b0 con 0x7fb2140b6770 2022-07-12T08:41:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21988a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb2140b4720 0x7fb214146570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41220/0 (socket says 172.21.15.61:41220) 2022-07-12T08:41:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21988a700 1 -- 172.21.15.61:0/2124093701 learned_addr learned my addr 172.21.15.61:0/2124093701 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:19.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a88c700 1 --2- 172.21.15.61:0/2124093701 >> v2:172.21.15.61:3300/0 conn(0x7fb21405cdd0 0x7fb214141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a08b700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b6770 0x7fb214146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb219089700 1 -- 172.21.15.61:0/2124093701 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249504132 0 0) 0x7fb2140b6b50 con 0x7fb2140b4720 2022-07-12T08:41:19.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb219089700 1 -- 172.21.15.61:0/2124093701 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200003260 con 0x7fb2140b4720 2022-07-12T08:41:19.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a88c700 1 -- 172.21.15.61:0/2124093701 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b6770 msgr2=0x7fb214146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:19.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a88c700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b6770 0x7fb214146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb219089700 1 -- 172.21.15.61:0/2124093701 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 687125358 0 0) 0x7fb200003260 con 0x7fb2140b4720 2022-07-12T08:41:19.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a88c700 1 -- 172.21.15.61:0/2124093701 >> v1:172.21.15.61:6789/0 conn(0x7fb2140b4720 legacy=0x7fb214146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a88c700 1 -- 172.21.15.61:0/2124093701 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2140b5450 con 0x7fb21405cdd0 2022-07-12T08:41:19.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a08b700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b6770 0x7fb214146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:19.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb21a88c700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21405cdd0 0x7fb214141a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb204001d00 tx=0x7fb20c006df0).ready entity=mon.0 client_cookie=a2a01c6870d42210 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb219089700 1 -- 172.21.15.61:0/2124093701 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb204001b60 con 0x7fb21405cdd0 2022-07-12T08:41:19.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb219089700 1 -- 172.21.15.61:0/2124093701 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c000e90 con 0x7fb21405cdd0 2022-07-12T08:41:19.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.508+0000 7fb219089700 1 -- 172.21.15.61:0/2124093701 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20c012460 con 0x7fb21405cdd0 2022-07-12T08:41:19.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- 172.21.15.61:0/2124093701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21405cdd0 msgr2=0x7fb214141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21405cdd0 0x7fb214141a90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fb204001d00 tx=0x7fb20c006df0).stop 2022-07-12T08:41:19.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- 172.21.15.61:0/2124093701 shutdown_connections 2022-07-12T08:41:19.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21405cdd0 0x7fb214141a90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 --2- 172.21.15.61:0/2124093701 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b6770 0x7fb214146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- 172.21.15.61:0/2124093701 >> 172.21.15.61:0/2124093701 conn(0x7fb2140b9d50 msgr2=0x7fb2140b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- 172.21.15.61:0/2124093701 shutdown_connections 2022-07-12T08:41:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- 172.21.15.61:0/2124093701 wait complete. 2022-07-12T08:41:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 Processor -- start 2022-07-12T08:41:19.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- start start 2022-07-12T08:41:19.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb21405cdd0 0x7fb214143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b4720 0x7fb214143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2140b6770 0x7fb2140b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214147fc0 con 0x7fb2140b6770 2022-07-12T08:41:19.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140b29b0 con 0x7fb21405cdd0 2022-07-12T08:41:19.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21b08d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2140b2670 con 0x7fb2140b4720 2022-07-12T08:41:19.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21a88c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2140b6770 0x7fb2140b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21a88c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2140b6770 0x7fb2140b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53012/0 (socket says 172.21.15.61:53012) 2022-07-12T08:41:19.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21a88c700 1 -- 172.21.15.61:0/4257312770 learned_addr learned my addr 172.21.15.61:0/4257312770 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:19.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21a08b700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb21405cdd0 0x7fb214143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.512+0000 7fb21988a700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b4720 0x7fb214143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21a88c700 1 -- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b4720 msgr2=0x7fb214143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21a88c700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2140b4720 0x7fb214143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21a88c700 1 -- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb21405cdd0 msgr2=0x7fb214143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21a88c700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb21405cdd0 0x7fb214143520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21a88c700 1 -- 172.21.15.61:0/4257312770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20800b180 con 0x7fb2140b6770 2022-07-12T08:41:19.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21a88c700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2140b6770 0x7fb2140b2e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb214148260 tx=0x7fb20c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20c00d7d0 con 0x7fb2140b6770 2022-07-12T08:41:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c00d930 con 0x7fb2140b6770 2022-07-12T08:41:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c00b180 con 0x7fb2140b6770 2022-07-12T08:41:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20c00dc90 con 0x7fb2140b6770 2022-07-12T08:41:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb214145dd0 con 0x7fb2140b6770 2022-07-12T08:41:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2140b2af0 con 0x7fb2140b6770 2022-07-12T08:41:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb20c025030 con 0x7fb2140b6770 2022-07-12T08:41:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb2137fe700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb1f4053940 0x7fb1f4055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.516+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb20c06d2d0 con 0x7fb2140b6770 2022-07-12T08:41:19.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb2137fe700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fb1f405ad40 0x7fb1f405d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 --> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb1f405d820 con 0x7fb1f405ad40 2022-07-12T08:41:19.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb21a08b700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb1f4053940 0x7fb1f4055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb20c06d5a0 con 0x7fb2140b6770 2022-07-12T08:41:19.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb21988a700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fb1f405ad40 0x7fb1f405d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb21a08b700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb1f4053940 0x7fb1f4055e00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb20800b000 tx=0x7fb208006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb21988a700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fb1f405ad40 0x7fb1f405d1a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.520+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== osd.3 v2:172.21.15.61:6824/21976 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fb1f405d820 con 0x7fb1f405ad40 2022-07-12T08:41:19.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.536+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 --> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb214057460 con 0x7fb1f405ad40 2022-07-12T08:41:19.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.536+0000 7fb2137fe700 1 -- 172.21.15.61:0/4257312770 <== osd.3 v2:172.21.15.61:6824/21976 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb214057460 con 0x7fb1f405ad40 2022-07-12T08:41:19.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fb1f405ad40 msgr2=0x7fb1f405d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fb1f405ad40 0x7fb1f405d1a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb1f4053940 msgr2=0x7fb1f4055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb1f4053940 0x7fb1f4055e00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb20800b000 tx=0x7fb208006df0).stop 2022-07-12T08:41:19.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2140b6770 msgr2=0x7fb2140b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 --2- 172.21.15.61:0/4257312770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2140b6770 0x7fb2140b2e00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb214148260 tx=0x7fb20c014000).stop 2022-07-12T08:41:19.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21a88c700 1 -- 172.21.15.61:0/4257312770 reap_dead start 2022-07-12T08:41:19.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 shutdown_connections 2022-07-12T08:41:19.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 >> 172.21.15.61:0/4257312770 conn(0x7fb2140b9d50 msgr2=0x7fb2140b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 shutdown_connections 2022-07-12T08:41:19.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.540+0000 7fb21b08d700 1 -- 172.21.15.61:0/4257312770 wait complete. 2022-07-12T08:41:19.591 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:19.593 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-07-12T08:41:19.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 Processor -- start 2022-07-12T08:41:19.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 -- start start 2022-07-12T08:41:19.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40aca50 0x7f98b40aee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98b40b1660 0x7f98b4143080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b40ace30 con 0x7f98b40b1200 2022-07-12T08:41:19.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b40ae420 con 0x7f98b40b1660 2022-07-12T08:41:19.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98bb5ac700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40ae6c0 con 0x7f98b40aca50 2022-07-12T08:41:19.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98b9b49700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98b40b1660 0x7f98b4143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98b8b47700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f98b40b1200 0x7f98b4142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:54838/0 (socket says 172.21.15.61:54838) 2022-07-12T08:41:19.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98b9348700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40aca50 0x7f98b40aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.712+0000 7f98b8b47700 1 -- 172.21.15.61:0/4287118321 learned_addr learned my addr 172.21.15.61:0/4287118321 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:19.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98b9b49700 1 -- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40aca50 msgr2=0x7f98b40aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:19.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98b9b49700 1 --2- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40aca50 0x7f98b40aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98b9b49700 1 -- 172.21.15.61:0/4287118321 >> v1:172.21.15.111:6789/0 conn(0x7f98b40b1200 legacy=0x7f98b4142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98b9b49700 1 -- 172.21.15.61:0/4287118321 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40ae960 con 0x7f98b40b1660 2022-07-12T08:41:19.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98b9b49700 1 --2- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 0x7f98b4143080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f98ac0032f0 tx=0x7f98ac00b4b0).ready entity=mon.1 client_cookie=909293654bcdebae server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98a3fff700 1 -- 172.21.15.61:0/4287118321 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98ac00de70 con 0x7f98b40b1660 2022-07-12T08:41:19.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98a3fff700 1 -- 172.21.15.61:0/4287118321 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac0153f0 con 0x7f98b40b1660 2022-07-12T08:41:19.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98a3fff700 1 -- 172.21.15.61:0/4287118321 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98ac015750 con 0x7f98b40b1660 2022-07-12T08:41:19.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 -- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 msgr2=0x7f98b4143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 --2- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 0x7f98b4143080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f98ac0032f0 tx=0x7f98ac00b4b0).stop 2022-07-12T08:41:19.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 -- 172.21.15.61:0/4287118321 shutdown_connections 2022-07-12T08:41:19.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 --2- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 0x7f98b4143080 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 --2- 172.21.15.61:0/4287118321 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40aca50 0x7f98b40aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 -- 172.21.15.61:0/4287118321 >> 172.21.15.61:0/4287118321 conn(0x7f98b4062c00 msgr2=0x7f98b4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 -- 172.21.15.61:0/4287118321 shutdown_connections 2022-07-12T08:41:19.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.716+0000 7f98bb5ac700 1 -- 172.21.15.61:0/4287118321 wait complete. 2022-07-12T08:41:19.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 Processor -- start 2022-07-12T08:41:19.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 -- start start 2022-07-12T08:41:19.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40aca50 0x7f98b4139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b1200 0x7f98b4155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 0x7f98b41561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4139670 con 0x7f98b40aca50 2022-07-12T08:41:19.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40a3480 con 0x7f98b40b1660 2022-07-12T08:41:19.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98bb5ac700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4067470 con 0x7f98b40b1200 2022-07-12T08:41:19.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98b9348700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40aca50 0x7f98b4139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98b8b47700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b1200 0x7f98b4155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98b9348700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40aca50 0x7f98b4139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53024/0 (socket says 172.21.15.61:53024) 2022-07-12T08:41:19.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98b9348700 1 -- 172.21.15.61:0/2062913429 learned_addr learned my addr 172.21.15.61:0/2062913429 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:19.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.720+0000 7f98b9b49700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 0x7f98b41561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b9348700 1 -- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b1200 msgr2=0x7f98b4155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b9348700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b1200 0x7f98b4155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b9348700 1 -- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 msgr2=0x7f98b41561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b9348700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b40b1660 0x7f98b41561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b9348700 1 -- 172.21.15.61:0/2062913429 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac00b710 con 0x7f98b40aca50 2022-07-12T08:41:19.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b8b47700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b1200 0x7f98b4155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:19.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98b9348700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40aca50 0x7f98b4139ef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f98b000c000 tx=0x7f98b0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98b0010780 con 0x7f98b40aca50 2022-07-12T08:41:19.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b00108e0 con 0x7f98b40aca50 2022-07-12T08:41:19.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b40a26c0 con 0x7f98b40aca50 2022-07-12T08:41:19.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98b0010c40 con 0x7f98b40aca50 2022-07-12T08:41:19.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b40667d0 con 0x7f98b40aca50 2022-07-12T08:41:19.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98b40a3300 con 0x7f98b40aca50 2022-07-12T08:41:19.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f98b001c460 con 0x7f98b40aca50 2022-07-12T08:41:19.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98a27fc700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98940539b0 0x7f9894055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.724+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f98b006c350 con 0x7f98b40aca50 2022-07-12T08:41:19.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98a27fc700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f989405ad10 0x7f989405d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98b8b47700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98940539b0 0x7f9894055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 --> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f989405d7f0 con 0x7f989405ad10 2022-07-12T08:41:19.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b006e8d0 con 0x7f98b40aca50 2022-07-12T08:41:19.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98b9b49700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f989405ad10 0x7f989405d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98b8b47700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98940539b0 0x7f9894055e70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f98a400d000 tx=0x7f98a4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98b9b49700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f989405ad10 0x7f989405d170 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.728+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== osd.4 v2:172.21.15.111:6802/21886 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f989405d7f0 con 0x7f989405ad10 2022-07-12T08:41:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.744+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 --> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f98b40672f0 con 0x7f989405ad10 2022-07-12T08:41:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.744+0000 7f98a27fc700 1 -- 172.21.15.61:0/2062913429 <== osd.4 v2:172.21.15.111:6802/21886 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f98ac015940 con 0x7f989405ad10 2022-07-12T08:41:19.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f989405ad10 msgr2=0x7f989405d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f989405ad10 0x7f989405d170 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98940539b0 msgr2=0x7f9894055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98940539b0 0x7f9894055e70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f98a400d000 tx=0x7f98a4009040).stop 2022-07-12T08:41:19.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40aca50 msgr2=0x7f98b4139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 --2- 172.21.15.61:0/2062913429 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40aca50 0x7f98b4139ef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f98b000c000 tx=0x7f98b0006df0).stop 2022-07-12T08:41:19.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98b9b49700 1 -- 172.21.15.61:0/2062913429 reap_dead start 2022-07-12T08:41:19.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 shutdown_connections 2022-07-12T08:41:19.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 >> 172.21.15.61:0/2062913429 conn(0x7f98b4062c00 msgr2=0x7f98b40b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 shutdown_connections 2022-07-12T08:41:19.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.748+0000 7f98bb5ac700 1 -- 172.21.15.61:0/2062913429 wait complete. 2022-07-12T08:41:19.788 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:19.789 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-07-12T08:41:19.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.904+0000 7f69761f6700 1 Processor -- start 2022-07-12T08:41:19.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- start start 2022-07-12T08:41:19.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700b56c0 con 0x7f6970141e50 2022-07-12T08:41:19.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69700b4bd0 con 0x7f69700b52e0 2022-07-12T08:41:19.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6970142f00 con 0x7f69700b4050 2022-07-12T08:41:19.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6970141e50 0x7f69701422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41256/0 (socket says 172.21.15.61:41256) 2022-07-12T08:41:19.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696ffff700 1 -- 172.21.15.61:0/1603413926 learned_addr learned my addr 172.21.15.61:0/1603413926 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:19.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696effd700 1 --2- 172.21.15.61:0/1603413926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696e7fc700 1 -- 172.21.15.61:0/1603413926 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786567314 0 0) 0x7f69700b56c0 con 0x7f6970141e50 2022-07-12T08:41:19.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696e7fc700 1 -- 172.21.15.61:0/1603413926 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f695c003260 con 0x7f6970141e50 2022-07-12T08:41:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696effd700 1 -- 172.21.15.61:0/1603413926 >> v1:172.21.15.111:6789/0 conn(0x7f69700b52e0 legacy=0x7f69700b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696e7fc700 1 -- 172.21.15.61:0/1603413926 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519994031 0 0) 0x7f695c003260 con 0x7f6970141e50 2022-07-12T08:41:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696effd700 1 -- 172.21.15.61:0/1603413926 >> v1:172.21.15.61:6789/0 conn(0x7f6970141e50 legacy=0x7f69701422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696effd700 1 -- 172.21.15.61:0/1603413926 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69700b0e80 con 0x7f69700b4050 2022-07-12T08:41:19.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696effd700 1 --2- 172.21.15.61:0/1603413926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700b28c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f6968001b80 tx=0x7f6964006df0).ready entity=mon.2 client_cookie=6f2598d7de1670b1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696e7fc700 1 -- 172.21.15.61:0/1603413926 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69680019e0 con 0x7f69700b4050 2022-07-12T08:41:19.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696e7fc700 1 -- 172.21.15.61:0/1603413926 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6964000e90 con 0x7f69700b4050 2022-07-12T08:41:19.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f696e7fc700 1 -- 172.21.15.61:0/1603413926 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6964011400 con 0x7f69700b4050 2022-07-12T08:41:19.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- 172.21.15.61:0/1603413926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 msgr2=0x7f69700b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 --2- 172.21.15.61:0/1603413926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700b28c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f6968001b80 tx=0x7f6964006df0).stop 2022-07-12T08:41:19.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- 172.21.15.61:0/1603413926 shutdown_connections 2022-07-12T08:41:19.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 --2- 172.21.15.61:0/1603413926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700b28c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- 172.21.15.61:0/1603413926 >> 172.21.15.61:0/1603413926 conn(0x7f6970067020 msgr2=0x7f6970067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- 172.21.15.61:0/1603413926 shutdown_connections 2022-07-12T08:41:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.908+0000 7f69761f6700 1 -- 172.21.15.61:0/1603413926 wait complete. 2022-07-12T08:41:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 Processor -- start 2022-07-12T08:41:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 -- start start 2022-07-12T08:41:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69700b52e0 0x7f69700af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970141e50 0x7f6970155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6970147e60 con 0x7f69700b52e0 2022-07-12T08:41:19.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69701556c0 con 0x7f6970141e50 2022-07-12T08:41:19.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6970155960 con 0x7f69700b4050 2022-07-12T08:41:19.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69700b52e0 0x7f69700af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69700b52e0 0x7f69700af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53040/0 (socket says 172.21.15.61:53040) 2022-07-12T08:41:19.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 -- 172.21.15.61:0/4081114863 learned_addr learned my addr 172.21.15.61:0/4081114863 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696ffff700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970141e50 0x7f6970155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696f7fe700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 -- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 msgr2=0x7f69700acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69700b4050 0x7f69700acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 -- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970141e50 msgr2=0x7f6970155e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6970141e50 0x7f6970155e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 -- 172.21.15.61:0/4081114863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696400b180 con 0x7f69700b52e0 2022-07-12T08:41:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696effd700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69700b52e0 0x7f69700af750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f6968001b80 tx=0x7f69700b2020).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6964015070 con 0x7f69700b52e0 2022-07-12T08:41:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696400d760 con 0x7f69700b52e0 2022-07-12T08:41:19.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.912+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69700ad230 con 0x7f69700b52e0 2022-07-12T08:41:19.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f696400dac0 con 0x7f69700b52e0 2022-07-12T08:41:19.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69701562f0 con 0x7f69700b52e0 2022-07-12T08:41:19.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6970147cd0 con 0x7f69700b52e0 2022-07-12T08:41:19.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6964024030 con 0x7f69700b52e0 2022-07-12T08:41:19.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6950053a60 0x7f6950055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f696406cfa0 con 0x7f69700b52e0 2022-07-12T08:41:19.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696f7fe700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6950053a60 0x7f6950055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7f695005ae40 0x7f695005d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:19.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 --> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f695005d880 con 0x7f695005ae40 2022-07-12T08:41:19.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f696406d2b0 con 0x7f69700b52e0 2022-07-12T08:41:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696f7fe700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6950053a60 0x7f6950055f20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f696000b000 tx=0x7f6960009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696ffff700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7f695005ae40 0x7f695005d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:19.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.916+0000 7f696ffff700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7f695005ae40 0x7f695005d2a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:19.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.920+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== osd.5 v2:172.21.15.111:6810/21888 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f695005d880 con 0x7f695005ae40 2022-07-12T08:41:19.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 --> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6970155b20 con 0x7f695005ae40 2022-07-12T08:41:19.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f696cff9700 1 -- 172.21.15.61:0/4081114863 <== osd.5 v2:172.21.15.111:6810/21888 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6970155b20 con 0x7f695005ae40 2022-07-12T08:41:19.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7f695005ae40 msgr2=0x7f695005d2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7f695005ae40 0x7f695005d2a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:19.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6950053a60 msgr2=0x7f6950055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6950053a60 0x7f6950055f20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f696000b000 tx=0x7f6960009040).stop 2022-07-12T08:41:19.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69700b52e0 msgr2=0x7f69700af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:19.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 --2- 172.21.15.61:0/4081114863 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69700b52e0 0x7f69700af750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f6968001b80 tx=0x7f69700b2020).stop 2022-07-12T08:41:19.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f696ffff700 1 -- 172.21.15.61:0/4081114863 reap_dead start 2022-07-12T08:41:19.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 shutdown_connections 2022-07-12T08:41:19.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 >> 172.21.15.61:0/4081114863 conn(0x7f6970067020 msgr2=0x7f697005bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:19.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 shutdown_connections 2022-07-12T08:41:19.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:19.936+0000 7f69761f6700 1 -- 172.21.15.61:0/4081114863 wait complete. 2022-07-12T08:41:19.976 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:19.977 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-07-12T08:41:20.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.096+0000 7fc22e79f700 1 Processor -- start 2022-07-12T08:41:20.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- start start 2022-07-12T08:41:20.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2280ab670 0x7fc22805f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc2280afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2280abc70 con 0x7fc2280ab290 2022-07-12T08:41:20.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2280aba90 con 0x7fc2280ab670 2022-07-12T08:41:20.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22805c720 con 0x7fc2280afa60 2022-07-12T08:41:20.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22d79d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc2280ab290 0x7fc22805ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41270/0 (socket says 172.21.15.61:41270) 2022-07-12T08:41:20.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22d79d700 1 -- 172.21.15.61:0/315827914 learned_addr learned my addr 172.21.15.61:0/315827914 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:20.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22df9e700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc2280afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/315827914 >> v2:172.21.15.111:3300/0 conn(0x7fc2280ab670 0x7fc22805f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc21ffff700 1 -- 172.21.15.61:0/315827914 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589109890 0 0) 0x7fc2280abc70 con 0x7fc2280ab290 2022-07-12T08:41:20.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc21ffff700 1 -- 172.21.15.61:0/315827914 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc20c003260 con 0x7fc2280ab290 2022-07-12T08:41:20.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22cf9c700 1 -- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 msgr2=0x7fc2280afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:20.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc2280afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22cf9c700 1 -- 172.21.15.61:0/315827914 >> v1:172.21.15.61:6789/0 conn(0x7fc2280ab290 legacy=0x7fc22805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22cf9c700 1 -- 172.21.15.61:0/315827914 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22805c9c0 con 0x7fc2280ab670 2022-07-12T08:41:20.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22df9e700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc2280afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:20.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc22805f140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc224002f70 tx=0x7fc224006df0).ready entity=mon.1 client_cookie=6d1cf02ad9452599 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc21ffff700 1 -- 172.21.15.61:0/315827914 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22400d4a0 con 0x7fc2280ab670 2022-07-12T08:41:20.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc21ffff700 1 -- 172.21.15.61:0/315827914 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc224000e90 con 0x7fc2280ab670 2022-07-12T08:41:20.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc21ffff700 1 -- 172.21.15.61:0/315827914 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc22400d7f0 con 0x7fc2280ab670 2022-07-12T08:41:20.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 msgr2=0x7fc22805f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc22805f140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fc224002f70 tx=0x7fc224006df0).stop 2022-07-12T08:41:20.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- 172.21.15.61:0/315827914 shutdown_connections 2022-07-12T08:41:20.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc2280afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 --2- 172.21.15.61:0/315827914 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc22805f140 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- 172.21.15.61:0/315827914 >> 172.21.15.61:0/315827914 conn(0x7fc2280b1950 msgr2=0x7fc2280b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- 172.21.15.61:0/315827914 shutdown_connections 2022-07-12T08:41:20.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.100+0000 7fc22e79f700 1 -- 172.21.15.61:0/315827914 wait complete. 2022-07-12T08:41:20.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 Processor -- start 2022-07-12T08:41:20.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 -- start start 2022-07-12T08:41:20.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2280ab290 0x7fc2280667b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc228066f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc228148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc228148810 con 0x7fc2280ab290 2022-07-12T08:41:20.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc228149560 con 0x7fc2280ab670 2022-07-12T08:41:20.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22814b760 con 0x7fc2280afa60 2022-07-12T08:41:20.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2280ab290 0x7fc2280667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2280ab290 0x7fc2280667b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53056/0 (socket says 172.21.15.61:53056) 2022-07-12T08:41:20.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 -- 172.21.15.61:0/373433866 learned_addr learned my addr 172.21.15.61:0/373433866 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:20.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22df9e700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc228148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc228066f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 -- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 msgr2=0x7fc228148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc228148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 -- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 msgr2=0x7fc228066f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc228066f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 -- 172.21.15.61:0/373433866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc220009470 con 0x7fc2280ab290 2022-07-12T08:41:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22df9e700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2280afa60 0x7fc228148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2280ab670 0x7fc228066f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22d79d700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2280ab290 0x7fc2280667b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fc218001110 tx=0x7fc218007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2180072e0 con 0x7fc2280ab290 2022-07-12T08:41:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc218007440 con 0x7fc2280ab290 2022-07-12T08:41:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc218005800 con 0x7fc2280ab290 2022-07-12T08:41:20.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22400b180 con 0x7fc2280ab290 2022-07-12T08:41:20.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.104+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22814aaf0 con 0x7fc2280ab290 2022-07-12T08:41:20.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc2280665f0 con 0x7fc2280ab290 2022-07-12T08:41:20.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc218010070 con 0x7fc2280ab290 2022-07-12T08:41:20.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc21e7fc700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc210053af0 0x7fc210055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc21806b7e0 con 0x7fc2280ab290 2022-07-12T08:41:20.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc21e7fc700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fc21005ae50 0x7fc21005d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc210053af0 0x7fc210055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 --> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc21005d890 con 0x7fc21005ae50 2022-07-12T08:41:20.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc21806dd60 con 0x7fc2280ab290 2022-07-12T08:41:20.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.108+0000 7fc22df9e700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fc21005ae50 0x7fc21005d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.112+0000 7fc22cf9c700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc210053af0 0x7fc210055fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc224015000 tx=0x7fc224014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.112+0000 7fc22df9e700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fc21005ae50 0x7fc21005d2b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.112+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== osd.6 v2:172.21.15.111:6818/22012 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fc21005d890 con 0x7fc21005ae50 2022-07-12T08:41:20.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 --> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc22814b380 con 0x7fc21005ae50 2022-07-12T08:41:20.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc21e7fc700 1 -- 172.21.15.61:0/373433866 <== osd.6 v2:172.21.15.111:6818/22012 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc22814b380 con 0x7fc21005ae50 2022-07-12T08:41:20.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fc21005ae50 msgr2=0x7fc21005d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fc21005ae50 0x7fc21005d2b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc210053af0 msgr2=0x7fc210055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc210053af0 0x7fc210055fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc224015000 tx=0x7fc224014000).stop 2022-07-12T08:41:20.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2280ab290 msgr2=0x7fc2280667b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 --2- 172.21.15.61:0/373433866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2280ab290 0x7fc2280667b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fc218001110 tx=0x7fc218007b80).stop 2022-07-12T08:41:20.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22df9e700 1 -- 172.21.15.61:0/373433866 reap_dead start 2022-07-12T08:41:20.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 shutdown_connections 2022-07-12T08:41:20.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 >> 172.21.15.61:0/373433866 conn(0x7fc2280b1950 msgr2=0x7fc2280b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 shutdown_connections 2022-07-12T08:41:20.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.128+0000 7fc22e79f700 1 -- 172.21.15.61:0/373433866 wait complete. 2022-07-12T08:41:20.148 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:20.149 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-07-12T08:41:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f584a4f6700 1 Processor -- start 2022-07-12T08:41:20.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f584a4f6700 1 -- start start 2022-07-12T08:41:20.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f584a4f6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f584405f140 0x7f5844151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f584a4f6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58440abc70 con 0x7f58440ab290 2022-07-12T08:41:20.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f584a4f6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584405c720 con 0x7f58440ab670 2022-07-12T08:41:20.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f584a4f6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58440aba90 con 0x7f584405f140 2022-07-12T08:41:20.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f5849cf5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f584405f140 0x7f5844151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f58494f4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f58440ab290 0x7f584405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41284/0 (socket says 172.21.15.61:41284) 2022-07-12T08:41:20.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f58494f4700 1 -- 172.21.15.61:0/3232291450 learned_addr learned my addr 172.21.15.61:0/3232291450 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:20.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f5849cf5700 1 --2- 172.21.15.61:0/3232291450 >> v2:172.21.15.61:3300/0 conn(0x7f584405f140 0x7f5844151bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53070/0 (socket says 172.21.15.61:53070) 2022-07-12T08:41:20.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f5849cf5700 1 -- 172.21.15.61:0/3232291450 >> v1:172.21.15.111:6789/0 conn(0x7f58440ab670 legacy=0x7f58440afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f5849cf5700 1 -- 172.21.15.61:0/3232291450 >> v1:172.21.15.61:6789/0 conn(0x7f58440ab290 legacy=0x7f584405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f583bfff700 1 -- 172.21.15.61:0/3232291450 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780446825 0 0) 0x7f58440abc70 con 0x7f58440ab290 2022-07-12T08:41:20.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f5849cf5700 1 -- 172.21.15.61:0/3232291450 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584405c9c0 con 0x7f584405f140 2022-07-12T08:41:20.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f5849cf5700 1 --2- 172.21.15.61:0/3232291450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f584405f140 0x7f5844151bf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f583c0032f0 tx=0x7f583c0091d0).ready entity=mon.0 client_cookie=6be859ed97303e15 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f583bfff700 1 -- 172.21.15.61:0/3232291450 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f583c009ea0 con 0x7f584405f140 2022-07-12T08:41:20.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f583bfff700 1 -- 172.21.15.61:0/3232291450 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c00b8a0 con 0x7f584405f140 2022-07-12T08:41:20.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.292+0000 7f583bfff700 1 -- 172.21.15.61:0/3232291450 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f583c00ba10 con 0x7f584405f140 2022-07-12T08:41:20.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- 172.21.15.61:0/3232291450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f584405f140 msgr2=0x7f5844151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 --2- 172.21.15.61:0/3232291450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f584405f140 0x7f5844151bf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f583c0032f0 tx=0x7f583c0091d0).stop 2022-07-12T08:41:20.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- 172.21.15.61:0/3232291450 shutdown_connections 2022-07-12T08:41:20.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 --2- 172.21.15.61:0/3232291450 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f584405f140 0x7f5844151bf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- 172.21.15.61:0/3232291450 >> 172.21.15.61:0/3232291450 conn(0x7f58440b1950 msgr2=0x7f58440b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- 172.21.15.61:0/3232291450 shutdown_connections 2022-07-12T08:41:20.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- 172.21.15.61:0/3232291450 wait complete. 2022-07-12T08:41:20.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 Processor -- start 2022-07-12T08:41:20.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- start start 2022-07-12T08:41:20.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f584405f140 0x7f5844063f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58440ab290 0x7f58441592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58440ab670 0x7f58441597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5844148ab0 con 0x7f58440ab670 2022-07-12T08:41:20.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5844066710 con 0x7f58440ab290 2022-07-12T08:41:20.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f584a4f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58440669b0 con 0x7f584405f140 2022-07-12T08:41:20.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f5849cf5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58440ab670 0x7f58441597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f5849cf5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58440ab670 0x7f58441597d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53076/0 (socket says 172.21.15.61:53076) 2022-07-12T08:41:20.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f5849cf5700 1 -- 172.21.15.61:0/1037406075 learned_addr learned my addr 172.21.15.61:0/1037406075 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:20.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f58494f4700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f584405f140 0x7f5844063f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.296+0000 7f5848cf3700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58440ab290 0x7f58441592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5849cf5700 1 -- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f584405f140 msgr2=0x7f5844063f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5849cf5700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f584405f140 0x7f5844063f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5849cf5700 1 -- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58440ab290 msgr2=0x7f58441592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5849cf5700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58440ab290 0x7f58441592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5849cf5700 1 -- 172.21.15.61:0/1037406075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c009470 con 0x7f58440ab670 2022-07-12T08:41:20.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5848cf3700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58440ab290 0x7f58441592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:20.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f5849cf5700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58440ab670 0x7f58441597d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f583c0032f0 tx=0x7f583c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f583c009ea0 con 0x7f58440ab670 2022-07-12T08:41:20.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c01f5e0 con 0x7f58440ab670 2022-07-12T08:41:20.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58441498f0 con 0x7f58440ab670 2022-07-12T08:41:20.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f583c01f940 con 0x7f58440ab670 2022-07-12T08:41:20.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5844148830 con 0x7f58440ab670 2022-07-12T08:41:20.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5844149a80 con 0x7f58440ab670 2022-07-12T08:41:20.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f583c026030 con 0x7f58440ab670 2022-07-12T08:41:20.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f582c053ad0 0x7f582c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f583c06ced0 con 0x7f58440ab670 2022-07-12T08:41:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f582c05ae30 0x7f582c05d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 --> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f582c05d870 con 0x7f582c05ae30 2022-07-12T08:41:20.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f583c06f490 con 0x7f58440ab670 2022-07-12T08:41:20.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.300+0000 7f58494f4700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f582c053ad0 0x7f582c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.304+0000 7f5848cf3700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f582c05ae30 0x7f582c05d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.304+0000 7f5848cf3700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f582c05ae30 0x7f582c05d290 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.304+0000 7f58494f4700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f582c053ad0 0x7f582c055f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f583400b000 tx=0x7f5834009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.304+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== osd.7 v2:172.21.15.111:6826/22094 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f582c05d870 con 0x7f582c05ae30 2022-07-12T08:41:20.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 --> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5844159fb0 con 0x7f582c05ae30 2022-07-12T08:41:20.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f583a7fc700 1 -- 172.21.15.61:0/1037406075 <== osd.7 v2:172.21.15.111:6826/22094 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5844159fb0 con 0x7f582c05ae30 2022-07-12T08:41:20.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f582c05ae30 msgr2=0x7f582c05d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f582c05ae30 0x7f582c05d290 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f582c053ad0 msgr2=0x7f582c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f582c053ad0 0x7f582c055f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f583400b000 tx=0x7f5834009040).stop 2022-07-12T08:41:20.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58440ab670 msgr2=0x7f58441597d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 --2- 172.21.15.61:0/1037406075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58440ab670 0x7f58441597d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f583c0032f0 tx=0x7f583c016000).stop 2022-07-12T08:41:20.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f5849cf5700 1 -- 172.21.15.61:0/1037406075 reap_dead start 2022-07-12T08:41:20.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 shutdown_connections 2022-07-12T08:41:20.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 >> 172.21.15.61:0/1037406075 conn(0x7f58440b1950 msgr2=0x7f58440b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.320+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 shutdown_connections 2022-07-12T08:41:20.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.324+0000 7f584a4f6700 1 -- 172.21.15.61:0/1037406075 wait complete. 2022-07-12T08:41:20.492 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:20.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-07-12T08:41:20.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8f0ea700 1 Processor -- start 2022-07-12T08:41:20.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8f0ea700 1 -- start start 2022-07-12T08:41:20.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8f0ea700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9a880b01f0 0x7f9a880ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8f0ea700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880ad2f0 con 0x7f9a8814a8f0 2022-07-12T08:41:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8f0ea700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8805c430 con 0x7f9a8805c050 2022-07-12T08:41:20.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8f0ea700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a880b0a40 con 0x7f9a880b01f0 2022-07-12T08:41:20.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8d687700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9a8814a8f0 0x7f9a881443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41304/0 (socket says 172.21.15.61:41304) 2022-07-12T08:41:20.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.624+0000 7f9a8d687700 1 -- 172.21.15.61:0/4059420705 learned_addr learned my addr 172.21.15.61:0/4059420705 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:20.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a87fff700 1 --2- 172.21.15.61:0/4059420705 >> v2:172.21.15.111:3300/0 conn(0x7f9a880b01f0 0x7f9a880ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a877fe700 1 -- 172.21.15.61:0/4059420705 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209370598 0 0) 0x7f9a880ad2f0 con 0x7f9a8814a8f0 2022-07-12T08:41:20.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a877fe700 1 -- 172.21.15.61:0/4059420705 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a74003260 con 0x7f9a8814a8f0 2022-07-12T08:41:20.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a87fff700 1 -- 172.21.15.61:0/4059420705 >> v1:172.21.15.111:6789/0 conn(0x7f9a8805c050 legacy=0x7f9a8805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a877fe700 1 -- 172.21.15.61:0/4059420705 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937635845 0 0) 0x7f9a74003260 con 0x7f9a8814a8f0 2022-07-12T08:41:20.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a87fff700 1 -- 172.21.15.61:0/4059420705 >> v1:172.21.15.61:6789/0 conn(0x7f9a8814a8f0 legacy=0x7f9a881443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a87fff700 1 -- 172.21.15.61:0/4059420705 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880b2420 con 0x7f9a880b01f0 2022-07-12T08:41:20.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a87fff700 1 --2- 172.21.15.61:0/4059420705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a880b01f0 0x7f9a880ac710 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9a80001b80 tx=0x7f9a78006df0).ready entity=mon.1 client_cookie=51741be575d7dafd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a877fe700 1 -- 172.21.15.61:0/4059420705 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a800019e0 con 0x7f9a880b01f0 2022-07-12T08:41:20.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a877fe700 1 -- 172.21.15.61:0/4059420705 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a78000e90 con 0x7f9a880b01f0 2022-07-12T08:41:20.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a877fe700 1 -- 172.21.15.61:0/4059420705 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a7800d4f0 con 0x7f9a880b01f0 2022-07-12T08:41:20.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/4059420705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a880b01f0 msgr2=0x7f9a880ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/4059420705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a880b01f0 0x7f9a880ac710 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9a80001b80 tx=0x7f9a78006df0).stop 2022-07-12T08:41:20.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/4059420705 shutdown_connections 2022-07-12T08:41:20.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/4059420705 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a880b01f0 0x7f9a880ac710 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/4059420705 >> 172.21.15.61:0/4059420705 conn(0x7f9a88062c00 msgr2=0x7f9a88062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/4059420705 shutdown_connections 2022-07-12T08:41:20.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/4059420705 wait complete. 2022-07-12T08:41:20.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.628+0000 7f9a8f0ea700 1 Processor -- start 2022-07-12T08:41:20.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 -- start start 2022-07-12T08:41:20.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a8805c050 0x7f9a881444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a880b01f0 0x7f9a88141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 0x7f9a88141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88142ee0 con 0x7f9a8814a8f0 2022-07-12T08:41:20.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88143180 con 0x7f9a8805c050 2022-07-12T08:41:20.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88143420 con 0x7f9a880b01f0 2022-07-12T08:41:20.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 0x7f9a88141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 0x7f9a88141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53092/0 (socket says 172.21.15.61:53092) 2022-07-12T08:41:20.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 -- 172.21.15.61:0/1416158238 learned_addr learned my addr 172.21.15.61:0/1416158238 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:20.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a87fff700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a880b01f0 0x7f9a88141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8ce86700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a8805c050 0x7f9a881444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 -- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a880b01f0 msgr2=0x7f9a88141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a880b01f0 0x7f9a88141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 -- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a8805c050 msgr2=0x7f9a881444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a8805c050 0x7f9a881444c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 -- 172.21.15.61:0/1416158238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7800b180 con 0x7f9a8814a8f0 2022-07-12T08:41:20.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8ce86700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a8805c050 0x7f9a881444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:20.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8d687700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 0x7f9a88141880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f9a80001c40 tx=0x7f9a80002160).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a80004ab0 con 0x7f9a8814a8f0 2022-07-12T08:41:20.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a80004c10 con 0x7f9a8814a8f0 2022-07-12T08:41:20.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a80007170 con 0x7f9a8814a8f0 2022-07-12T08:41:20.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a88141d70 con 0x7f9a8814a8f0 2022-07-12T08:41:20.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.632+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8814d190 con 0x7f9a8814a8f0 2022-07-12T08:41:20.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.636+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a881435e0 con 0x7f9a8814a8f0 2022-07-12T08:41:20.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.636+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9a80010070 con 0x7f9a8814a8f0 2022-07-12T08:41:20.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.636+0000 7f9a85ffb700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a68053ad0 0x7f9a68055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.636+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9a8006bb40 con 0x7f9a8814a8f0 2022-07-12T08:41:20.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.636+0000 7f9a8ce86700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a68053ad0 0x7f9a68055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.636+0000 7f9a8ce86700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a68053ad0 0x7f9a68055f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9a7c008000 tx=0x7f9a7c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.640+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a80042990 con 0x7f9a8814a8f0 2022-07-12T08:41:20.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.776+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9a88057460 con 0x7f9a8814a8f0 2022-07-12T08:41:20.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.776+0000 7f9a85ffb700 1 -- 172.21.15.61:0/1416158238 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a80019020 con 0x7f9a8814a8f0 2022-07-12T08:41:20.777 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:20.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.776+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a68053ad0 msgr2=0x7f9a68055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.776+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a68053ad0 0x7f9a68055f90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f9a7c008000 tx=0x7f9a7c009040).stop 2022-07-12T08:41:20.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.776+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 msgr2=0x7f9a88141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.776+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 0x7f9a88141880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f9a80001c40 tx=0x7f9a80002160).stop 2022-07-12T08:41:20.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 shutdown_connections 2022-07-12T08:41:20.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a8814a8f0 0x7f9a88141880 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a880b01f0 0x7f9a88141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a68053ad0 0x7f9a68055f90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 --2- 172.21.15.61:0/1416158238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a8805c050 0x7f9a881444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 >> 172.21.15.61:0/1416158238 conn(0x7f9a88062c00 msgr2=0x7f9a880b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 shutdown_connections 2022-07-12T08:41:20.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.780+0000 7f9a8f0ea700 1 -- 172.21.15.61:0/1416158238 wait complete. 2022-07-12T08:41:20.808 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2022-07-12T08:41:20.809 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-07-12T08:41:20.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.924+0000 7fe85e82d700 1 Processor -- start 2022-07-12T08:41:20.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.924+0000 7fe85e82d700 1 -- start start 2022-07-12T08:41:20.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8580b1f80 0x7fe8580ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8580afc70 0x7fe85805c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85805d480 con 0x7fe8580abdf0 2022-07-12T08:41:20.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8580b0fc0 con 0x7fe8580afc70 2022-07-12T08:41:20.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8580b2360 con 0x7fe8580b1f80 2022-07-12T08:41:20.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8580afc70 0x7fe85805c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8580afc70 0x7fe85805c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53100/0 (socket says 172.21.15.61:53100) 2022-07-12T08:41:20.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 -- 172.21.15.61:0/4156406077 learned_addr learned my addr 172.21.15.61:0/4156406077 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:20.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe857fff700 1 --2- 172.21.15.61:0/4156406077 >> v2:172.21.15.111:3300/0 conn(0x7fe8580b1f80 0x7fe8580ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 -- 172.21.15.61:0/4156406077 >> v2:172.21.15.111:3300/0 conn(0x7fe8580b1f80 msgr2=0x7fe8580ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:20.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 --2- 172.21.15.61:0/4156406077 >> v2:172.21.15.111:3300/0 conn(0x7fe8580b1f80 0x7fe8580ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 -- 172.21.15.61:0/4156406077 >> v1:172.21.15.111:6789/0 conn(0x7fe8580abdf0 legacy=0x7fe85805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 -- 172.21.15.61:0/4156406077 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8580b16d0 con 0x7fe8580afc70 2022-07-12T08:41:20.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85cdca700 1 --2- 172.21.15.61:0/4156406077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe85805c420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe850002f70 tx=0x7fe850006eb0).ready entity=mon.0 client_cookie=bffdcab7de1b9350 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe856ffd700 1 -- 172.21.15.61:0/4156406077 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe85000d610 con 0x7fe8580afc70 2022-07-12T08:41:20.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe856ffd700 1 -- 172.21.15.61:0/4156406077 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe850000e90 con 0x7fe8580afc70 2022-07-12T08:41:20.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe856ffd700 1 -- 172.21.15.61:0/4156406077 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe85000d9a0 con 0x7fe8580afc70 2022-07-12T08:41:20.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- 172.21.15.61:0/4156406077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 msgr2=0x7fe85805c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 --2- 172.21.15.61:0/4156406077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe85805c420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe850002f70 tx=0x7fe850006eb0).stop 2022-07-12T08:41:20.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- 172.21.15.61:0/4156406077 shutdown_connections 2022-07-12T08:41:20.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 --2- 172.21.15.61:0/4156406077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe85805c420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 --2- 172.21.15.61:0/4156406077 >> v2:172.21.15.111:3300/0 conn(0x7fe8580b1f80 0x7fe8580ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- 172.21.15.61:0/4156406077 >> 172.21.15.61:0/4156406077 conn(0x7fe858062c00 msgr2=0x7fe858062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:20.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- 172.21.15.61:0/4156406077 shutdown_connections 2022-07-12T08:41:20.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- 172.21.15.61:0/4156406077 wait complete. 2022-07-12T08:41:20.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 Processor -- start 2022-07-12T08:41:20.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.928+0000 7fe85e82d700 1 -- start start 2022-07-12T08:41:20.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8580abdf0 0x7fe858152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe858066ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8580b1f80 0x7fe858147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8581524b0 con 0x7fe8580afc70 2022-07-12T08:41:20.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe858152750 con 0x7fe8580b1f80 2022-07-12T08:41:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8581530e0 con 0x7fe8580abdf0 2022-07-12T08:41:20.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe858066ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe858066ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53104/0 (socket says 172.21.15.61:53104) 2022-07-12T08:41:20.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 -- 172.21.15.61:0/921111286 learned_addr learned my addr 172.21.15.61:0/921111286 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:20.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe857fff700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8580abdf0 0x7fe858152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85cdca700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8580b1f80 0x7fe858147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 -- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8580abdf0 msgr2=0x7fe858152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8580abdf0 0x7fe858152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 -- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8580b1f80 msgr2=0x7fe858147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:20.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8580b1f80 0x7fe858147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:20.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 -- 172.21.15.61:0/921111286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85000b1e0 con 0x7fe8580afc70 2022-07-12T08:41:20.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8577fe700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe858066ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe8580b62e0 tx=0x7fe848001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84800e4a0 con 0x7fe8580afc70 2022-07-12T08:41:20.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84800e600 con 0x7fe8580afc70 2022-07-12T08:41:20.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8581521c0 con 0x7fe8580afc70 2022-07-12T08:41:20.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84800e960 con 0x7fe8580afc70 2022-07-12T08:41:20.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8581492a0 con 0x7fe8580afc70 2022-07-12T08:41:20.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.932+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe848019030 con 0x7fe8580afc70 2022-07-12T08:41:20.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.936+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe858152910 con 0x7fe8580afc70 2022-07-12T08:41:20.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.936+0000 7fe8557fa700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe83c053ad0 0x7fe83c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:20.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.936+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe84806b780 con 0x7fe8580afc70 2022-07-12T08:41:20.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.936+0000 7fe857fff700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe83c053ad0 0x7fe83c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:20.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.936+0000 7fe857fff700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe83c053ad0 0x7fe83c055f90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe84000c000 tx=0x7fe840008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:20.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:20.940+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8480426e0 con 0x7fe8580afc70 2022-07-12T08:41:21.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.068+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe858057460 con 0x7fe8580afc70 2022-07-12T08:41:21.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.068+0000 7fe8557fa700 1 -- 172.21.15.61:0/921111286 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe848045d00 con 0x7fe8580afc70 2022-07-12T08:41:21.068 INFO:teuthology.orchestra.run.smithi061.stdout:55834574852 2022-07-12T08:41:21.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe83c053ad0 msgr2=0x7fe83c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe83c053ad0 0x7fe83c055f90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe84000c000 tx=0x7fe840008040).stop 2022-07-12T08:41:21.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 msgr2=0x7fe858066ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe858066ca0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe8580b62e0 tx=0x7fe848001400).stop 2022-07-12T08:41:21.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 shutdown_connections 2022-07-12T08:41:21.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe83c053ad0 0x7fe83c055f90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8580b1f80 0x7fe858147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8580afc70 0x7fe858066ca0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 --2- 172.21.15.61:0/921111286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8580abdf0 0x7fe858152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 >> 172.21.15.61:0/921111286 conn(0x7fe858062c00 msgr2=0x7fe8580b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 shutdown_connections 2022-07-12T08:41:21.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.072+0000 7fe85e82d700 1 -- 172.21.15.61:0/921111286 wait complete. 2022-07-12T08:41:21.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2022-07-12T08:41:21.090 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-07-12T08:41:21.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 Processor -- start 2022-07-12T08:41:21.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 -- start start 2022-07-12T08:41:21.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdb1005c050 0x7fdb100ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdb1014a8f0 0x7fdb101443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb100ad2f0 con 0x7fdb100b01f0 2022-07-12T08:41:21.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb100b0a40 con 0x7fdb1005c050 2022-07-12T08:41:21.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb165b2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb1005c430 con 0x7fdb1014a8f0 2022-07-12T08:41:21.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdb1005c050 0x7fdb100ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdb1005c050 0x7fdb100ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53110/0 (socket says 172.21.15.61:53110) 2022-07-12T08:41:21.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 -- 172.21.15.61:0/1364771542 learned_addr learned my addr 172.21.15.61:0/1364771542 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:21.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb14b4f700 1 --2- 172.21.15.61:0/1364771542 >> v2:172.21.15.111:3300/0 conn(0x7fdb1014a8f0 0x7fdb101443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 -- 172.21.15.61:0/1364771542 >> v2:172.21.15.111:3300/0 conn(0x7fdb1014a8f0 msgr2=0x7fdb101443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:21.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 --2- 172.21.15.61:0/1364771542 >> v2:172.21.15.111:3300/0 conn(0x7fdb1014a8f0 0x7fdb101443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 -- 172.21.15.61:0/1364771542 >> v1:172.21.15.111:6789/0 conn(0x7fdb100b01f0 legacy=0x7fdb1005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 -- 172.21.15.61:0/1364771542 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb100b2420 con 0x7fdb1005c050 2022-07-12T08:41:21.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0effd700 1 -- 172.21.15.61:0/1364771542 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596448846 0 0) 0x7fdb100ad2f0 con 0x7fdb100b01f0 2022-07-12T08:41:21.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0ffff700 1 --2- 172.21.15.61:0/1364771542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb100ac710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdb04002f70 tx=0x7fdb04006eb0).ready entity=mon.0 client_cookie=863f3c5e2a2de26b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.216+0000 7fdb0effd700 1 -- 172.21.15.61:0/1364771542 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb0400d5c0 con 0x7fdb1005c050 2022-07-12T08:41:21.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb0effd700 1 -- 172.21.15.61:0/1364771542 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb04000e90 con 0x7fdb1005c050 2022-07-12T08:41:21.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb0effd700 1 -- 172.21.15.61:0/1364771542 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb0400d910 con 0x7fdb1005c050 2022-07-12T08:41:21.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 -- 172.21.15.61:0/1364771542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 msgr2=0x7fdb100ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 --2- 172.21.15.61:0/1364771542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb100ac710 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fdb04002f70 tx=0x7fdb04006eb0).stop 2022-07-12T08:41:21.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 -- 172.21.15.61:0/1364771542 shutdown_connections 2022-07-12T08:41:21.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 --2- 172.21.15.61:0/1364771542 >> v2:172.21.15.111:3300/0 conn(0x7fdb1014a8f0 0x7fdb101443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 --2- 172.21.15.61:0/1364771542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb100ac710 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 -- 172.21.15.61:0/1364771542 >> 172.21.15.61:0/1364771542 conn(0x7fdb10062c00 msgr2=0x7fdb10062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 -- 172.21.15.61:0/1364771542 shutdown_connections 2022-07-12T08:41:21.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.220+0000 7fdb165b2700 1 -- 172.21.15.61:0/1364771542 wait complete. 2022-07-12T08:41:21.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 Processor -- start 2022-07-12T08:41:21.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 -- start start 2022-07-12T08:41:21.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb10141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb100b01f0 0x7fdb100ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb1014a8f0 0x7fdb10156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb100af520 con 0x7fdb1005c050 2022-07-12T08:41:21.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb100add50 con 0x7fdb100b01f0 2022-07-12T08:41:21.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb101432b0 con 0x7fdb1014a8f0 2022-07-12T08:41:21.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb10141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb10141800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53116/0 (socket says 172.21.15.61:53116) 2022-07-12T08:41:21.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 -- 172.21.15.61:0/3062923869 learned_addr learned my addr 172.21.15.61:0/3062923869 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:21.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb14b4f700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb1014a8f0 0x7fdb10156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0f7fe700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb100b01f0 0x7fdb100ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 -- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb1014a8f0 msgr2=0x7fdb10156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb1014a8f0 0x7fdb10156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 -- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb100b01f0 msgr2=0x7fdb100ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb100b01f0 0x7fdb100ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 -- 172.21.15.61:0/3062923869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0400b1e0 con 0x7fdb1005c050 2022-07-12T08:41:21.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0ffff700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb10141800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdb04002f70 tx=0x7fdb04015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb04006b40 con 0x7fdb1005c050 2022-07-12T08:41:21.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0400de40 con 0x7fdb1005c050 2022-07-12T08:41:21.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb100aee20 con 0x7fdb1005c050 2022-07-12T08:41:21.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.224+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb04011de0 con 0x7fdb1005c050 2022-07-12T08:41:21.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.228+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb100a1d90 con 0x7fdb1005c050 2022-07-12T08:41:21.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.228+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdb04025030 con 0x7fdb1005c050 2022-07-12T08:41:21.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.228+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb10143470 con 0x7fdb1005c050 2022-07-12T08:41:21.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.228+0000 7fdb0d7fa700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdaf0053a60 0x7fdaf0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.228+0000 7fdb0f7fe700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdaf0053a60 0x7fdaf0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.228+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb0406cea0 con 0x7fdb1005c050 2022-07-12T08:41:21.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.232+0000 7fdb0f7fe700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdaf0053a60 0x7fdaf0055f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdb0000d000 tx=0x7fdb00007e10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.232+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb04043e00 con 0x7fdb1005c050 2022-07-12T08:41:21.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.360+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdb100adf10 con 0x7fdb1005c050 2022-07-12T08:41:21.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb0d7fa700 1 -- 172.21.15.61:0/3062923869 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb04023090 con 0x7fdb1005c050 2022-07-12T08:41:21.362 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdaf0053a60 msgr2=0x7fdaf0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdaf0053a60 0x7fdaf0055f20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdb0000d000 tx=0x7fdb00007e10).stop 2022-07-12T08:41:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 msgr2=0x7fdb10141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb10141800 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdb04002f70 tx=0x7fdb04015000).stop 2022-07-12T08:41:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 shutdown_connections 2022-07-12T08:41:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb1014a8f0 0x7fdb10156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdaf0053a60 0x7fdaf0055f20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb100b01f0 0x7fdb100ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 --2- 172.21.15.61:0/3062923869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb1005c050 0x7fdb10141800 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 >> 172.21.15.61:0/3062923869 conn(0x7fdb10062c00 msgr2=0x7fdb100b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 shutdown_connections 2022-07-12T08:41:21.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.364+0000 7fdb165b2700 1 -- 172.21.15.61:0/3062923869 wait complete. 2022-07-12T08:41:21.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.2 2022-07-12T08:41:21.384 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-07-12T08:41:21.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 Processor -- start 2022-07-12T08:41:21.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- start start 2022-07-12T08:41:21.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f73cc05c050 0x7f73cc0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc0b01f0 0x7f73cc14a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73cc0ad2f0 con 0x7f73cc05cb00 2022-07-12T08:41:21.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73cc0b0a40 con 0x7f73cc05c050 2022-07-12T08:41:21.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc05c430 con 0x7f73cc0b01f0 2022-07-12T08:41:21.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f73cc05c050 0x7f73cc0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f73cc05c050 0x7f73cc0ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53124/0 (socket says 172.21.15.61:53124) 2022-07-12T08:41:21.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 -- 172.21.15.61:0/3834829247 learned_addr learned my addr 172.21.15.61:0/3834829247 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:21.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73ca59c700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc0b01f0 0x7f73cc14a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 -- 172.21.15.61:0/3834829247 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc0b01f0 msgr2=0x7f73cc14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:21.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc0b01f0 0x7f73cc14a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 -- 172.21.15.61:0/3834829247 >> v1:172.21.15.111:6789/0 conn(0x7f73cc05cb00 legacy=0x7f73cc1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 -- 172.21.15.61:0/3834829247 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc0b2420 con 0x7f73cc05c050 2022-07-12T08:41:21.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73ca59c700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc0b01f0 0x7f73cc14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:21.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73cad9d700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc05c050 0x7f73cc0ac710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f73b4002f70 tx=0x7f73b4006eb0).ready entity=mon.0 client_cookie=f889389e8fa74a63 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73c9d9b700 1 -- 172.21.15.61:0/3834829247 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73b400d5c0 con 0x7f73cc05c050 2022-07-12T08:41:21.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73c9d9b700 1 -- 172.21.15.61:0/3834829247 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b4000e90 con 0x7f73cc05c050 2022-07-12T08:41:21.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73c9d9b700 1 -- 172.21.15.61:0/3834829247 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73b400d950 con 0x7f73cc05c050 2022-07-12T08:41:21.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- 172.21.15.61:0/3834829247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc05c050 msgr2=0x7f73cc0ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc05c050 0x7f73cc0ac710 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f73b4002f70 tx=0x7f73b4006eb0).stop 2022-07-12T08:41:21.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- 172.21.15.61:0/3834829247 shutdown_connections 2022-07-12T08:41:21.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc0b01f0 0x7f73cc14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 --2- 172.21.15.61:0/3834829247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc05c050 0x7f73cc0ac710 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- 172.21.15.61:0/3834829247 >> 172.21.15.61:0/3834829247 conn(0x7f73cc062c00 msgr2=0x7f73cc062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- 172.21.15.61:0/3834829247 shutdown_connections 2022-07-12T08:41:21.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.496+0000 7f73d1035700 1 -- 172.21.15.61:0/3834829247 wait complete. 2022-07-12T08:41:21.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 Processor -- start 2022-07-12T08:41:21.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 -- start start 2022-07-12T08:41:21.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc05c050 0x7f73cc14db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73cc05cb00 0x7f73cc0b3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 0x7f73cc144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0661a0 con 0x7f73cc0b01f0 2022-07-12T08:41:21.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc066440 con 0x7f73cc05cb00 2022-07-12T08:41:21.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc0666e0 con 0x7f73cc05c050 2022-07-12T08:41:21.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 0x7f73cc144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 0x7f73cc144880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53130/0 (socket says 172.21.15.61:53130) 2022-07-12T08:41:21.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 -- 172.21.15.61:0/973098725 learned_addr learned my addr 172.21.15.61:0/973098725 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:21.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cad9d700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc05c050 0x7f73cc14db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73ca59c700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73cc05cb00 0x7f73cc0b3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 -- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc05c050 msgr2=0x7f73cc14db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc05c050 0x7f73cc14db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 -- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73cc05cb00 msgr2=0x7f73cc0b3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73cc05cb00 0x7f73cc0b3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 -- 172.21.15.61:0/973098725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b400b1e0 con 0x7f73cc0b01f0 2022-07-12T08:41:21.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73ca59c700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73cc05cb00 0x7f73cc0b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:21.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cad9d700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc05c050 0x7f73cc14db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:21.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73cb59e700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 0x7f73cc144880 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f73cc0b30c0 tx=0x7f73bc007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73bc00ce30 con 0x7f73cc0b01f0 2022-07-12T08:41:21.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc00f470 con 0x7f73cc0b01f0 2022-07-12T08:41:21.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73cc066b80 con 0x7f73cc0b01f0 2022-07-12T08:41:21.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73bc00f7d0 con 0x7f73cc0b01f0 2022-07-12T08:41:21.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.500+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73cc14e660 con 0x7f73cc0b01f0 2022-07-12T08:41:21.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.504+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f73bc019030 con 0x7f73cc0b01f0 2022-07-12T08:41:21.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.504+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73cc065ff0 con 0x7f73cc0b01f0 2022-07-12T08:41:21.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.504+0000 7f73c3fff700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73b00539b0 0x7f73b0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.504+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f73bc06b250 con 0x7f73cc0b01f0 2022-07-12T08:41:21.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.504+0000 7f73cad9d700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73b00539b0 0x7f73b0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.504+0000 7f73cad9d700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73b00539b0 0x7f73b0055e70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f73b401a000 tx=0x7f73b4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.508+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73bc0421b0 con 0x7f73cc0b01f0 2022-07-12T08:41:21.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.632+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f73cc057460 con 0x7f73cc0b01f0 2022-07-12T08:41:21.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73c3fff700 1 -- 172.21.15.61:0/973098725 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f73bc0457d0 con 0x7f73cc0b01f0 2022-07-12T08:41:21.646 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:21.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73b00539b0 msgr2=0x7f73b0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73b00539b0 0x7f73b0055e70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f73b401a000 tx=0x7f73b4015000).stop 2022-07-12T08:41:21.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 msgr2=0x7f73cc144880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 0x7f73cc144880 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f73cc0b30c0 tx=0x7f73bc007bc0).stop 2022-07-12T08:41:21.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 shutdown_connections 2022-07-12T08:41:21.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73cc0b01f0 0x7f73cc144880 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73b00539b0 0x7f73b0055e70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f73cc05cb00 0x7f73cc0b3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 --2- 172.21.15.61:0/973098725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73cc05c050 0x7f73cc14db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 >> 172.21.15.61:0/973098725 conn(0x7f73cc062c00 msgr2=0x7f73cc147370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 shutdown_connections 2022-07-12T08:41:21.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.636+0000 7f73d1035700 1 -- 172.21.15.61:0/973098725 wait complete. 2022-07-12T08:41:21.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.3 2022-07-12T08:41:21.683 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-07-12T08:41:21.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.828+0000 7f3d4f59e700 1 Processor -- start 2022-07-12T08:41:21.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.828+0000 7f3d4f59e700 1 -- start start 2022-07-12T08:41:21.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d50146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d500b4720 0x7f3d50146570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3d50146a60 0x7f3d5005bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d500b6b50 con 0x7f3d500b4720 2022-07-12T08:41:21.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d500b4f10 con 0x7f3d50146a60 2022-07-12T08:41:21.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500b51b0 con 0x7f3d500b6770 2022-07-12T08:41:21.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d500b4720 0x7f3d50146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d500b4720 0x7f3d50146570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53138/0 (socket says 172.21.15.61:53138) 2022-07-12T08:41:21.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 -- 172.21.15.61:0/3215787679 learned_addr learned my addr 172.21.15.61:0/3215787679 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:21.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4e59c700 1 --2- 172.21.15.61:0/3215787679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d50146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4ed9d700 1 --2- 172.21.15.61:0/3215787679 >> v2:172.21.15.111:3300/0 conn(0x7f3d50146a60 0x7f3d5005bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 -- 172.21.15.61:0/3215787679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 msgr2=0x7f3d50146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:21.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 --2- 172.21.15.61:0/3215787679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d50146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 -- 172.21.15.61:0/3215787679 >> v2:172.21.15.111:3300/0 conn(0x7f3d50146a60 msgr2=0x7f3d5005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:21.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 --2- 172.21.15.61:0/3215787679 >> v2:172.21.15.111:3300/0 conn(0x7f3d50146a60 0x7f3d5005bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 -- 172.21.15.61:0/3215787679 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500b5450 con 0x7f3d500b4720 2022-07-12T08:41:21.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4dd9b700 1 --2- 172.21.15.61:0/3215787679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d500b4720 0x7f3d50146570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3d38002f70 tx=0x7f3d38006e50).ready entity=mon.0 client_cookie=f4efd1148553bace server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4d59a700 1 -- 172.21.15.61:0/3215787679 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d3800d570 con 0x7f3d500b4720 2022-07-12T08:41:21.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4d59a700 1 -- 172.21.15.61:0/3215787679 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d38000e90 con 0x7f3d500b4720 2022-07-12T08:41:21.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4d59a700 1 -- 172.21.15.61:0/3215787679 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d3800d900 con 0x7f3d500b4720 2022-07-12T08:41:21.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- 172.21.15.61:0/3215787679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d500b4720 msgr2=0x7f3d50146570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/3215787679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d500b4720 0x7f3d50146570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f3d38002f70 tx=0x7f3d38006e50).stop 2022-07-12T08:41:21.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- 172.21.15.61:0/3215787679 shutdown_connections 2022-07-12T08:41:21.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/3215787679 >> v2:172.21.15.111:3300/0 conn(0x7f3d50146a60 0x7f3d5005bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/3215787679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d500b4720 0x7f3d50146570 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/3215787679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d50146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- 172.21.15.61:0/3215787679 >> 172.21.15.61:0/3215787679 conn(0x7f3d500b9d50 msgr2=0x7f3d500b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- 172.21.15.61:0/3215787679 shutdown_connections 2022-07-12T08:41:21.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.832+0000 7f3d4f59e700 1 -- 172.21.15.61:0/3215787679 wait complete. 2022-07-12T08:41:21.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 Processor -- start 2022-07-12T08:41:21.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 -- start start 2022-07-12T08:41:21.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d500b4720 0x7f3d50142480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d5013c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 0x7f3d5013c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500b06c0 con 0x7f3d50146a60 2022-07-12T08:41:21.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5009c3e0 con 0x7f3d500b4720 2022-07-12T08:41:21.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5009c680 con 0x7f3d500b6770 2022-07-12T08:41:21.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 0x7f3d5013c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 0x7f3d5013c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53146/0 (socket says 172.21.15.61:53146) 2022-07-12T08:41:21.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 -- 172.21.15.61:0/4273394842 learned_addr learned my addr 172.21.15.61:0/4273394842 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:21.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4e59c700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d500b4720 0x7f3d50142480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4dd9b700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d5013c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 -- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 msgr2=0x7f3d5013c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d5013c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 -- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d500b4720 msgr2=0x7f3d50142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d500b4720 0x7f3d50142480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 -- 172.21.15.61:0/4273394842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3800b1e0 con 0x7f3d50146a60 2022-07-12T08:41:21.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4dd9b700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d5013c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:21.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4e59c700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d500b4720 0x7f3d50142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4ed9d700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 0x7f3d5013c9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3d48006e00 tx=0x7f3d480091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d4800c670 con 0x7f3d50146a60 2022-07-12T08:41:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4800c7d0 con 0x7f3d50146a60 2022-07-12T08:41:21.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d4800cb30 con 0x7f3d50146a60 2022-07-12T08:41:21.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5009c920 con 0x7f3d50146a60 2022-07-12T08:41:21.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.836+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d500b2e60 con 0x7f3d50146a60 2022-07-12T08:41:21.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.840+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5009cab0 con 0x7f3d50146a60 2022-07-12T08:41:21.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.840+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3d4800cc90 con 0x7f3d50146a60 2022-07-12T08:41:21.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.840+0000 7f3d477fe700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d280539b0 0x7f3d28055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:21.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.840+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3d4806c1b0 con 0x7f3d50146a60 2022-07-12T08:41:21.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.840+0000 7f3d4e59c700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d280539b0 0x7f3d28055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:21.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.840+0000 7f3d4e59c700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d280539b0 0x7f3d28055e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f3d4000b000 tx=0x7f3d40006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:21.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.848+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d48043000 con 0x7f3d50146a60 2022-07-12T08:41:21.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3d50057460 con 0x7f3d50146a60 2022-07-12T08:41:21.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d477fe700 1 -- 172.21.15.61:0/4273394842 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d48046620 con 0x7f3d50146a60 2022-07-12T08:41:21.983 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:21.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d280539b0 msgr2=0x7f3d28055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d280539b0 0x7f3d28055e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f3d4000b000 tx=0x7f3d40006df0).stop 2022-07-12T08:41:21.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 msgr2=0x7f3d5013c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:21.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 0x7f3d5013c9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f3d48006e00 tx=0x7f3d480091d0).stop 2022-07-12T08:41:21.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 shutdown_connections 2022-07-12T08:41:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d50146a60 0x7f3d5013c9d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d500b6770 0x7f3d5013c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d280539b0 0x7f3d28055e70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 --2- 172.21.15.61:0/4273394842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d500b4720 0x7f3d50142480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 >> 172.21.15.61:0/4273394842 conn(0x7f3d500b9d50 msgr2=0x7f3d5005c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:21.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 shutdown_connections 2022-07-12T08:41:21.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:21.976+0000 7f3d4f59e700 1 -- 172.21.15.61:0/4273394842 wait complete. 2022-07-12T08:41:21.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574853 for osd.4 2022-07-12T08:41:21.991 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-07-12T08:41:22.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 Processor -- start 2022-07-12T08:41:22.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 -- start start 2022-07-12T08:41:22.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65a009bce0 0x7f65a00af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a00aecb0 con 0x7f65a00af280 2022-07-12T08:41:22.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a0143b00 con 0x7f65a009bce0 2022-07-12T08:41:22.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.116+0000 7f65a5876700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01435c0 con 0x7f65a00b3f60 2022-07-12T08:41:22.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65a009bce0 0x7f65a00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65a009bce0 0x7f65a00af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53150/0 (socket says 172.21.15.61:53150) 2022-07-12T08:41:22.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 -- 172.21.15.61:0/4263431305 learned_addr learned my addr 172.21.15.61:0/4263431305 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:22.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5075700 1 --2- 172.21.15.61:0/4263431305 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 -- 172.21.15.61:0/4263431305 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 msgr2=0x7f65a00b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:22.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 --2- 172.21.15.61:0/4263431305 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a00b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 -- 172.21.15.61:0/4263431305 >> v1:172.21.15.111:6789/0 conn(0x7f65a00af280 legacy=0x7f65a0060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 -- 172.21.15.61:0/4263431305 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a0143860 con 0x7f65a009bce0 2022-07-12T08:41:22.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f6597fff700 1 --2- 172.21.15.61:0/4263431305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a009bce0 0x7f65a00af660 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6590002f70 tx=0x7f6590006eb0).ready entity=mon.0 client_cookie=375f966aeb9af807 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65977fe700 1 -- 172.21.15.61:0/4263431305 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659000d570 con 0x7f65a009bce0 2022-07-12T08:41:22.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65977fe700 1 -- 172.21.15.61:0/4263431305 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590000e90 con 0x7f65a009bce0 2022-07-12T08:41:22.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65977fe700 1 -- 172.21.15.61:0/4263431305 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659000d900 con 0x7f65a009bce0 2022-07-12T08:41:22.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 -- 172.21.15.61:0/4263431305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a009bce0 msgr2=0x7f65a00af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 --2- 172.21.15.61:0/4263431305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a009bce0 0x7f65a00af660 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f6590002f70 tx=0x7f6590006eb0).stop 2022-07-12T08:41:22.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 -- 172.21.15.61:0/4263431305 shutdown_connections 2022-07-12T08:41:22.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 --2- 172.21.15.61:0/4263431305 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a00b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 --2- 172.21.15.61:0/4263431305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a009bce0 0x7f65a00af660 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 -- 172.21.15.61:0/4263431305 >> 172.21.15.61:0/4263431305 conn(0x7f65a00b92c0 msgr2=0x7f65a00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:22.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 -- 172.21.15.61:0/4263431305 shutdown_connections 2022-07-12T08:41:22.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 -- 172.21.15.61:0/4263431305 wait complete. 2022-07-12T08:41:22.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 Processor -- start 2022-07-12T08:41:22.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.120+0000 7f65a5876700 1 -- start start 2022-07-12T08:41:22.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a009bce0 0x7f65a0153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 0x7f65a0142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a009e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a009c8d0 con 0x7f65a00af280 2022-07-12T08:41:22.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a009cb70 con 0x7f65a009bce0 2022-07-12T08:41:22.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0151d10 con 0x7f65a00b3f60 2022-07-12T08:41:22.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 0x7f65a0142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 0x7f65a0142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53158/0 (socket says 172.21.15.61:53158) 2022-07-12T08:41:22.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 -- 172.21.15.61:0/1725829771 learned_addr learned my addr 172.21.15.61:0/1725829771 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:22.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a4874700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a009bce0 0x7f65a0153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5075700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a009e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 -- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 msgr2=0x7f65a009e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a009e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 -- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a009bce0 msgr2=0x7f65a0153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a009bce0 0x7f65a0153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 -- 172.21.15.61:0/1725829771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659000b1e0 con 0x7f65a00af280 2022-07-12T08:41:22.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6597fff700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 0x7f65a0142a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f6590002f70 tx=0x7f6590015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65900117e0 con 0x7f65a00af280 2022-07-12T08:41:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590011940 con 0x7f65a00af280 2022-07-12T08:41:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a009c350 con 0x7f65a00af280 2022-07-12T08:41:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6590011ca0 con 0x7f65a00af280 2022-07-12T08:41:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a01522d0 con 0x7f65a00af280 2022-07-12T08:41:22.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.124+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a0057460 con 0x7f65a00af280 2022-07-12T08:41:22.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.128+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6590025030 con 0x7f65a00af280 2022-07-12T08:41:22.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.128+0000 7f6595ffb700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65800539b0 0x7f6580055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.128+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f659006d1d0 con 0x7f65a00af280 2022-07-12T08:41:22.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.128+0000 7f65a4874700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65800539b0 0x7f6580055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.128+0000 7f65a4874700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65800539b0 0x7f6580055e70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f659c00b000 tx=0x7f659c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.132+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6590044090 con 0x7f65a00af280 2022-07-12T08:41:22.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.260+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f65a00afeb0 con 0x7f65a00af280 2022-07-12T08:41:22.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.260+0000 7f6595ffb700 1 -- 172.21.15.61:0/1725829771 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6590023090 con 0x7f65a00af280 2022-07-12T08:41:22.264 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:22.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65800539b0 msgr2=0x7f6580055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65800539b0 0x7f6580055e70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f659c00b000 tx=0x7f659c009040).stop 2022-07-12T08:41:22.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 msgr2=0x7f65a0142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 0x7f65a0142a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f6590002f70 tx=0x7f6590015000).stop 2022-07-12T08:41:22.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 shutdown_connections 2022-07-12T08:41:22.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65a00b3f60 0x7f65a009e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65a00af280 0x7f65a0142a40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65800539b0 0x7f6580055e70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 --2- 172.21.15.61:0/1725829771 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65a009bce0 0x7f65a0153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 >> 172.21.15.61:0/1725829771 conn(0x7f65a00b92c0 msgr2=0x7f65a013e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:22.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 shutdown_connections 2022-07-12T08:41:22.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.264+0000 7f65a5876700 1 -- 172.21.15.61:0/1725829771 wait complete. 2022-07-12T08:41:22.282 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2022-07-12T08:41:22.282 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-07-12T08:41:22.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 Processor -- start 2022-07-12T08:41:22.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 -- start start 2022-07-12T08:41:22.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c013ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd2c00b0350 0x7fd2c00b0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c00afb60 con 0x7fd2c00af780 2022-07-12T08:41:22.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2c01435a0 con 0x7fd2c00b0350 2022-07-12T08:41:22.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.388+0000 7fd2bffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0143840 con 0x7fd2c00613a0 2022-07-12T08:41:22.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bf7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd2c00b0350 0x7fd2c00b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2beffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd2c00af780 0x7fd2c013c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41382/0 (socket says 172.21.15.61:41382) 2022-07-12T08:41:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2beffd700 1 -- 172.21.15.61:0/3846339783 learned_addr learned my addr 172.21.15.61:0/3846339783 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2b7fff700 1 --2- 172.21.15.61:0/3846339783 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c013ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bf7fe700 1 -- 172.21.15.61:0/3846339783 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 msgr2=0x7fd2c013ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bf7fe700 1 --2- 172.21.15.61:0/3846339783 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c013ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bf7fe700 1 -- 172.21.15.61:0/3846339783 >> v1:172.21.15.61:6789/0 conn(0x7fd2c00af780 legacy=0x7fd2c013c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2be7fc700 1 -- 172.21.15.61:0/3846339783 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340589169 0 0) 0x7fd2c00afb60 con 0x7fd2c00af780 2022-07-12T08:41:22.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bf7fe700 1 -- 172.21.15.61:0/3846339783 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0067430 con 0x7fd2c00b0350 2022-07-12T08:41:22.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bf7fe700 1 --2- 172.21.15.61:0/3846339783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00b0350 0x7fd2c00b0790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd2b80032f0 tx=0x7fd2b80091d0).ready entity=mon.0 client_cookie=308d7605ec49e4d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2be7fc700 1 -- 172.21.15.61:0/3846339783 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2b800b6c0 con 0x7fd2c00b0350 2022-07-12T08:41:22.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2be7fc700 1 -- 172.21.15.61:0/3846339783 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b800b820 con 0x7fd2c00b0350 2022-07-12T08:41:22.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2be7fc700 1 -- 172.21.15.61:0/3846339783 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2b800b990 con 0x7fd2c00b0350 2022-07-12T08:41:22.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 -- 172.21.15.61:0/3846339783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00b0350 msgr2=0x7fd2c00b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 --2- 172.21.15.61:0/3846339783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00b0350 0x7fd2c00b0790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fd2b80032f0 tx=0x7fd2b80091d0).stop 2022-07-12T08:41:22.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 -- 172.21.15.61:0/3846339783 shutdown_connections 2022-07-12T08:41:22.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 --2- 172.21.15.61:0/3846339783 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00b0350 0x7fd2c00b0790 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 --2- 172.21.15.61:0/3846339783 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c013ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 -- 172.21.15.61:0/3846339783 >> 172.21.15.61:0/3846339783 conn(0x7fd2c00b5780 msgr2=0x7fd2c00b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:22.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 -- 172.21.15.61:0/3846339783 shutdown_connections 2022-07-12T08:41:22.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 -- 172.21.15.61:0/3846339783 wait complete. 2022-07-12T08:41:22.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.392+0000 7fd2bffff700 1 Processor -- start 2022-07-12T08:41:22.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 -- start start 2022-07-12T08:41:22.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c0065d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 0x7fd2c00ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c00b0350 0x7fd2c0144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0067090 con 0x7fd2c00af780 2022-07-12T08:41:22.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c01460a0 con 0x7fd2c00b0350 2022-07-12T08:41:22.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2c0066df0 con 0x7fd2c00613a0 2022-07-12T08:41:22.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 0x7fd2c00ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 0x7fd2c00ae440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53172/0 (socket says 172.21.15.61:53172) 2022-07-12T08:41:22.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 -- 172.21.15.61:0/1537709824 learned_addr learned my addr 172.21.15.61:0/1537709824 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:22.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2beffd700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c0065d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bf7fe700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c00b0350 0x7fd2c0144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 -- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 msgr2=0x7fd2c0065d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c0065d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 -- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c00b0350 msgr2=0x7fd2c0144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c00b0350 0x7fd2c0144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 -- 172.21.15.61:0/1537709824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b8009470 con 0x7fd2c00af780 2022-07-12T08:41:22.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2b7fff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 0x7fd2c00ae440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fd2a800c000 tx=0x7fd2a8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2a8010780 con 0x7fd2c00af780 2022-07-12T08:41:22.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a80108e0 con 0x7fd2c00af780 2022-07-12T08:41:22.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c0064d40 con 0x7fd2c00af780 2022-07-12T08:41:22.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2a8010c40 con 0x7fd2c00af780 2022-07-12T08:41:22.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.396+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c0146580 con 0x7fd2c00af780 2022-07-12T08:41:22.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.400+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c0057460 con 0x7fd2c00af780 2022-07-12T08:41:22.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.400+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd2a801c460 con 0x7fd2c00af780 2022-07-12T08:41:22.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.400+0000 7fd2bcff9700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2a4053a40 0x7fd2a4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.400+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd2a806c470 con 0x7fd2c00af780 2022-07-12T08:41:22.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.400+0000 7fd2beffd700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2a4053a40 0x7fd2a4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.400+0000 7fd2beffd700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2a4053a40 0x7fd2a4055f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fd2b000d000 tx=0x7fd2b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.408+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd2a80431e0 con 0x7fd2c00af780 2022-07-12T08:41:22.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.536+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd2c00b1730 con 0x7fd2c00af780 2022-07-12T08:41:22.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.536+0000 7fd2bcff9700 1 -- 172.21.15.61:0/1537709824 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2a8046800 con 0x7fd2c00af780 2022-07-12T08:41:22.541 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:22.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2a4053a40 msgr2=0x7fd2a4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2a4053a40 0x7fd2a4055f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fd2b000d000 tx=0x7fd2b0009040).stop 2022-07-12T08:41:22.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 msgr2=0x7fd2c00ae440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 0x7fd2c00ae440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fd2a800c000 tx=0x7fd2a8006df0).stop 2022-07-12T08:41:22.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 shutdown_connections 2022-07-12T08:41:22.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2a4053a40 0x7fd2a4055f00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd2c00b0350 0x7fd2c0144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2c00af780 0x7fd2c00ae440 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 --2- 172.21.15.61:0/1537709824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd2c00613a0 0x7fd2c0065d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 >> 172.21.15.61:0/1537709824 conn(0x7fd2c00b5780 msgr2=0x7fd2c00b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:22.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 shutdown_connections 2022-07-12T08:41:22.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.540+0000 7fd2bffff700 1 -- 172.21.15.61:0/1537709824 wait complete. 2022-07-12T08:41:22.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2022-07-12T08:41:22.556 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-07-12T08:41:22.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 Processor -- start 2022-07-12T08:41:22.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 -- start start 2022-07-12T08:41:22.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680aca50 0x7fdc680aee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdc680b1660 0x7fdc68143080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc680ace30 con 0x7fdc680b1200 2022-07-12T08:41:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc680ae420 con 0x7fdc680b1660 2022-07-12T08:41:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.664+0000 7fdc6fae6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680ae6c0 con 0x7fdc680aca50 2022-07-12T08:41:22.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d081700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdc680b1200 0x7fdc68142970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41396/0 (socket says 172.21.15.61:41396) 2022-07-12T08:41:22.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d081700 1 -- 172.21.15.61:0/3971169864 learned_addr learned my addr 172.21.15.61:0/3971169864 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:22.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6e083700 1 --2- 172.21.15.61:0/3971169864 >> v2:172.21.15.61:3300/0 conn(0x7fdc680b1660 0x7fdc68143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d882700 1 --2- 172.21.15.61:0/3971169864 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680aca50 0x7fdc680aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6c880700 1 -- 172.21.15.61:0/3971169864 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205663682 0 0) 0x7fdc680ace30 con 0x7fdc680b1200 2022-07-12T08:41:22.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6c880700 1 -- 172.21.15.61:0/3971169864 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc50003260 con 0x7fdc680b1200 2022-07-12T08:41:22.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d882700 1 -- 172.21.15.61:0/3971169864 >> v2:172.21.15.61:3300/0 conn(0x7fdc680b1660 msgr2=0x7fdc68143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:22.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d882700 1 --2- 172.21.15.61:0/3971169864 >> v2:172.21.15.61:3300/0 conn(0x7fdc680b1660 0x7fdc68143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d882700 1 -- 172.21.15.61:0/3971169864 >> v1:172.21.15.61:6789/0 conn(0x7fdc680b1200 legacy=0x7fdc68142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d882700 1 -- 172.21.15.61:0/3971169864 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc680ae960 con 0x7fdc680aca50 2022-07-12T08:41:22.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6d882700 1 --2- 172.21.15.61:0/3971169864 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680aca50 0x7fdc680aee00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fdc5c002f70 tx=0x7fdc5c006df0).ready entity=mon.2 client_cookie=d3f9b6dd0ec55cce server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6c880700 1 -- 172.21.15.61:0/3971169864 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc5c00d4a0 con 0x7fdc680aca50 2022-07-12T08:41:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6c880700 1 -- 172.21.15.61:0/3971169864 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5c000e90 con 0x7fdc680aca50 2022-07-12T08:41:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6c880700 1 -- 172.21.15.61:0/3971169864 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc5c00d7f0 con 0x7fdc680aca50 2022-07-12T08:41:22.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 -- 172.21.15.61:0/3971169864 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680aca50 msgr2=0x7fdc680aee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/3971169864 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680aca50 0x7fdc680aee00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fdc5c002f70 tx=0x7fdc5c006df0).stop 2022-07-12T08:41:22.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 -- 172.21.15.61:0/3971169864 shutdown_connections 2022-07-12T08:41:22.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/3971169864 >> v2:172.21.15.61:3300/0 conn(0x7fdc680b1660 0x7fdc68143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/3971169864 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680aca50 0x7fdc680aee00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 -- 172.21.15.61:0/3971169864 >> 172.21.15.61:0/3971169864 conn(0x7fdc68062c00 msgr2=0x7fdc68062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:22.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 -- 172.21.15.61:0/3971169864 shutdown_connections 2022-07-12T08:41:22.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.668+0000 7fdc6fae6700 1 -- 172.21.15.61:0/3971169864 wait complete. 2022-07-12T08:41:22.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 Processor -- start 2022-07-12T08:41:22.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 -- start start 2022-07-12T08:41:22.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc680aca50 0x7fdc68142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 0x7fdc68066f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680b1660 0x7fdc68067470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc68066b00 con 0x7fdc680b1200 2022-07-12T08:41:22.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc680a0080 con 0x7fdc680aca50 2022-07-12T08:41:22.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc68065750 con 0x7fdc680b1660 2022-07-12T08:41:22.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 0x7fdc68066f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 0x7fdc68066f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53188/0 (socket says 172.21.15.61:53188) 2022-07-12T08:41:22.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 -- 172.21.15.61:0/2817735453 learned_addr learned my addr 172.21.15.61:0/2817735453 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:22.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d882700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc680aca50 0x7fdc68142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6e083700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680b1660 0x7fdc68067470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 -- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680b1660 msgr2=0x7fdc68067470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680b1660 0x7fdc68067470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 -- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc680aca50 msgr2=0x7fdc68142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc680aca50 0x7fdc68142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 -- 172.21.15.61:0/2817735453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c00b180 con 0x7fdc680b1200 2022-07-12T08:41:22.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6e083700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680b1660 0x7fdc68067470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:22.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6d081700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 0x7fdc68066f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fdc64001110 tx=0x7fdc64007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc6400ce50 con 0x7fdc680b1200 2022-07-12T08:41:22.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc640072e0 con 0x7fdc680b1200 2022-07-12T08:41:22.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc68067960 con 0x7fdc680b1200 2022-07-12T08:41:22.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc64005800 con 0x7fdc680b1200 2022-07-12T08:41:22.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.672+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc681393b0 con 0x7fdc680b1200 2022-07-12T08:41:22.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.676+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6809fec0 con 0x7fdc680b1200 2022-07-12T08:41:22.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.676+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdc6401a060 con 0x7fdc680b1200 2022-07-12T08:41:22.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.676+0000 7fdc5affd700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdc48053a40 0x7fdc48055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.676+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdc6406b340 con 0x7fdc680b1200 2022-07-12T08:41:22.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.676+0000 7fdc6d882700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdc48053a40 0x7fdc48055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.676+0000 7fdc6d882700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdc48053a40 0x7fdc48055f00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdc5c01a000 tx=0x7fdc5c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.680+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc640422a0 con 0x7fdc680b1200 2022-07-12T08:41:22.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.804+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdc68057460 con 0x7fdc680b1200 2022-07-12T08:41:22.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.804+0000 7fdc5affd700 1 -- 172.21.15.61:0/2817735453 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc64018020 con 0x7fdc680b1200 2022-07-12T08:41:22.804 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-12T08:41:22.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdc48053a40 msgr2=0x7fdc48055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdc48053a40 0x7fdc48055f00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fdc5c01a000 tx=0x7fdc5c014000).stop 2022-07-12T08:41:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 msgr2=0x7fdc68066f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 0x7fdc68066f80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fdc64001110 tx=0x7fdc64007b80).stop 2022-07-12T08:41:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 shutdown_connections 2022-07-12T08:41:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdc680b1660 0x7fdc68067470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc680b1200 0x7fdc68066f80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdc48053a40 0x7fdc48055f00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 --2- 172.21.15.61:0/2817735453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdc680aca50 0x7fdc68142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 >> 172.21.15.61:0/2817735453 conn(0x7fdc68062c00 msgr2=0x7fdc680b5660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 shutdown_connections 2022-07-12T08:41:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.808+0000 7fdc6fae6700 1 -- 172.21.15.61:0/2817735453 wait complete. 2022-07-12T08:41:22.824 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2022-07-12T08:41:22.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-07-12T08:41:22.825 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-12T08:41:22.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 Processor -- start 2022-07-12T08:41:22.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 -- start start 2022-07-12T08:41:22.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0ab290 0x7f27ec0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f27ec0afa60 0x7f27ec0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:22.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec0abc70 con 0x7f27ec05f140 2022-07-12T08:41:22.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ec05f580 con 0x7f27ec0afa60 2022-07-12T08:41:22.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec05c720 con 0x7f27ec0ab290 2022-07-12T08:41:22.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27ebfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f27ec05f140 0x7f27ec05ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41412/0 (socket says 172.21.15.61:41412) 2022-07-12T08:41:22.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27ebfff700 1 -- 172.21.15.61:0/2680838098 learned_addr learned my addr 172.21.15.61:0/2680838098 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:22.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f0ebc700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0ab290 0x7f27ec0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f16bd700 1 --2- 172.21.15.61:0/2680838098 >> v2:172.21.15.111:3300/0 conn(0x7f27ec0afa60 0x7f27ec0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:22.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27eb7fe700 1 -- 172.21.15.61:0/2680838098 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751684555 0 0) 0x7f27ec0abc70 con 0x7f27ec05f140 2022-07-12T08:41:22.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27eb7fe700 1 -- 172.21.15.61:0/2680838098 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc003260 con 0x7f27ec05f140 2022-07-12T08:41:22.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27eb7fe700 1 -- 172.21.15.61:0/2680838098 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1075908978 0 0) 0x7f27cc003260 con 0x7f27ec05f140 2022-07-12T08:41:22.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f16bd700 1 -- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0ab290 msgr2=0x7f27ec0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:22.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f16bd700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0ab290 0x7f27ec0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:22.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f16bd700 1 -- 172.21.15.61:0/2680838098 >> v1:172.21.15.61:6789/0 conn(0x7f27ec05f140 legacy=0x7f27ec05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f16bd700 1 -- 172.21.15.61:0/2680838098 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec05c9c0 con 0x7f27ec0afa60 2022-07-12T08:41:22.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f0ebc700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0ab290 0x7f27ec0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:22.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f16bd700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec0afa60 0x7f27ec0aff10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f27e4001b80 tx=0x7f27dc006df0).ready entity=mon.1 client_cookie=a082aa9343113953 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:22.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27eb7fe700 1 -- 172.21.15.61:0/2680838098 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27e40019e0 con 0x7f27ec0afa60 2022-07-12T08:41:22.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27eb7fe700 1 -- 172.21.15.61:0/2680838098 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc000e90 con 0x7f27ec0afa60 2022-07-12T08:41:22.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27eb7fe700 1 -- 172.21.15.61:0/2680838098 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27dc012460 con 0x7f27ec0afa60 2022-07-12T08:41:22.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 -- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec0afa60 msgr2=0x7f27ec0aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:22.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.940+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec0afa60 0x7f27ec0aff10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f27e4001b80 tx=0x7f27dc006df0).stop 2022-07-12T08:41:23.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- 172.21.15.61:0/2680838098 shutdown_connections 2022-07-12T08:41:23.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec0afa60 0x7f27ec0aff10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/2680838098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0ab290 0x7f27ec0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- 172.21.15.61:0/2680838098 >> 172.21.15.61:0/2680838098 conn(0x7f27ec0b1950 msgr2=0x7f27ec0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- 172.21.15.61:0/2680838098 shutdown_connections 2022-07-12T08:41:23.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- 172.21.15.61:0/2680838098 wait complete. 2022-07-12T08:41:23.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 Processor -- start 2022-07-12T08:41:23.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- start start 2022-07-12T08:41:23.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec05f140 0x7f27ec08fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 0x7f27ec066bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0afa60 0x7f27ec0970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec0923d0 con 0x7f27ec0ab290 2022-07-12T08:41:23.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec092670 con 0x7f27ec05f140 2022-07-12T08:41:23.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f1ebe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec092910 con 0x7f27ec0afa60 2022-07-12T08:41:23.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 0x7f27ec066bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 0x7f27ec066bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53198/0 (socket says 172.21.15.61:53198) 2022-07-12T08:41:23.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27ebfff700 1 -- 172.21.15.61:0/554862820 learned_addr learned my addr 172.21.15.61:0/554862820 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:23.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f16bd700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0afa60 0x7f27ec0970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.944+0000 7f27f0ebc700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec05f140 0x7f27ec08fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27ebfff700 1 -- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0afa60 msgr2=0x7f27ec0970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27ebfff700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0afa60 0x7f27ec0970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27ebfff700 1 -- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec05f140 msgr2=0x7f27ec08fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27ebfff700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec05f140 0x7f27ec08fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27ebfff700 1 -- 172.21.15.61:0/554862820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d400b180 con 0x7f27ec0ab290 2022-07-12T08:41:23.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27f16bd700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0afa60 0x7f27ec0970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:23.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27ebfff700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 0x7f27ec066bd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f27e4001c40 tx=0x7f27e4001640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27e4004ab0 con 0x7f27ec0ab290 2022-07-12T08:41:23.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e4004c10 con 0x7f27ec0ab290 2022-07-12T08:41:23.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc00b180 con 0x7f27ec0ab290 2022-07-12T08:41:23.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27e4007170 con 0x7f27ec0ab290 2022-07-12T08:41:23.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec067140 con 0x7f27ec0ab290 2022-07-12T08:41:23.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec08fc70 con 0x7f27ec0ab290 2022-07-12T08:41:23.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f27e4010070 con 0x7f27ec0ab290 2022-07-12T08:41:23.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27e9ffb700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f27d0053ad0 0x7f27d0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f27e406bcc0 con 0x7f27ec0ab290 2022-07-12T08:41:23.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.948+0000 7f27f0ebc700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f27d0053ad0 0x7f27d0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.952+0000 7f27f0ebc700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f27d0053ad0 0x7f27d0055f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f27d400b000 tx=0x7f27d4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:22.956+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f27e4042c20 con 0x7f27ec0ab290 2022-07-12T08:41:23.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.076+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27ec057460 con 0x7f27d0053ad0 2022-07-12T08:41:23.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.084+0000 7f27e9ffb700 1 -- 172.21.15.61:0/554862820 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82241 (secure 0 0 0) 0x7f27ec057460 con 0x7f27d0053ad0 2022-07-12T08:41:23.084 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:23.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f27d0053ad0 msgr2=0x7f27d0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f27d0053ad0 0x7f27d0055f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f27d400b000 tx=0x7f27d4006df0).stop 2022-07-12T08:41:23.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 msgr2=0x7f27ec066bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 0x7f27ec066bd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f27e4001c40 tx=0x7f27e4001640).stop 2022-07-12T08:41:23.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 shutdown_connections 2022-07-12T08:41:23.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27ec0afa60 0x7f27ec0970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f27ec0ab290 0x7f27ec066bd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f27d0053ad0 0x7f27d0055f90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 --2- 172.21.15.61:0/554862820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27ec05f140 0x7f27ec08fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 >> 172.21.15.61:0/554862820 conn(0x7f27ec0b1950 msgr2=0x7f27ec0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 shutdown_connections 2022-07-12T08:41:23.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.088+0000 7f27f1ebe700 1 -- 172.21.15.61:0/554862820 wait complete. 2022-07-12T08:41:23.097 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-12T08:41:23.107 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2022-07-12T08:41:21.085285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6366896,"kb_used_data":743228176,"kb_used_omap":569,"kb_used_meta":0,"kb_avail":743228144,"statfs":{"total":767585320960,"available":761065619456,"internally_reserved":32768,"allocated":761065652224,"data_stored":761065652224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":583091,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":160,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001857"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592471+0000","last_change":"2022-07-12T08:41:14.592304+0000","last_active":"2022-07-12T08:41:14.592471+0000","last_peered":"2022-07-12T08:41:14.592471+0000","last_clean":"2022-07-12T08:41:14.592471+0000","last_became_active":"2022-07-12T08:41:14.591935+0000","last_became_peered":"2022-07-12T08:41:14.591935+0000","last_unstale":"2022-07-12T08:41:14.592471+0000","last_undegraded":"2022-07-12T08:41:14.592471+0000","last_fullsized":"2022-07-12T08:41:14.592471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593820+0000","last_change":"2022-07-12T08:41:04.955012+0000","last_active":"2022-07-12T08:41:14.593820+0000","last_peered":"2022-07-12T08:41:14.593820+0000","last_clean":"2022-07-12T08:41:14.593820+0000","last_became_active":"2022-07-12T08:41:04.954339+0000","last_became_peered":"2022-07-12T08:41:04.954339+0000","last_unstale":"2022-07-12T08:41:14.593820+0000","last_undegraded":"2022-07-12T08:41:14.593820+0000","last_fullsized":"2022-07-12T08:41:14.593820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_clean_scrub_stamp":"2022-07-12T08:40:58.647421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.585823+0000","last_change":"2022-07-12T08:41:11.067479+0000","last_active":"2022-07-12T08:41:14.585823+0000","last_peered":"2022-07-12T08:41:14.585823+0000","last_clean":"2022-07-12T08:41:14.585823+0000","last_became_active":"2022-07-12T08:41:11.067215+0000","last_became_peered":"2022-07-12T08:41:11.067215+0000","last_unstale":"2022-07-12T08:41:14.585823+0000","last_undegraded":"2022-07-12T08:41:14.585823+0000","last_fullsized":"2022-07-12T08:41:14.585823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.600509+0000","last_change":"2022-07-12T08:41:14.252385+0000","last_active":"2022-07-12T08:41:14.600509+0000","last_peered":"2022-07-12T08:41:14.600509+0000","last_clean":"2022-07-12T08:41:14.600509+0000","last_became_active":"2022-07-12T08:41:14.252022+0000","last_became_peered":"2022-07-12T08:41:14.252022+0000","last_unstale":"2022-07-12T08:41:14.600509+0000","last_undegraded":"2022-07-12T08:41:14.600509+0000","last_fullsized":"2022-07-12T08:41:14.600509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593852+0000","last_change":"2022-07-12T08:41:14.593713+0000","last_active":"2022-07-12T08:41:14.593852+0000","last_peered":"2022-07-12T08:41:14.593852+0000","last_clean":"2022-07-12T08:41:14.593852+0000","last_became_active":"2022-07-12T08:41:14.593266+0000","last_became_peered":"2022-07-12T08:41:14.593266+0000","last_unstale":"2022-07-12T08:41:14.593852+0000","last_undegraded":"2022-07-12T08:41:14.593852+0000","last_fullsized":"2022-07-12T08:41:14.593852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593698+0000","last_change":"2022-07-12T08:41:11.064070+0000","last_active":"2022-07-12T08:41:14.593698+0000","last_peered":"2022-07-12T08:41:14.593698+0000","last_clean":"2022-07-12T08:41:14.593698+0000","last_became_active":"2022-07-12T08:41:11.063329+0000","last_became_peered":"2022-07-12T08:41:11.063329+0000","last_unstale":"2022-07-12T08:41:14.593698+0000","last_undegraded":"2022-07-12T08:41:14.593698+0000","last_fullsized":"2022-07-12T08:41:14.593698+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.590378+0000","last_change":"2022-07-12T08:41:14.254382+0000","last_active":"2022-07-12T08:41:14.590378+0000","last_peered":"2022-07-12T08:41:14.590378+0000","last_clean":"2022-07-12T08:41:14.590378+0000","last_became_active":"2022-07-12T08:41:14.254057+0000","last_became_peered":"2022-07-12T08:41:14.254057+0000","last_unstale":"2022-07-12T08:41:14.590378+0000","last_undegraded":"2022-07-12T08:41:14.590378+0000","last_fullsized":"2022-07-12T08:41:14.590378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592135+0000","last_change":"2022-07-12T08:41:14.591841+0000","last_active":"2022-07-12T08:41:14.592135+0000","last_peered":"2022-07-12T08:41:14.592135+0000","last_clean":"2022-07-12T08:41:14.592135+0000","last_became_active":"2022-07-12T08:41:14.591365+0000","last_became_peered":"2022-07-12T08:41:14.591365+0000","last_unstale":"2022-07-12T08:41:14.592135+0000","last_undegraded":"2022-07-12T08:41:14.592135+0000","last_fullsized":"2022-07-12T08:41:14.592135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.601087+0000","last_change":"2022-07-12T08:41:11.071303+0000","last_active":"2022-07-12T08:41:14.601087+0000","last_peered":"2022-07-12T08:41:14.601087+0000","last_clean":"2022-07-12T08:41:14.601087+0000","last_became_active":"2022-07-12T08:41:11.070549+0000","last_became_peered":"2022-07-12T08:41:11.070549+0000","last_unstale":"2022-07-12T08:41:14.601087+0000","last_undegraded":"2022-07-12T08:41:14.601087+0000","last_fullsized":"2022-07-12T08:41:14.601087+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.632648+0000","last_change":"2022-07-12T08:41:14.247604+0000","last_active":"2022-07-12T08:41:14.632648+0000","last_peered":"2022-07-12T08:41:14.632648+0000","last_clean":"2022-07-12T08:41:14.632648+0000","last_became_active":"2022-07-12T08:41:14.246709+0000","last_became_peered":"2022-07-12T08:41:14.246709+0000","last_unstale":"2022-07-12T08:41:14.632648+0000","last_undegraded":"2022-07-12T08:41:14.632648+0000","last_fullsized":"2022-07-12T08:41:14.632648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.605431+0000","last_change":"2022-07-12T08:41:14.605187+0000","last_active":"2022-07-12T08:41:14.605431+0000","last_peered":"2022-07-12T08:41:14.605431+0000","last_clean":"2022-07-12T08:41:14.605431+0000","last_became_active":"2022-07-12T08:41:14.604591+0000","last_became_peered":"2022-07-12T08:41:14.604591+0000","last_unstale":"2022-07-12T08:41:14.605431+0000","last_undegraded":"2022-07-12T08:41:14.605431+0000","last_fullsized":"2022-07-12T08:41:14.605431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.585342+0000","last_change":"2022-07-12T08:41:11.066220+0000","last_active":"2022-07-12T08:41:14.585342+0000","last_peered":"2022-07-12T08:41:14.585342+0000","last_clean":"2022-07-12T08:41:14.585342+0000","last_became_active":"2022-07-12T08:41:11.065911+0000","last_became_peered":"2022-07-12T08:41:11.065911+0000","last_unstale":"2022-07-12T08:41:14.585342+0000","last_undegraded":"2022-07-12T08:41:14.585342+0000","last_fullsized":"2022-07-12T08:41:14.585342+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.597764+0000","last_change":"2022-07-12T08:41:14.250011+0000","last_active":"2022-07-12T08:41:14.597764+0000","last_peered":"2022-07-12T08:41:14.597764+0000","last_clean":"2022-07-12T08:41:14.597764+0000","last_became_active":"2022-07-12T08:41:14.249561+0000","last_became_peered":"2022-07-12T08:41:14.249561+0000","last_unstale":"2022-07-12T08:41:14.597764+0000","last_undegraded":"2022-07-12T08:41:14.597764+0000","last_fullsized":"2022-07-12T08:41:14.597764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592130+0000","last_change":"2022-07-12T08:41:14.591768+0000","last_active":"2022-07-12T08:41:14.592130+0000","last_peered":"2022-07-12T08:41:14.592130+0000","last_clean":"2022-07-12T08:41:14.592130+0000","last_became_active":"2022-07-12T08:41:14.591339+0000","last_became_peered":"2022-07-12T08:41:14.591339+0000","last_unstale":"2022-07-12T08:41:14.592130+0000","last_undegraded":"2022-07-12T08:41:14.592130+0000","last_fullsized":"2022-07-12T08:41:14.592130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.598813+0000","last_change":"2022-07-12T08:41:14.250649+0000","last_active":"2022-07-12T08:41:14.598813+0000","last_peered":"2022-07-12T08:41:14.598813+0000","last_clean":"2022-07-12T08:41:14.598813+0000","last_became_active":"2022-07-12T08:41:14.250173+0000","last_became_peered":"2022-07-12T08:41:14.250173+0000","last_unstale":"2022-07-12T08:41:14.598813+0000","last_undegraded":"2022-07-12T08:41:14.598813+0000","last_fullsized":"2022-07-12T08:41:14.598813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.586480+0000","last_change":"2022-07-12T08:41:11.958581+0000","last_active":"2022-07-12T08:41:14.586480+0000","last_peered":"2022-07-12T08:41:14.586480+0000","last_clean":"2022-07-12T08:41:14.586480+0000","last_became_active":"2022-07-12T08:41:11.958006+0000","last_became_peered":"2022-07-12T08:41:11.958006+0000","last_unstale":"2022-07-12T08:41:14.586480+0000","last_undegraded":"2022-07-12T08:41:14.586480+0000","last_fullsized":"2022-07-12T08:41:14.586480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.596255+0000","last_change":"2022-07-12T08:41:14.596037+0000","last_active":"2022-07-12T08:41:14.596255+0000","last_peered":"2022-07-12T08:41:14.596255+0000","last_clean":"2022-07-12T08:41:14.596255+0000","last_became_active":"2022-07-12T08:41:14.595564+0000","last_became_peered":"2022-07-12T08:41:14.595564+0000","last_unstale":"2022-07-12T08:41:14.596255+0000","last_undegraded":"2022-07-12T08:41:14.596255+0000","last_fullsized":"2022-07-12T08:41:14.596255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.606301+0000","last_change":"2022-07-12T08:41:14.247806+0000","last_active":"2022-07-12T08:41:14.606301+0000","last_peered":"2022-07-12T08:41:14.606301+0000","last_clean":"2022-07-12T08:41:14.606301+0000","last_became_active":"2022-07-12T08:41:14.246935+0000","last_became_peered":"2022-07-12T08:41:14.246935+0000","last_unstale":"2022-07-12T08:41:14.606301+0000","last_undegraded":"2022-07-12T08:41:14.606301+0000","last_fullsized":"2022-07-12T08:41:14.606301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593221+0000","last_change":"2022-07-12T08:41:11.954669+0000","last_active":"2022-07-12T08:41:14.593221+0000","last_peered":"2022-07-12T08:41:14.593221+0000","last_clean":"2022-07-12T08:41:14.593221+0000","last_became_active":"2022-07-12T08:41:11.954063+0000","last_became_peered":"2022-07-12T08:41:11.954063+0000","last_unstale":"2022-07-12T08:41:14.593221+0000","last_undegraded":"2022-07-12T08:41:14.593221+0000","last_fullsized":"2022-07-12T08:41:14.593221+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.602106+0000","last_change":"2022-07-12T08:41:14.599984+0000","last_active":"2022-07-12T08:41:14.602106+0000","last_peered":"2022-07-12T08:41:14.602106+0000","last_clean":"2022-07-12T08:41:14.602106+0000","last_became_active":"2022-07-12T08:41:14.599622+0000","last_became_peered":"2022-07-12T08:41:14.599622+0000","last_unstale":"2022-07-12T08:41:14.602106+0000","last_undegraded":"2022-07-12T08:41:14.602106+0000","last_fullsized":"2022-07-12T08:41:14.602106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.601989+0000","last_change":"2022-07-12T08:41:14.254163+0000","last_active":"2022-07-12T08:41:14.601989+0000","last_peered":"2022-07-12T08:41:14.601989+0000","last_clean":"2022-07-12T08:41:14.601989+0000","last_became_active":"2022-07-12T08:41:14.253755+0000","last_became_peered":"2022-07-12T08:41:14.253755+0000","last_unstale":"2022-07-12T08:41:14.601989+0000","last_undegraded":"2022-07-12T08:41:14.601989+0000","last_fullsized":"2022-07-12T08:41:14.601989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.585970+0000","last_change":"2022-07-12T08:41:11.069495+0000","last_active":"2022-07-12T08:41:14.585970+0000","last_peered":"2022-07-12T08:41:14.585970+0000","last_clean":"2022-07-12T08:41:14.585970+0000","last_became_active":"2022-07-12T08:41:11.068746+0000","last_became_peered":"2022-07-12T08:41:11.068746+0000","last_unstale":"2022-07-12T08:41:14.585970+0000","last_undegraded":"2022-07-12T08:41:14.585970+0000","last_fullsized":"2022-07-12T08:41:14.585970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592105+0000","last_change":"2022-07-12T08:41:14.591852+0000","last_active":"2022-07-12T08:41:14.592105+0000","last_peered":"2022-07-12T08:41:14.592105+0000","last_clean":"2022-07-12T08:41:14.592105+0000","last_became_active":"2022-07-12T08:41:14.591399+0000","last_became_peered":"2022-07-12T08:41:14.591399+0000","last_unstale":"2022-07-12T08:41:14.592105+0000","last_undegraded":"2022-07-12T08:41:14.592105+0000","last_fullsized":"2022-07-12T08:41:14.592105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-12T08:41:16.064057+0000","last_change":"2022-07-12T08:41:14.255857+0000","last_active":"2022-07-12T08:41:16.064057+0000","last_peered":"2022-07-12T08:41:16.064057+0000","last_clean":"2022-07-12T08:41:16.064057+0000","last_became_active":"2022-07-12T08:41:14.255277+0000","last_became_peered":"2022-07-12T08:41:14.255277+0000","last_unstale":"2022-07-12T08:41:16.064057+0000","last_undegraded":"2022-07-12T08:41:16.064057+0000","last_fullsized":"2022-07-12T08:41:16.064057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.590028+0000","last_change":"2022-07-12T08:41:11.064835+0000","last_active":"2022-07-12T08:41:14.590028+0000","last_peered":"2022-07-12T08:41:14.590028+0000","last_clean":"2022-07-12T08:41:14.590028+0000","last_became_active":"2022-07-12T08:41:11.064473+0000","last_became_peered":"2022-07-12T08:41:11.064473+0000","last_unstale":"2022-07-12T08:41:14.590028+0000","last_undegraded":"2022-07-12T08:41:14.590028+0000","last_fullsized":"2022-07-12T08:41:14.590028+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903532,"kb_used_omap":63,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":0,"allocated":95133216768,"data_stored":95133216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64800,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795840,"kb_used_data":92903540,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92903540,"statfs":{"total":95948165120,"available":95133224960,"internally_reserved":0,"allocated":95133224960,"data_stored":95133224960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47791,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795844,"kb_used_data":92903536,"kb_used_omap":61,"kb_used_meta":0,"kb_avail":92903536,"statfs":{"total":95948165120,"available":95133220864,"internally_reserved":0,"allocated":95133220864,"data_stored":95133220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63030,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903524,"kb_used_omap":65,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":0,"allocated":95133208576,"data_stored":95133208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67510,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795852,"kb_used_data":92903528,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":92903528,"statfs":{"total":95948165120,"available":95133212672,"internally_reserved":0,"allocated":95133212672,"data_stored":95133212672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70328,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795908,"kb_used_data":92903488,"kb_used_omap":102,"kb_used_meta":0,"kb_avail":92903472,"statfs":{"total":95948165120,"available":95133155328,"internally_reserved":16384,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":105427,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795900,"kb_used_data":92903496,"kb_used_omap":82,"kb_used_meta":0,"kb_avail":92903480,"statfs":{"total":95948165120,"available":95133163520,"internally_reserved":16384,"allocated":95133179904,"data_stored":95133179904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":84168,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903532,"kb_used_omap":78,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":0,"allocated":95133216768,"data_stored":95133216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80037,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]}]}]}} 2022-07-12T08:41:23.110 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-12T08:41:23.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.228+0000 7faabba82700 1 Processor -- start 2022-07-12T08:41:23.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- start start 2022-07-12T08:41:23.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faab413ce60 0x7faab40b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faab40b47f0 0x7faab40b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab413d240 con 0x7faab4142d50 2022-07-12T08:41:23.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40b24c0 con 0x7faab40b47f0 2022-07-12T08:41:23.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab41431f0 con 0x7faab413ce60 2022-07-12T08:41:23.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faab901d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7faab4142d50 0x7faab40b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41424/0 (socket says 172.21.15.61:41424) 2022-07-12T08:41:23.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faab901d700 1 -- 172.21.15.61:0/3343774459 learned_addr learned my addr 172.21.15.61:0/3343774459 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faaba01f700 1 --2- 172.21.15.61:0/3343774459 >> v2:172.21.15.61:3300/0 conn(0x7faab40b47f0 0x7faab40b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faab981e700 1 --2- 172.21.15.61:0/3343774459 >> v2:172.21.15.111:3300/0 conn(0x7faab413ce60 0x7faab40b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faaba01f700 1 -- 172.21.15.61:0/3343774459 >> v2:172.21.15.111:3300/0 conn(0x7faab413ce60 msgr2=0x7faab40b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faaba01f700 1 --2- 172.21.15.61:0/3343774459 >> v2:172.21.15.111:3300/0 conn(0x7faab413ce60 0x7faab40b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faaba01f700 1 -- 172.21.15.61:0/3343774459 >> v1:172.21.15.61:6789/0 conn(0x7faab4142d50 legacy=0x7faab40b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faaba01f700 1 -- 172.21.15.61:0/3343774459 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4143490 con 0x7faab40b47f0 2022-07-12T08:41:23.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faaba01f700 1 --2- 172.21.15.61:0/3343774459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b47f0 0x7faab40b4c70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7faaac0032f0 tx=0x7faaac0091d0).ready entity=mon.0 client_cookie=9e85ef075c0a5cd1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faab881c700 1 -- 172.21.15.61:0/3343774459 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaac00b680 con 0x7faab40b47f0 2022-07-12T08:41:23.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faab881c700 1 -- 172.21.15.61:0/3343774459 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac00b7e0 con 0x7faab40b47f0 2022-07-12T08:41:23.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faab881c700 1 -- 172.21.15.61:0/3343774459 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaac00b950 con 0x7faab40b47f0 2022-07-12T08:41:23.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- 172.21.15.61:0/3343774459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b47f0 msgr2=0x7faab40b4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 --2- 172.21.15.61:0/3343774459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b47f0 0x7faab40b4c70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7faaac0032f0 tx=0x7faaac0091d0).stop 2022-07-12T08:41:23.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- 172.21.15.61:0/3343774459 shutdown_connections 2022-07-12T08:41:23.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 --2- 172.21.15.61:0/3343774459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b47f0 0x7faab40b4c70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 --2- 172.21.15.61:0/3343774459 >> v2:172.21.15.111:3300/0 conn(0x7faab413ce60 0x7faab40b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- 172.21.15.61:0/3343774459 >> 172.21.15.61:0/3343774459 conn(0x7faab405f9f0 msgr2=0x7faab4066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- 172.21.15.61:0/3343774459 shutdown_connections 2022-07-12T08:41:23.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.232+0000 7faabba82700 1 -- 172.21.15.61:0/3343774459 wait complete. 2022-07-12T08:41:23.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.236+0000 7faabba82700 1 Processor -- start 2022-07-12T08:41:23.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.236+0000 7faabba82700 1 -- start start 2022-07-12T08:41:23.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b47f0 0x7faab4147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab413ce60 0x7faab413f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 0x7faab413d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab413ecb0 con 0x7faab4142d50 2022-07-12T08:41:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab413eee0 con 0x7faab40b47f0 2022-07-12T08:41:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab413f180 con 0x7faab413ce60 2022-07-12T08:41:23.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 0x7faab413d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 0x7faab413d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53216/0 (socket says 172.21.15.61:53216) 2022-07-12T08:41:23.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 -- 172.21.15.61:0/324228522 learned_addr learned my addr 172.21.15.61:0/324228522 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:23.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faab981e700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b47f0 0x7faab4147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faab901d700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab413ce60 0x7faab413f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 -- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab413ce60 msgr2=0x7faab413f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab413ce60 0x7faab413f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 -- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b47f0 msgr2=0x7faab4147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b47f0 0x7faab4147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 -- 172.21.15.61:0/324228522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac009470 con 0x7faab4142d50 2022-07-12T08:41:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaba01f700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 0x7faab413d3f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7faaac0032f0 tx=0x7faaac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaac00b680 con 0x7faab4142d50 2022-07-12T08:41:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac021630 con 0x7faab4142d50 2022-07-12T08:41:23.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab413d8e0 con 0x7faab4142d50 2022-07-12T08:41:23.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaac021990 con 0x7faab4142d50 2022-07-12T08:41:23.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab41552d0 con 0x7faab4142d50 2022-07-12T08:41:23.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab413dde0 con 0x7faab4142d50 2022-07-12T08:41:23.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.240+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faaac027020 con 0x7faab4142d50 2022-07-12T08:41:23.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.244+0000 7faaa6ffd700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa94053a40 0x7faa94055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.244+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faaac06d390 con 0x7faab4142d50 2022-07-12T08:41:23.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.244+0000 7faab981e700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa94053a40 0x7faa94055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.244+0000 7faab981e700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa94053a40 0x7faa94055f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7faaa800b000 tx=0x7faaa8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.248+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faaac044250 con 0x7faab4142d50 2022-07-12T08:41:23.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.372+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faab413f340 con 0x7faa94053a40 2022-07-12T08:41:23.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faaa6ffd700 1 -- 172.21.15.61:0/324228522 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82241 (secure 0 0 0) 0x7faab413f340 con 0x7faa94053a40 2022-07-12T08:41:23.409 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:23.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa94053a40 msgr2=0x7faa94055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa94053a40 0x7faa94055f00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7faaa800b000 tx=0x7faaa8006df0).stop 2022-07-12T08:41:23.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 msgr2=0x7faab413d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 0x7faab413d3f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7faaac0032f0 tx=0x7faaac016000).stop 2022-07-12T08:41:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 shutdown_connections 2022-07-12T08:41:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab4142d50 0x7faab413d3f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab413ce60 0x7faab413f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa94053a40 0x7faa94055f00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 --2- 172.21.15.61:0/324228522 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b47f0 0x7faab4147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 >> 172.21.15.61:0/324228522 conn(0x7faab405f9f0 msgr2=0x7faab40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 shutdown_connections 2022-07-12T08:41:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.380+0000 7faabba82700 1 -- 172.21.15.61:0/324228522 wait complete. 2022-07-12T08:41:23.439 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-12T08:41:23.440 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2022-07-12T08:41:23.085912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6366896,"kb_used_data":743228176,"kb_used_omap":569,"kb_used_meta":0,"kb_avail":743228144,"statfs":{"total":767585320960,"available":761065619456,"internally_reserved":32768,"allocated":761065652224,"data_stored":761065652224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":583091,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":160,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002484"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592471+0000","last_change":"2022-07-12T08:41:14.592304+0000","last_active":"2022-07-12T08:41:14.592471+0000","last_peered":"2022-07-12T08:41:14.592471+0000","last_clean":"2022-07-12T08:41:14.592471+0000","last_became_active":"2022-07-12T08:41:14.591935+0000","last_became_peered":"2022-07-12T08:41:14.591935+0000","last_unstale":"2022-07-12T08:41:14.592471+0000","last_undegraded":"2022-07-12T08:41:14.592471+0000","last_fullsized":"2022-07-12T08:41:14.592471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593820+0000","last_change":"2022-07-12T08:41:04.955012+0000","last_active":"2022-07-12T08:41:14.593820+0000","last_peered":"2022-07-12T08:41:14.593820+0000","last_clean":"2022-07-12T08:41:14.593820+0000","last_became_active":"2022-07-12T08:41:04.954339+0000","last_became_peered":"2022-07-12T08:41:04.954339+0000","last_unstale":"2022-07-12T08:41:14.593820+0000","last_undegraded":"2022-07-12T08:41:14.593820+0000","last_fullsized":"2022-07-12T08:41:14.593820+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_clean_scrub_stamp":"2022-07-12T08:40:58.647421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.585823+0000","last_change":"2022-07-12T08:41:11.067479+0000","last_active":"2022-07-12T08:41:14.585823+0000","last_peered":"2022-07-12T08:41:14.585823+0000","last_clean":"2022-07-12T08:41:14.585823+0000","last_became_active":"2022-07-12T08:41:11.067215+0000","last_became_peered":"2022-07-12T08:41:11.067215+0000","last_unstale":"2022-07-12T08:41:14.585823+0000","last_undegraded":"2022-07-12T08:41:14.585823+0000","last_fullsized":"2022-07-12T08:41:14.585823+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.600509+0000","last_change":"2022-07-12T08:41:14.252385+0000","last_active":"2022-07-12T08:41:14.600509+0000","last_peered":"2022-07-12T08:41:14.600509+0000","last_clean":"2022-07-12T08:41:14.600509+0000","last_became_active":"2022-07-12T08:41:14.252022+0000","last_became_peered":"2022-07-12T08:41:14.252022+0000","last_unstale":"2022-07-12T08:41:14.600509+0000","last_undegraded":"2022-07-12T08:41:14.600509+0000","last_fullsized":"2022-07-12T08:41:14.600509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593852+0000","last_change":"2022-07-12T08:41:14.593713+0000","last_active":"2022-07-12T08:41:14.593852+0000","last_peered":"2022-07-12T08:41:14.593852+0000","last_clean":"2022-07-12T08:41:14.593852+0000","last_became_active":"2022-07-12T08:41:14.593266+0000","last_became_peered":"2022-07-12T08:41:14.593266+0000","last_unstale":"2022-07-12T08:41:14.593852+0000","last_undegraded":"2022-07-12T08:41:14.593852+0000","last_fullsized":"2022-07-12T08:41:14.593852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593698+0000","last_change":"2022-07-12T08:41:11.064070+0000","last_active":"2022-07-12T08:41:14.593698+0000","last_peered":"2022-07-12T08:41:14.593698+0000","last_clean":"2022-07-12T08:41:14.593698+0000","last_became_active":"2022-07-12T08:41:11.063329+0000","last_became_peered":"2022-07-12T08:41:11.063329+0000","last_unstale":"2022-07-12T08:41:14.593698+0000","last_undegraded":"2022-07-12T08:41:14.593698+0000","last_fullsized":"2022-07-12T08:41:14.593698+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.590378+0000","last_change":"2022-07-12T08:41:14.254382+0000","last_active":"2022-07-12T08:41:14.590378+0000","last_peered":"2022-07-12T08:41:14.590378+0000","last_clean":"2022-07-12T08:41:14.590378+0000","last_became_active":"2022-07-12T08:41:14.254057+0000","last_became_peered":"2022-07-12T08:41:14.254057+0000","last_unstale":"2022-07-12T08:41:14.590378+0000","last_undegraded":"2022-07-12T08:41:14.590378+0000","last_fullsized":"2022-07-12T08:41:14.590378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592135+0000","last_change":"2022-07-12T08:41:14.591841+0000","last_active":"2022-07-12T08:41:14.592135+0000","last_peered":"2022-07-12T08:41:14.592135+0000","last_clean":"2022-07-12T08:41:14.592135+0000","last_became_active":"2022-07-12T08:41:14.591365+0000","last_became_peered":"2022-07-12T08:41:14.591365+0000","last_unstale":"2022-07-12T08:41:14.592135+0000","last_undegraded":"2022-07-12T08:41:14.592135+0000","last_fullsized":"2022-07-12T08:41:14.592135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.601087+0000","last_change":"2022-07-12T08:41:11.071303+0000","last_active":"2022-07-12T08:41:14.601087+0000","last_peered":"2022-07-12T08:41:14.601087+0000","last_clean":"2022-07-12T08:41:14.601087+0000","last_became_active":"2022-07-12T08:41:11.070549+0000","last_became_peered":"2022-07-12T08:41:11.070549+0000","last_unstale":"2022-07-12T08:41:14.601087+0000","last_undegraded":"2022-07-12T08:41:14.601087+0000","last_fullsized":"2022-07-12T08:41:14.601087+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.632648+0000","last_change":"2022-07-12T08:41:14.247604+0000","last_active":"2022-07-12T08:41:14.632648+0000","last_peered":"2022-07-12T08:41:14.632648+0000","last_clean":"2022-07-12T08:41:14.632648+0000","last_became_active":"2022-07-12T08:41:14.246709+0000","last_became_peered":"2022-07-12T08:41:14.246709+0000","last_unstale":"2022-07-12T08:41:14.632648+0000","last_undegraded":"2022-07-12T08:41:14.632648+0000","last_fullsized":"2022-07-12T08:41:14.632648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.605431+0000","last_change":"2022-07-12T08:41:14.605187+0000","last_active":"2022-07-12T08:41:14.605431+0000","last_peered":"2022-07-12T08:41:14.605431+0000","last_clean":"2022-07-12T08:41:14.605431+0000","last_became_active":"2022-07-12T08:41:14.604591+0000","last_became_peered":"2022-07-12T08:41:14.604591+0000","last_unstale":"2022-07-12T08:41:14.605431+0000","last_undegraded":"2022-07-12T08:41:14.605431+0000","last_fullsized":"2022-07-12T08:41:14.605431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.585342+0000","last_change":"2022-07-12T08:41:11.066220+0000","last_active":"2022-07-12T08:41:14.585342+0000","last_peered":"2022-07-12T08:41:14.585342+0000","last_clean":"2022-07-12T08:41:14.585342+0000","last_became_active":"2022-07-12T08:41:11.065911+0000","last_became_peered":"2022-07-12T08:41:11.065911+0000","last_unstale":"2022-07-12T08:41:14.585342+0000","last_undegraded":"2022-07-12T08:41:14.585342+0000","last_fullsized":"2022-07-12T08:41:14.585342+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.597764+0000","last_change":"2022-07-12T08:41:14.250011+0000","last_active":"2022-07-12T08:41:14.597764+0000","last_peered":"2022-07-12T08:41:14.597764+0000","last_clean":"2022-07-12T08:41:14.597764+0000","last_became_active":"2022-07-12T08:41:14.249561+0000","last_became_peered":"2022-07-12T08:41:14.249561+0000","last_unstale":"2022-07-12T08:41:14.597764+0000","last_undegraded":"2022-07-12T08:41:14.597764+0000","last_fullsized":"2022-07-12T08:41:14.597764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592130+0000","last_change":"2022-07-12T08:41:14.591768+0000","last_active":"2022-07-12T08:41:14.592130+0000","last_peered":"2022-07-12T08:41:14.592130+0000","last_clean":"2022-07-12T08:41:14.592130+0000","last_became_active":"2022-07-12T08:41:14.591339+0000","last_became_peered":"2022-07-12T08:41:14.591339+0000","last_unstale":"2022-07-12T08:41:14.592130+0000","last_undegraded":"2022-07-12T08:41:14.592130+0000","last_fullsized":"2022-07-12T08:41:14.592130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.598813+0000","last_change":"2022-07-12T08:41:14.250649+0000","last_active":"2022-07-12T08:41:14.598813+0000","last_peered":"2022-07-12T08:41:14.598813+0000","last_clean":"2022-07-12T08:41:14.598813+0000","last_became_active":"2022-07-12T08:41:14.250173+0000","last_became_peered":"2022-07-12T08:41:14.250173+0000","last_unstale":"2022-07-12T08:41:14.598813+0000","last_undegraded":"2022-07-12T08:41:14.598813+0000","last_fullsized":"2022-07-12T08:41:14.598813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.586480+0000","last_change":"2022-07-12T08:41:11.958581+0000","last_active":"2022-07-12T08:41:14.586480+0000","last_peered":"2022-07-12T08:41:14.586480+0000","last_clean":"2022-07-12T08:41:14.586480+0000","last_became_active":"2022-07-12T08:41:11.958006+0000","last_became_peered":"2022-07-12T08:41:11.958006+0000","last_unstale":"2022-07-12T08:41:14.586480+0000","last_undegraded":"2022-07-12T08:41:14.586480+0000","last_fullsized":"2022-07-12T08:41:14.586480+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.596255+0000","last_change":"2022-07-12T08:41:14.596037+0000","last_active":"2022-07-12T08:41:14.596255+0000","last_peered":"2022-07-12T08:41:14.596255+0000","last_clean":"2022-07-12T08:41:14.596255+0000","last_became_active":"2022-07-12T08:41:14.595564+0000","last_became_peered":"2022-07-12T08:41:14.595564+0000","last_unstale":"2022-07-12T08:41:14.596255+0000","last_undegraded":"2022-07-12T08:41:14.596255+0000","last_fullsized":"2022-07-12T08:41:14.596255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.606301+0000","last_change":"2022-07-12T08:41:14.247806+0000","last_active":"2022-07-12T08:41:14.606301+0000","last_peered":"2022-07-12T08:41:14.606301+0000","last_clean":"2022-07-12T08:41:14.606301+0000","last_became_active":"2022-07-12T08:41:14.246935+0000","last_became_peered":"2022-07-12T08:41:14.246935+0000","last_unstale":"2022-07-12T08:41:14.606301+0000","last_undegraded":"2022-07-12T08:41:14.606301+0000","last_fullsized":"2022-07-12T08:41:14.606301+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.593221+0000","last_change":"2022-07-12T08:41:11.954669+0000","last_active":"2022-07-12T08:41:14.593221+0000","last_peered":"2022-07-12T08:41:14.593221+0000","last_clean":"2022-07-12T08:41:14.593221+0000","last_became_active":"2022-07-12T08:41:11.954063+0000","last_became_peered":"2022-07-12T08:41:11.954063+0000","last_unstale":"2022-07-12T08:41:14.593221+0000","last_undegraded":"2022-07-12T08:41:14.593221+0000","last_fullsized":"2022-07-12T08:41:14.593221+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.602106+0000","last_change":"2022-07-12T08:41:14.599984+0000","last_active":"2022-07-12T08:41:14.602106+0000","last_peered":"2022-07-12T08:41:14.602106+0000","last_clean":"2022-07-12T08:41:14.602106+0000","last_became_active":"2022-07-12T08:41:14.599622+0000","last_became_peered":"2022-07-12T08:41:14.599622+0000","last_unstale":"2022-07-12T08:41:14.602106+0000","last_undegraded":"2022-07-12T08:41:14.602106+0000","last_fullsized":"2022-07-12T08:41:14.602106+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.601989+0000","last_change":"2022-07-12T08:41:14.254163+0000","last_active":"2022-07-12T08:41:14.601989+0000","last_peered":"2022-07-12T08:41:14.601989+0000","last_clean":"2022-07-12T08:41:14.601989+0000","last_became_active":"2022-07-12T08:41:14.253755+0000","last_became_peered":"2022-07-12T08:41:14.253755+0000","last_unstale":"2022-07-12T08:41:14.601989+0000","last_undegraded":"2022-07-12T08:41:14.601989+0000","last_fullsized":"2022-07-12T08:41:14.601989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.585970+0000","last_change":"2022-07-12T08:41:11.069495+0000","last_active":"2022-07-12T08:41:14.585970+0000","last_peered":"2022-07-12T08:41:14.585970+0000","last_clean":"2022-07-12T08:41:14.585970+0000","last_became_active":"2022-07-12T08:41:11.068746+0000","last_became_peered":"2022-07-12T08:41:11.068746+0000","last_unstale":"2022-07-12T08:41:14.585970+0000","last_undegraded":"2022-07-12T08:41:14.585970+0000","last_fullsized":"2022-07-12T08:41:14.585970+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.592105+0000","last_change":"2022-07-12T08:41:14.591852+0000","last_active":"2022-07-12T08:41:14.592105+0000","last_peered":"2022-07-12T08:41:14.592105+0000","last_clean":"2022-07-12T08:41:14.592105+0000","last_became_active":"2022-07-12T08:41:14.591399+0000","last_became_peered":"2022-07-12T08:41:14.591399+0000","last_unstale":"2022-07-12T08:41:14.592105+0000","last_undegraded":"2022-07-12T08:41:14.592105+0000","last_fullsized":"2022-07-12T08:41:14.592105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:14.236926+0000","last_clean_scrub_stamp":"2022-07-12T08:41:14.236926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-12T08:41:16.064057+0000","last_change":"2022-07-12T08:41:14.255857+0000","last_active":"2022-07-12T08:41:16.064057+0000","last_peered":"2022-07-12T08:41:16.064057+0000","last_clean":"2022-07-12T08:41:16.064057+0000","last_became_active":"2022-07-12T08:41:14.255277+0000","last_became_peered":"2022-07-12T08:41:14.255277+0000","last_unstale":"2022-07-12T08:41:16.064057+0000","last_undegraded":"2022-07-12T08:41:16.064057+0000","last_fullsized":"2022-07-12T08:41:16.064057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:13.234053+0000","last_clean_scrub_stamp":"2022-07-12T08:41:13.234053+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-12T08:41:14.590028+0000","last_change":"2022-07-12T08:41:11.064835+0000","last_active":"2022-07-12T08:41:14.590028+0000","last_peered":"2022-07-12T08:41:14.590028+0000","last_clean":"2022-07-12T08:41:14.590028+0000","last_became_active":"2022-07-12T08:41:11.064473+0000","last_became_peered":"2022-07-12T08:41:11.064473+0000","last_unstale":"2022-07-12T08:41:14.590028+0000","last_undegraded":"2022-07-12T08:41:14.590028+0000","last_fullsized":"2022-07-12T08:41:14.590028+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903532,"kb_used_omap":63,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":0,"allocated":95133216768,"data_stored":95133216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64800,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795840,"kb_used_data":92903540,"kb_used_omap":46,"kb_used_meta":0,"kb_avail":92903540,"statfs":{"total":95948165120,"available":95133224960,"internally_reserved":0,"allocated":95133224960,"data_stored":95133224960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47791,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795844,"kb_used_data":92903536,"kb_used_omap":61,"kb_used_meta":0,"kb_avail":92903536,"statfs":{"total":95948165120,"available":95133220864,"internally_reserved":0,"allocated":95133220864,"data_stored":95133220864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63030,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903524,"kb_used_omap":65,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":0,"allocated":95133208576,"data_stored":95133208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67510,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795852,"kb_used_data":92903528,"kb_used_omap":68,"kb_used_meta":0,"kb_avail":92903528,"statfs":{"total":95948165120,"available":95133212672,"internally_reserved":0,"allocated":95133212672,"data_stored":95133212672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70328,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795908,"kb_used_data":92903488,"kb_used_omap":102,"kb_used_meta":0,"kb_avail":92903472,"statfs":{"total":95948165120,"available":95133155328,"internally_reserved":16384,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":105427,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3640000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1419999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795900,"kb_used_data":92903496,"kb_used_omap":82,"kb_used_meta":0,"kb_avail":92903480,"statfs":{"total":95948165120,"available":95133163520,"internally_reserved":16384,"allocated":95133179904,"data_stored":95133179904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":84168,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903532,"kb_used_omap":78,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":0,"allocated":95133216768,"data_stored":95133216768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80037,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.298}]}]}]}} 2022-07-12T08:41:23.452 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-07-12T08:41:23.453 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-07-12T08:41:23.453 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-07-12T08:41:23.454 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-07-12T08:41:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 Processor -- start 2022-07-12T08:41:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 -- start start 2022-07-12T08:41:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd45c067a00 0x7fd45c143010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c05bdc0 0x7fd45c143500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd45c05f140 0x7fd45c1460d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c0ae7c0 con 0x7fd45c05f140 2022-07-12T08:41:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c05f930 con 0x7fd45c067a00 2022-07-12T08:41:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.564+0000 7fd461e15700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c067de0 con 0x7fd45c05bdc0 2022-07-12T08:41:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd45c05f140 0x7fd45c1460d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd45c05f140 0x7fd45c1460d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53224/0 (socket says 172.21.15.61:53224) 2022-07-12T08:41:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 -- 172.21.15.61:0/2982300286 learned_addr learned my addr 172.21.15.61:0/2982300286 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd460e13700 1 --2- 172.21.15.61:0/2982300286 >> v2:172.21.15.111:3300/0 conn(0x7fd45c067a00 0x7fd45c143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd45bfff700 1 --2- 172.21.15.61:0/2982300286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c05bdc0 0x7fd45c143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 -- 172.21.15.61:0/2982300286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c05bdc0 msgr2=0x7fd45c143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:23.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 --2- 172.21.15.61:0/2982300286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c05bdc0 0x7fd45c143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 -- 172.21.15.61:0/2982300286 >> v2:172.21.15.111:3300/0 conn(0x7fd45c067a00 msgr2=0x7fd45c143010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:23.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 --2- 172.21.15.61:0/2982300286 >> v2:172.21.15.111:3300/0 conn(0x7fd45c067a00 0x7fd45c143010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 -- 172.21.15.61:0/2982300286 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c0ae9e0 con 0x7fd45c05f140 2022-07-12T08:41:23.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461614700 1 --2- 172.21.15.61:0/2982300286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05f140 0x7fd45c1460d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd4540032f0 tx=0x7fd454009290).ready entity=mon.0 client_cookie=f073d544c7a1bdba server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd45b7fe700 1 -- 172.21.15.61:0/2982300286 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd45400b680 con 0x7fd45c05f140 2022-07-12T08:41:23.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd45b7fe700 1 -- 172.21.15.61:0/2982300286 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45400b7e0 con 0x7fd45c05f140 2022-07-12T08:41:23.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd45b7fe700 1 -- 172.21.15.61:0/2982300286 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd45400b950 con 0x7fd45c05f140 2022-07-12T08:41:23.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 -- 172.21.15.61:0/2982300286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05f140 msgr2=0x7fd45c1460d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 --2- 172.21.15.61:0/2982300286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05f140 0x7fd45c1460d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fd4540032f0 tx=0x7fd454009290).stop 2022-07-12T08:41:23.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 -- 172.21.15.61:0/2982300286 shutdown_connections 2022-07-12T08:41:23.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 --2- 172.21.15.61:0/2982300286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c05bdc0 0x7fd45c143500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 --2- 172.21.15.61:0/2982300286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05f140 0x7fd45c1460d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 --2- 172.21.15.61:0/2982300286 >> v2:172.21.15.111:3300/0 conn(0x7fd45c067a00 0x7fd45c143010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 -- 172.21.15.61:0/2982300286 >> 172.21.15.61:0/2982300286 conn(0x7fd45c05c5a0 msgr2=0x7fd45c0631c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 -- 172.21.15.61:0/2982300286 shutdown_connections 2022-07-12T08:41:23.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.568+0000 7fd461e15700 1 -- 172.21.15.61:0/2982300286 wait complete. 2022-07-12T08:41:23.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 Processor -- start 2022-07-12T08:41:23.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 -- start start 2022-07-12T08:41:23.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 0x7fd45c13e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45c05f140 0x7fd45c148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c067a00 0x7fd45c152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c064b20 con 0x7fd45c05bdc0 2022-07-12T08:41:23.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c0645e0 con 0x7fd45c05f140 2022-07-12T08:41:23.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c064880 con 0x7fd45c067a00 2022-07-12T08:41:23.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 0x7fd45c13e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 0x7fd45c13e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53226/0 (socket says 172.21.15.61:53226) 2022-07-12T08:41:23.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 -- 172.21.15.61:0/1947542684 learned_addr learned my addr 172.21.15.61:0/1947542684 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:23.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd45bfff700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45c05f140 0x7fd45c148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461614700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c067a00 0x7fd45c152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 -- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c067a00 msgr2=0x7fd45c152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c067a00 0x7fd45c152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 -- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45c05f140 msgr2=0x7fd45c148360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45c05f140 0x7fd45c148360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 -- 172.21.15.61:0/1947542684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454009530 con 0x7fd45c05bdc0 2022-07-12T08:41:23.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd460e13700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 0x7fd45c13e880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fd45000c000 tx=0x7fd450006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd450010780 con 0x7fd45c05bdc0 2022-07-12T08:41:23.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4500108e0 con 0x7fd45c05bdc0 2022-07-12T08:41:23.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45c148850 con 0x7fd45c05bdc0 2022-07-12T08:41:23.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd450010c40 con 0x7fd45c05bdc0 2022-07-12T08:41:23.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.572+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45c064f90 con 0x7fd45c05bdc0 2022-07-12T08:41:23.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.576+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd45001c460 con 0x7fd45c05bdc0 2022-07-12T08:41:23.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.576+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45c057460 con 0x7fd45c05bdc0 2022-07-12T08:41:23.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.576+0000 7fd459ffb700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd43c0539d0 0x7fd43c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.576+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd45006c240 con 0x7fd45c05bdc0 2022-07-12T08:41:23.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.576+0000 7fd45bfff700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd43c0539d0 0x7fd43c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.576+0000 7fd45bfff700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd43c0539d0 0x7fd43c055e90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd44c00b000 tx=0x7fd44c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.584+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd450043510 con 0x7fd45c05bdc0 2022-07-12T08:41:23.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.748+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd45c063f90 con 0x7fd45c05bdc0 2022-07-12T08:41:23.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.748+0000 7fd459ffb700 1 -- 172.21.15.61:0/1947542684 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7fd450046760 con 0x7fd45c05bdc0 2022-07-12T08:41:23.767 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:23.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd43c0539d0 msgr2=0x7fd43c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd43c0539d0 0x7fd43c055e90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd44c00b000 tx=0x7fd44c006df0).stop 2022-07-12T08:41:23.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 msgr2=0x7fd45c13e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 0x7fd45c13e880 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fd45000c000 tx=0x7fd450006df0).stop 2022-07-12T08:41:23.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 shutdown_connections 2022-07-12T08:41:23.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd45c067a00 0x7fd45c152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd43c0539d0 0x7fd43c055e90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd45c05f140 0x7fd45c148360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 --2- 172.21.15.61:0/1947542684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd45c05bdc0 0x7fd45c13e880 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 >> 172.21.15.61:0/1947542684 conn(0x7fd45c05c5a0 msgr2=0x7fd45c0b5800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 shutdown_connections 2022-07-12T08:41:23.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.752+0000 7fd461e15700 1 -- 172.21.15.61:0/1947542684 wait complete. 2022-07-12T08:41:23.772 INFO:teuthology.orchestra.run.smithi061.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-12T08:41:23.773 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-07-12T08:41:23.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:23.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 Processor -- start 2022-07-12T08:41:23.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 -- start start 2022-07-12T08:41:23.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040ab290 0x7ff0040ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff00405f140 0x7ff00405ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff0040afa60 0x7ff0040afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00405f580 con 0x7ff00405f140 2022-07-12T08:41:23.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0040abc70 con 0x7ff0040afa60 2022-07-12T08:41:23.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff003fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00405c720 con 0x7ff0040ab290 2022-07-12T08:41:23.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff00405f140 0x7ff00405ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff00405f140 0x7ff00405ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53234/0 (socket says 172.21.15.61:53234) 2022-07-12T08:41:23.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 -- 172.21.15.61:0/3570121529 learned_addr learned my addr 172.21.15.61:0/3570121529 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:23.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff002ffd700 1 --2- 172.21.15.61:0/3570121529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040ab290 0x7ff0040ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 -- 172.21.15.61:0/3570121529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040ab290 msgr2=0x7ff0040ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0037fe700 1 --2- 172.21.15.61:0/3570121529 >> v2:172.21.15.111:3300/0 conn(0x7ff0040afa60 0x7ff0040afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 --2- 172.21.15.61:0/3570121529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040ab290 0x7ff0040ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 -- 172.21.15.61:0/3570121529 >> v2:172.21.15.111:3300/0 conn(0x7ff0040afa60 msgr2=0x7ff0040afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 --2- 172.21.15.61:0/3570121529 >> v2:172.21.15.111:3300/0 conn(0x7ff0040afa60 0x7ff0040afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 -- 172.21.15.61:0/3570121529 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00405c9c0 con 0x7ff00405f140 2022-07-12T08:41:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0037fe700 1 --2- 172.21.15.61:0/3570121529 >> v2:172.21.15.111:3300/0 conn(0x7ff0040afa60 0x7ff0040afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:23.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff0027fc700 1 --2- 172.21.15.61:0/3570121529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff00405ce60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7feff4002f70 tx=0x7feff4006eb0).ready entity=mon.0 client_cookie=c4cb1cb5d26a8750 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff001ffb700 1 -- 172.21.15.61:0/3570121529 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff400d570 con 0x7ff00405f140 2022-07-12T08:41:23.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff001ffb700 1 -- 172.21.15.61:0/3570121529 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff4000e90 con 0x7ff00405f140 2022-07-12T08:41:23.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.896+0000 7ff001ffb700 1 -- 172.21.15.61:0/3570121529 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff400d900 con 0x7ff00405f140 2022-07-12T08:41:23.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- 172.21.15.61:0/3570121529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 msgr2=0x7ff00405ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- 172.21.15.61:0/3570121529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff00405ce60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7feff4002f70 tx=0x7feff4006eb0).stop 2022-07-12T08:41:23.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- 172.21.15.61:0/3570121529 shutdown_connections 2022-07-12T08:41:23.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- 172.21.15.61:0/3570121529 >> v2:172.21.15.111:3300/0 conn(0x7ff0040afa60 0x7ff0040afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- 172.21.15.61:0/3570121529 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff00405ce60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- 172.21.15.61:0/3570121529 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040ab290 0x7ff0040ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- 172.21.15.61:0/3570121529 >> 172.21.15.61:0/3570121529 conn(0x7ff0040b1950 msgr2=0x7ff0040b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:23.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- 172.21.15.61:0/3570121529 shutdown_connections 2022-07-12T08:41:23.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- 172.21.15.61:0/3570121529 wait complete. 2022-07-12T08:41:23.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 Processor -- start 2022-07-12T08:41:23.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- start start 2022-07-12T08:41:23.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff004067270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0040ab290 0x7ff0040640e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040afa60 0x7ff0041488e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00414a450 con 0x7ff00405f140 2022-07-12T08:41:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0040678e0 con 0x7ff0040ab290 2022-07-12T08:41:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff003fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff004067020 con 0x7ff0040afa60 2022-07-12T08:41:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff004067270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff004067270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53240/0 (socket says 172.21.15.61:53240) 2022-07-12T08:41:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 -- 172.21.15.61:0/4093602927 learned_addr learned my addr 172.21.15.61:0/4093602927 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:23.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff0037fe700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040afa60 0x7ff0041488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff0027fc700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0040ab290 0x7ff0040640e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:23.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 -- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040afa60 msgr2=0x7ff0041488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040afa60 0x7ff0041488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 -- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0040ab290 msgr2=0x7ff0040640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:23.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0040ab290 0x7ff0040640e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:23.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.900+0000 7ff002ffd700 1 -- 172.21.15.61:0/4093602927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff400b1e0 con 0x7ff00405f140 2022-07-12T08:41:23.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7ff002ffd700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff004067270 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7feff80030b0 tx=0x7feff8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:23.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff8011780 con 0x7ff00405f140 2022-07-12T08:41:23.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff80118e0 con 0x7ff00405f140 2022-07-12T08:41:23.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff004066ae0 con 0x7ff00405f140 2022-07-12T08:41:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff8011c40 con 0x7ff00405f140 2022-07-12T08:41:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff004149890 con 0x7ff00405f140 2022-07-12T08:41:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff004057460 con 0x7ff00405f140 2022-07-12T08:41:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feff801c460 con 0x7ff00405f140 2022-07-12T08:41:24.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7feff3fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fefe0053920 0x7fefe0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feff806bf90 con 0x7ff00405f140 2022-07-12T08:41:24.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.904+0000 7ff0027fc700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fefe0053920 0x7fefe0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.908+0000 7ff0027fc700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fefe0053920 0x7fefe0055de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7feff401c000 tx=0x7feff4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:23.912+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feff8042e90 con 0x7ff00405f140 2022-07-12T08:41:24.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.040+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff00406b2b0 con 0x7ff00405f140 2022-07-12T08:41:24.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.044+0000 7feff3fff700 1 -- 172.21.15.61:0/4093602927 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7feff80464b0 con 0x7ff00405f140 2022-07-12T08:41:24.049 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:24.050 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:24.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fefe0053920 msgr2=0x7fefe0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fefe0053920 0x7fefe0055de0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7feff401c000 tx=0x7feff4015000).stop 2022-07-12T08:41:24.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 msgr2=0x7ff004067270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff004067270 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7feff80030b0 tx=0x7feff8008040).stop 2022-07-12T08:41:24.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 shutdown_connections 2022-07-12T08:41:24.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0040afa60 0x7ff0041488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fefe0053920 0x7fefe0055de0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0040ab290 0x7ff0040640e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 --2- 172.21.15.61:0/4093602927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff00405f140 0x7ff004067270 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 >> 172.21.15.61:0/4093602927 conn(0x7ff0040b1950 msgr2=0x7ff0040b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 shutdown_connections 2022-07-12T08:41:24.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.048+0000 7ff003fff700 1 -- 172.21.15.61:0/4093602927 wait complete. 2022-07-12T08:41:24.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:24.066 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:24.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 Processor -- start 2022-07-12T08:41:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 -- start start 2022-07-12T08:41:24.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87740634e0 0x7f8774140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8774067b90 0x7f87741413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8774060860 con 0x7f8774098720 2022-07-12T08:41:24.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8774140ad0 con 0x7f8774067b90 2022-07-12T08:41:24.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8773fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8774144220 con 0x7f87740634e0 2022-07-12T08:41:24.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f87737fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8774067b90 0x7f87741413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f87737fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8774067b90 0x7f87741413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53252/0 (socket says 172.21.15.61:53252) 2022-07-12T08:41:24.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f87737fe700 1 -- 172.21.15.61:0/4061542545 learned_addr learned my addr 172.21.15.61:0/4061542545 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:24.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f8772ffd700 1 --1- 172.21.15.61:0/4061542545 >> v1:172.21.15.61:6789/0 conn(0x7f8774098720 0x7f8774060150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41466/0 (socket says 172.21.15.61:41466) 2022-07-12T08:41:24.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.180+0000 7f87727fc700 1 --2- 172.21.15.61:0/4061542545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87740634e0 0x7f8774140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f87737fe700 1 -- 172.21.15.61:0/4061542545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87740634e0 msgr2=0x7f8774140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:24.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f87737fe700 1 --2- 172.21.15.61:0/4061542545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87740634e0 0x7f8774140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f87737fe700 1 -- 172.21.15.61:0/4061542545 >> v1:172.21.15.61:6789/0 conn(0x7f8774098720 legacy=0x7f8774060150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f87737fe700 1 -- 172.21.15.61:0/4061542545 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87740b0a30 con 0x7f8774067b90 2022-07-12T08:41:24.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f87737fe700 1 --2- 172.21.15.61:0/4061542545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8774067b90 0x7f87741413b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f876c0032f0 tx=0x7f876c009290).ready entity=mon.0 client_cookie=67eba9a221eea1ab server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8771ffb700 1 -- 172.21.15.61:0/4061542545 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f876c00b8a0 con 0x7f8774067b90 2022-07-12T08:41:24.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8771ffb700 1 -- 172.21.15.61:0/4061542545 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876c00ba00 con 0x7f8774067b90 2022-07-12T08:41:24.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8771ffb700 1 -- 172.21.15.61:0/4061542545 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f876c00bb70 con 0x7f8774067b90 2022-07-12T08:41:24.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 -- 172.21.15.61:0/4061542545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8774067b90 msgr2=0x7f87741413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 --2- 172.21.15.61:0/4061542545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8774067b90 0x7f87741413b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f876c0032f0 tx=0x7f876c009290).stop 2022-07-12T08:41:24.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 -- 172.21.15.61:0/4061542545 shutdown_connections 2022-07-12T08:41:24.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 --2- 172.21.15.61:0/4061542545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8774067b90 0x7f87741413b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 --2- 172.21.15.61:0/4061542545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87740634e0 0x7f8774140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 -- 172.21.15.61:0/4061542545 >> 172.21.15.61:0/4061542545 conn(0x7f87740b3600 msgr2=0x7f8774060d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 -- 172.21.15.61:0/4061542545 shutdown_connections 2022-07-12T08:41:24.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 -- 172.21.15.61:0/4061542545 wait complete. 2022-07-12T08:41:24.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 Processor -- start 2022-07-12T08:41:24.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.184+0000 7f8773fff700 1 -- start start 2022-07-12T08:41:24.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 0x7f877414fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8774067b90 0x7f8774150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8774098720 0x7f8774146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87741394f0 con 0x7f87740634e0 2022-07-12T08:41:24.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b5410 con 0x7f8774067b90 2022-07-12T08:41:24.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87740b56b0 con 0x7f8774098720 2022-07-12T08:41:24.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 0x7f877414fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 0x7f877414fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53254/0 (socket says 172.21.15.61:53254) 2022-07-12T08:41:24.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 -- 172.21.15.61:0/73708061 learned_addr learned my addr 172.21.15.61:0/73708061 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:24.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f87737fe700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8774098720 0x7f8774146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f87727fc700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8774067b90 0x7f8774150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 -- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8774098720 msgr2=0x7f8774146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8774098720 0x7f8774146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 -- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8774067b90 msgr2=0x7f8774150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8774067b90 0x7f8774150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 -- 172.21.15.61:0/73708061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c009530 con 0x7f87740634e0 2022-07-12T08:41:24.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f87727fc700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8774067b90 0x7f8774150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:24.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8772ffd700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 0x7f877414fbe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8774062070 tx=0x7f8764007c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f876400ce30 con 0x7f87740634e0 2022-07-12T08:41:24.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87640072e0 con 0x7f87740634e0 2022-07-12T08:41:24.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8774150b20 con 0x7f87740634e0 2022-07-12T08:41:24.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8764005740 con 0x7f87740634e0 2022-07-12T08:41:24.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8774146750 con 0x7f87740634e0 2022-07-12T08:41:24.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877413aa80 con 0x7f87740634e0 2022-07-12T08:41:24.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.188+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8764010070 con 0x7f87740634e0 2022-07-12T08:41:24.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.192+0000 7f876bfff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87580539b0 0x7f8758055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.192+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f876406b450 con 0x7f87740634e0 2022-07-12T08:41:24.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.192+0000 7f87727fc700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87580539b0 0x7f8758055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.192+0000 7f87727fc700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87580539b0 0x7f8758055e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f875c00b000 tx=0x7f875c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.196+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f87640423b0 con 0x7f87740634e0 2022-07-12T08:41:24.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.320+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8774057460 con 0x7f87740634e0 2022-07-12T08:41:24.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f876bfff700 1 -- 172.21.15.61:0/73708061 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f87640459d0 con 0x7f87740634e0 2022-07-12T08:41:24.394 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:24.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87580539b0 msgr2=0x7f8758055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87580539b0 0x7f8758055e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f875c00b000 tx=0x7f875c008040).stop 2022-07-12T08:41:24.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 msgr2=0x7f877414fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 0x7f877414fbe0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8774062070 tx=0x7f8764007c60).stop 2022-07-12T08:41:24.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 shutdown_connections 2022-07-12T08:41:24.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8774098720 0x7f8774146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87580539b0 0x7f8758055e70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8774067b90 0x7f8774150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 --2- 172.21.15.61:0/73708061 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87740634e0 0x7f877414fbe0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.324+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 >> 172.21.15.61:0/73708061 conn(0x7f87740b3600 msgr2=0x7f87741445b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.328+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 shutdown_connections 2022-07-12T08:41:24.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.328+0000 7f8773fff700 1 -- 172.21.15.61:0/73708061 wait complete. 2022-07-12T08:41:24.399 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:24.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.548+0000 7f6e30ff9700 1 Processor -- start 2022-07-12T08:41:24.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.548+0000 7f6e30ff9700 1 -- start start 2022-07-12T08:41:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6e2c0aad90 0x7f6e2c0ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c0ab170 con 0x7f6e2c05bd10 2022-07-12T08:41:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2c061b90 con 0x7f6e2c0b2e70 2022-07-12T08:41:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2c1435c0 con 0x7f6e2c0aad90 2022-07-12T08:41:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6e2c05bd10 0x7f6e2c141a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41484/0 (socket says 172.21.15.61:41484) 2022-07-12T08:41:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2bfff700 1 -- 172.21.15.61:0/1007905212 learned_addr learned my addr 172.21.15.61:0/1007905212 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2b7fe700 1 --2- 172.21.15.61:0/1007905212 >> v2:172.21.15.111:3300/0 conn(0x7f6e2c0aad90 0x7f6e2c0ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234427522 0 0) 0x7f6e2c0ab170 con 0x7f6e2c05bd10 2022-07-12T08:41:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c003260 con 0x7f6e2c05bd10 2022-07-12T08:41:24.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343956455 0 0) 0x7f6e2c061b90 con 0x7f6e2c0b2e70 2022-07-12T08:41:24.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c0ab170 con 0x7f6e2c0b2e70 2022-07-12T08:41:24.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/1007905212 >> v1:172.21.15.111:6789/0 conn(0x7f6e2c0b2e70 legacy=0x7f6e2c0603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/1007905212 >> v1:172.21.15.61:6789/0 conn(0x7f6e2c05bd10 legacy=0x7f6e2c141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/1007905212 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c143860 con 0x7f6e2c0aad90 2022-07-12T08:41:24.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2b7fe700 1 --2- 172.21.15.61:0/1007905212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0aad90 0x7f6e2c0ab490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f6e1c002f70 tx=0x7f6e1c006df0).ready entity=mon.1 client_cookie=c2e691cd534564aa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e1c00d540 con 0x7f6e2c0aad90 2022-07-12T08:41:24.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c000e90 con 0x7f6e2c0aad90 2022-07-12T08:41:24.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e2affd700 1 -- 172.21.15.61:0/1007905212 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e1c00d890 con 0x7f6e2c0aad90 2022-07-12T08:41:24.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- 172.21.15.61:0/1007905212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0aad90 msgr2=0x7f6e2c0ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/1007905212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0aad90 0x7f6e2c0ab490 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f6e1c002f70 tx=0x7f6e1c006df0).stop 2022-07-12T08:41:24.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- 172.21.15.61:0/1007905212 shutdown_connections 2022-07-12T08:41:24.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/1007905212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0aad90 0x7f6e2c0ab490 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- 172.21.15.61:0/1007905212 >> 172.21.15.61:0/1007905212 conn(0x7f6e2c0b57c0 msgr2=0x7f6e2c0b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- 172.21.15.61:0/1007905212 shutdown_connections 2022-07-12T08:41:24.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- 172.21.15.61:0/1007905212 wait complete. 2022-07-12T08:41:24.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 Processor -- start 2022-07-12T08:41:24.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.552+0000 7f6e30ff9700 1 -- start start 2022-07-12T08:41:24.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 0x7f6e2c13f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6e2c0aad90 0x7f6e2c1389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0b2e70 0x7f6e2c138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c1465e0 con 0x7f6e2c05bd10 2022-07-12T08:41:24.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c146880 con 0x7f6e2c0b2e70 2022-07-12T08:41:24.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2c146b20 con 0x7f6e2c0aad90 2022-07-12T08:41:24.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 0x7f6e2c13f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 0x7f6e2c13f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53268/0 (socket says 172.21.15.61:53268) 2022-07-12T08:41:24.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/3553796332 learned_addr learned my addr 172.21.15.61:0/3553796332 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:24.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2bfff700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0b2e70 0x7f6e2c138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e23fff700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6e2c0aad90 0x7f6e2c1389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6e2c0aad90 msgr2=0x7f6e2c1389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6e2c0aad90 0x7f6e2c1389b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0b2e70 msgr2=0x7f6e2c138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0b2e70 0x7f6e2c138ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 -- 172.21.15.61:0/3553796332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c00b180 con 0x7f6e2c05bd10 2022-07-12T08:41:24.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2bfff700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0b2e70 0x7f6e2c138ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:24.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e2b7fe700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 0x7f6e2c13f3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f6e1c002f70 tx=0x7f6e1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e1c016070 con 0x7f6e2c05bd10 2022-07-12T08:41:24.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2c139390 con 0x7f6e2c05bd10 2022-07-12T08:41:24.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1c013850 con 0x7f6e2c05bd10 2022-07-12T08:41:24.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e1c013bb0 con 0x7f6e2c05bd10 2022-07-12T08:41:24.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2c13ed30 con 0x7f6e2c05bd10 2022-07-12T08:41:24.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.556+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e2c145900 con 0x7f6e2c05bd10 2022-07-12T08:41:24.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.560+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6e1c025030 con 0x7f6e2c05bd10 2022-07-12T08:41:24.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.560+0000 7f6e297fa700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6e10053ad0 0x7f6e10055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.560+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e1c06cdb0 con 0x7f6e2c05bd10 2022-07-12T08:41:24.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.560+0000 7f6e23fff700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6e10053ad0 0x7f6e10055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.560+0000 7f6e23fff700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6e10053ad0 0x7f6e10055f90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f6e1400d000 tx=0x7f6e14007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.568+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e2c145900 con 0x7f6e2c05bd10 2022-07-12T08:41:24.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6e2c145900 con 0x7f6e2c05bd10 2022-07-12T08:41:24.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e297fa700 1 -- 172.21.15.61:0/3553796332 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f6e2c145900 con 0x7f6e2c05bd10 2022-07-12T08:41:24.711 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:24.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6e10053ad0 msgr2=0x7f6e10055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6e10053ad0 0x7f6e10055f90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f6e1400d000 tx=0x7f6e14007ad0).stop 2022-07-12T08:41:24.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 msgr2=0x7f6e2c13f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 0x7f6e2c13f3c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f6e1c002f70 tx=0x7f6e1c014000).stop 2022-07-12T08:41:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 shutdown_connections 2022-07-12T08:41:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6e10053ad0 0x7f6e10055f90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6e2c0b2e70 0x7f6e2c138ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6e2c0aad90 0x7f6e2c1389b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 --2- 172.21.15.61:0/3553796332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e2c05bd10 0x7f6e2c13f3c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 >> 172.21.15.61:0/3553796332 conn(0x7f6e2c0b57c0 msgr2=0x7f6e2c0b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 shutdown_connections 2022-07-12T08:41:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.696+0000 7f6e30ff9700 1 -- 172.21.15.61:0/3553796332 wait complete. 2022-07-12T08:41:24.728 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:24.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa698f77700 1 Processor -- start 2022-07-12T08:41:24.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa698f77700 1 -- start start 2022-07-12T08:41:24.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa698f77700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa694144860 0x7fa69414b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa698f77700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6940b0500 con 0x7fa6940b0120 2022-07-12T08:41:24.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa698f77700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69405d0b0 con 0x7fa6940ab730 2022-07-12T08:41:24.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa698f77700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940b1040 con 0x7fa694144860 2022-07-12T08:41:24.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa69259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa6940b0120 0x7fa694144150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41494/0 (socket says 172.21.15.61:41494) 2022-07-12T08:41:24.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa69259c700 1 -- 172.21.15.61:0/2107502513 learned_addr learned my addr 172.21.15.61:0/2107502513 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:24.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa692d9d700 1 --2- 172.21.15.61:0/2107502513 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa694144860 0x7fa69414b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa691d9b700 1 -- 172.21.15.61:0/2107502513 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178908538 0 0) 0x7fa6940b0500 con 0x7fa6940b0120 2022-07-12T08:41:24.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa691d9b700 1 -- 172.21.15.61:0/2107502513 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa674003260 con 0x7fa6940b0120 2022-07-12T08:41:24.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.844+0000 7fa692d9d700 1 -- 172.21.15.61:0/2107502513 >> v1:172.21.15.111:6789/0 conn(0x7fa6940ab730 legacy=0x7fa69414aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa692d9d700 1 -- 172.21.15.61:0/2107502513 >> v1:172.21.15.61:6789/0 conn(0x7fa6940b0120 legacy=0x7fa694144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa692d9d700 1 -- 172.21.15.61:0/2107502513 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69405d350 con 0x7fa694144860 2022-07-12T08:41:24.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa692d9d700 1 --2- 172.21.15.61:0/2107502513 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa694144860 0x7fa69414b1e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa67c001110 tx=0x7fa68c0091d0).ready entity=mon.2 client_cookie=478bbddf308f4f72 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa691d9b700 1 -- 172.21.15.61:0/2107502513 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa68c00b580 con 0x7fa694144860 2022-07-12T08:41:24.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa691d9b700 1 -- 172.21.15.61:0/2107502513 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa68c00b6e0 con 0x7fa694144860 2022-07-12T08:41:24.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa691d9b700 1 -- 172.21.15.61:0/2107502513 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa68c00b850 con 0x7fa694144860 2022-07-12T08:41:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 -- 172.21.15.61:0/2107502513 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa694144860 msgr2=0x7fa69414b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 --2- 172.21.15.61:0/2107502513 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa694144860 0x7fa69414b1e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fa67c001110 tx=0x7fa68c0091d0).stop 2022-07-12T08:41:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 -- 172.21.15.61:0/2107502513 shutdown_connections 2022-07-12T08:41:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 --2- 172.21.15.61:0/2107502513 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa694144860 0x7fa69414b1e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 -- 172.21.15.61:0/2107502513 >> 172.21.15.61:0/2107502513 conn(0x7fa6940b5290 msgr2=0x7fa6940b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 -- 172.21.15.61:0/2107502513 shutdown_connections 2022-07-12T08:41:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.848+0000 7fa698f77700 1 -- 172.21.15.61:0/2107502513 wait complete. 2022-07-12T08:41:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 Processor -- start 2022-07-12T08:41:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 -- start start 2022-07-12T08:41:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6940ab730 0x7fa69414a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 0x7fa69415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa694144860 0x7fa6941425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6940af310 con 0x7fa6940b0120 2022-07-12T08:41:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69415a7f0 con 0x7fa694144860 2022-07-12T08:41:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa694067670 con 0x7fa6940ab730 2022-07-12T08:41:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 0x7fa69415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 0x7fa69415a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53284/0 (socket says 172.21.15.61:53284) 2022-07-12T08:41:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 -- 172.21.15.61:0/204851121 learned_addr learned my addr 172.21.15.61:0/204851121 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa69259c700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6940ab730 0x7fa69414a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa692d9d700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa694144860 0x7fa6941425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 -- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6940ab730 msgr2=0x7fa69414a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6940ab730 0x7fa69414a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 -- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa694144860 msgr2=0x7fa6941425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa694144860 0x7fa6941425f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 -- 172.21.15.61:0/204851121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68c009470 con 0x7fa6940b0120 2022-07-12T08:41:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa692d9d700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa694144860 0x7fa6941425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa689d9b700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 0x7fa69415a300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa694156350 tx=0x7fa67c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa67c007240 con 0x7fa6940b0120 2022-07-12T08:41:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c0073a0 con 0x7fa6940b0120 2022-07-12T08:41:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa694067910 con 0x7fa6940b0120 2022-07-12T08:41:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa67c006700 con 0x7fa6940b0120 2022-07-12T08:41:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.852+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6940ad1c0 con 0x7fa6940b0120 2022-07-12T08:41:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.856+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa67c010030 con 0x7fa6940b0120 2022-07-12T08:41:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.856+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6940aca10 con 0x7fa6940b0120 2022-07-12T08:41:24.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.856+0000 7fa68bfff700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa678053ad0 0x7fa678055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:24.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.856+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa67c06bd90 con 0x7fa6940b0120 2022-07-12T08:41:24.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.856+0000 7fa69259c700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa678053ad0 0x7fa678055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:24.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.856+0000 7fa69259c700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa678053ad0 0x7fa678055f90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa68400d000 tx=0x7fa684009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:24.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.860+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa6940aca10 con 0x7fa6940b0120 2022-07-12T08:41:24.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.988+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa6940aca10 con 0x7fa6940b0120 2022-07-12T08:41:24.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.988+0000 7fa68bfff700 1 -- 172.21.15.61:0/204851121 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa6940aca10 con 0x7fa6940b0120 2022-07-12T08:41:24.988 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:24.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa678053ad0 msgr2=0x7fa678055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa678053ad0 0x7fa678055f90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa68400d000 tx=0x7fa684009040).stop 2022-07-12T08:41:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 msgr2=0x7fa69415a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 0x7fa69415a300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa694156350 tx=0x7fa67c007c00).stop 2022-07-12T08:41:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 shutdown_connections 2022-07-12T08:41:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa678053ad0 0x7fa678055f90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa694144860 0x7fa6941425f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6940b0120 0x7fa69415a300 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 --2- 172.21.15.61:0/204851121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6940ab730 0x7fa69414a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:24.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 >> 172.21.15.61:0/204851121 conn(0x7fa6940b5290 msgr2=0x7fa6940b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:24.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 shutdown_connections 2022-07-12T08:41:24.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:24.992+0000 7fa698f77700 1 -- 172.21.15.61:0/204851121 wait complete. 2022-07-12T08:41:25.028 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:25.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.160+0000 7fbd6c9f4700 1 Processor -- start 2022-07-12T08:41:25.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.160+0000 7fbd6c9f4700 1 -- start start 2022-07-12T08:41:25.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbd68141ec0 0x7fbd681422e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6805bdb0 con 0x7fbd680b7bd0 2022-07-12T08:41:25.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd680b7fb0 con 0x7fbd68141ec0 2022-07-12T08:41:25.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680b2030 con 0x7fbd680ad4d0 2022-07-12T08:41:25.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbd68141ec0 0x7fbd681422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbd68141ec0 0x7fbd681422e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53294/0 (socket says 172.21.15.61:53294) 2022-07-12T08:41:25.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 -- 172.21.15.61:0/1047930753 learned_addr learned my addr 172.21.15.61:0/1047930753 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:25.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd5effd700 1 --2- 172.21.15.61:0/1047930753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 -- 172.21.15.61:0/1047930753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 msgr2=0x7fbd681419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:25.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 --2- 172.21.15.61:0/1047930753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 -- 172.21.15.61:0/1047930753 >> v1:172.21.15.111:6789/0 conn(0x7fbd680b7bd0 legacy=0x7fbd680acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 -- 172.21.15.61:0/1047930753 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd681427d0 con 0x7fbd68141ec0 2022-07-12T08:41:25.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd67fff700 1 --2- 172.21.15.61:0/1047930753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd681422e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fbd600032f0 tx=0x7fbd60009290).ready entity=mon.0 client_cookie=311619ca7f96c9af server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:25.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd66ffd700 1 -- 172.21.15.61:0/1047930753 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd6000b8a0 con 0x7fbd68141ec0 2022-07-12T08:41:25.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd66ffd700 1 -- 172.21.15.61:0/1047930753 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd6000ba00 con 0x7fbd68141ec0 2022-07-12T08:41:25.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd66ffd700 1 -- 172.21.15.61:0/1047930753 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd6000bb70 con 0x7fbd68141ec0 2022-07-12T08:41:25.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/1047930753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 msgr2=0x7fbd681422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/1047930753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd681422e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fbd600032f0 tx=0x7fbd60009290).stop 2022-07-12T08:41:25.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/1047930753 shutdown_connections 2022-07-12T08:41:25.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/1047930753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd681422e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/1047930753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/1047930753 >> 172.21.15.61:0/1047930753 conn(0x7fbd680655e0 msgr2=0x7fbd680b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:25.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/1047930753 shutdown_connections 2022-07-12T08:41:25.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/1047930753 wait complete. 2022-07-12T08:41:25.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 Processor -- start 2022-07-12T08:41:25.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.164+0000 7fbd6c9f4700 1 -- start start 2022-07-12T08:41:25.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd680b7bd0 0x7fbd68152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd68152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68151cf0 con 0x7fbd68141ec0 2022-07-12T08:41:25.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680b0e50 con 0x7fbd680b7bd0 2022-07-12T08:41:25.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6813f9d0 con 0x7fbd680ad4d0 2022-07-12T08:41:25.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd68152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd680b7bd0 0x7fbd68152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd677fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd68152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53300/0 (socket says 172.21.15.61:53300) 2022-07-12T08:41:25.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd67fff700 1 -- 172.21.15.61:0/3752594841 learned_addr learned my addr 172.21.15.61:0/3752594841 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:25.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 -- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 msgr2=0x7fbd681411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 -- 172.21.15.61:0/3752594841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 msgr2=0x7fbd68152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd68152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 -- 172.21.15.61:0/3752594841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd60009530 con 0x7fbd680b7bd0 2022-07-12T08:41:25.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd5effd700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd680b7bd0 0x7fbd68152370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fbd5000c000 tx=0x7fbd50006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:25.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd50010780 con 0x7fbd680b7bd0 2022-07-12T08:41:25.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd500108e0 con 0x7fbd680b7bd0 2022-07-12T08:41:25.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd68152d50 con 0x7fbd680b7bd0 2022-07-12T08:41:25.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd50010c40 con 0x7fbd680b7bd0 2022-07-12T08:41:25.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd680aed10 con 0x7fbd680b7bd0 2022-07-12T08:41:25.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.168+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd68057460 con 0x7fbd680b7bd0 2022-07-12T08:41:25.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.172+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbd5001c460 con 0x7fbd680b7bd0 2022-07-12T08:41:25.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.172+0000 7fbd657fa700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbd4c053ad0 0x7fbd4c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.172+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbd5006c200 con 0x7fbd680b7bd0 2022-07-12T08:41:25.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.172+0000 7fbd677fe700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbd4c053ad0 0x7fbd4c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.172+0000 7fbd677fe700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbd4c053ad0 0x7fbd4c055f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbd5800d000 tx=0x7fbd58000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:25.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.180+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbd50042ff0 con 0x7fbd680b7bd0 2022-07-12T08:41:25.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.304+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd6806b2b0 con 0x7fbd680b7bd0 2022-07-12T08:41:25.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd657fa700 1 -- 172.21.15.61:0/3752594841 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fbd50046610 con 0x7fbd680b7bd0 2022-07-12T08:41:25.382 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:25.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbd4c053ad0 msgr2=0x7fbd4c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbd4c053ad0 0x7fbd4c055f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fbd5800d000 tx=0x7fbd58000bc0).stop 2022-07-12T08:41:25.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd680b7bd0 msgr2=0x7fbd68152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd680b7bd0 0x7fbd68152370 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fbd5000c000 tx=0x7fbd50006df0).stop 2022-07-12T08:41:25.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 shutdown_connections 2022-07-12T08:41:25.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbd68141ec0 0x7fbd68152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbd4c053ad0 0x7fbd4c055f90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbd680b7bd0 0x7fbd68152370 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.312+0000 7fbd6c9f4700 1 --2- 172.21.15.61:0/3752594841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbd680ad4d0 0x7fbd681411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.316+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 >> 172.21.15.61:0/3752594841 conn(0x7fbd680655e0 msgr2=0x7fbd68148060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:25.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.316+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 shutdown_connections 2022-07-12T08:41:25.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.316+0000 7fbd6c9f4700 1 -- 172.21.15.61:0/3752594841 wait complete. 2022-07-12T08:41:25.415 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:25.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 Processor -- start 2022-07-12T08:41:25.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 -- start start 2022-07-12T08:41:25.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f21c01440a0 0x7f21c005bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c0144480 0x7f21c01413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21c01418d0 0x7f21c013ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c00624b0 con 0x7f21c01418d0 2022-07-12T08:41:25.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c00b3010 con 0x7f21c01440a0 2022-07-12T08:41:25.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c72c2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c0144da0 con 0x7f21c0144480 2022-07-12T08:41:25.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c6ac1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21c01418d0 0x7f21c013ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c6ac1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21c01418d0 0x7f21c013ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53308/0 (socket says 172.21.15.61:53308) 2022-07-12T08:41:25.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c6ac1700 1 -- 172.21.15.61:0/1453788032 learned_addr learned my addr 172.21.15.61:0/1453788032 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:25.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c62c0700 1 --2- 172.21.15.61:0/1453788032 >> v2:172.21.15.111:3300/0 conn(0x7f21c01440a0 0x7f21c005bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.536+0000 7f21c5abf700 1 --2- 172.21.15.61:0/1453788032 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c0144480 0x7f21c01413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c6ac1700 1 -- 172.21.15.61:0/1453788032 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c0144480 msgr2=0x7f21c01413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:25.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c6ac1700 1 --2- 172.21.15.61:0/1453788032 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c0144480 0x7f21c01413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c6ac1700 1 -- 172.21.15.61:0/1453788032 >> v2:172.21.15.111:3300/0 conn(0x7f21c01440a0 msgr2=0x7f21c005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:25.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c6ac1700 1 --2- 172.21.15.61:0/1453788032 >> v2:172.21.15.111:3300/0 conn(0x7f21c01440a0 0x7f21c005bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c6ac1700 1 -- 172.21.15.61:0/1453788032 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00b2d70 con 0x7f21c01418d0 2022-07-12T08:41:25.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c6ac1700 1 --2- 172.21.15.61:0/1453788032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c01418d0 0x7f21c013ff10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f21bc002f70 tx=0x7f21bc006eb0).ready entity=mon.0 client_cookie=180dd0cb85d6d7a6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:25.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c52be700 1 -- 172.21.15.61:0/1453788032 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21bc00d570 con 0x7f21c01418d0 2022-07-12T08:41:25.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c52be700 1 -- 172.21.15.61:0/1453788032 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc000e90 con 0x7f21c01418d0 2022-07-12T08:41:25.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c52be700 1 -- 172.21.15.61:0/1453788032 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21bc00d900 con 0x7f21c01418d0 2022-07-12T08:41:25.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 -- 172.21.15.61:0/1453788032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c01418d0 msgr2=0x7f21c013ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1453788032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c01418d0 0x7f21c013ff10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f21bc002f70 tx=0x7f21bc006eb0).stop 2022-07-12T08:41:25.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 -- 172.21.15.61:0/1453788032 shutdown_connections 2022-07-12T08:41:25.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1453788032 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c0144480 0x7f21c01413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1453788032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c01418d0 0x7f21c013ff10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1453788032 >> v2:172.21.15.111:3300/0 conn(0x7f21c01440a0 0x7f21c005bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 -- 172.21.15.61:0/1453788032 >> 172.21.15.61:0/1453788032 conn(0x7f21c00b70b0 msgr2=0x7f21c00b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:25.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 -- 172.21.15.61:0/1453788032 shutdown_connections 2022-07-12T08:41:25.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 -- 172.21.15.61:0/1453788032 wait complete. 2022-07-12T08:41:25.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 Processor -- start 2022-07-12T08:41:25.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.540+0000 7f21c72c2700 1 -- start start 2022-07-12T08:41:25.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 0x7f21c0143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 0x7f21c013c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 0x7f21c013c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c0151bd0 con 0x7f21c0144480 2022-07-12T08:41:25.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00ac070 con 0x7f21c01440a0 2022-07-12T08:41:25.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c01428f0 con 0x7f21c01418d0 2022-07-12T08:41:25.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c6ac1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 0x7f21c013c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c62c0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 0x7f21c0143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 0x7f21c013c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 0x7f21c013c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:60118/0 (socket says 172.21.15.61:60118) 2022-07-12T08:41:25.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c62c0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 0x7f21c0143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:35506/0 (socket says 172.21.15.61:35506) 2022-07-12T08:41:25.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 -- 172.21.15.61:0/1694388958 learned_addr learned my addr 172.21.15.61:0/1694388958 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:25.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c6ac1700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 0x7f21c013c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53314/0 (socket says 172.21.15.61:53314) 2022-07-12T08:41:25.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 -- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 msgr2=0x7f21c0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 0x7f21c0143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 -- 172.21.15.61:0/1694388958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 msgr2=0x7f21c013c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 0x7f21c013c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 -- 172.21.15.61:0/1694388958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc00b1e0 con 0x7f21c01440a0 2022-07-12T08:41:25.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c62c0700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 0x7f21c0143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:25.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c6ac1700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 0x7f21c013c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:25.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c5abf700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 0x7f21c013c230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f21b000c000 tx=0x7f21b0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:25.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21b0011750 con 0x7f21c01440a0 2022-07-12T08:41:25.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b00118b0 con 0x7f21c01440a0 2022-07-12T08:41:25.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c009c350 con 0x7f21c01440a0 2022-07-12T08:41:25.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21b0011c10 con 0x7f21c01440a0 2022-07-12T08:41:25.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c013cc10 con 0x7f21c01440a0 2022-07-12T08:41:25.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.544+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c01430d0 con 0x7f21c01440a0 2022-07-12T08:41:25.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.548+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f21b0011d70 con 0x7f21c01440a0 2022-07-12T08:41:25.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.548+0000 7f21af7fe700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21a00539b0 0x7f21a0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:25.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.548+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f21b006c390 con 0x7f21c01440a0 2022-07-12T08:41:25.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.548+0000 7f21c62c0700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21a00539b0 0x7f21a0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:25.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.548+0000 7f21c62c0700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21a00539b0 0x7f21a0055e70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f21b800b000 tx=0x7f21b80091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:25.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.556+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21b00431f0 con 0x7f21c01440a0 2022-07-12T08:41:25.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.724+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21c0151d40 con 0x7f21c01440a0 2022-07-12T08:41:25.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.724+0000 7f21af7fe700 1 -- 172.21.15.61:0/1694388958 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+2137 (secure 0 0 0) 0x7f21b0046810 con 0x7f21c01440a0 2022-07-12T08:41:25.727 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:25.746 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:15.584363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.111:6835/1570404457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1570404457},{"type":"v1","addr":"172.21.15.111:6835","nonce":1570404457}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:25.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.724+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21a00539b0 msgr2=0x7f21a0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.724+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21a00539b0 0x7f21a0055e70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f21b800b000 tx=0x7f21b80091d0).stop 2022-07-12T08:41:25.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.724+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 msgr2=0x7f21c013c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:25.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.724+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 0x7f21c013c230 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f21b000c000 tx=0x7f21b0006df0).stop 2022-07-12T08:41:25.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 shutdown_connections 2022-07-12T08:41:25.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21c0144480 0x7f21c013c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21a00539b0 0x7f21a0055e70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21c01440a0 0x7f21c013c230 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 --2- 172.21.15.61:0/1694388958 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21c01418d0 0x7f21c0143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:25.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 >> 172.21.15.61:0/1694388958 conn(0x7f21c00b70b0 msgr2=0x7f21c005c280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:25.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 shutdown_connections 2022-07-12T08:41:25.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:25.728+0000 7f21c72c2700 1 -- 172.21.15.61:0/1694388958 wait complete. 2022-07-12T08:41:25.788 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2022-07-12T08:41:25.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:41:14.580428+0000', 'modified': '2022-07-12T08:41:15.584363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.111:6835/1570404457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 1570404457}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 1570404457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:41:25.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:41:25.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:41:26.140 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":4282,"want_state":"up:active","state":"up:active","rank_uptime":11.543703144,"mdsmap_epoch":5,"osdmap_epoch":22,"osdmap_epoch_barrier":21,"uptime":14.882618731999999} 2022-07-12T08:41:26.145 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 4282, "mdsmap_epoch": 5, "osdmap_epoch": 22, "osdmap_epoch_barrier": 21, "rank_uptime": 11.543703144, "state": "up:active", "uptime": 14.882618732, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:41:26.146 INFO:teuthology.run_tasks:Running task ceph-fuse... 2022-07-12T08:41:26.226 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2022-07-12T08:41:26.227 INFO:tasks.ceph_fuse:config is None 2022-07-12T08:41:26.227 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'client.2': None} 2022-07-12T08:41:26.228 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2022-07-12T08:41:26.228 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 Processor -- start 2022-07-12T08:41:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 -- start start 2022-07-12T08:41:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37680af280 0x7f3768060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f37680b3f60 0x7f37680b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37680aecb0 con 0x7f376809bce0 2022-07-12T08:41:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37680afd70 con 0x7f37680b3f60 2022-07-12T08:41:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376edd6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3768143b00 con 0x7f37680af280 2022-07-12T08:41:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376d5d3700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f376809bce0 0x7f37680af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41538/0 (socket says 172.21.15.61:41538) 2022-07-12T08:41:26.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376d5d3700 1 -- 172.21.15.61:0/1939103230 learned_addr learned my addr 172.21.15.61:0/1939103230 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:26.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376ddd4700 1 --2- 172.21.15.61:0/1939103230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37680af280 0x7f3768060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376e5d5700 1 --2- 172.21.15.61:0/1939103230 >> v2:172.21.15.111:3300/0 conn(0x7f37680b3f60 0x7f37680b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376cdd2700 1 -- 172.21.15.61:0/1939103230 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091877522 0 0) 0x7f37680aecb0 con 0x7f376809bce0 2022-07-12T08:41:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376cdd2700 1 -- 172.21.15.61:0/1939103230 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3754003260 con 0x7f376809bce0 2022-07-12T08:41:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376ddd4700 1 -- 172.21.15.61:0/1939103230 >> v2:172.21.15.111:3300/0 conn(0x7f37680b3f60 msgr2=0x7f37680b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376ddd4700 1 --2- 172.21.15.61:0/1939103230 >> v2:172.21.15.111:3300/0 conn(0x7f37680b3f60 0x7f37680b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376ddd4700 1 -- 172.21.15.61:0/1939103230 >> v1:172.21.15.61:6789/0 conn(0x7f376809bce0 legacy=0x7f37680af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376ddd4700 1 -- 172.21.15.61:0/1939103230 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37681435c0 con 0x7f37680af280 2022-07-12T08:41:26.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376ddd4700 1 --2- 172.21.15.61:0/1939103230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37680af280 0x7f3768060680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f37600032f0 tx=0x7f37600091a0).ready entity=mon.2 client_cookie=465c0afcf03bf2e7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:26.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376cdd2700 1 -- 172.21.15.61:0/1939103230 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f376000b600 con 0x7f37680af280 2022-07-12T08:41:26.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376cdd2700 1 -- 172.21.15.61:0/1939103230 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376000b760 con 0x7f37680af280 2022-07-12T08:41:26.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.364+0000 7f376cdd2700 1 -- 172.21.15.61:0/1939103230 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f376000b8d0 con 0x7f37680af280 2022-07-12T08:41:26.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1939103230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37680af280 msgr2=0x7f3768060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 --2- 172.21.15.61:0/1939103230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37680af280 0x7f3768060680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f37600032f0 tx=0x7f37600091a0).stop 2022-07-12T08:41:26.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1939103230 shutdown_connections 2022-07-12T08:41:26.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 --2- 172.21.15.61:0/1939103230 >> v2:172.21.15.111:3300/0 conn(0x7f37680b3f60 0x7f37680b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 --2- 172.21.15.61:0/1939103230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37680af280 0x7f3768060680 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1939103230 >> 172.21.15.61:0/1939103230 conn(0x7f37680b92c0 msgr2=0x7f37680b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:26.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1939103230 shutdown_connections 2022-07-12T08:41:26.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1939103230 wait complete. 2022-07-12T08:41:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 Processor -- start 2022-07-12T08:41:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- start start 2022-07-12T08:41:26.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f376809bce0 0x7f37681443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37680af280 0x7f37680b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 0x7f37680b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37681429d0 con 0x7f37680b3f60 2022-07-12T08:41:26.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37680b3660 con 0x7f37680af280 2022-07-12T08:41:26.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3768141740 con 0x7f376809bce0 2022-07-12T08:41:26.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376e5d5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 0x7f37680b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376e5d5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 0x7f37680b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53326/0 (socket says 172.21.15.61:53326) 2022-07-12T08:41:26.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376e5d5700 1 -- 172.21.15.61:0/1206716330 learned_addr learned my addr 172.21.15.61:0/1206716330 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:26.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f376809bce0 0x7f37681443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376d5d3700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37680af280 0x7f37680b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 -- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37680af280 msgr2=0x7f37680b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37680af280 0x7f37680b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 -- 172.21.15.61:0/1206716330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 msgr2=0x7f37680b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 0x7f37680b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 -- 172.21.15.61:0/1206716330 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3760009420 con 0x7f376809bce0 2022-07-12T08:41:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376e5d5700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 0x7f37680b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376ddd4700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f376809bce0 0x7f37681443b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f37600032f0 tx=0x7f3760016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:26.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3760006440 con 0x7f376809bce0 2022-07-12T08:41:26.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37600065a0 con 0x7f376809bce0 2022-07-12T08:41:26.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3768141c60 con 0x7f376809bce0 2022-07-12T08:41:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3760006900 con 0x7f376809bce0 2022-07-12T08:41:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.368+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376809f020 con 0x7f376809bce0 2022-07-12T08:41:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.372+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3768141900 con 0x7f376809bce0 2022-07-12T08:41:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.372+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f376000f070 con 0x7f376809bce0 2022-07-12T08:41:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.372+0000 7f3766ffd700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3748053a40 0x7f3748055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.372+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f376006d090 con 0x7f376809bce0 2022-07-12T08:41:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.372+0000 7f376d5d3700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3748053a40 0x7f3748055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.372+0000 7f376d5d3700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3748053a40 0x7f3748055f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f375c00d000 tx=0x7f375c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.384+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3760043ff0 con 0x7f376809bce0 2022-07-12T08:41:26.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.512+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3768057460 con 0x7f376809bce0 2022-07-12T08:41:26.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f3766ffd700 1 -- 172.21.15.61:0/1206716330 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f3760015020 con 0x7f376809bce0 2022-07-12T08:41:26.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:26.530 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:26.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3748053a40 msgr2=0x7f3748055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3748053a40 0x7f3748055f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f375c00d000 tx=0x7f375c009040).stop 2022-07-12T08:41:26.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f376809bce0 msgr2=0x7f37681443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f376809bce0 0x7f37681443b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f37600032f0 tx=0x7f3760016000).stop 2022-07-12T08:41:26.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 shutdown_connections 2022-07-12T08:41:26.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37680b3f60 0x7f37680b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3748053a40 0x7f3748055f00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37680af280 0x7f37680b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 --2- 172.21.15.61:0/1206716330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f376809bce0 0x7f37681443b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 >> 172.21.15.61:0/1206716330 conn(0x7f37680b92c0 msgr2=0x7f37681480f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:26.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 shutdown_connections 2022-07-12T08:41:26.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.516+0000 7f376edd6700 1 -- 172.21.15.61:0/1206716330 wait complete. 2022-07-12T08:41:26.548 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:26.574 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 Processor -- start 2022-07-12T08:41:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- start start 2022-07-12T08:41:26.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f838c0981e0 0x7f838c147430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c147920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c060520 con 0x7f838c0b02f0 2022-07-12T08:41:26.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c0985c0 con 0x7f838c0981e0 2022-07-12T08:41:26.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0b14b0 con 0x7f838c0673f0 2022-07-12T08:41:26.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8391837700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c147920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392038700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f838c0981e0 0x7f838c147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392839700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f838c0b02f0 0x7f838c14a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:55156/0 (socket says 172.21.15.61:55156) 2022-07-12T08:41:26.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392839700 1 -- 172.21.15.61:0/1515826997 learned_addr learned my addr 172.21.15.61:0/1515826997 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:26.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392038700 1 -- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 msgr2=0x7f838c147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:26.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392038700 1 --2- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c147920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392038700 1 -- 172.21.15.61:0/1515826997 >> v1:172.21.15.111:6789/0 conn(0x7f838c0b02f0 legacy=0x7f838c14a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8391036700 1 -- 172.21.15.61:0/1515826997 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802213283 0 0) 0x7f838c060520 con 0x7f838c0b02f0 2022-07-12T08:41:26.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392038700 1 -- 172.21.15.61:0/1515826997 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0b19f0 con 0x7f838c0981e0 2022-07-12T08:41:26.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8391837700 1 --2- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:26.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8392038700 1 --2- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0981e0 0x7f838c147430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8388002f70 tx=0x7f8388006df0).ready entity=mon.1 client_cookie=8c9b5e9ca655ca82 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:26.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8391036700 1 -- 172.21.15.61:0/1515826997 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838800d570 con 0x7f838c0981e0 2022-07-12T08:41:26.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8391036700 1 -- 172.21.15.61:0/1515826997 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8388000e90 con 0x7f838c0981e0 2022-07-12T08:41:26.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f8391036700 1 -- 172.21.15.61:0/1515826997 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838800d8c0 con 0x7f838c0981e0 2022-07-12T08:41:26.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0981e0 msgr2=0x7f838c147430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 --2- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0981e0 0x7f838c147430 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8388002f70 tx=0x7f8388006df0).stop 2022-07-12T08:41:26.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- 172.21.15.61:0/1515826997 shutdown_connections 2022-07-12T08:41:26.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 --2- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 --2- 172.21.15.61:0/1515826997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0981e0 0x7f838c147430 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- 172.21.15.61:0/1515826997 >> 172.21.15.61:0/1515826997 conn(0x7f838c0b23f0 msgr2=0x7f838c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:26.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- 172.21.15.61:0/1515826997 shutdown_connections 2022-07-12T08:41:26.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.728+0000 7f839429c700 1 -- 172.21.15.61:0/1515826997 wait complete. 2022-07-12T08:41:26.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 Processor -- start 2022-07-12T08:41:26.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 -- start start 2022-07-12T08:41:26.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f838c0981e0 0x7f838c13fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0b02f0 0x7f838c1429c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c143f30 con 0x7f838c0981e0 2022-07-12T08:41:26.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c09b410 con 0x7f838c0b02f0 2022-07-12T08:41:26.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c09b6b0 con 0x7f838c0673f0 2022-07-12T08:41:26.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8391837700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f838c0981e0 0x7f838c13fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c148840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:35534/0 (socket says 172.21.15.61:35534) 2022-07-12T08:41:26.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 -- 172.21.15.61:0/2037755874 learned_addr learned my addr 172.21.15.61:0/2037755874 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:26.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392839700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0b02f0 0x7f838c1429c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 -- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0b02f0 msgr2=0x7f838c1429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0b02f0 0x7f838c1429c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 -- 172.21.15.61:0/2037755874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f838c0981e0 msgr2=0x7f838c13fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f838c0981e0 0x7f838c13fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 -- 172.21.15.61:0/2037755874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c00b180 con 0x7f838c0673f0 2022-07-12T08:41:26.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f8392038700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c148840 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f838c140a00 tx=0x7f8388014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:26.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8388016070 con 0x7f838c0673f0 2022-07-12T08:41:26.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8388013700 con 0x7f838c0673f0 2022-07-12T08:41:26.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838800b180 con 0x7f838c0673f0 2022-07-12T08:41:26.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8388013a60 con 0x7f838c0673f0 2022-07-12T08:41:26.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.732+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c143b70 con 0x7f838c0673f0 2022-07-12T08:41:26.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.736+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c143d60 con 0x7f838c0673f0 2022-07-12T08:41:26.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.736+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8388025030 con 0x7f838c0673f0 2022-07-12T08:41:26.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.736+0000 7f837b7fe700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f836c053a40 0x7f836c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:26.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.736+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f838806ce60 con 0x7f838c0673f0 2022-07-12T08:41:26.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.736+0000 7f8391837700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f836c053a40 0x7f836c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:26.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.736+0000 7f8391837700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f836c053a40 0x7f836c055f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f837c00b000 tx=0x7f837c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:26.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.744+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8388043cb0 con 0x7f838c0673f0 2022-07-12T08:41:26.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.872+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f838c057460 con 0x7f838c0673f0 2022-07-12T08:41:26.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.872+0000 7f837b7fe700 1 -- 172.21.15.61:0/2037755874 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8388023070 con 0x7f838c0673f0 2022-07-12T08:41:26.877 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:26.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.872+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f836c053a40 msgr2=0x7f836c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.872+0000 7f839429c700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f836c053a40 0x7f836c055f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f837c00b000 tx=0x7f837c006df0).stop 2022-07-12T08:41:26.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.872+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 msgr2=0x7f838c148840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:26.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.872+0000 7f839429c700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c148840 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f838c140a00 tx=0x7f8388014000).stop 2022-07-12T08:41:26.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 shutdown_connections 2022-07-12T08:41:26.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f836c053a40 0x7f836c055f00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f838c0b02f0 0x7f838c1429c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f838c0981e0 0x7f838c13fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 --2- 172.21.15.61:0/2037755874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f838c0673f0 0x7f838c148840 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:26.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 >> 172.21.15.61:0/2037755874 conn(0x7f838c0b23f0 msgr2=0x7f838c147bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:26.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 shutdown_connections 2022-07-12T08:41:26.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:26.876+0000 7f839429c700 1 -- 172.21.15.61:0/2037755874 wait complete. 2022-07-12T08:41:26.895 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:27.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.052+0000 7fa1c293f700 1 Processor -- start 2022-07-12T08:41:27.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.052+0000 7fa1c293f700 1 -- start start 2022-07-12T08:41:27.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc0b5920 0x7fa1bc0b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1bc0987f0 0x7fa1bc098c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc097c80 con 0x7fa1bc143580 2022-07-12T08:41:27.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0b5d00 con 0x7fa1bc0987f0 2022-07-12T08:41:27.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc05c7f0 con 0x7fa1bc0b5920 2022-07-12T08:41:27.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1bc0987f0 0x7fa1bc098c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1bc0987f0 0x7fa1bc098c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53346/0 (socket says 172.21.15.61:53346) 2022-07-12T08:41:27.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 -- 172.21.15.61:0/52003070 learned_addr learned my addr 172.21.15.61:0/52003070 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:27.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c193d700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc0b5920 0x7fa1bc0b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 -- 172.21.15.61:0/52003070 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc0b5920 msgr2=0x7fa1bc0b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:27.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc0b5920 0x7fa1bc0b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 -- 172.21.15.61:0/52003070 >> v1:172.21.15.111:6789/0 conn(0x7fa1bc143580 legacy=0x7fa1bc05faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 -- 172.21.15.61:0/52003070 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc05cbf0 con 0x7fa1bc0987f0 2022-07-12T08:41:27.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c193d700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc0b5920 0x7fa1bc0b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:27.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c113c700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc098c50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa1b80032f0 tx=0x7fa1b8009290).ready entity=mon.0 client_cookie=4e0188e8d58224f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c093b700 1 -- 172.21.15.61:0/52003070 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1b800b8a0 con 0x7fa1bc0987f0 2022-07-12T08:41:27.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c093b700 1 -- 172.21.15.61:0/52003070 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b800ba00 con 0x7fa1bc0987f0 2022-07-12T08:41:27.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c093b700 1 -- 172.21.15.61:0/52003070 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1b800bb70 con 0x7fa1bc0987f0 2022-07-12T08:41:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- 172.21.15.61:0/52003070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 msgr2=0x7fa1bc098c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc098c50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fa1b80032f0 tx=0x7fa1b8009290).stop 2022-07-12T08:41:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- 172.21.15.61:0/52003070 shutdown_connections 2022-07-12T08:41:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc098c50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 --2- 172.21.15.61:0/52003070 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc0b5920 0x7fa1bc0b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- 172.21.15.61:0/52003070 >> 172.21.15.61:0/52003070 conn(0x7fa1bc063340 msgr2=0x7fa1bc0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- 172.21.15.61:0/52003070 shutdown_connections 2022-07-12T08:41:27.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.056+0000 7fa1c293f700 1 -- 172.21.15.61:0/52003070 wait complete. 2022-07-12T08:41:27.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 Processor -- start 2022-07-12T08:41:27.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 -- start start 2022-07-12T08:41:27.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc0ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1bc0b5920 0x7fa1bc066da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc143580 0x7fa1bc067290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc063920 con 0x7fa1bc0987f0 2022-07-12T08:41:27.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc063ff0 con 0x7fa1bc0b5920 2022-07-12T08:41:27.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c293f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc065430 con 0x7fa1bc143580 2022-07-12T08:41:27.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c193d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc0ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c193d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc0ac6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53352/0 (socket says 172.21.15.61:53352) 2022-07-12T08:41:27.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.060+0000 7fa1c193d700 1 -- 172.21.15.61:0/2550904266 learned_addr learned my addr 172.21.15.61:0/2550904266 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:27.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c113c700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1bc0b5920 0x7fa1bc066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c213e700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc143580 0x7fa1bc067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c193d700 1 -- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc143580 msgr2=0x7fa1bc067290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c193d700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc143580 0x7fa1bc067290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c193d700 1 -- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1bc0b5920 msgr2=0x7fa1bc066da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c193d700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1bc0b5920 0x7fa1bc066da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c193d700 1 -- 172.21.15.61:0/2550904266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b8009530 con 0x7fa1bc0987f0 2022-07-12T08:41:27.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c113c700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1bc0b5920 0x7fa1bc066da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:27.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c193d700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc0ac6d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa1ac006940 tx=0x7fa1ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ac011850 con 0x7fa1bc0987f0 2022-07-12T08:41:27.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac0119b0 con 0x7fa1bc0987f0 2022-07-12T08:41:27.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1ac011d10 con 0x7fa1bc0987f0 2022-07-12T08:41:27.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc063b50 con 0x7fa1bc0987f0 2022-07-12T08:41:27.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc0656d0 con 0x7fa1bc0987f0 2022-07-12T08:41:27.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc057460 con 0x7fa1bc0987f0 2022-07-12T08:41:27.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.064+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa1ac012420 con 0x7fa1bc0987f0 2022-07-12T08:41:27.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.068+0000 7fa1aaffd700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa19c053920 0x7fa19c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.068+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa1ac06bec0 con 0x7fa1bc0987f0 2022-07-12T08:41:27.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.068+0000 7fa1c113c700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa19c053920 0x7fa19c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.068+0000 7fa1c113c700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa19c053920 0x7fa19c055de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa1b8019000 tx=0x7fa1b8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.072+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1ac042d20 con 0x7fa1bc0987f0 2022-07-12T08:41:27.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa1bc063ce0 con 0x7fa1bc0987f0 2022-07-12T08:41:27.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1aaffd700 1 -- 172.21.15.61:0/2550904266 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa1ac046340 con 0x7fa1bc0987f0 2022-07-12T08:41:27.200 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:27.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa19c053920 msgr2=0x7fa19c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa19c053920 0x7fa19c055de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa1b8019000 tx=0x7fa1b8016000).stop 2022-07-12T08:41:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 msgr2=0x7fa1bc0ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc0ac6d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa1ac006940 tx=0x7fa1ac008040).stop 2022-07-12T08:41:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 shutdown_connections 2022-07-12T08:41:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1bc143580 0x7fa1bc067290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa19c053920 0x7fa19c055de0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1bc0b5920 0x7fa1bc066da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 --2- 172.21.15.61:0/2550904266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1bc0987f0 0x7fa1bc0ac6d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 >> 172.21.15.61:0/2550904266 conn(0x7fa1bc063340 msgr2=0x7fa1bc143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 shutdown_connections 2022-07-12T08:41:27.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.200+0000 7fa1c293f700 1 -- 172.21.15.61:0/2550904266 wait complete. 2022-07-12T08:41:27.212 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:27.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 Processor -- start 2022-07-12T08:41:27.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- start start 2022-07-12T08:41:27.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10fc0b5920 0x7f10fc14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f10fc0aa830 0x7f10fc0aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10fc05c3e0 con 0x7f10fc0b0280 2022-07-12T08:41:27.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc0b5d00 con 0x7f10fc0b5920 2022-07-12T08:41:27.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10fc05c610 con 0x7f10fc0aa830 2022-07-12T08:41:27.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10fc0b5920 0x7f10fc14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10fc0b5920 0x7f10fc14a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53360/0 (socket says 172.21.15.61:53360) 2022-07-12T08:41:27.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 -- 172.21.15.61:0/719696036 learned_addr learned my addr 172.21.15.61:0/719696036 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:27.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fb59e700 1 --2- 172.21.15.61:0/719696036 >> v2:172.21.15.111:3300/0 conn(0x7f10fc0aa830 0x7f10fc0aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 -- 172.21.15.61:0/719696036 >> v2:172.21.15.111:3300/0 conn(0x7f10fc0aa830 msgr2=0x7f10fc0aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:27.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 --2- 172.21.15.61:0/719696036 >> v2:172.21.15.111:3300/0 conn(0x7f10fc0aa830 0x7f10fc0aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 -- 172.21.15.61:0/719696036 >> v1:172.21.15.111:6789/0 conn(0x7f10fc0b0280 legacy=0x7f10fc14aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 -- 172.21.15.61:0/719696036 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc05c8b0 con 0x7f10fc0b5920 2022-07-12T08:41:27.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10fad9d700 1 --2- 172.21.15.61:0/719696036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc14a9e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f10f40032f0 tx=0x7f10f4009290).ready entity=mon.0 client_cookie=aeb1631244fb9995 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10f9d9b700 1 -- 172.21.15.61:0/719696036 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400b870 con 0x7f10fc0b5920 2022-07-12T08:41:27.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10f9d9b700 1 -- 172.21.15.61:0/719696036 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f400b9d0 con 0x7f10fc0b5920 2022-07-12T08:41:27.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f10f9d9b700 1 -- 172.21.15.61:0/719696036 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10f400bb40 con 0x7f10fc0b5920 2022-07-12T08:41:27.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- 172.21.15.61:0/719696036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 msgr2=0x7f10fc14a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 --2- 172.21.15.61:0/719696036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc14a9e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f10f40032f0 tx=0x7f10f4009290).stop 2022-07-12T08:41:27.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- 172.21.15.61:0/719696036 shutdown_connections 2022-07-12T08:41:27.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 --2- 172.21.15.61:0/719696036 >> v2:172.21.15.111:3300/0 conn(0x7f10fc0aa830 0x7f10fc0aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 --2- 172.21.15.61:0/719696036 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc14a9e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- 172.21.15.61:0/719696036 >> 172.21.15.61:0/719696036 conn(0x7f10fc063340 msgr2=0x7f10fc0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- 172.21.15.61:0/719696036 shutdown_connections 2022-07-12T08:41:27.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.328+0000 7f11008b8700 1 -- 172.21.15.61:0/719696036 wait complete. 2022-07-12T08:41:27.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 Processor -- start 2022-07-12T08:41:27.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 -- start start 2022-07-12T08:41:27.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10fc0aa830 0x7f10fc099650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10fc0b0280 0x7f10fc149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc063880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0acbf0 con 0x7f10fc0b5920 2022-07-12T08:41:27.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc148290 con 0x7f10fc0aa830 2022-07-12T08:41:27.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10fc0aedd0 con 0x7f10fc0b0280 2022-07-12T08:41:27.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc063880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc063880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53370/0 (socket says 172.21.15.61:53370) 2022-07-12T08:41:27.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 -- 172.21.15.61:0/2928781870 learned_addr learned my addr 172.21.15.61:0/2928781870 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:27.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fad9d700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10fc0aa830 0x7f10fc099650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fa59c700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10fc0b0280 0x7f10fc149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 -- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10fc0b0280 msgr2=0x7f10fc149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10fc0b0280 0x7f10fc149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 -- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10fc0aa830 msgr2=0x7f10fc099650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10fc0aa830 0x7f10fc099650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 -- 172.21.15.61:0/2928781870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4009530 con 0x7f10fc0b5920 2022-07-12T08:41:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10fb59e700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc063880 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f10ec00a000 tx=0x7f10ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10ec011780 con 0x7f10fc0b5920 2022-07-12T08:41:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ec0118e0 con 0x7f10fc0b5920 2022-07-12T08:41:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10fc0af070 con 0x7f10fc0b5920 2022-07-12T08:41:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10ec011c40 con 0x7f10fc0b5920 2022-07-12T08:41:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.332+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10fc0ac870 con 0x7f10fc0b5920 2022-07-12T08:41:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.336+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10fc0af200 con 0x7f10fc0b5920 2022-07-12T08:41:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.336+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f10ec01c460 con 0x7f10fc0b5920 2022-07-12T08:41:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.336+0000 7f10f3fff700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f10e0053ad0 0x7f10e0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.336+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f10ec06c130 con 0x7f10fc0b5920 2022-07-12T08:41:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.336+0000 7f10fad9d700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f10e0053ad0 0x7f10e0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.336+0000 7f10fad9d700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f10e0053ad0 0x7f10e0055f90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f10f401b000 tx=0x7f10f4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.340+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10ec043030 con 0x7f10fc0b5920 2022-07-12T08:41:27.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10fc06b2b0 con 0x7f10fc0b5920 2022-07-12T08:41:27.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f10f3fff700 1 -- 172.21.15.61:0/2928781870 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f10ec046650 con 0x7f10fc0b5920 2022-07-12T08:41:27.469 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:27.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f10e0053ad0 msgr2=0x7f10e0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f10e0053ad0 0x7f10e0055f90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f10f401b000 tx=0x7f10f4016000).stop 2022-07-12T08:41:27.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 msgr2=0x7f10fc063880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc063880 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f10ec00a000 tx=0x7f10ec008040).stop 2022-07-12T08:41:27.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 shutdown_connections 2022-07-12T08:41:27.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10fc0b5920 0x7f10fc063880 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10fc0b0280 0x7f10fc149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f10e0053ad0 0x7f10e0055f90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 --2- 172.21.15.61:0/2928781870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10fc0aa830 0x7f10fc099650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 >> 172.21.15.61:0/2928781870 conn(0x7f10fc063340 msgr2=0x7f10fc0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 shutdown_connections 2022-07-12T08:41:27.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.468+0000 7f11008b8700 1 -- 172.21.15.61:0/2928781870 wait complete. 2022-07-12T08:41:27.504 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:27.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.632+0000 7fa1781de700 1 Processor -- start 2022-07-12T08:41:27.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.632+0000 7fa1781de700 1 -- start start 2022-07-12T08:41:27.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1700b3860 0x7fa1700b2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa1700b2780 0x7fa170144b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700b2c10 con 0x7fa1700a9cc0 2022-07-12T08:41:27.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1700b1e30 con 0x7fa1700b3860 2022-07-12T08:41:27.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1700b3c40 con 0x7fa1700b2780 2022-07-12T08:41:27.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa175779700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa1700a9cc0 0x7fa170144410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41594/0 (socket says 172.21.15.61:41594) 2022-07-12T08:41:27.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa175779700 1 -- 172.21.15.61:0/2627434618 learned_addr learned my addr 172.21.15.61:0/2627434618 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:27.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa175f7a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1700b3860 0x7fa1700b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa17677b700 1 --2- 172.21.15.61:0/2627434618 >> v2:172.21.15.111:3300/0 conn(0x7fa1700b2780 0x7fa170144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa174f78700 1 -- 172.21.15.61:0/2627434618 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570801903 0 0) 0x7fa16c001110 con 0x7fa1700a9cc0 2022-07-12T08:41:27.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa174f78700 1 -- 172.21.15.61:0/2627434618 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa158003260 con 0x7fa1700a9cc0 2022-07-12T08:41:27.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa17677b700 1 -- 172.21.15.61:0/2627434618 >> v2:172.21.15.61:3300/0 conn(0x7fa1700b3860 msgr2=0x7fa1700b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:27.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa17677b700 1 --2- 172.21.15.61:0/2627434618 >> v2:172.21.15.61:3300/0 conn(0x7fa1700b3860 0x7fa1700b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa17677b700 1 -- 172.21.15.61:0/2627434618 >> v1:172.21.15.61:6789/0 conn(0x7fa1700a9cc0 legacy=0x7fa170144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa17677b700 1 -- 172.21.15.61:0/2627434618 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17005d230 con 0x7fa1700b2780 2022-07-12T08:41:27.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa17677b700 1 --2- 172.21.15.61:0/2627434618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa170144b20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa1680032f0 tx=0x7fa1680091d0).ready entity=mon.1 client_cookie=e9ee33708331321c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa174f78700 1 -- 172.21.15.61:0/2627434618 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16800b600 con 0x7fa1700b2780 2022-07-12T08:41:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa174f78700 1 -- 172.21.15.61:0/2627434618 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16800b760 con 0x7fa1700b2780 2022-07-12T08:41:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa174f78700 1 -- 172.21.15.61:0/2627434618 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16800b8d0 con 0x7fa1700b2780 2022-07-12T08:41:27.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- 172.21.15.61:0/2627434618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 msgr2=0x7fa170144b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 --2- 172.21.15.61:0/2627434618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa170144b20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa1680032f0 tx=0x7fa1680091d0).stop 2022-07-12T08:41:27.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- 172.21.15.61:0/2627434618 shutdown_connections 2022-07-12T08:41:27.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 --2- 172.21.15.61:0/2627434618 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa170144b20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 --2- 172.21.15.61:0/2627434618 >> v2:172.21.15.61:3300/0 conn(0x7fa1700b3860 0x7fa1700b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- 172.21.15.61:0/2627434618 >> 172.21.15.61:0/2627434618 conn(0x7fa1700b6e40 msgr2=0x7fa1700b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- 172.21.15.61:0/2627434618 shutdown_connections 2022-07-12T08:41:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.636+0000 7fa1781de700 1 -- 172.21.15.61:0/2627434618 wait complete. 2022-07-12T08:41:27.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 Processor -- start 2022-07-12T08:41:27.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 -- start start 2022-07-12T08:41:27.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 0x7fa17013f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa1700ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1700b3860 0x7fa1700ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170150480 con 0x7fa1700a9cc0 2022-07-12T08:41:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa170150720 con 0x7fa1700b2780 2022-07-12T08:41:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1701509c0 con 0x7fa1700b3860 2022-07-12T08:41:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 0x7fa17013f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 0x7fa17013f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53384/0 (socket says 172.21.15.61:53384) 2022-07-12T08:41:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa17677b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1700b3860 0x7fa1700ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 -- 172.21.15.61:0/3749923539 learned_addr learned my addr 172.21.15.61:0/3749923539 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175779700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa1700ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 -- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1700b3860 msgr2=0x7fa1700ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1700b3860 0x7fa1700ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 -- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 msgr2=0x7fa1700ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa1700ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 -- 172.21.15.61:0/3749923539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168009470 con 0x7fa1700a9cc0 2022-07-12T08:41:27.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa175f7a700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 0x7fa17013f110 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fa16000c000 tx=0x7fa160006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa160010780 con 0x7fa1700a9cc0 2022-07-12T08:41:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1600108e0 con 0x7fa1700a9cc0 2022-07-12T08:41:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa160010c40 con 0x7fa1700a9cc0 2022-07-12T08:41:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17013f660 con 0x7fa1700a9cc0 2022-07-12T08:41:27.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.640+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa170150050 con 0x7fa1700a9cc0 2022-07-12T08:41:27.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.644+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa16001c460 con 0x7fa1700a9cc0 2022-07-12T08:41:27.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.644+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa170150b80 con 0x7fa1700a9cc0 2022-07-12T08:41:27.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.644+0000 7fa15effd700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1500539b0 0x7fa150055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.644+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa16006c350 con 0x7fa1700a9cc0 2022-07-12T08:41:27.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.644+0000 7fa175779700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1500539b0 0x7fa150055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.644+0000 7fa175779700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1500539b0 0x7fa150055e70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa16c00d000 tx=0x7fa16c0078a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.652+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa160043140 con 0x7fa1700a9cc0 2022-07-12T08:41:27.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa170057460 con 0x7fa1700a9cc0 2022-07-12T08:41:27.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa15effd700 1 -- 172.21.15.61:0/3749923539 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa160046760 con 0x7fa1700a9cc0 2022-07-12T08:41:27.782 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:27.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1500539b0 msgr2=0x7fa150055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1500539b0 0x7fa150055e70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fa16c00d000 tx=0x7fa16c0078a0).stop 2022-07-12T08:41:27.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 msgr2=0x7fa17013f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 0x7fa17013f110 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fa16000c000 tx=0x7fa160006df0).stop 2022-07-12T08:41:27.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 shutdown_connections 2022-07-12T08:41:27.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1700b3860 0x7fa1700ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1500539b0 0x7fa150055e70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1700b2780 0x7fa1700ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 --2- 172.21.15.61:0/3749923539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700a9cc0 0x7fa17013f110 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 >> 172.21.15.61:0/3749923539 conn(0x7fa1700b6e40 msgr2=0x7fa170140420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 shutdown_connections 2022-07-12T08:41:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.780+0000 7fa1781de700 1 -- 172.21.15.61:0/3749923539 wait complete. 2022-07-12T08:41:27.791 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:41:27.791 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:27.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 Processor -- start 2022-07-12T08:41:27.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- start start 2022-07-12T08:41:27.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48280b6be0 0x7f482814aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f482814b3e0 0x7f482805cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482805ff20 con 0x7f482814afa0 2022-07-12T08:41:27.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482814b850 con 0x7f482814b3e0 2022-07-12T08:41:27.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280b1870 con 0x7f48280b6be0 2022-07-12T08:41:27.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f48277fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f482814afa0 0x7f482805c3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41606/0 (socket says 172.21.15.61:41606) 2022-07-12T08:41:27.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f48277fe700 1 -- 172.21.15.61:0/1283730874 learned_addr learned my addr 172.21.15.61:0/1283730874 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:27.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482ccca700 1 --2- 172.21.15.61:0/1283730874 >> v2:172.21.15.61:3300/0 conn(0x7f482814b3e0 0x7f482805cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4827fff700 1 --2- 172.21.15.61:0/1283730874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48280b6be0 0x7f482814aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4826ffd700 1 -- 172.21.15.61:0/1283730874 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003082091 0 0) 0x7f482805ff20 con 0x7f482814afa0 2022-07-12T08:41:27.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4826ffd700 1 -- 172.21.15.61:0/1283730874 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4814003260 con 0x7f482814afa0 2022-07-12T08:41:27.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4827fff700 1 -- 172.21.15.61:0/1283730874 >> v2:172.21.15.61:3300/0 conn(0x7f482814b3e0 msgr2=0x7f482805cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:27.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4827fff700 1 --2- 172.21.15.61:0/1283730874 >> v2:172.21.15.61:3300/0 conn(0x7f482814b3e0 0x7f482805cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4827fff700 1 -- 172.21.15.61:0/1283730874 >> v1:172.21.15.61:6789/0 conn(0x7f482814afa0 legacy=0x7f482805c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4827fff700 1 -- 172.21.15.61:0/1283730874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48280b6fc0 con 0x7f48280b6be0 2022-07-12T08:41:27.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482ccca700 1 --2- 172.21.15.61:0/1283730874 >> v2:172.21.15.61:3300/0 conn(0x7f482814b3e0 0x7f482805cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:27.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4827fff700 1 --2- 172.21.15.61:0/1283730874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48280b6be0 0x7f482814aab0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f481c002f70 tx=0x7f481c006df0).ready entity=mon.2 client_cookie=e57e6db208b188fe server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4826ffd700 1 -- 172.21.15.61:0/1283730874 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481c00d4a0 con 0x7f48280b6be0 2022-07-12T08:41:27.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4826ffd700 1 -- 172.21.15.61:0/1283730874 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c000e90 con 0x7f48280b6be0 2022-07-12T08:41:27.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f4826ffd700 1 -- 172.21.15.61:0/1283730874 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481c00d7f0 con 0x7f48280b6be0 2022-07-12T08:41:27.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- 172.21.15.61:0/1283730874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48280b6be0 msgr2=0x7f482814aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 --2- 172.21.15.61:0/1283730874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48280b6be0 0x7f482814aab0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f481c002f70 tx=0x7f481c006df0).stop 2022-07-12T08:41:27.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- 172.21.15.61:0/1283730874 shutdown_connections 2022-07-12T08:41:27.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 --2- 172.21.15.61:0/1283730874 >> v2:172.21.15.61:3300/0 conn(0x7f482814b3e0 0x7f482805cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 --2- 172.21.15.61:0/1283730874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48280b6be0 0x7f482814aab0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- 172.21.15.61:0/1283730874 >> 172.21.15.61:0/1283730874 conn(0x7f48280645c0 msgr2=0x7f48280b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:27.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- 172.21.15.61:0/1283730874 shutdown_connections 2022-07-12T08:41:27.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.904+0000 7f482e72d700 1 -- 172.21.15.61:0/1283730874 wait complete. 2022-07-12T08:41:27.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 Processor -- start 2022-07-12T08:41:27.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 -- start start 2022-07-12T08:41:27.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 0x7f48280b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f482814afa0 0x7f4828159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482814b3e0 0x7f482815a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48280ae860 con 0x7f48280b6be0 2022-07-12T08:41:27.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48280aeb00 con 0x7f482814b3e0 2022-07-12T08:41:27.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48280ad870 con 0x7f482814afa0 2022-07-12T08:41:27.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 0x7f48280b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 0x7f48280b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53394/0 (socket says 172.21.15.61:53394) 2022-07-12T08:41:27.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 -- 172.21.15.61:0/176108903 learned_addr learned my addr 172.21.15.61:0/176108903 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:27.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f48277fe700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f482814afa0 0x7f4828159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482ccca700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482814b3e0 0x7f482815a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 -- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f482814afa0 msgr2=0x7f4828159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f482814afa0 0x7f4828159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 -- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482814b3e0 msgr2=0x7f482815a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:27.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482814b3e0 0x7f482815a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:27.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 -- 172.21.15.61:0/176108903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481800b180 con 0x7f48280b6be0 2022-07-12T08:41:27.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482ccca700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482814b3e0 0x7f482815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:27.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f4827fff700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 0x7f48280b03f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f481c002f70 tx=0x7f481c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481c0139c0 con 0x7f48280b6be0 2022-07-12T08:41:27.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c013b20 con 0x7f48280b6be0 2022-07-12T08:41:27.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c00b180 con 0x7f48280b6be0 2022-07-12T08:41:27.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f481c013e80 con 0x7f48280b6be0 2022-07-12T08:41:27.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.908+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48280ae480 con 0x7f48280b6be0 2022-07-12T08:41:27.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.912+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48280ae6c0 con 0x7f48280b6be0 2022-07-12T08:41:27.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.912+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f481c01c070 con 0x7f48280b6be0 2022-07-12T08:41:27.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.912+0000 7f48257fa700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4808053a60 0x7f4808055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:27.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.912+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f481c06c8c0 con 0x7f48280b6be0 2022-07-12T08:41:27.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.912+0000 7f48277fe700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4808053a60 0x7f4808055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:27.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.916+0000 7f48277fe700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4808053a60 0x7f4808055f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f482000d000 tx=0x7f4820009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:27.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:27.916+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f48280ae6c0 con 0x7f48280b6be0 2022-07-12T08:41:28.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.080+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48280ae6c0 con 0x7f48280b6be0 2022-07-12T08:41:28.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f48257fa700 1 -- 172.21.15.61:0/176108903 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+2137 (secure 0 0 0) 0x7f48280ae6c0 con 0x7f48280b6be0 2022-07-12T08:41:28.085 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:28.085 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:15.584363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.111:6835/1570404457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1570404457},{"type":"v1","addr":"172.21.15.111:6835","nonce":1570404457}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:28.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4808053a60 msgr2=0x7f4808055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:28.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4808053a60 0x7f4808055f20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f482000d000 tx=0x7f4820009040).stop 2022-07-12T08:41:28.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 msgr2=0x7f48280b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:28.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 0x7f48280b03f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f481c002f70 tx=0x7f481c014000).stop 2022-07-12T08:41:28.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 shutdown_connections 2022-07-12T08:41:28.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4808053a60 0x7f4808055f20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:28.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f482814b3e0 0x7f482815a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:28.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f482814afa0 0x7f4828159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:28.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 --2- 172.21.15.61:0/176108903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48280b6be0 0x7f48280b03f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:28.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 >> 172.21.15.61:0/176108903 conn(0x7f48280645c0 msgr2=0x7f48280b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:28.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 shutdown_connections 2022-07-12T08:41:28.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:28.084+0000 7f482e72d700 1 -- 172.21.15.61:0/176108903 wait complete. 2022-07-12T08:41:28.131 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2022-07-12T08:41:28.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:41:14.580428+0000', 'modified': '2022-07-12T08:41:15.584363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.111:6835/1570404457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 1570404457}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 1570404457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:41:28.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:41:28.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:41:28.460 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":4282,"want_state":"up:active","state":"up:active","rank_uptime":13.868707019,"mdsmap_epoch":5,"osdmap_epoch":22,"osdmap_epoch_barrier":21,"uptime":17.207622303000001} 2022-07-12T08:41:28.461 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 4282, "mdsmap_epoch": 5, "osdmap_epoch": 22, "osdmap_epoch_barrier": 21, "rank_uptime": 13.868707019, "state": "up:active", "uptime": 17.207622303, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:41:28.461 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:41:28.475 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:41:28.476 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:41:28.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:28.477 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:28.528 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:41:28.599 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:28.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:28.620 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:28.621 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:28.622 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:41:28.680 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:41:28.681 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:28.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:41:28.711 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:28.711 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:28.768 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:28.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:28.771 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:41:28.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[20990]: starting ceph client 2022-07-12T08:41:28.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:41:28.774+0000 7fad9862f080 -1 init, newargv = 0x5589baa0f530 newargc=9 2022-07-12T08:41:28.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[20990]: starting fuse 2022-07-12T08:41:29.898 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:41:29.929 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:29.930 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:29.987 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:29.988 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:29.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:29.990 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:41:29.998 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:41:29.999 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:41:30.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:30.001 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:41:30.001 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:41:30.002 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:41:30.003 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:41:30.004 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:41:30.005 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:41:30.005 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:41:30.022 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:41:30.023 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:41:30.023 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:41:30.024 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:41:30.024 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:41:30.025 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:41:30.025 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:41:30.026 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:41:30.027 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:41:30.028 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:41:30.028 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:41:30.040 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:41:30.040 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:41:30.041 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:41:30.042 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:41:30.042 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:41:30.101 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.20990.asok 2022-07-12T08:41:30.104 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.20990.asok 2022-07-12T08:41:30.105 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:30.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.20990.asok status 2022-07-12T08:41:30.251 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:41:30.252 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:41:30.252 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:41:30.252 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:41:30.253 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:41:30.253 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:41:30.254 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:41:30.254 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "20990", 2022-07-12T08:41:30.254 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:41:30.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:41:30.255 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:41:30.255 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:41:30.258 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4487, 2022-07-12T08:41:30.259 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:41:30.259 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:41:30.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:41:30.260 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4487 2022-07-12T08:41:30.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:41:30.262 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:41:30.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:41:30.263 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:41:30.264 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 760255951 2022-07-12T08:41:30.264 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:41:30.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:41:30.265 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:41:30.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:41:30.266 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:41:30.267 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 760255951 2022-07-12T08:41:30.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:41:30.268 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4487 172.21.15.121:0/760255951", 2022-07-12T08:41:30.269 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/760255951", 2022-07-12T08:41:30.269 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:41:30.269 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 5, 2022-07-12T08:41:30.270 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 22, 2022-07-12T08:41:30.270 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:41:30.270 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:41:30.271 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:41:30.271 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:41:30.274 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:30.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 Processor -- start 2022-07-12T08:41:30.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 -- start start 2022-07-12T08:41:30.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f118c0ab670 0x7f118c05f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f118c0afa60 0x7f118c0afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c0abc70 con 0x7f118c0ab290 2022-07-12T08:41:30.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c0aba90 con 0x7f118c0afa60 2022-07-12T08:41:30.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.392+0000 7f1191868700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f118c05c720 con 0x7f118c0ab670 2022-07-12T08:41:30.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191067700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f118c0afa60 0x7f118c0afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1190866700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f118c0ab290 0x7f118c05ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41620/0 (socket says 172.21.15.61:41620) 2022-07-12T08:41:30.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1190866700 1 -- 172.21.15.61:0/1008365669 learned_addr learned my addr 172.21.15.61:0/1008365669 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:30.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118bfff700 1 --2- 172.21.15.61:0/1008365669 >> v2:172.21.15.111:3300/0 conn(0x7f118c0ab670 0x7f118c05f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118bfff700 1 -- 172.21.15.61:0/1008365669 >> v2:172.21.15.61:3300/0 conn(0x7f118c0afa60 msgr2=0x7f118c0afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:30.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118bfff700 1 --2- 172.21.15.61:0/1008365669 >> v2:172.21.15.61:3300/0 conn(0x7f118c0afa60 0x7f118c0afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118bfff700 1 -- 172.21.15.61:0/1008365669 >> v1:172.21.15.61:6789/0 conn(0x7f118c0ab290 legacy=0x7f118c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118b7fe700 1 -- 172.21.15.61:0/1008365669 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136376646 0 0) 0x7f118c0abc70 con 0x7f118c0ab290 2022-07-12T08:41:30.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118bfff700 1 -- 172.21.15.61:0/1008365669 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c05c9c0 con 0x7f118c0ab670 2022-07-12T08:41:30.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191067700 1 --2- 172.21.15.61:0/1008365669 >> v2:172.21.15.61:3300/0 conn(0x7f118c0afa60 0x7f118c0afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:30.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118bfff700 1 --2- 172.21.15.61:0/1008365669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0ab670 0x7f118c05f140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f117c0032f0 tx=0x7f117c0091d0).ready entity=mon.1 client_cookie=bdb75cbaab774ed9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118b7fe700 1 -- 172.21.15.61:0/1008365669 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f117c00b680 con 0x7f118c0ab670 2022-07-12T08:41:30.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118b7fe700 1 -- 172.21.15.61:0/1008365669 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c00b7e0 con 0x7f118c0ab670 2022-07-12T08:41:30.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f118b7fe700 1 -- 172.21.15.61:0/1008365669 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f117c00b950 con 0x7f118c0ab670 2022-07-12T08:41:30.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 -- 172.21.15.61:0/1008365669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0ab670 msgr2=0x7f118c05f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 --2- 172.21.15.61:0/1008365669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0ab670 0x7f118c05f140 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f117c0032f0 tx=0x7f117c0091d0).stop 2022-07-12T08:41:30.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 -- 172.21.15.61:0/1008365669 shutdown_connections 2022-07-12T08:41:30.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 --2- 172.21.15.61:0/1008365669 >> v2:172.21.15.61:3300/0 conn(0x7f118c0afa60 0x7f118c0afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 --2- 172.21.15.61:0/1008365669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0ab670 0x7f118c05f140 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 -- 172.21.15.61:0/1008365669 >> 172.21.15.61:0/1008365669 conn(0x7f118c0b1950 msgr2=0x7f118c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:30.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 -- 172.21.15.61:0/1008365669 shutdown_connections 2022-07-12T08:41:30.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.396+0000 7f1191868700 1 -- 172.21.15.61:0/1008365669 wait complete. 2022-07-12T08:41:30.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 Processor -- start 2022-07-12T08:41:30.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 -- start start 2022-07-12T08:41:30.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 0x7f118c149ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f118c0ab670 0x7f118c148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0afa60 0x7f118c0ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c064ba0 con 0x7f118c0ab290 2022-07-12T08:41:30.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0ad0d0 con 0x7f118c0afa60 2022-07-12T08:41:30.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1191868700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c14a090 con 0x7f118c0ab670 2022-07-12T08:41:30.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1190866700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 0x7f118c149ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1190866700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 0x7f118c149ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53408/0 (socket says 172.21.15.61:53408) 2022-07-12T08:41:30.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.400+0000 7f1190866700 1 -- 172.21.15.61:0/3611043781 learned_addr learned my addr 172.21.15.61:0/3611043781 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:30.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f118bfff700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f118c0ab670 0x7f118c148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1191067700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0afa60 0x7f118c0ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1190866700 1 -- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f118c0ab670 msgr2=0x7f118c148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1190866700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f118c0ab670 0x7f118c148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1190866700 1 -- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0afa60 msgr2=0x7f118c0ad570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1190866700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0afa60 0x7f118c0ad570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1190866700 1 -- 172.21.15.61:0/3611043781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118400b180 con 0x7f118c0ab290 2022-07-12T08:41:30.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f118bfff700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f118c0ab670 0x7f118c148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:30.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1191067700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0afa60 0x7f118c0ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:30.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1190866700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 0x7f118c149ba0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1174000bc0 tx=0x7f1174007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11740072e0 con 0x7f118c0ab290 2022-07-12T08:41:30.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174007440 con 0x7f118c0ab290 2022-07-12T08:41:30.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c009470 con 0x7f118c0ab290 2022-07-12T08:41:30.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1174005800 con 0x7f118c0ab290 2022-07-12T08:41:30.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c149020 con 0x7f118c0ab290 2022-07-12T08:41:30.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c0ad290 con 0x7f118c0ab290 2022-07-12T08:41:30.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.404+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f117401a060 con 0x7f118c0ab290 2022-07-12T08:41:30.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.408+0000 7f1189ffb700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1170053ad0 0x7f1170055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.408+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f117406b770 con 0x7f118c0ab290 2022-07-12T08:41:30.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.408+0000 7f118bfff700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1170053ad0 0x7f1170055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.408+0000 7f118bfff700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1170053ad0 0x7f1170055f90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f117c00b000 tx=0x7f117c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.416+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11740426d0 con 0x7f118c0ab290 2022-07-12T08:41:30.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.540+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f118c057460 con 0x7f118c0ab290 2022-07-12T08:41:30.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.540+0000 7f1189ffb700 1 -- 172.21.15.61:0/3611043781 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f1174018020 con 0x7f118c0ab290 2022-07-12T08:41:30.545 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:30.546 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1170053ad0 msgr2=0x7f1170055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1170053ad0 0x7f1170055f90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f117c00b000 tx=0x7f117c016000).stop 2022-07-12T08:41:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 msgr2=0x7f118c149ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 0x7f118c149ba0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1174000bc0 tx=0x7f1174007af0).stop 2022-07-12T08:41:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 shutdown_connections 2022-07-12T08:41:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1170053ad0 0x7f1170055f90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f118c0afa60 0x7f118c0ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f118c0ab670 0x7f118c148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 --2- 172.21.15.61:0/3611043781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118c0ab290 0x7f118c149ba0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 >> 172.21.15.61:0/3611043781 conn(0x7f118c0b1950 msgr2=0x7f118c05bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 shutdown_connections 2022-07-12T08:41:30.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.544+0000 7f1191868700 1 -- 172.21.15.61:0/3611043781 wait complete. 2022-07-12T08:41:30.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:30.553 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:30.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 Processor -- start 2022-07-12T08:41:30.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 -- start start 2022-07-12T08:41:30.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b4805d060 0x7f2b480b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2b48142ab0 0x7f2b480afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b481433c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b48143d80 con 0x7f2b4805d060 2022-07-12T08:41:30.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4805d440 con 0x7f2b48142ab0 2022-07-12T08:41:30.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b4c9f0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480674b0 con 0x7f2b48142f10 2022-07-12T08:41:30.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b4805d060 0x7f2b480b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b4805d060 0x7f2b480b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53416/0 (socket says 172.21.15.61:53416) 2022-07-12T08:41:30.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 -- 172.21.15.61:0/1424881542 learned_addr learned my addr 172.21.15.61:0/1424881542 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:30.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b46ffd700 1 --2- 172.21.15.61:0/1424881542 >> v2:172.21.15.111:3300/0 conn(0x7f2b48142ab0 0x7f2b480afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b47fff700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b481433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 -- 172.21.15.61:0/1424881542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 msgr2=0x7f2b481433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:30.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b481433c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 -- 172.21.15.61:0/1424881542 >> v2:172.21.15.111:3300/0 conn(0x7f2b48142ab0 msgr2=0x7f2b480afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:30.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 --2- 172.21.15.61:0/1424881542 >> v2:172.21.15.111:3300/0 conn(0x7f2b48142ab0 0x7f2b480afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 -- 172.21.15.61:0/1424881542 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48067750 con 0x7f2b4805d060 2022-07-12T08:41:30.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b47fff700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b481433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:30.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.667+0000 7f2b477fe700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b0130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2b3c002f70 tx=0x7f2b3c006eb0).ready entity=mon.0 client_cookie=892b1951346c1ca8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b467fc700 1 -- 172.21.15.61:0/1424881542 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3c00d570 con 0x7f2b4805d060 2022-07-12T08:41:30.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b467fc700 1 -- 172.21.15.61:0/1424881542 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c000e90 con 0x7f2b4805d060 2022-07-12T08:41:30.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b467fc700 1 -- 172.21.15.61:0/1424881542 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3c00d900 con 0x7f2b4805d060 2022-07-12T08:41:30.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/1424881542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 msgr2=0x7f2b480b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b0130 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2b3c002f70 tx=0x7f2b3c006eb0).stop 2022-07-12T08:41:30.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/1424881542 shutdown_connections 2022-07-12T08:41:30.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b481433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/1424881542 >> v2:172.21.15.111:3300/0 conn(0x7f2b48142ab0 0x7f2b480afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/1424881542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b0130 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/1424881542 >> 172.21.15.61:0/1424881542 conn(0x7f2b480b6900 msgr2=0x7f2b48066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:30.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/1424881542 shutdown_connections 2022-07-12T08:41:30.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/1424881542 wait complete. 2022-07-12T08:41:30.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 Processor -- start 2022-07-12T08:41:30.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- start start 2022-07-12T08:41:30.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b480b1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b48147d70 0x7f2b48151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4809c650 con 0x7f2b4805d060 2022-07-12T08:41:30.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4809c8f0 con 0x7f2b48147d70 2022-07-12T08:41:30.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b4c9f0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4809cb90 con 0x7f2b48142f10 2022-07-12T08:41:30.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53422/0 (socket says 172.21.15.61:53422) 2022-07-12T08:41:30.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 -- 172.21.15.61:0/2884990779 learned_addr learned my addr 172.21.15.61:0/2884990779 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:30.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b46ffd700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b480b1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b47fff700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b48147d70 0x7f2b48151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 -- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 msgr2=0x7f2b480b1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b480b1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 -- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b48147d70 msgr2=0x7f2b48151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b48147d70 0x7f2b48151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 -- 172.21.15.61:0/2884990779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c00b1e0 con 0x7f2b4805d060 2022-07-12T08:41:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.671+0000 7f2b477fe700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b4cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2b3c002f70 tx=0x7f2b3c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3c006be0 con 0x7f2b4805d060 2022-07-12T08:41:30.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c011c30 con 0x7f2b4805d060 2022-07-12T08:41:30.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b48148220 con 0x7f2b4805d060 2022-07-12T08:41:30.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b3c01e460 con 0x7f2b4805d060 2022-07-12T08:41:30.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b48152390 con 0x7f2b4805d060 2022-07-12T08:41:30.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b48057460 con 0x7f2b4805d060 2022-07-12T08:41:30.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b3c025030 con 0x7f2b4805d060 2022-07-12T08:41:30.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b44ff9700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b280539d0 0x7f2b28055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2b3c06cf80 con 0x7f2b4805d060 2022-07-12T08:41:30.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b46ffd700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b280539d0 0x7f2b28055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.675+0000 7f2b46ffd700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b280539d0 0x7f2b28055e90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f2b3800b000 tx=0x7f2b38006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.683+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b3c043ee0 con 0x7f2b4805d060 2022-07-12T08:41:30.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2b481528c0 con 0x7f2b4805d060 2022-07-12T08:41:30.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b44ff9700 1 -- 172.21.15.61:0/2884990779 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2b3c023070 con 0x7f2b4805d060 2022-07-12T08:41:30.810 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:30.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b280539d0 msgr2=0x7f2b28055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b280539d0 0x7f2b28055e90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f2b3800b000 tx=0x7f2b38006df0).stop 2022-07-12T08:41:30.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 msgr2=0x7f2b480b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b4cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f2b3c002f70 tx=0x7f2b3c015000).stop 2022-07-12T08:41:30.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 shutdown_connections 2022-07-12T08:41:30.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b280539d0 0x7f2b28055e90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b48147d70 0x7f2b48151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b48142f10 0x7f2b480b1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 --2- 172.21.15.61:0/2884990779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b4805d060 0x7f2b480b4cd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 >> 172.21.15.61:0/2884990779 conn(0x7f2b480b6900 msgr2=0x7f2b4805f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:30.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 shutdown_connections 2022-07-12T08:41:30.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.811+0000 7f2b4c9f0700 1 -- 172.21.15.61:0/2884990779 wait complete. 2022-07-12T08:41:30.838 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:30.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 Processor -- start 2022-07-12T08:41:30.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 -- start start 2022-07-12T08:41:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09100ad7f0 0x7f0910067b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0910142e60 0x7f09100af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f091005c590 con 0x7f09100b1530 2022-07-12T08:41:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f091013f750 con 0x7f09100ad7f0 2022-07-12T08:41:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f0914bd7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09100b1910 con 0x7f0910142e60 2022-07-12T08:41:30.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f090f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f09100b1530 0x7f0910067430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41648/0 (socket says 172.21.15.61:41648) 2022-07-12T08:41:30.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.971+0000 7f090f7fe700 1 -- 172.21.15.61:0/3843653079 learned_addr learned my addr 172.21.15.61:0/3843653079 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:30.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090ffff700 1 --2- 172.21.15.61:0/3843653079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0910142e60 0x7f09100af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090effd700 1 --2- 172.21.15.61:0/3843653079 >> v2:172.21.15.111:3300/0 conn(0x7f09100ad7f0 0x7f0910067b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090e7fc700 1 -- 172.21.15.61:0/3843653079 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411401564 0 0) 0x7f091005c590 con 0x7f09100b1530 2022-07-12T08:41:30.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090e7fc700 1 -- 172.21.15.61:0/3843653079 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08fc003260 con 0x7f09100b1530 2022-07-12T08:41:30.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090ffff700 1 -- 172.21.15.61:0/3843653079 >> v2:172.21.15.111:3300/0 conn(0x7f09100ad7f0 msgr2=0x7f0910067b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:30.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090ffff700 1 --2- 172.21.15.61:0/3843653079 >> v2:172.21.15.111:3300/0 conn(0x7f09100ad7f0 0x7f0910067b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090ffff700 1 -- 172.21.15.61:0/3843653079 >> v1:172.21.15.61:6789/0 conn(0x7f09100b1530 legacy=0x7f0910067430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090ffff700 1 -- 172.21.15.61:0/3843653079 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091005d370 con 0x7f0910142e60 2022-07-12T08:41:30.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090ffff700 1 --2- 172.21.15.61:0/3843653079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0910142e60 0x7f09100af1d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f09080032f0 tx=0x7f09080091d0).ready entity=mon.2 client_cookie=4a07597528e80d5a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:30.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090e7fc700 1 -- 172.21.15.61:0/3843653079 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f090800b600 con 0x7f0910142e60 2022-07-12T08:41:30.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090e7fc700 1 -- 172.21.15.61:0/3843653079 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f090800b760 con 0x7f0910142e60 2022-07-12T08:41:30.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f090e7fc700 1 -- 172.21.15.61:0/3843653079 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f090800b8d0 con 0x7f0910142e60 2022-07-12T08:41:30.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 -- 172.21.15.61:0/3843653079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0910142e60 msgr2=0x7f09100af1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:30.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 --2- 172.21.15.61:0/3843653079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0910142e60 0x7f09100af1d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f09080032f0 tx=0x7f09080091d0).stop 2022-07-12T08:41:30.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 -- 172.21.15.61:0/3843653079 shutdown_connections 2022-07-12T08:41:30.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 --2- 172.21.15.61:0/3843653079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0910142e60 0x7f09100af1d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 --2- 172.21.15.61:0/3843653079 >> v2:172.21.15.111:3300/0 conn(0x7f09100ad7f0 0x7f0910067b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:30.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 -- 172.21.15.61:0/3843653079 >> 172.21.15.61:0/3843653079 conn(0x7f0910063c50 msgr2=0x7f091005f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:30.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 -- 172.21.15.61:0/3843653079 shutdown_connections 2022-07-12T08:41:30.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 -- 172.21.15.61:0/3843653079 wait complete. 2022-07-12T08:41:30.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 Processor -- start 2022-07-12T08:41:30.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.975+0000 7f0914bd7700 1 -- start start 2022-07-12T08:41:30.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09100ad7f0 0x7f09100b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09100b1530 0x7f0910145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 0x7f091013e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:30.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910151c00 con 0x7f0910142e60 2022-07-12T08:41:30.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09100ab460 con 0x7f09100ad7f0 2022-07-12T08:41:30.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09100ab700 con 0x7f09100b1530 2022-07-12T08:41:30.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 0x7f091013e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 0x7f091013e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53440/0 (socket says 172.21.15.61:53440) 2022-07-12T08:41:30.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09100ad7f0 0x7f09100b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:30.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 -- 172.21.15.61:0/4073584370 learned_addr learned my addr 172.21.15.61:0/4073584370 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090effd700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09100b1530 0x7f0910145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 -- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09100b1530 msgr2=0x7f0910145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09100b1530 0x7f0910145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 -- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09100ad7f0 msgr2=0x7f09100b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09100ad7f0 0x7f09100b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 -- 172.21.15.61:0/4073584370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0908009470 con 0x7f0910142e60 2022-07-12T08:41:31.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090ffff700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 0x7f091013e900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f09080032f0 tx=0x7f0908016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f090800b600 con 0x7f0910142e60 2022-07-12T08:41:31.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908011c10 con 0x7f0910142e60 2022-07-12T08:41:31.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09100ab9a0 con 0x7f0910142e60 2022-07-12T08:41:31.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f090801f3f0 con 0x7f0910142e60 2022-07-12T08:41:31.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09100b6450 con 0x7f0910142e60 2022-07-12T08:41:31.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0908026030 con 0x7f0910142e60 2022-07-12T08:41:31.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.979+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09100b3f90 con 0x7f0910142e60 2022-07-12T08:41:31.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.983+0000 7f090cff9700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08f0053af0 0x7f08f0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.983+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f090806c830 con 0x7f0910142e60 2022-07-12T08:41:31.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.983+0000 7f090f7fe700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08f0053af0 0x7f08f0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.983+0000 7f090f7fe700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08f0053af0 0x7f08f0055fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f090000d000 tx=0x7f0900009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:30.987+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0908043790 con 0x7f0910142e60 2022-07-12T08:41:31.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.115+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0910057460 con 0x7f0910142e60 2022-07-12T08:41:31.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.115+0000 7f090cff9700 1 -- 172.21.15.61:0/4073584370 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f09080150a0 con 0x7f0910142e60 2022-07-12T08:41:31.116 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:31.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08f0053af0 msgr2=0x7f08f0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08f0053af0 0x7f08f0055fb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f090000d000 tx=0x7f0900009040).stop 2022-07-12T08:41:31.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 msgr2=0x7f091013e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 0x7f091013e900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f09080032f0 tx=0x7f0908016000).stop 2022-07-12T08:41:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 shutdown_connections 2022-07-12T08:41:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0910142e60 0x7f091013e900 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09100b1530 0x7f0910145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08f0053af0 0x7f08f0055fb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 --2- 172.21.15.61:0/4073584370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09100ad7f0 0x7f09100b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 >> 172.21.15.61:0/4073584370 conn(0x7f0910063c50 msgr2=0x7f09101423f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 shutdown_connections 2022-07-12T08:41:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.119+0000 7f0914bd7700 1 -- 172.21.15.61:0/4073584370 wait complete. 2022-07-12T08:41:31.156 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:31.242 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:31.257 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:31.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:31.266 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:31.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:41:31.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 Processor -- start 2022-07-12T08:41:31.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 -- start start 2022-07-12T08:41:31.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c065490 0x7f969c0b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f969c0b6550 0x7f969c13fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c05f1e0 con 0x7f969c0b0df0 2022-07-12T08:41:31.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c146850 con 0x7f969c0b6550 2022-07-12T08:41:31.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0b11d0 con 0x7f969c065490 2022-07-12T08:41:31.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f969c0b6550 0x7f969c13fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f969c0b6550 0x7f969c13fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53448/0 (socket says 172.21.15.61:53448) 2022-07-12T08:41:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 -- 172.21.15.61:0/3650075954 learned_addr learned my addr 172.21.15.61:0/3650075954 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969a7fc700 1 --2- 172.21.15.61:0/3650075954 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c065490 0x7f969c0b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 -- 172.21.15.61:0/3650075954 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c065490 msgr2=0x7f969c0b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 --2- 172.21.15.61:0/3650075954 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c065490 0x7f969c0b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 -- 172.21.15.61:0/3650075954 >> v1:172.21.15.111:6789/0 conn(0x7f969c0b0df0 legacy=0x7f969c146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.275+0000 7f969b7fe700 1 -- 172.21.15.61:0/3650075954 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c060170 con 0x7f969c0b6550 2022-07-12T08:41:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969b7fe700 1 --2- 172.21.15.61:0/3650075954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b6550 0x7f969c13fcb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f96940032f0 tx=0x7f9694009290).ready entity=mon.0 client_cookie=8c88f2a5b3c33009 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f9699ffb700 1 -- 172.21.15.61:0/3650075954 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f969400b8a0 con 0x7f969c0b6550 2022-07-12T08:41:31.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f9699ffb700 1 -- 172.21.15.61:0/3650075954 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969400ba00 con 0x7f969c0b6550 2022-07-12T08:41:31.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f9699ffb700 1 -- 172.21.15.61:0/3650075954 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f969400bb70 con 0x7f969c0b6550 2022-07-12T08:41:31.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 -- 172.21.15.61:0/3650075954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b6550 msgr2=0x7f969c13fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 --2- 172.21.15.61:0/3650075954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b6550 0x7f969c13fcb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f96940032f0 tx=0x7f9694009290).stop 2022-07-12T08:41:31.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 -- 172.21.15.61:0/3650075954 shutdown_connections 2022-07-12T08:41:31.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 --2- 172.21.15.61:0/3650075954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b6550 0x7f969c13fcb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 --2- 172.21.15.61:0/3650075954 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c065490 0x7f969c0b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 -- 172.21.15.61:0/3650075954 >> 172.21.15.61:0/3650075954 conn(0x7f969c063ad0 msgr2=0x7f969c063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:31.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 -- 172.21.15.61:0/3650075954 shutdown_connections 2022-07-12T08:41:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 -- 172.21.15.61:0/3650075954 wait complete. 2022-07-12T08:41:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 Processor -- start 2022-07-12T08:41:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.279+0000 7f969bfff700 1 -- start start 2022-07-12T08:41:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 0x7f969c145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c0b6550 0x7f969c0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c142640 0x7f969c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0ae860 con 0x7f969c0b0df0 2022-07-12T08:41:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c146400 con 0x7f969c142640 2022-07-12T08:41:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c144310 con 0x7f969c0b6550 2022-07-12T08:41:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 0x7f969c145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 0x7f969c145140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53450/0 (socket says 172.21.15.61:53450) 2022-07-12T08:41:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 -- 172.21.15.61:0/507003706 learned_addr learned my addr 172.21.15.61:0/507003706 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969a7fc700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c0b6550 0x7f969c0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969b7fe700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c142640 0x7f969c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 -- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c0b6550 msgr2=0x7f969c0ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c0b6550 0x7f969c0ad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 -- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c142640 msgr2=0x7f969c151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c142640 0x7f969c151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 -- 172.21.15.61:0/507003706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9694009530 con 0x7f969c0b0df0 2022-07-12T08:41:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969affd700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 0x7f969c145140 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f969c05d670 tx=0x7f968c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f968c00ce30 con 0x7f969c0b0df0 2022-07-12T08:41:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c00f470 con 0x7f969c0b0df0 2022-07-12T08:41:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0ae160 con 0x7f969c0b0df0 2022-07-12T08:41:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f968c00f7d0 con 0x7f969c0b0df0 2022-07-12T08:41:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c0b0300 con 0x7f969c0b0df0 2022-07-12T08:41:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c0adfa0 con 0x7f969c0b0df0 2022-07-12T08:41:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.283+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f968c019030 con 0x7f969c0b0df0 2022-07-12T08:41:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.287+0000 7f968bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9678053ad0 0x7f9678055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.287+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f968c06b380 con 0x7f969c0b0df0 2022-07-12T08:41:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.287+0000 7f969a7fc700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9678053ad0 0x7f9678055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.287+0000 7f969a7fc700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9678053ad0 0x7f9678055f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f969000c000 tx=0x7f9690008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.291+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f968c0422e0 con 0x7f969c0b0df0 2022-07-12T08:41:31.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.419+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f969c057460 con 0x7f969c0b0df0 2022-07-12T08:41:31.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.419+0000 7f968bfff700 1 -- 172.21.15.61:0/507003706 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f968c045900 con 0x7f969c0b0df0 2022-07-12T08:41:31.426 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:31.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9678053ad0 msgr2=0x7f9678055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9678053ad0 0x7f9678055f90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f969000c000 tx=0x7f9690008040).stop 2022-07-12T08:41:31.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 msgr2=0x7f969c145140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 0x7f969c145140 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f969c05d670 tx=0x7f968c007c50).stop 2022-07-12T08:41:31.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 shutdown_connections 2022-07-12T08:41:31.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9678053ad0 0x7f9678055f90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f969c142640 0x7f969c151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f969c0b6550 0x7f969c0ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 --2- 172.21.15.61:0/507003706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f969c0b0df0 0x7f969c145140 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 >> 172.21.15.61:0/507003706 conn(0x7f969c063ad0 msgr2=0x7f969c066380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 shutdown_connections 2022-07-12T08:41:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.423+0000 7f969bfff700 1 -- 172.21.15.61:0/507003706 wait complete. 2022-07-12T08:41:31.444 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:31.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 Processor -- start 2022-07-12T08:41:31.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 -- start start 2022-07-12T08:41:31.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdffc0b0120 0x7fdffc144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdffc0b0500 con 0x7fdffc0ab730 2022-07-12T08:41:31.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdffc0b1040 con 0x7fdffc0b0120 2022-07-12T08:41:31.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fe001d69700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc144d50 con 0x7fdffc14aad0 2022-07-12T08:41:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffb7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdffc0b0120 0x7fdffc144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:35656/0 (socket says 172.21.15.61:35656) 2022-07-12T08:41:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 -- 172.21.15.61:0/4042661896 learned_addr learned my addr 172.21.15.61:0/4042661896 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffa7fc700 1 -- 172.21.15.61:0/4042661896 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025494905 0 0) 0x7fdffc0b0500 con 0x7fdffc0ab730 2022-07-12T08:41:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffa7fc700 1 -- 172.21.15.61:0/4042661896 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe8003260 con 0x7fdffc0ab730 2022-07-12T08:41:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 -- 172.21.15.61:0/4042661896 >> v2:172.21.15.111:3300/0 conn(0x7fdffc0b0120 msgr2=0x7fdffc144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 --2- 172.21.15.61:0/4042661896 >> v2:172.21.15.111:3300/0 conn(0x7fdffc0b0120 0x7fdffc144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 -- 172.21.15.61:0/4042661896 >> v1:172.21.15.111:6789/0 conn(0x7fdffc0ab730 legacy=0x7fdffc144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.567+0000 7fdffbfff700 1 -- 172.21.15.61:0/4042661896 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc05d0b0 con 0x7fdffc14aad0 2022-07-12T08:41:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffbfff700 1 --2- 172.21.15.61:0/4042661896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc14af10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fdff4002f70 tx=0x7fdff4006eb0).ready entity=mon.2 client_cookie=7e426e2f7d97f1fc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffa7fc700 1 -- 172.21.15.61:0/4042661896 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdff400d5c0 con 0x7fdffc14aad0 2022-07-12T08:41:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffa7fc700 1 -- 172.21.15.61:0/4042661896 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff4000e90 con 0x7fdffc14aad0 2022-07-12T08:41:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffa7fc700 1 -- 172.21.15.61:0/4042661896 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdff400d950 con 0x7fdffc14aad0 2022-07-12T08:41:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- 172.21.15.61:0/4042661896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 msgr2=0x7fdffc14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 --2- 172.21.15.61:0/4042661896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc14af10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fdff4002f70 tx=0x7fdff4006eb0).stop 2022-07-12T08:41:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- 172.21.15.61:0/4042661896 shutdown_connections 2022-07-12T08:41:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 --2- 172.21.15.61:0/4042661896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc14af10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 --2- 172.21.15.61:0/4042661896 >> v2:172.21.15.111:3300/0 conn(0x7fdffc0b0120 0x7fdffc144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- 172.21.15.61:0/4042661896 >> 172.21.15.61:0/4042661896 conn(0x7fdffc0b5290 msgr2=0x7fdffc0b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- 172.21.15.61:0/4042661896 shutdown_connections 2022-07-12T08:41:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- 172.21.15.61:0/4042661896 wait complete. 2022-07-12T08:41:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 Processor -- start 2022-07-12T08:41:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- start start 2022-07-12T08:41:31.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdffc0ab730 0x7fdffc0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 0x7fdffc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc13f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc145be0 con 0x7fdffc0b0120 2022-07-12T08:41:31.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc142180 con 0x7fdffc0ab730 2022-07-12T08:41:31.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fe001d69700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc0ae610 con 0x7fdffc14aad0 2022-07-12T08:41:31.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 0x7fdffc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 0x7fdffc13f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53468/0 (socket says 172.21.15.61:53468) 2022-07-12T08:41:31.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 -- 172.21.15.61:0/350956073 learned_addr learned my addr 172.21.15.61:0/350956073 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffb7fe700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdffc0ab730 0x7fdffc0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffbfff700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc13f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 -- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 msgr2=0x7fdffc13f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc13f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 -- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdffc0ab730 msgr2=0x7fdffc0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdffc0ab730 0x7fdffc0aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.571+0000 7fdffaffd700 1 -- 172.21.15.61:0/350956073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff400b1e0 con 0x7fdffc0b0120 2022-07-12T08:41:31.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdffaffd700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 0x7fdffc13f4d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdfec001490 tx=0x7fdfec007e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfec002240 con 0x7fdffc0b0120 2022-07-12T08:41:31.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec0023a0 con 0x7fdffc0b0120 2022-07-12T08:41:31.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc0ae8b0 con 0x7fdffc0b0120 2022-07-12T08:41:31.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfec001650 con 0x7fdffc0b0120 2022-07-12T08:41:31.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc145860 con 0x7fdffc0b0120 2022-07-12T08:41:31.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdfec010030 con 0x7fdffc0b0120 2022-07-12T08:41:31.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc057460 con 0x7fdffc0b0120 2022-07-12T08:41:31.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdff8ff9700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfdc053ad0 0x7fdfdc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdfec06be50 con 0x7fdffc0b0120 2022-07-12T08:41:31.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.575+0000 7fdffb7fe700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfdc053ad0 0x7fdfdc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.579+0000 7fdffb7fe700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfdc053ad0 0x7fdfdc055f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdff000b000 tx=0x7fdff0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.583+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdfec042d10 con 0x7fdffc0b0120 2022-07-12T08:41:31.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.707+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdffc0447a0 con 0x7fdffc0b0120 2022-07-12T08:41:31.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.707+0000 7fdff8ff9700 1 -- 172.21.15.61:0/350956073 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdfec046330 con 0x7fdffc0b0120 2022-07-12T08:41:31.715 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:31.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfdc053ad0 msgr2=0x7fdfdc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfdc053ad0 0x7fdfdc055f90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fdff000b000 tx=0x7fdff0006df0).stop 2022-07-12T08:41:31.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 msgr2=0x7fdffc13f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 0x7fdffc13f4d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fdfec001490 tx=0x7fdfec007e10).stop 2022-07-12T08:41:31.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 shutdown_connections 2022-07-12T08:41:31.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdffc14aad0 0x7fdffc13f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdffc0b0120 0x7fdffc13f4d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdfdc053ad0 0x7fdfdc055f90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 --2- 172.21.15.61:0/350956073 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdffc0ab730 0x7fdffc0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 >> 172.21.15.61:0/350956073 conn(0x7fdffc0b5290 msgr2=0x7fdffc0b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:31.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 shutdown_connections 2022-07-12T08:41:31.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.711+0000 7fe001d69700 1 -- 172.21.15.61:0/350956073 wait complete. 2022-07-12T08:41:31.726 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:41:31.727 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:31.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 Processor -- start 2022-07-12T08:41:31.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 -- start start 2022-07-12T08:41:31.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940ab290 0x7f3d940ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d9405f140 0x7f3d9405ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d940abc70 con 0x7f3d940afa60 2022-07-12T08:41:31.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9405f580 con 0x7f3d9405f140 2022-07-12T08:41:31.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d9a793700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9405c720 con 0x7f3d940ab290 2022-07-12T08:41:31.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d9405f140 0x7f3d9405ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d9405f140 0x7f3d9405ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53474/0 (socket says 172.21.15.61:53474) 2022-07-12T08:41:31.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 -- 172.21.15.61:0/2591519692 learned_addr learned my addr 172.21.15.61:0/2591519692 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d99791700 1 --2- 172.21.15.61:0/2591519692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940ab290 0x7f3d940ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 -- 172.21.15.61:0/2591519692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940ab290 msgr2=0x7f3d940ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:31.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 --2- 172.21.15.61:0/2591519692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940ab290 0x7f3d940ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 -- 172.21.15.61:0/2591519692 >> v1:172.21.15.111:6789/0 conn(0x7f3d940afa60 legacy=0x7f3d9414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 -- 172.21.15.61:0/2591519692 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9405c9c0 con 0x7f3d9405f140 2022-07-12T08:41:31.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d98f90700 1 --2- 172.21.15.61:0/2591519692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d9405ce60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3d88002f70 tx=0x7f3d88006eb0).ready entity=mon.0 client_cookie=a5aa479605e03e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d93fff700 1 -- 172.21.15.61:0/2591519692 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8800d570 con 0x7f3d9405f140 2022-07-12T08:41:31.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d93fff700 1 -- 172.21.15.61:0/2591519692 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d88000e90 con 0x7f3d9405f140 2022-07-12T08:41:31.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.831+0000 7f3d93fff700 1 -- 172.21.15.61:0/2591519692 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8800d900 con 0x7f3d9405f140 2022-07-12T08:41:31.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- 172.21.15.61:0/2591519692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 msgr2=0x7f3d9405ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 --2- 172.21.15.61:0/2591519692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d9405ce60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f3d88002f70 tx=0x7f3d88006eb0).stop 2022-07-12T08:41:31.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- 172.21.15.61:0/2591519692 shutdown_connections 2022-07-12T08:41:31.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 --2- 172.21.15.61:0/2591519692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d9405ce60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 --2- 172.21.15.61:0/2591519692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940ab290 0x7f3d940ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- 172.21.15.61:0/2591519692 >> 172.21.15.61:0/2591519692 conn(0x7f3d940b1950 msgr2=0x7f3d940b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:31.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- 172.21.15.61:0/2591519692 shutdown_connections 2022-07-12T08:41:31.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- 172.21.15.61:0/2591519692 wait complete. 2022-07-12T08:41:31.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 Processor -- start 2022-07-12T08:41:31.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- start start 2022-07-12T08:41:31.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d94147240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d940ab290 0x7f3d940667c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940afa60 0x7f3d94159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940660f0 con 0x7f3d9405f140 2022-07-12T08:41:31.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d94066320 con 0x7f3d940ab290 2022-07-12T08:41:31.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d9a793700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d940ae2a0 con 0x7f3d940afa60 2022-07-12T08:41:31.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d94147240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d94147240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53478/0 (socket says 172.21.15.61:53478) 2022-07-12T08:41:31.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 -- 172.21.15.61:0/1249013665 learned_addr learned my addr 172.21.15.61:0/1249013665 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:31.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99f92700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940afa60 0x7f3d94159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d98f90700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d940ab290 0x7f3d940667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 -- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940afa60 msgr2=0x7f3d94159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940afa60 0x7f3d94159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 -- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d940ab290 msgr2=0x7f3d940667c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d940ab290 0x7f3d940667c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 -- 172.21.15.61:0/1249013665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8800b1e0 con 0x7f3d9405f140 2022-07-12T08:41:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.835+0000 7f3d99791700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d94147240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3d8c00c000 tx=0x7f3d8c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8c00c850 con 0x7f3d9405f140 2022-07-12T08:41:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d8c00c9b0 con 0x7f3d9405f140 2022-07-12T08:41:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d940af1d0 con 0x7f3d9405f140 2022-07-12T08:41:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d8c00cd10 con 0x7f3d9405f140 2022-07-12T08:41:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d94159950 con 0x7f3d9405f140 2022-07-12T08:41:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d94057460 con 0x7f3d9405f140 2022-07-12T08:41:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3d8c022070 con 0x7f3d9405f140 2022-07-12T08:41:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d927fc700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d74053ad0 0x7f3d74055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3d8c06c5f0 con 0x7f3d9405f140 2022-07-12T08:41:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d98f90700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d74053ad0 0x7f3d74055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.839+0000 7f3d98f90700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d74053ad0 0x7f3d74055f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3d8801a000 tx=0x7f3d88015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:31.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:31.847+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d8c0434b0 con 0x7f3d9405f140 2022-07-12T08:41:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.003+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d94066490 con 0x7f3d9405f140 2022-07-12T08:41:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d927fc700 1 -- 172.21.15.61:0/1249013665 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+2137 (secure 0 0 0) 0x7f3d8c046ad0 con 0x7f3d9405f140 2022-07-12T08:41:32.011 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:32.012 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:15.584363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.111:6835/1570404457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1570404457},{"type":"v1","addr":"172.21.15.111:6835","nonce":1570404457}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d74053ad0 msgr2=0x7f3d74055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d74053ad0 0x7f3d74055f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f3d8801a000 tx=0x7f3d88015000).stop 2022-07-12T08:41:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 msgr2=0x7f3d94147240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d94147240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f3d8c00c000 tx=0x7f3d8c0091d0).stop 2022-07-12T08:41:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 shutdown_connections 2022-07-12T08:41:32.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3d940afa60 0x7f3d94159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:32.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3d74053ad0 0x7f3d74055f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:32.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3d940ab290 0x7f3d940667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:32.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 --2- 172.21.15.61:0/1249013665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d9405f140 0x7f3d94147240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:32.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 >> 172.21.15.61:0/1249013665 conn(0x7f3d940b1950 msgr2=0x7f3d940b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:32.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 shutdown_connections 2022-07-12T08:41:32.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:32.007+0000 7f3d9a793700 1 -- 172.21.15.61:0/1249013665 wait complete. 2022-07-12T08:41:32.046 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2022-07-12T08:41:32.256 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:41:14.580428+0000', 'modified': '2022-07-12T08:41:15.584363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.111:6835/1570404457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 1570404457}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 1570404457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:41:32.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:41:32.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:41:32.379 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":4282,"want_state":"up:active","state":"up:active","rank_uptime":17.785489597000002,"mdsmap_epoch":5,"osdmap_epoch":22,"osdmap_epoch_barrier":21,"uptime":21.12440269} 2022-07-12T08:41:32.380 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 4282, "mdsmap_epoch": 5, "osdmap_epoch": 22, "osdmap_epoch_barrier": 21, "rank_uptime": 17.785489597, "state": "up:active", "uptime": 21.12440269, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:41:32.380 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:41:32.380 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:41:32.381 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:41:32.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:32.381 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:32.404 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:41:32.444 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:32.445 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:32.530 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:32.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:32.540 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:41:32.597 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:41:32.597 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:32.619 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:41:32.645 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:32.645 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:32.710 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:32.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:32.719 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:41:32.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[20947]: starting ceph client 2022-07-12T08:41:32.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:41:32.718+0000 7f610759f080 -1 init, newargv = 0x55a1bb9f56e0 newargc=9 2022-07-12T08:41:32.739 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[20947]: starting fuse 2022-07-12T08:41:33.831 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:41:33.867 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:33.867 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:33.938 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:33.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:33.940 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:41:33.958 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:41:33.959 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:41:33.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:33.960 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:41:33.961 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:41:33.961 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:41:33.961 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:41:33.963 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:41:33.964 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:41:33.964 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:41:33.964 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:41:33.965 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:41:33.965 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:41:33.965 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:41:33.966 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:41:33.966 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:41:33.967 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:41:33.967 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:41:33.968 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:41:33.968 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:41:33.968 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:41:33.969 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:41:33.970 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:41:33.971 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:41:33.971 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:41:33.972 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:41:34.062 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.20947.asok 2022-07-12T08:41:34.071 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.20947.asok 2022-07-12T08:41:34.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:34.073 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.20947.asok status 2022-07-12T08:41:34.198 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:41:34.199 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:41:34.199 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:41:34.204 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:41:34.204 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:41:34.205 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:41:34.205 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:41:34.212 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "20947", 2022-07-12T08:41:34.213 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:41:34.213 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:41:34.214 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:41:34.214 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:41:34.215 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 4632, 2022-07-12T08:41:34.215 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:41:34.216 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:41:34.216 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:41:34.217 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 4632 2022-07-12T08:41:34.217 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:41:34.217 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:41:34.218 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:41:34.219 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:41:34.219 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2979239422 2022-07-12T08:41:34.220 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:41:34.220 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:41:34.221 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:41:34.221 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:41:34.222 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:41:34.222 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2979239422 2022-07-12T08:41:34.229 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:41:34.230 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.4632 172.21.15.130:0/2979239422", 2022-07-12T08:41:34.231 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/2979239422", 2022-07-12T08:41:34.248 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:41:34.249 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 5, 2022-07-12T08:41:34.249 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 22, 2022-07-12T08:41:34.254 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:41:34.257 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:41:34.257 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:41:34.258 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:41:34.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:34.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b95dd3700 1 Processor -- start 2022-07-12T08:41:34.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b95dd3700 1 -- start start 2022-07-12T08:41:34.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b95dd3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7b900b01f0 0x7f7b900ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b95dd3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900ad2f0 con 0x7f7b9014a8f0 2022-07-12T08:41:34.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b95dd3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b9005c430 con 0x7f7b9005c050 2022-07-12T08:41:34.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b95dd3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b900b0a40 con 0x7f7b900b01f0 2022-07-12T08:41:34.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7b9014a8f0 0x7f7b901443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41708/0 (socket says 172.21.15.61:41708) 2022-07-12T08:41:34.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8ffff700 1 -- 172.21.15.61:0/970959112 learned_addr learned my addr 172.21.15.61:0/970959112 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:34.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8effd700 1 --2- 172.21.15.61:0/970959112 >> v2:172.21.15.111:3300/0 conn(0x7f7b900b01f0 0x7f7b900ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609391721 0 0) 0x7f7b900ad2f0 con 0x7f7b9014a8f0 2022-07-12T08:41:34.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b70003260 con 0x7f7b9014a8f0 2022-07-12T08:41:34.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380229330 0 0) 0x7f7b9005c430 con 0x7f7b9005c050 2022-07-12T08:41:34.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b900ad2f0 con 0x7f7b9005c050 2022-07-12T08:41:34.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8effd700 1 -- 172.21.15.61:0/970959112 >> v1:172.21.15.111:6789/0 conn(0x7f7b9005c050 legacy=0x7f7b9005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8effd700 1 -- 172.21.15.61:0/970959112 >> v1:172.21.15.61:6789/0 conn(0x7f7b9014a8f0 legacy=0x7f7b901443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8effd700 1 -- 172.21.15.61:0/970959112 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900b2420 con 0x7f7b900b01f0 2022-07-12T08:41:34.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8effd700 1 --2- 172.21.15.61:0/970959112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b900ac710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7b78002f70 tx=0x7f7b78006df0).ready entity=mon.1 client_cookie=618fc13dc3720098 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b7800d540 con 0x7f7b900b01f0 2022-07-12T08:41:34.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b78000e90 con 0x7f7b900b01f0 2022-07-12T08:41:34.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.382+0000 7f7b8e7fc700 1 -- 172.21.15.61:0/970959112 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b7800d890 con 0x7f7b900b01f0 2022-07-12T08:41:34.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 -- 172.21.15.61:0/970959112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 msgr2=0x7f7b900ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/970959112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b900ac710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f7b78002f70 tx=0x7f7b78006df0).stop 2022-07-12T08:41:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 -- 172.21.15.61:0/970959112 shutdown_connections 2022-07-12T08:41:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/970959112 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b900ac710 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 -- 172.21.15.61:0/970959112 >> 172.21.15.61:0/970959112 conn(0x7f7b90062c00 msgr2=0x7f7b90062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:34.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 -- 172.21.15.61:0/970959112 shutdown_connections 2022-07-12T08:41:34.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 -- 172.21.15.61:0/970959112 wait complete. 2022-07-12T08:41:34.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 Processor -- start 2022-07-12T08:41:34.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.386+0000 7f7b95dd3700 1 -- start start 2022-07-12T08:41:34.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b9005c050 0x7f7b900ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b90141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 0x7f7b90141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90141d70 con 0x7f7b9014a8f0 2022-07-12T08:41:34.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015a560 con 0x7f7b900b01f0 2022-07-12T08:41:34.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015a990 con 0x7f7b9005c050 2022-07-12T08:41:34.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 0x7f7b90141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 0x7f7b90141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53496/0 (socket says 172.21.15.61:53496) 2022-07-12T08:41:34.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 -- 172.21.15.61:0/2719804092 learned_addr learned my addr 172.21.15.61:0/2719804092 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:34.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8f7fe700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b9005c050 0x7f7b900ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8effd700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b90141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 -- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b9005c050 msgr2=0x7f7b900ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b9005c050 0x7f7b900ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 -- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 msgr2=0x7f7b90141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b90141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 -- 172.21.15.61:0/2719804092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7800b180 con 0x7f7b9014a8f0 2022-07-12T08:41:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8effd700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b90141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8ffff700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 0x7f7b90141880 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f7b900ade80 tx=0x7f7b88007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b88005ba0 con 0x7f7b9014a8f0 2022-07-12T08:41:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88005d00 con 0x7f7b9014a8f0 2022-07-12T08:41:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b900aeb70 con 0x7f7b9014a8f0 2022-07-12T08:41:34.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b880072c0 con 0x7f7b9014a8f0 2022-07-12T08:41:34.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.390+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b900ae7e0 con 0x7f7b9014a8f0 2022-07-12T08:41:34.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.394+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b900acb20 con 0x7f7b9014a8f0 2022-07-12T08:41:34.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.394+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7b8801a060 con 0x7f7b9014a8f0 2022-07-12T08:41:34.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.394+0000 7f7b8cff9700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b740539b0 0x7f7b74055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.394+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7b8806b520 con 0x7f7b9014a8f0 2022-07-12T08:41:34.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.394+0000 7f7b8f7fe700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b740539b0 0x7f7b74055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.394+0000 7f7b8f7fe700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b740539b0 0x7f7b74055e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7b8000d000 tx=0x7f7b80007a60).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.398+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7b88042480 con 0x7f7b9014a8f0 2022-07-12T08:41:34.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.530+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b90057460 con 0x7f7b9014a8f0 2022-07-12T08:41:34.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.530+0000 7f7b8cff9700 1 -- 172.21.15.61:0/2719804092 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f7b88018020 con 0x7f7b9014a8f0 2022-07-12T08:41:34.533 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:34.533 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:34.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.530+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b740539b0 msgr2=0x7f7b74055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.530+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b740539b0 0x7f7b74055e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7b8000d000 tx=0x7f7b80007a60).stop 2022-07-12T08:41:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.530+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 msgr2=0x7f7b90141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.530+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 0x7f7b90141880 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f7b900ade80 tx=0x7f7b88007e00).stop 2022-07-12T08:41:34.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 shutdown_connections 2022-07-12T08:41:34.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b9014a8f0 0x7f7b90141880 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b740539b0 0x7f7b74055e70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b900b01f0 0x7f7b90141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 --2- 172.21.15.61:0/2719804092 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b9005c050 0x7f7b900ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 >> 172.21.15.61:0/2719804092 conn(0x7f7b90062c00 msgr2=0x7f7b900b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:34.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 shutdown_connections 2022-07-12T08:41:34.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.534+0000 7f7b95dd3700 1 -- 172.21.15.61:0/2719804092 wait complete. 2022-07-12T08:41:34.551 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:34.553 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:34.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 Processor -- start 2022-07-12T08:41:34.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- start start 2022-07-12T08:41:34.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8684060c10 0x7f868414b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86840b1000 0x7f86840af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f86840afdc0 0x7f868409a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8684099010 con 0x7f86840b1000 2022-07-12T08:41:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8684060ff0 con 0x7f86840afdc0 2022-07-12T08:41:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868414a930 con 0x7f8684060c10 2022-07-12T08:41:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86840b1000 0x7f86840af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86840b1000 0x7f86840af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53502/0 (socket says 172.21.15.61:53502) 2022-07-12T08:41:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 -- 172.21.15.61:0/2658553927 learned_addr learned my addr 172.21.15.61:0/2658553927 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f86897d7700 1 --2- 172.21.15.61:0/2658553927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8684060c10 0x7f868414b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8689fd8700 1 --2- 172.21.15.61:0/2658553927 >> v2:172.21.15.111:3300/0 conn(0x7f86840afdc0 0x7f868409a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 -- 172.21.15.61:0/2658553927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8684060c10 msgr2=0x7f868414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:34.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 --2- 172.21.15.61:0/2658553927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8684060c10 0x7f868414b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 -- 172.21.15.61:0/2658553927 >> v2:172.21.15.111:3300/0 conn(0x7f86840afdc0 msgr2=0x7f868409a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 --2- 172.21.15.61:0/2658553927 >> v2:172.21.15.111:3300/0 conn(0x7f86840afdc0 0x7f868409a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 -- 172.21.15.61:0/2658553927 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868414abd0 con 0x7f86840b1000 2022-07-12T08:41:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8689fd8700 1 --2- 172.21.15.61:0/2658553927 >> v2:172.21.15.111:3300/0 conn(0x7f86840afdc0 0x7f868409a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:34.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f8688fd6700 1 --2- 172.21.15.61:0/2658553927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840af8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f8674002f70 tx=0x7f8674006eb0).ready entity=mon.0 client_cookie=9af9acbd8f9351c9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f867bfff700 1 -- 172.21.15.61:0/2658553927 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f867400d570 con 0x7f86840b1000 2022-07-12T08:41:34.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f867bfff700 1 -- 172.21.15.61:0/2658553927 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8674000e90 con 0x7f86840b1000 2022-07-12T08:41:34.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f867bfff700 1 -- 172.21.15.61:0/2658553927 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f867400d900 con 0x7f86840b1000 2022-07-12T08:41:34.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- 172.21.15.61:0/2658553927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 msgr2=0x7f86840af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- 172.21.15.61:0/2658553927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840af8d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f8674002f70 tx=0x7f8674006eb0).stop 2022-07-12T08:41:34.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- 172.21.15.61:0/2658553927 shutdown_connections 2022-07-12T08:41:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- 172.21.15.61:0/2658553927 >> v2:172.21.15.111:3300/0 conn(0x7f86840afdc0 0x7f868409a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- 172.21.15.61:0/2658553927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840af8d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 --2- 172.21.15.61:0/2658553927 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8684060c10 0x7f868414b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- 172.21.15.61:0/2658553927 >> 172.21.15.61:0/2658553927 conn(0x7f868405bd10 msgr2=0x7f868405c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.665+0000 7f868a7d9700 1 -- 172.21.15.61:0/2658553927 shutdown_connections 2022-07-12T08:41:34.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 -- 172.21.15.61:0/2658553927 wait complete. 2022-07-12T08:41:34.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 Processor -- start 2022-07-12T08:41:34.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 -- start start 2022-07-12T08:41:34.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8684060c10 0x7f86840b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f86840afdc0 0x7f8684065b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8684066030 con 0x7f86840b1000 2022-07-12T08:41:34.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8684062cb0 con 0x7f8684060c10 2022-07-12T08:41:34.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f868a7d9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8684065900 con 0x7f86840afdc0 2022-07-12T08:41:34.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840627c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53510/0 (socket says 172.21.15.61:53510) 2022-07-12T08:41:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 -- 172.21.15.61:0/1060901357 learned_addr learned my addr 172.21.15.61:0/1060901357 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f86897d7700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8684060c10 0x7f86840b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8688fd6700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f86840afdc0 0x7f8684065b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 -- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f86840afdc0 msgr2=0x7f8684065b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f86840afdc0 0x7f8684065b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 -- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8684060c10 msgr2=0x7f86840b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8684060c10 0x7f86840b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 -- 172.21.15.61:0/1060901357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867400b1e0 con 0x7f86840b1000 2022-07-12T08:41:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f86897d7700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8684060c10 0x7f86840b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f8689fd8700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840627c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f867c006cc0 tx=0x7f867c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.669+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f867c013620 con 0x7f86840b1000 2022-07-12T08:41:34.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867c013780 con 0x7f86840b1000 2022-07-12T08:41:34.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86840905e0 con 0x7f86840b1000 2022-07-12T08:41:34.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f867c013ae0 con 0x7f86840b1000 2022-07-12T08:41:34.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86840655f0 con 0x7f86840b1000 2022-07-12T08:41:34.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f867c013c40 con 0x7f86840b1000 2022-07-12T08:41:34.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f867a7fc700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f866c0539b0 0x7f866c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8684057460 con 0x7f86840b1000 2022-07-12T08:41:34.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f867c06bd90 con 0x7f86840b1000 2022-07-12T08:41:34.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f86897d7700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f866c0539b0 0x7f866c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.673+0000 7f86897d7700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f866c0539b0 0x7f866c055e70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f868000c000 tx=0x7f8680008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.681+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f867c042be0 con 0x7f86840b1000 2022-07-12T08:41:34.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.805+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f868406b2b0 con 0x7f86840b1000 2022-07-12T08:41:34.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.805+0000 7f867a7fc700 1 -- 172.21.15.61:0/1060901357 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f867c00f040 con 0x7f86840b1000 2022-07-12T08:41:34.812 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:34.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f866c0539b0 msgr2=0x7f866c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f866c0539b0 0x7f866c055e70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f868000c000 tx=0x7f8680008040).stop 2022-07-12T08:41:34.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 msgr2=0x7f86840627c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840627c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f867c006cc0 tx=0x7f867c008040).stop 2022-07-12T08:41:34.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 shutdown_connections 2022-07-12T08:41:34.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86840b1000 0x7f86840627c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f86840afdc0 0x7f8684065b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f866c0539b0 0x7f866c055e70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 --2- 172.21.15.61:0/1060901357 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8684060c10 0x7f86840b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 >> 172.21.15.61:0/1060901357 conn(0x7f868405bd10 msgr2=0x7f868409a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:34.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 shutdown_connections 2022-07-12T08:41:34.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.809+0000 7f868a7d9700 1 -- 172.21.15.61:0/1060901357 wait complete. 2022-07-12T08:41:34.821 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:34.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580dcd1700 1 Processor -- start 2022-07-12T08:41:34.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580dcd1700 1 -- start start 2022-07-12T08:41:34.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580dcd1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f580805f140 0x7f58080600c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580dcd1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580805f650 con 0x7f58080b3120 2022-07-12T08:41:34.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580dcd1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580805f8f0 con 0x7f58080b2310 2022-07-12T08:41:34.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580dcd1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f580805fb90 con 0x7f580805f140 2022-07-12T08:41:34.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f5807fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f580805f140 0x7f58080600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580cccf700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f58080b3120 0x7f5808146770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41732/0 (socket says 172.21.15.61:41732) 2022-07-12T08:41:34.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f580cccf700 1 -- 172.21.15.61:0/2430017795 learned_addr learned my addr 172.21.15.61:0/2430017795 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:34.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f5807fff700 1 -- 172.21.15.61:0/2430017795 >> v1:172.21.15.111:6789/0 conn(0x7f58080b2310 legacy=0x7f58080605b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:34.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f5807fff700 1 -- 172.21.15.61:0/2430017795 >> v1:172.21.15.61:6789/0 conn(0x7f58080b3120 legacy=0x7f5808146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f58077fe700 1 -- 172.21.15.61:0/2430017795 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845133148 0 0) 0x7f580805f650 con 0x7f58080b3120 2022-07-12T08:41:34.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f5807fff700 1 -- 172.21.15.61:0/2430017795 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58080accf0 con 0x7f580805f140 2022-07-12T08:41:34.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f5807fff700 1 --2- 172.21.15.61:0/2430017795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f580805f140 0x7f58080600c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f57f80032f0 tx=0x7f57f80091d0).ready entity=mon.0 client_cookie=8759185eb73bbd31 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f58077fe700 1 -- 172.21.15.61:0/2430017795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f8009ea0 con 0x7f580805f140 2022-07-12T08:41:34.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f58077fe700 1 -- 172.21.15.61:0/2430017795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f800b8a0 con 0x7f580805f140 2022-07-12T08:41:34.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.937+0000 7f58077fe700 1 -- 172.21.15.61:0/2430017795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f800ba10 con 0x7f580805f140 2022-07-12T08:41:34.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- 172.21.15.61:0/2430017795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f580805f140 msgr2=0x7f58080600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 --2- 172.21.15.61:0/2430017795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f580805f140 0x7f58080600c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f57f80032f0 tx=0x7f57f80091d0).stop 2022-07-12T08:41:34.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- 172.21.15.61:0/2430017795 shutdown_connections 2022-07-12T08:41:34.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 --2- 172.21.15.61:0/2430017795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f580805f140 0x7f58080600c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- 172.21.15.61:0/2430017795 >> 172.21.15.61:0/2430017795 conn(0x7f58080b4650 msgr2=0x7f580805bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:34.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- 172.21.15.61:0/2430017795 shutdown_connections 2022-07-12T08:41:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- 172.21.15.61:0/2430017795 wait complete. 2022-07-12T08:41:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 Processor -- start 2022-07-12T08:41:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- start start 2022-07-12T08:41:34.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f580805f140 0x7f580809ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58080b2310 0x7f580809cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 0x7f5808097b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580809ab40 con 0x7f58080b3120 2022-07-12T08:41:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580815a900 con 0x7f580805f140 2022-07-12T08:41:34.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580dcd1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580809c5c0 con 0x7f58080b2310 2022-07-12T08:41:34.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580d4d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 0x7f5808097b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580d4d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 0x7f5808097b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53524/0 (socket says 172.21.15.61:53524) 2022-07-12T08:41:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.941+0000 7f580d4d0700 1 -- 172.21.15.61:0/3090285838 learned_addr learned my addr 172.21.15.61:0/3090285838 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5807fff700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58080b2310 0x7f580809cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580cccf700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f580805f140 0x7f580809ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580d4d0700 1 -- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58080b2310 msgr2=0x7f580809cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580d4d0700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58080b2310 0x7f580809cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580d4d0700 1 -- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f580805f140 msgr2=0x7f580809ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580d4d0700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f580805f140 0x7f580809ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580d4d0700 1 -- 172.21.15.61:0/3090285838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f8009470 con 0x7f58080b3120 2022-07-12T08:41:34.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5807fff700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58080b2310 0x7f580809cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:34.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580d4d0700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 0x7f5808097b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f58080ae530 tx=0x7f5800007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5800012400 con 0x7f58080b3120 2022-07-12T08:41:34.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5800012560 con 0x7f58080b3120 2022-07-12T08:41:34.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580809c860 con 0x7f58080b3120 2022-07-12T08:41:34.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f58000128c0 con 0x7f58080b3120 2022-07-12T08:41:34.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58080980a0 con 0x7f58080b3120 2022-07-12T08:41:34.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580809c9f0 con 0x7f58080b3120 2022-07-12T08:41:34.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5800019030 con 0x7f58080b3120 2022-07-12T08:41:34.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5805ffb700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec053ad0 0x7f57ec055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:34.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.945+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f580006b1d0 con 0x7f58080b3120 2022-07-12T08:41:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.949+0000 7f580cccf700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec053ad0 0x7f57ec055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.949+0000 7f580cccf700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec053ad0 0x7f57ec055f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f57f000d000 tx=0x7f57f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:34.953+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58000420d0 con 0x7f58080b3120 2022-07-12T08:41:35.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.077+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5808057460 con 0x7f58080b3120 2022-07-12T08:41:35.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f5805ffb700 1 -- 172.21.15.61:0/3090285838 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f58000456f0 con 0x7f58080b3120 2022-07-12T08:41:35.088 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:35.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec053ad0 msgr2=0x7f57ec055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec053ad0 0x7f57ec055f90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f57f000d000 tx=0x7f57f0009040).stop 2022-07-12T08:41:35.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 msgr2=0x7f5808097b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 0x7f5808097b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f58080ae530 tx=0x7f5800007bc0).stop 2022-07-12T08:41:35.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 shutdown_connections 2022-07-12T08:41:35.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58080b3120 0x7f5808097b30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f58080b2310 0x7f580809cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec053ad0 0x7f57ec055f90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 --2- 172.21.15.61:0/3090285838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f580805f140 0x7f580809ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 >> 172.21.15.61:0/3090285838 conn(0x7f58080b4650 msgr2=0x7f580805bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:35.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 shutdown_connections 2022-07-12T08:41:35.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.081+0000 7f580dcd1700 1 -- 172.21.15.61:0/3090285838 wait complete. 2022-07-12T08:41:35.112 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:35.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72288d3700 1 Processor -- start 2022-07-12T08:41:35.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72288d3700 1 -- start start 2022-07-12T08:41:35.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72288d3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7224142eb0 0x7f722405c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72288d3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7224143500 con 0x7f72240b6130 2022-07-12T08:41:35.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72288d3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7224143290 con 0x7f722405d130 2022-07-12T08:41:35.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72288d3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72240b65c0 con 0x7f7224142eb0 2022-07-12T08:41:35.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72237fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7224142eb0 0x7f722405c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f7223fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f72240b6130 0x7f72240b1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41748/0 (socket says 172.21.15.61:41748) 2022-07-12T08:41:35.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f7223fff700 1 -- 172.21.15.61:0/102709247 learned_addr learned my addr 172.21.15.61:0/102709247 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:35.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411792848 0 0) 0x7f7224143500 con 0x7f72240b6130 2022-07-12T08:41:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.229+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7204003260 con 0x7f72240b6130 2022-07-12T08:41:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737876971 0 0) 0x7f7224143290 con 0x7f722405d130 2022-07-12T08:41:35.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7224143500 con 0x7f722405d130 2022-07-12T08:41:35.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72237fe700 1 -- 172.21.15.61:0/102709247 >> v1:172.21.15.111:6789/0 conn(0x7f722405d130 legacy=0x7f72240b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72237fe700 1 -- 172.21.15.61:0/102709247 >> v1:172.21.15.61:6789/0 conn(0x7f72240b6130 legacy=0x7f72240b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72237fe700 1 -- 172.21.15.61:0/102709247 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7224143760 con 0x7f7224142eb0 2022-07-12T08:41:35.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72237fe700 1 --2- 172.21.15.61:0/102709247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7224142eb0 0x7f722405c6d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7224143c30 tx=0x7f720c006df0).ready entity=mon.0 client_cookie=9efabf3c4459ab9b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f720c00d4a0 con 0x7f7224142eb0 2022-07-12T08:41:35.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c000e90 con 0x7f7224142eb0 2022-07-12T08:41:35.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72227fc700 1 -- 172.21.15.61:0/102709247 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f720c00d7f0 con 0x7f7224142eb0 2022-07-12T08:41:35.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 -- 172.21.15.61:0/102709247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7224142eb0 msgr2=0x7f722405c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 --2- 172.21.15.61:0/102709247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7224142eb0 0x7f722405c6d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7224143c30 tx=0x7f720c006df0).stop 2022-07-12T08:41:35.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 -- 172.21.15.61:0/102709247 shutdown_connections 2022-07-12T08:41:35.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 --2- 172.21.15.61:0/102709247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7224142eb0 0x7f722405c6d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 -- 172.21.15.61:0/102709247 >> 172.21.15.61:0/102709247 conn(0x7f7224066d70 msgr2=0x7f7224067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:35.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 -- 172.21.15.61:0/102709247 shutdown_connections 2022-07-12T08:41:35.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 -- 172.21.15.61:0/102709247 wait complete. 2022-07-12T08:41:35.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 Processor -- start 2022-07-12T08:41:35.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.233+0000 7f72288d3700 1 -- start start 2022-07-12T08:41:35.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 0x7f7224146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72240b6130 0x7f72240aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7224142eb0 0x7f72240aef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7224145700 con 0x7f722405d130 2022-07-12T08:41:35.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72240ade70 con 0x7f72240b6130 2022-07-12T08:41:35.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72240adc60 con 0x7f7224142eb0 2022-07-12T08:41:35.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 0x7f7224146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 0x7f7224146740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53534/0 (socket says 172.21.15.61:53534) 2022-07-12T08:41:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 -- 172.21.15.61:0/2444292340 learned_addr learned my addr 172.21.15.61:0/2444292340 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:35.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7223fff700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7224142eb0 0x7f72240aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7222ffd700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72240b6130 0x7f72240aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 -- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7224142eb0 msgr2=0x7f72240aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7224142eb0 0x7f72240aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 -- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72240b6130 msgr2=0x7f72240aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72240b6130 0x7f72240aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 -- 172.21.15.61:0/2444292340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c00b180 con 0x7f722405d130 2022-07-12T08:41:35.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72237fe700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 0x7f7224146740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7224143c30 tx=0x7f720c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f720c013ac0 con 0x7f722405d130 2022-07-12T08:41:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720c013c20 con 0x7f722405d130 2022-07-12T08:41:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72240ada00 con 0x7f722405d130 2022-07-12T08:41:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f720c01e400 con 0x7f722405d130 2022-07-12T08:41:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72240ad560 con 0x7f722405d130 2022-07-12T08:41:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224057460 con 0x7f722405d130 2022-07-12T08:41:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f720c025030 con 0x7f722405d130 2022-07-12T08:41:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.237+0000 7f7220ff9700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7208053a20 0x7f7208055ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.241+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f720c06cc10 con 0x7f722405d130 2022-07-12T08:41:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.241+0000 7f7222ffd700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7208053a20 0x7f7208055ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.241+0000 7f7222ffd700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7208053a20 0x7f7208055ee0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f721400d000 tx=0x7f7214007a90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.245+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f720c043b70 con 0x7f722405d130 2022-07-12T08:41:35.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.369+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f722406b2b0 con 0x7f722405d130 2022-07-12T08:41:35.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f7220ff9700 1 -- 172.21.15.61:0/2444292340 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f720c047190 con 0x7f722405d130 2022-07-12T08:41:35.379 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:35.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7208053a20 msgr2=0x7f7208055ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7208053a20 0x7f7208055ee0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f721400d000 tx=0x7f7214007a90).stop 2022-07-12T08:41:35.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 msgr2=0x7f7224146740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 0x7f7224146740 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7224143c30 tx=0x7f720c014000).stop 2022-07-12T08:41:35.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 shutdown_connections 2022-07-12T08:41:35.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7224142eb0 0x7f72240aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7208053a20 0x7f7208055ee0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72240b6130 0x7f72240aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 --2- 172.21.15.61:0/2444292340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f722405d130 0x7f7224146740 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 >> 172.21.15.61:0/2444292340 conn(0x7f7224066d70 msgr2=0x7f722405c460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:35.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 shutdown_connections 2022-07-12T08:41:35.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.373+0000 7f72288d3700 1 -- 172.21.15.61:0/2444292340 wait complete. 2022-07-12T08:41:35.414 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:35.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71bf59e700 1 Processor -- start 2022-07-12T08:41:35.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71bf59e700 1 -- start start 2022-07-12T08:41:35.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71bf59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71c00af8a0 0x7f71c01413d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71bf59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c00b2600 con 0x7f71c005c5a0 2022-07-12T08:41:35.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71bf59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c01409f0 con 0x7f71c00b21c0 2022-07-12T08:41:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71bf59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c0060590 con 0x7f71c00af8a0 2022-07-12T08:41:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.537+0000 7f71be59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71c00af8a0 0x7f71c01413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bdd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f71c005c5a0 0x7f71c013a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41760/0 (socket says 172.21.15.61:41760) 2022-07-12T08:41:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bdd9b700 1 -- 172.21.15.61:0/2377908946 learned_addr learned my addr 172.21.15.61:0/2377908946 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71be59c700 1 -- 172.21.15.61:0/2377908946 >> v1:172.21.15.111:6789/0 conn(0x7f71c00b21c0 legacy=0x7f71c005f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bd59a700 1 -- 172.21.15.61:0/2377908946 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769240093 0 0) 0x7f71c01409f0 con 0x7f71c00b21c0 2022-07-12T08:41:35.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71be59c700 1 -- 172.21.15.61:0/2377908946 >> v1:172.21.15.61:6789/0 conn(0x7f71c005c5a0 legacy=0x7f71c013a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71be59c700 1 -- 172.21.15.61:0/2377908946 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0140c90 con 0x7f71c00af8a0 2022-07-12T08:41:35.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71be59c700 1 --2- 172.21.15.61:0/2377908946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c01413d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f71b0001110 tx=0x7f71a8006df0).ready entity=mon.0 client_cookie=314a2f78b5bd1830 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bd59a700 1 -- 172.21.15.61:0/2377908946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71a800d5c0 con 0x7f71c00af8a0 2022-07-12T08:41:35.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bd59a700 1 -- 172.21.15.61:0/2377908946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a8000e90 con 0x7f71c00af8a0 2022-07-12T08:41:35.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bd59a700 1 -- 172.21.15.61:0/2377908946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71a800d910 con 0x7f71c00af8a0 2022-07-12T08:41:35.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- 172.21.15.61:0/2377908946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 msgr2=0x7f71c01413d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 --2- 172.21.15.61:0/2377908946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c01413d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f71b0001110 tx=0x7f71a8006df0).stop 2022-07-12T08:41:35.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- 172.21.15.61:0/2377908946 shutdown_connections 2022-07-12T08:41:35.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 --2- 172.21.15.61:0/2377908946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c01413d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- 172.21.15.61:0/2377908946 >> 172.21.15.61:0/2377908946 conn(0x7f71c0061f50 msgr2=0x7f71c0063a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:35.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- 172.21.15.61:0/2377908946 shutdown_connections 2022-07-12T08:41:35.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- 172.21.15.61:0/2377908946 wait complete. 2022-07-12T08:41:35.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 Processor -- start 2022-07-12T08:41:35.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- start start 2022-07-12T08:41:35.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71c005c5a0 0x7f71c00b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c0153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71c00b21c0 0x7f71c0153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c0143520 con 0x7f71c00af8a0 2022-07-12T08:41:35.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c0142670 con 0x7f71c00b21c0 2022-07-12T08:41:35.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.541+0000 7f71bf59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c0152fd0 con 0x7f71c005c5a0 2022-07-12T08:41:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c0153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c0153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53548/0 (socket says 172.21.15.61:53548) 2022-07-12T08:41:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 -- 172.21.15.61:0/1678411435 learned_addr learned my addr 172.21.15.61:0/1678411435 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bed9d700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71c00b21c0 0x7f71c0153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71be59c700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71c005c5a0 0x7f71c00b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 -- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71c005c5a0 msgr2=0x7f71c00b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71c005c5a0 0x7f71c00b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 -- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71c00b21c0 msgr2=0x7f71c0153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71c00b21c0 0x7f71c0153920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 -- 172.21.15.61:0/1678411435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a800b180 con 0x7f71c00af8a0 2022-07-12T08:41:35.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bdd9b700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c0153430 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f71c0146aa0 tx=0x7f71b0007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71b00070d0 con 0x7f71c00af8a0 2022-07-12T08:41:35.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b0007230 con 0x7f71c00af8a0 2022-07-12T08:41:35.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71b0005610 con 0x7f71c00af8a0 2022-07-12T08:41:35.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c01428a0 con 0x7f71c00af8a0 2022-07-12T08:41:35.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.545+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c0153e90 con 0x7f71c00af8a0 2022-07-12T08:41:35.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.549+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f71b0010070 con 0x7f71c00af8a0 2022-07-12T08:41:35.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.549+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c0057460 con 0x7f71c00af8a0 2022-07-12T08:41:35.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.549+0000 7f71b77fe700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71a40539d0 0x7f71a4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.549+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f71b006b740 con 0x7f71c00af8a0 2022-07-12T08:41:35.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.549+0000 7f71be59c700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71a40539d0 0x7f71a4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.549+0000 7f71be59c700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71a40539d0 0x7f71a4055e90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f71a801a000 tx=0x7f71a8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.553+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71b00426a0 con 0x7f71c00af8a0 2022-07-12T08:41:35.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71c00008d0 con 0x7f71c00af8a0 2022-07-12T08:41:35.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71b77fe700 1 -- 172.21.15.61:0/1678411435 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f71b0018070 con 0x7f71c00af8a0 2022-07-12T08:41:35.692 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:35.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71a40539d0 msgr2=0x7f71a4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71a40539d0 0x7f71a4055e90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f71a801a000 tx=0x7f71a8014000).stop 2022-07-12T08:41:35.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 msgr2=0x7f71c0153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c0153430 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f71c0146aa0 tx=0x7f71b0007bc0).stop 2022-07-12T08:41:35.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 shutdown_connections 2022-07-12T08:41:35.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71a40539d0 0x7f71a4055e90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71c00b21c0 0x7f71c0153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71c00af8a0 0x7f71c0153430 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 --2- 172.21.15.61:0/1678411435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71c005c5a0 0x7f71c00b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 >> 172.21.15.61:0/1678411435 conn(0x7f71c0061f50 msgr2=0x7f71c013bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:35.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 shutdown_connections 2022-07-12T08:41:35.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.681+0000 7f71bf59e700 1 -- 172.21.15.61:0/1678411435 wait complete. 2022-07-12T08:41:35.708 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:41:35.709 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.809+0000 7efe6d44c700 1 Processor -- start 2022-07-12T08:41:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.809+0000 7efe6d44c700 1 -- start start 2022-07-12T08:41:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe6d44c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efe680b01f0 0x7efe680ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe6d44c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe680ad2f0 con 0x7efe6814a8f0 2022-07-12T08:41:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe6d44c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6805c430 con 0x7efe6805c050 2022-07-12T08:41:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe6d44c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe680b0a40 con 0x7efe680b01f0 2022-07-12T08:41:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe677fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efe6814a8f0 0x7efe681443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41778/0 (socket says 172.21.15.61:41778) 2022-07-12T08:41:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe677fe700 1 -- 172.21.15.61:0/3753778110 learned_addr learned my addr 172.21.15.61:0/3753778110 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe667fc700 1 --2- 172.21.15.61:0/3753778110 >> v2:172.21.15.111:3300/0 conn(0x7efe680b01f0 0x7efe680ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843293177 0 0) 0x7efe6805c430 con 0x7efe6805c050 2022-07-12T08:41:35.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe48003260 con 0x7efe6805c050 2022-07-12T08:41:35.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164452411 0 0) 0x7efe680ad2f0 con 0x7efe6814a8f0 2022-07-12T08:41:35.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6805c430 con 0x7efe6814a8f0 2022-07-12T08:41:35.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe667fc700 1 -- 172.21.15.61:0/3753778110 >> v1:172.21.15.111:6789/0 conn(0x7efe6805c050 legacy=0x7efe6805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe667fc700 1 -- 172.21.15.61:0/3753778110 >> v1:172.21.15.61:6789/0 conn(0x7efe6814a8f0 legacy=0x7efe681443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe667fc700 1 -- 172.21.15.61:0/3753778110 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680b2420 con 0x7efe680b01f0 2022-07-12T08:41:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe667fc700 1 --2- 172.21.15.61:0/3753778110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe680b01f0 0x7efe680ac710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efe50002f70 tx=0x7efe50006df0).ready entity=mon.1 client_cookie=53c04ead9ddae328 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe5000d540 con 0x7efe680b01f0 2022-07-12T08:41:35.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe50000e90 con 0x7efe680b01f0 2022-07-12T08:41:35.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe65ffb700 1 -- 172.21.15.61:0/3753778110 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe5000d890 con 0x7efe680b01f0 2022-07-12T08:41:35.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe6d44c700 1 -- 172.21.15.61:0/3753778110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe680b01f0 msgr2=0x7efe680ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.813+0000 7efe6d44c700 1 --2- 172.21.15.61:0/3753778110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe680b01f0 0x7efe680ac710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7efe50002f70 tx=0x7efe50006df0).stop 2022-07-12T08:41:35.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 -- 172.21.15.61:0/3753778110 shutdown_connections 2022-07-12T08:41:35.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 --2- 172.21.15.61:0/3753778110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe680b01f0 0x7efe680ac710 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 -- 172.21.15.61:0/3753778110 >> 172.21.15.61:0/3753778110 conn(0x7efe68062c00 msgr2=0x7efe68062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:35.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 -- 172.21.15.61:0/3753778110 shutdown_connections 2022-07-12T08:41:35.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 -- 172.21.15.61:0/3753778110 wait complete. 2022-07-12T08:41:35.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 Processor -- start 2022-07-12T08:41:35.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.817+0000 7efe6d44c700 1 -- start start 2022-07-12T08:41:35.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 0x7efe68144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe680b01f0 0x7efe68159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe6814a8f0 0x7efe68159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680aeca0 con 0x7efe680b01f0 2022-07-12T08:41:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe680ae6a0 con 0x7efe6814a8f0 2022-07-12T08:41:35.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe680ae940 con 0x7efe6805c050 2022-07-12T08:41:35.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe667fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe680b01f0 0x7efe68159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe66ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 0x7efe68144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe6814a8f0 0x7efe68159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe66ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 0x7efe68144f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:35758/0 (socket says 172.21.15.61:35758) 2022-07-12T08:41:35.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe667fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe680b01f0 0x7efe68159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53564/0 (socket says 172.21.15.61:53564) 2022-07-12T08:41:35.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe66ffd700 1 -- 172.21.15.61:0/2707624630 learned_addr learned my addr 172.21.15.61:0/2707624630 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:35.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 -- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 msgr2=0x7efe68144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 0x7efe68144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 -- 172.21.15.61:0/2707624630 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe680b01f0 msgr2=0x7efe68159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe680b01f0 0x7efe68159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 -- 172.21.15.61:0/2707624630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5000b180 con 0x7efe6814a8f0 2022-07-12T08:41:35.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe66ffd700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 0x7efe68144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:35.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe677fe700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe6814a8f0 0x7efe68159f40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7efe60001110 tx=0x7efe60007a00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe6000cd50 con 0x7efe6814a8f0 2022-07-12T08:41:35.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe600070d0 con 0x7efe6814a8f0 2022-07-12T08:41:35.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe680aeed0 con 0x7efe6814a8f0 2022-07-12T08:41:35.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe60007430 con 0x7efe6814a8f0 2022-07-12T08:41:35.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe68143b70 con 0x7efe6814a8f0 2022-07-12T08:41:35.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.821+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe680af060 con 0x7efe6814a8f0 2022-07-12T08:41:35.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.825+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efe60010070 con 0x7efe6814a8f0 2022-07-12T08:41:35.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.825+0000 7efe5ffff700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe4c053ad0 0x7efe4c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:35.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.825+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efe6006b280 con 0x7efe6814a8f0 2022-07-12T08:41:35.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.825+0000 7efe66ffd700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe4c053ad0 0x7efe4c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:35.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.825+0000 7efe66ffd700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe4c053ad0 0x7efe4c055f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efe5800d000 tx=0x7efe58007e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:35.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.833+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe600421e0 con 0x7efe6814a8f0 2022-07-12T08:41:36.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.997+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe68057460 con 0x7efe6814a8f0 2022-07-12T08:41:36.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:35.997+0000 7efe5ffff700 1 -- 172.21.15.61:0/2707624630 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+2137 (secure 0 0 0) 0x7efe60045800 con 0x7efe6814a8f0 2022-07-12T08:41:36.002 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:36.003 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:15.584363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.111:6835/1570404457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1570404457},{"type":"v1","addr":"172.21.15.111:6835","nonce":1570404457}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:36.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe4c053ad0 msgr2=0x7efe4c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:36.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe4c053ad0 0x7efe4c055f90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efe5800d000 tx=0x7efe58007e80).stop 2022-07-12T08:41:36.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe6814a8f0 msgr2=0x7efe68159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:36.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe6814a8f0 0x7efe68159f40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7efe60001110 tx=0x7efe60007a00).stop 2022-07-12T08:41:36.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 shutdown_connections 2022-07-12T08:41:36.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe4c053ad0 0x7efe4c055f90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:36.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe6814a8f0 0x7efe68159f40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:36.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe680b01f0 0x7efe68159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:36.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 --2- 172.21.15.61:0/2707624630 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe6805c050 0x7efe68144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:36.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 >> 172.21.15.61:0/2707624630 conn(0x7efe68062c00 msgr2=0x7efe680b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:36.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 shutdown_connections 2022-07-12T08:41:36.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:36.001+0000 7efe6d44c700 1 -- 172.21.15.61:0/2707624630 wait complete. 2022-07-12T08:41:36.018 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2022-07-12T08:41:36.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:41:14.580428+0000', 'modified': '2022-07-12T08:41:15.584363+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.111:6835/1570404457', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 1570404457}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 1570404457}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:41:36.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:41:36.242 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:41:36.371 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":4282,"want_state":"up:active","state":"up:active","rank_uptime":21.781803793000002,"mdsmap_epoch":5,"osdmap_epoch":22,"osdmap_epoch_barrier":21,"uptime":25.120722350000001} 2022-07-12T08:41:36.372 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 4282, "mdsmap_epoch": 5, "osdmap_epoch": 22, "osdmap_epoch_barrier": 21, "rank_uptime": 21.781803793, "state": "up:active", "uptime": 25.12072235, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:41:36.373 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:41:36.378 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2022-07-12T08:41:36.379 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.2 at ubuntu@smithi137.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2... 2022-07-12T08:41:36.379 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.380 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:36.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2022-07-12T08:41:36.419 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:36.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:36.494 INFO:teuthology.orchestra.run.smithi137.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:36.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.495 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2022-07-12T08:41:36.509 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:41:36.512 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.2 /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:36.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2022-07-12T08:41:36.573 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:41:36.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:41:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:41:36.642 INFO:teuthology.orchestra.run.smithi137.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:41:36.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.643 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2022-07-12T08:41:36.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi137.stderr:2022-07-12T08:41:36.644+0000 7fabe94fb080 -1 init, newargv = 0x555adf5c7800 newargc=9 2022-07-12T08:41:36.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi137.stderr:ceph-fuse[20968]: starting ceph client 2022-07-12T08:41:36.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi137.stderr:ceph-fuse[20968]: starting fuse 2022-07-12T08:41:36.656 INFO:teuthology.orchestra.run.smithi137.stdout:52 2022-07-12T08:41:36.656 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:41:36.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo python3 -c ' 2022-07-12T08:41:36.659 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2022-07-12T08:41:36.659 DEBUG:teuthology.orchestra.run.smithi137:> import re 2022-07-12T08:41:36.660 DEBUG:teuthology.orchestra.run.smithi137:> import os 2022-07-12T08:41:36.661 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2022-07-12T08:41:36.661 DEBUG:teuthology.orchestra.run.smithi137:> 2022-07-12T08:41:36.662 DEBUG:teuthology.orchestra.run.smithi137:> def find_socket(client_name): 2022-07-12T08:41:36.662 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2022-07-12T08:41:36.663 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2022-07-12T08:41:36.663 DEBUG:teuthology.orchestra.run.smithi137:> 2022-07-12T08:41:36.664 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2022-07-12T08:41:36.665 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2022-07-12T08:41:36.665 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2022-07-12T08:41:36.666 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2022-07-12T08:41:36.666 DEBUG:teuthology.orchestra.run.smithi137:> 2022-07-12T08:41:36.667 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2022-07-12T08:41:36.668 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:41:36.669 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:41:36.669 DEBUG:teuthology.orchestra.run.smithi137:> return f 2022-07-12T08:41:36.670 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:41:36.670 DEBUG:teuthology.orchestra.run.smithi137:> 2022-07-12T08:41:36.671 DEBUG:teuthology.orchestra.run.smithi137:> print(find_socket("client.2")) 2022-07-12T08:41:36.672 DEBUG:teuthology.orchestra.run.smithi137:> ' 2022-07-12T08:41:36.745 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.2.20968.asok 2022-07-12T08:41:36.750 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.20968.asok 2022-07-12T08:41:36.751 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.20968.asok status 2022-07-12T08:41:36.848 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2022-07-12T08:41:36.848 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2022-07-12T08:41:36.849 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:41:36.849 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:41:36.850 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "2", 2022-07-12T08:41:36.850 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2022-07-12T08:41:36.850 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2022-07-12T08:41:36.851 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "20968", 2022-07-12T08:41:36.852 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2022-07-12T08:41:36.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2022-07-12T08:41:36.853 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2022-07-12T08:41:36.853 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2022-07-12T08:41:36.853 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4666, 2022-07-12T08:41:36.854 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2022-07-12T08:41:36.854 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2022-07-12T08:41:36.855 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2022-07-12T08:41:36.855 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4666 2022-07-12T08:41:36.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2022-07-12T08:41:36.856 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2022-07-12T08:41:36.856 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2022-07-12T08:41:36.856 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:0", 2022-07-12T08:41:36.857 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3159769360 2022-07-12T08:41:36.857 INFO:teuthology.orchestra.run.smithi137.stdout: } 2022-07-12T08:41:36.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2022-07-12T08:41:36.858 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2022-07-12T08:41:36.858 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2022-07-12T08:41:36.859 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:0", 2022-07-12T08:41:36.859 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3159769360 2022-07-12T08:41:36.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2022-07-12T08:41:36.860 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4666 172.21.15.137:0/3159769360", 2022-07-12T08:41:36.860 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "172.21.15.137:0/3159769360", 2022-07-12T08:41:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2022-07-12T08:41:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 5, 2022-07-12T08:41:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 22, 2022-07-12T08:41:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:41:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "blacklisted": false, 2022-07-12T08:41:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2022-07-12T08:41:36.873 INFO:teuthology.orchestra.run.smithi137.stdout:} 2022-07-12T08:41:36.876 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.877 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:36.878 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:36.887 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:41:36.891 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:41:36.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.892 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:36.918 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.919 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:36.919 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:36.932 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:41:36.933 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:41:36.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:36.935 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:37.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:37.007 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:37.009 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:37.020 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2022-07-12T08:41:37.026 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-07-12T08:41:37.027 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:37.029 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:37.089 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2022-07-12T08:41:37.239 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:37.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.364+0000 7fb79b36b700 1 Processor -- start 2022-07-12T08:41:37.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.364+0000 7fb79b36b700 1 -- start start 2022-07-12T08:41:37.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940b1f80 0x7fb7940ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb7940afc70 0x7fb79405c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79405d480 con 0x7fb7940abdf0 2022-07-12T08:41:37.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7940b0fc0 con 0x7fb7940afc70 2022-07-12T08:41:37.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7940b2360 con 0x7fb7940b1f80 2022-07-12T08:41:37.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb7940afc70 0x7fb79405c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb7940afc70 0x7fb79405c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53574/0 (socket says 172.21.15.61:53574) 2022-07-12T08:41:37.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 -- 172.21.15.61:0/1710838721 learned_addr learned my addr 172.21.15.61:0/1710838721 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:37.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb798906700 1 --1- 172.21.15.61:0/1710838721 >> v1:172.21.15.61:6789/0 conn(0x7fb7940abdf0 0x7fb79405bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41788/0 (socket says 172.21.15.61:41788) 2022-07-12T08:41:37.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799107700 1 --2- 172.21.15.61:0/1710838721 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940b1f80 0x7fb7940ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb78bfff700 1 -- 172.21.15.61:0/1710838721 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078855375 0 0) 0x7fb79405d480 con 0x7fb7940abdf0 2022-07-12T08:41:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 -- 172.21.15.61:0/1710838721 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940b1f80 msgr2=0x7fb7940ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 --2- 172.21.15.61:0/1710838721 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940b1f80 0x7fb7940ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 -- 172.21.15.61:0/1710838721 >> v1:172.21.15.61:6789/0 conn(0x7fb7940abdf0 legacy=0x7fb79405bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 -- 172.21.15.61:0/1710838721 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7940b16d0 con 0x7fb7940afc70 2022-07-12T08:41:37.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb799908700 1 --2- 172.21.15.61:0/1710838721 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb79405c420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb78c0032f0 tx=0x7fb78c009290).ready entity=mon.0 client_cookie=6ac96cd4f0ec2a5a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb78bfff700 1 -- 172.21.15.61:0/1710838721 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb78c00b8a0 con 0x7fb7940afc70 2022-07-12T08:41:37.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb78bfff700 1 -- 172.21.15.61:0/1710838721 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78c00ba00 con 0x7fb7940afc70 2022-07-12T08:41:37.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb78bfff700 1 -- 172.21.15.61:0/1710838721 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb78c00bb70 con 0x7fb7940afc70 2022-07-12T08:41:37.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- 172.21.15.61:0/1710838721 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 msgr2=0x7fb79405c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1710838721 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb79405c420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb78c0032f0 tx=0x7fb78c009290).stop 2022-07-12T08:41:37.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- 172.21.15.61:0/1710838721 shutdown_connections 2022-07-12T08:41:37.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1710838721 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb79405c420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1710838721 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940b1f80 0x7fb7940ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- 172.21.15.61:0/1710838721 >> 172.21.15.61:0/1710838721 conn(0x7fb794062c00 msgr2=0x7fb794062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:37.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- 172.21.15.61:0/1710838721 shutdown_connections 2022-07-12T08:41:37.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.368+0000 7fb79b36b700 1 -- 172.21.15.61:0/1710838721 wait complete. 2022-07-12T08:41:37.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.372+0000 7fb79b36b700 1 Processor -- start 2022-07-12T08:41:37.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.372+0000 7fb79b36b700 1 -- start start 2022-07-12T08:41:37.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.372+0000 7fb79b36b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940abdf0 0x7fb79414a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.372+0000 7fb79b36b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb7941555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7940b1f80 0x7fb794147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb794145670 con 0x7fb7940afc70 2022-07-12T08:41:37.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7941453d0 con 0x7fb7940b1f80 2022-07-12T08:41:37.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb794146bc0 con 0x7fb7940abdf0 2022-07-12T08:41:37.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb7941555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb7941555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53578/0 (socket says 172.21.15.61:53578) 2022-07-12T08:41:37.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 -- 172.21.15.61:0/1440668891 learned_addr learned my addr 172.21.15.61:0/1440668891 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb799908700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7940b1f80 0x7fb794147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb799107700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940abdf0 0x7fb79414a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 -- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940abdf0 msgr2=0x7fb79414a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940abdf0 0x7fb79414a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 -- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7940b1f80 msgr2=0x7fb794147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7940b1f80 0x7fb794147060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 -- 172.21.15.61:0/1440668891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c009530 con 0x7fb7940afc70 2022-07-12T08:41:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb798906700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb7941555c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb790000fe0 tx=0x7fb790007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7900070d0 con 0x7fb7940afc70 2022-07-12T08:41:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb790007230 con 0x7fb7940afc70 2022-07-12T08:41:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb794155ab0 con 0x7fb7940afc70 2022-07-12T08:41:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb790005740 con 0x7fb7940afc70 2022-07-12T08:41:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7940673a0 con 0x7fb7940afc70 2022-07-12T08:41:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.376+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb794057460 con 0x7fb7940afc70 2022-07-12T08:41:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.380+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb79001a060 con 0x7fb7940afc70 2022-07-12T08:41:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.380+0000 7fb78a7fc700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb77c053ad0 0x7fb77c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.380+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb79006b720 con 0x7fb7940afc70 2022-07-12T08:41:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.380+0000 7fb799107700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb77c053ad0 0x7fb77c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.380+0000 7fb799107700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb77c053ad0 0x7fb77c055f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb78400c000 tx=0x7fb784008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.384+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb790042570 con 0x7fb7940afc70 2022-07-12T08:41:37.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.516+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7940447a0 con 0x7fb7940afc70 2022-07-12T08:41:37.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.516+0000 7fb78a7fc700 1 -- 172.21.15.61:0/1440668891 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fb790018020 con 0x7fb7940afc70 2022-07-12T08:41:37.521 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:37.521 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:37.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb77c053ad0 msgr2=0x7fb77c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb77c053ad0 0x7fb77c055f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fb78400c000 tx=0x7fb784008040).stop 2022-07-12T08:41:37.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 msgr2=0x7fb7941555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb7941555c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb790000fe0 tx=0x7fb790007b40).stop 2022-07-12T08:41:37.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 shutdown_connections 2022-07-12T08:41:37.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb77c053ad0 0x7fb77c055f90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7940b1f80 0x7fb794147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7940afc70 0x7fb7941555c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 --2- 172.21.15.61:0/1440668891 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7940abdf0 0x7fb79414a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 >> 172.21.15.61:0/1440668891 conn(0x7fb794062c00 msgr2=0x7fb7940b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:37.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 shutdown_connections 2022-07-12T08:41:37.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.520+0000 7fb79b36b700 1 -- 172.21.15.61:0/1440668891 wait complete. 2022-07-12T08:41:37.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:37.540 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:37.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.656+0000 7ff90bfff700 1 Processor -- start 2022-07-12T08:41:37.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90bfff700 1 -- start start 2022-07-12T08:41:37.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff90c060360 0x7ff90c066c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c0b07e0 con 0x7ff90c05c120 2022-07-12T08:41:37.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c140bd0 con 0x7ff90c060c20 2022-07-12T08:41:37.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90bfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c061530 con 0x7ff90c060360 2022-07-12T08:41:37.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff903fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff90c05c120 0x7ff90c0b1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41804/0 (socket says 172.21.15.61:41804) 2022-07-12T08:41:37.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff903fff700 1 -- 172.21.15.61:0/4101478430 learned_addr learned my addr 172.21.15.61:0/4101478430 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:37.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90b7fe700 1 --2- 172.21.15.61:0/4101478430 >> v2:172.21.15.111:3300/0 conn(0x7ff90c060360 0x7ff90c066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344491187 0 0) 0x7ff90c0b07e0 con 0x7ff90c05c120 2022-07-12T08:41:37.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec003260 con 0x7ff90c05c120 2022-07-12T08:41:37.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898354241 0 0) 0x7ff8fc001110 con 0x7ff90c060c20 2022-07-12T08:41:37.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff90c0b07e0 con 0x7ff90c060c20 2022-07-12T08:41:37.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90b7fe700 1 -- 172.21.15.61:0/4101478430 >> v1:172.21.15.111:6789/0 conn(0x7ff90c060c20 legacy=0x7ff90c0641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90b7fe700 1 -- 172.21.15.61:0/4101478430 >> v1:172.21.15.61:6789/0 conn(0x7ff90c05c120 legacy=0x7ff90c0b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90b7fe700 1 -- 172.21.15.61:0/4101478430 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c0b0540 con 0x7ff90c060360 2022-07-12T08:41:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.660+0000 7ff90b7fe700 1 --2- 172.21.15.61:0/4101478430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c066c00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff9040032f0 tx=0x7ff9040091d0).ready entity=mon.1 client_cookie=5a589085ef8fbc01 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff90400b600 con 0x7ff90c060360 2022-07-12T08:41:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90400b760 con 0x7ff90c060360 2022-07-12T08:41:37.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90a7fc700 1 -- 172.21.15.61:0/4101478430 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff90400b8d0 con 0x7ff90c060360 2022-07-12T08:41:37.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 -- 172.21.15.61:0/4101478430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 msgr2=0x7ff90c066c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 --2- 172.21.15.61:0/4101478430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c066c00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7ff9040032f0 tx=0x7ff9040091d0).stop 2022-07-12T08:41:37.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 -- 172.21.15.61:0/4101478430 shutdown_connections 2022-07-12T08:41:37.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 --2- 172.21.15.61:0/4101478430 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c066c00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 -- 172.21.15.61:0/4101478430 >> 172.21.15.61:0/4101478430 conn(0x7ff90c05f140 msgr2=0x7ff90c05f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 -- 172.21.15.61:0/4101478430 shutdown_connections 2022-07-12T08:41:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 -- 172.21.15.61:0/4101478430 wait complete. 2022-07-12T08:41:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.664+0000 7ff90bfff700 1 Processor -- start 2022-07-12T08:41:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 -- start start 2022-07-12T08:41:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff90c05c120 0x7ff90c1533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c09e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 0x7ff90c146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c09e7b0 con 0x7ff90c060c20 2022-07-12T08:41:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c09e9d0 con 0x7ff90c060360 2022-07-12T08:41:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c09ec70 con 0x7ff90c05c120 2022-07-12T08:41:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 0x7ff90c146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff903fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c09e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 0x7ff90c146a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53594/0 (socket says 172.21.15.61:53594) 2022-07-12T08:41:37.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 -- 172.21.15.61:0/2763579058 learned_addr learned my addr 172.21.15.61:0/2763579058 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:37.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff903fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c09e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:60396/0 (socket says 172.21.15.61:60396) 2022-07-12T08:41:37.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90affd700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff90c05c120 0x7ff90c1533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 -- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff90c05c120 msgr2=0x7ff90c1533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff90c05c120 0x7ff90c1533e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 -- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 msgr2=0x7ff90c09e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c09e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 -- 172.21.15.61:0/2763579058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff904009470 con 0x7ff90c060c20 2022-07-12T08:41:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90b7fe700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 0x7ff90c146a50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff9040032f0 tx=0x7ff904016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff90400b600 con 0x7ff90c060c20 2022-07-12T08:41:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff904011e60 con 0x7ff90c060c20 2022-07-12T08:41:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff90401f5e0 con 0x7ff90c060c20 2022-07-12T08:41:37.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c0a02c0 con 0x7ff90c060c20 2022-07-12T08:41:37.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.668+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c143fc0 con 0x7ff90c060c20 2022-07-12T08:41:37.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.672+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff904026030 con 0x7ff90c060c20 2022-07-12T08:41:37.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.672+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c0a0450 con 0x7ff90c060c20 2022-07-12T08:41:37.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.672+0000 7ff908ff9700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8f00539d0 0x7ff8f0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.672+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff90406c8c0 con 0x7ff90c060c20 2022-07-12T08:41:37.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.672+0000 7ff90affd700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8f00539d0 0x7ff8f0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.672+0000 7ff90affd700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8f00539d0 0x7ff8f0055e90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff8fc011000 tx=0x7ff8fc0079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.676+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff904043820 con 0x7ff90c060c20 2022-07-12T08:41:37.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.800+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff90c057460 con 0x7ff90c060c20 2022-07-12T08:41:37.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff908ff9700 1 -- 172.21.15.61:0/2763579058 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff904024020 con 0x7ff90c060c20 2022-07-12T08:41:37.809 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:37.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8f00539d0 msgr2=0x7ff8f0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8f00539d0 0x7ff8f0055e90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff8fc011000 tx=0x7ff8fc0079e0).stop 2022-07-12T08:41:37.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 msgr2=0x7ff90c146a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 0x7ff90c146a50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff9040032f0 tx=0x7ff904016000).stop 2022-07-12T08:41:37.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 shutdown_connections 2022-07-12T08:41:37.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff90c060c20 0x7ff90c146a50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8f00539d0 0x7ff8f0055e90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff90c060360 0x7ff90c09e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 --2- 172.21.15.61:0/2763579058 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff90c05c120 0x7ff90c1533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 >> 172.21.15.61:0/2763579058 conn(0x7ff90c05f140 msgr2=0x7ff90c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:37.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 shutdown_connections 2022-07-12T08:41:37.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.804+0000 7ff90bfff700 1 -- 172.21.15.61:0/2763579058 wait complete. 2022-07-12T08:41:37.821 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:37.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce92fa700 1 Processor -- start 2022-07-12T08:41:37.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce92fa700 1 -- start start 2022-07-12T08:41:37.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce92fa700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcce40b1a70 0x7fcce4060da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce92fa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40624d0 con 0x7fcce40b2b20 2022-07-12T08:41:37.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce92fa700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40b31c0 con 0x7fcce40620f0 2022-07-12T08:41:37.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce92fa700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce40b3460 con 0x7fcce40b1a70 2022-07-12T08:41:37.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce8af9700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcce40b2b20 0x7fcce4141f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41820/0 (socket says 172.21.15.61:41820) 2022-07-12T08:41:37.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce8af9700 1 -- 172.21.15.61:0/2211474235 learned_addr learned my addr 172.21.15.61:0/2211474235 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:37.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 --2- 172.21.15.61:0/2211474235 >> v2:172.21.15.111:3300/0 conn(0x7fcce40b1a70 0x7fcce4060da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480628655 0 0) 0x7fcce40624d0 con 0x7fcce40b2b20 2022-07-12T08:41:37.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd0003260 con 0x7fcce40b2b20 2022-07-12T08:41:37.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2471725854 0 0) 0x7fccd0003260 con 0x7fcce40b2b20 2022-07-12T08:41:37.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 >> v2:172.21.15.111:3300/0 conn(0x7fcce40b1a70 msgr2=0x7fcce4060da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:37.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 --2- 172.21.15.61:0/2211474235 >> v2:172.21.15.111:3300/0 conn(0x7fcce40b1a70 0x7fcce4060da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 >> v1:172.21.15.111:6789/0 conn(0x7fcce40620f0 legacy=0x7fcce41464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40b3700 con 0x7fcce40b2b20 2022-07-12T08:41:37.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fccdc001a00 con 0x7fcce40b2b20 2022-07-12T08:41:37.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 >> v1:172.21.15.61:6789/0 conn(0x7fcce40b2b20 legacy=0x7fcce4141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccd0003750 0x7fccd0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccd0004500 0x7fccd0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004e70 0x7fccd000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40b3460 con 0x7fccd0004500 2022-07-12T08:41:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd000bd20 con 0x7fccd0003750 2022-07-12T08:41:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd000bee0 con 0x7fccd0004e70 2022-07-12T08:41:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccd0004500 0x7fccd0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce8af9700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004e70 0x7fccd000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce3fff700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccd0003750 0x7fccd0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 -- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004e70 msgr2=0x7fccd000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004e70 0x7fccd000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 -- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccd0003750 msgr2=0x7fccd0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:37.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fccd0003750 0x7fccd0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 -- 172.21.15.61:0/2211474235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd0003260 con 0x7fccd0004500 2022-07-12T08:41:37.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce8af9700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004e70 0x7fccd000b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:37.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce8af9700 1 -- 172.21.15.61:0/2211474235 reap_dead start 2022-07-12T08:41:37.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce37fe700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccd0004500 0x7fccd0004980 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fccd800c000 tx=0x7fccd8006df0).ready entity=mon.0 client_cookie=68532c6c37c7ce79 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccd800e830 con 0x7fccd0004500 2022-07-12T08:41:37.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd800e990 con 0x7fccd0004500 2022-07-12T08:41:37.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.932+0000 7fcce2ffd700 1 -- 172.21.15.61:0/2211474235 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccd800ecf0 con 0x7fccd0004500 2022-07-12T08:41:37.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- 172.21.15.61:0/2211474235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccd0004500 msgr2=0x7fccd0004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccd0004500 0x7fccd0004980 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fccd800c000 tx=0x7fccd8006df0).stop 2022-07-12T08:41:37.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- 172.21.15.61:0/2211474235 shutdown_connections 2022-07-12T08:41:37.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 --2- 172.21.15.61:0/2211474235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccd0004500 0x7fccd0004980 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- 172.21.15.61:0/2211474235 >> 172.21.15.61:0/2211474235 conn(0x7fcce40b70d0 msgr2=0x7fcce40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:37.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- 172.21.15.61:0/2211474235 shutdown_connections 2022-07-12T08:41:37.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- 172.21.15.61:0/2211474235 wait complete. 2022-07-12T08:41:37.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 Processor -- start 2022-07-12T08:41:37.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- start start 2022-07-12T08:41:37.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004500 0x7fcce4067190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcce4148e10 0x7fcce40aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 0x7fcce4144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce40b10a0 con 0x7fcce41484d0 2022-07-12T08:41:37.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce4148030 con 0x7fcce4148e10 2022-07-12T08:41:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce92fa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce41479b0 con 0x7fccd0004500 2022-07-12T08:41:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 0x7fcce4144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 0x7fcce4144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53612/0 (socket says 172.21.15.61:53612) 2022-07-12T08:41:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 -- 172.21.15.61:0/709526420 learned_addr learned my addr 172.21.15.61:0/709526420 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce37fe700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcce4148e10 0x7fcce40aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce3fff700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004500 0x7fcce4067190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 -- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004500 msgr2=0x7fcce4067190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004500 0x7fcce4067190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 -- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcce4148e10 msgr2=0x7fcce40aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:37.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcce4148e10 0x7fcce40aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:37.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.936+0000 7fcce8af9700 1 -- 172.21.15.61:0/709526420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4147200 con 0x7fcce41484d0 2022-07-12T08:41:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce8af9700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 0x7fcce4144960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fccdc001210 tx=0x7fcce40601a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc0107e0 con 0x7fcce41484d0 2022-07-12T08:41:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc010940 con 0x7fcce41484d0 2022-07-12T08:41:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc010ca0 con 0x7fcce41484d0 2022-07-12T08:41:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce40b0400 con 0x7fcce41484d0 2022-07-12T08:41:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce4148b20 con 0x7fcce41484d0 2022-07-12T08:41:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce41473f0 con 0x7fcce41484d0 2022-07-12T08:41:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fccdc01a020 con 0x7fcce41484d0 2022-07-12T08:41:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce17fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fccc4053ad0 0x7fccc4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fccdc06bfc0 con 0x7fcce41484d0 2022-07-12T08:41:37.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.940+0000 7fcce3fff700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fccc4053ad0 0x7fccc4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:37.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.944+0000 7fcce3fff700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fccc4053ad0 0x7fccc4055f90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fccd400d000 tx=0x7fccd4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:37.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:37.948+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fccdc042e50 con 0x7fcce41484d0 2022-07-12T08:41:38.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.068+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcce41473f0 con 0x7fcce41484d0 2022-07-12T08:41:38.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.068+0000 7fcce17fa700 1 -- 172.21.15.61:0/709526420 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcce41473f0 con 0x7fcce41484d0 2022-07-12T08:41:38.075 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.068+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fccc4053ad0 msgr2=0x7fccc4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.068+0000 7fcce92fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fccc4053ad0 0x7fccc4055f90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fccd400d000 tx=0x7fccd4009040).stop 2022-07-12T08:41:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.068+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 msgr2=0x7fcce4144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.068+0000 7fcce92fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 0x7fcce4144960 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fccdc001210 tx=0x7fcce40601a0).stop 2022-07-12T08:41:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 shutdown_connections 2022-07-12T08:41:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcce41484d0 0x7fcce4144960 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fccc4053ad0 0x7fccc4055f90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcce4148e10 0x7fcce40aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 --2- 172.21.15.61:0/709526420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fccd0004500 0x7fcce4067190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 >> 172.21.15.61:0/709526420 conn(0x7fcce40b70d0 msgr2=0x7fcce405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 shutdown_connections 2022-07-12T08:41:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.072+0000 7fcce92fa700 1 -- 172.21.15.61:0/709526420 wait complete. 2022-07-12T08:41:38.092 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:38.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd40d960700 1 Processor -- start 2022-07-12T08:41:38.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd40d960700 1 -- start start 2022-07-12T08:41:38.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd40d960700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd408147eb0 0x7fd40814a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd40d960700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080b04e0 con 0x7fd408061300 2022-07-12T08:41:38.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd40d960700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080616e0 con 0x7fd408061ac0 2022-07-12T08:41:38.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd40d960700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40813f2c0 con 0x7fd408147eb0 2022-07-12T08:41:38.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd406ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd408061300 0x7fd40813fbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41836/0 (socket says 172.21.15.61:41836) 2022-07-12T08:41:38.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd406ffd700 1 -- 172.21.15.61:0/2255948019 learned_addr learned my addr 172.21.15.61:0/2255948019 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd4077fe700 1 --2- 172.21.15.61:0/2255948019 >> v2:172.21.15.61:3300/0 conn(0x7fd408147eb0 0x7fd40814a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd405ffb700 1 -- 172.21.15.61:0/2255948019 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261640789 0 0) 0x7fd3f8001110 con 0x7fd408061300 2022-07-12T08:41:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd405ffb700 1 -- 172.21.15.61:0/2255948019 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3ec003260 con 0x7fd408061300 2022-07-12T08:41:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd4077fe700 1 -- 172.21.15.61:0/2255948019 >> v1:172.21.15.111:6789/0 conn(0x7fd408061ac0 legacy=0x7fd4080b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd4077fe700 1 -- 172.21.15.61:0/2255948019 >> v1:172.21.15.61:6789/0 conn(0x7fd408061300 legacy=0x7fd40813fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.208+0000 7fd4077fe700 1 -- 172.21.15.61:0/2255948019 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40813f4b0 con 0x7fd408147eb0 2022-07-12T08:41:38.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd4077fe700 1 --2- 172.21.15.61:0/2255948019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd40814a2a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd3f80012a0 tx=0x7fd40000a370).ready entity=mon.0 client_cookie=e13e462bf9ebc10 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd405ffb700 1 -- 172.21.15.61:0/2255948019 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40000cb60 con 0x7fd408147eb0 2022-07-12T08:41:38.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd405ffb700 1 -- 172.21.15.61:0/2255948019 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000ccc0 con 0x7fd408147eb0 2022-07-12T08:41:38.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd405ffb700 1 -- 172.21.15.61:0/2255948019 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd400010600 con 0x7fd408147eb0 2022-07-12T08:41:38.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 -- 172.21.15.61:0/2255948019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 msgr2=0x7fd40814a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 --2- 172.21.15.61:0/2255948019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd40814a2a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd3f80012a0 tx=0x7fd40000a370).stop 2022-07-12T08:41:38.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 -- 172.21.15.61:0/2255948019 shutdown_connections 2022-07-12T08:41:38.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 --2- 172.21.15.61:0/2255948019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd40814a2a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 -- 172.21.15.61:0/2255948019 >> 172.21.15.61:0/2255948019 conn(0x7fd4080b5a40 msgr2=0x7fd4080b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 -- 172.21.15.61:0/2255948019 shutdown_connections 2022-07-12T08:41:38.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.212+0000 7fd40d960700 1 -- 172.21.15.61:0/2255948019 wait complete. 2022-07-12T08:41:38.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 Processor -- start 2022-07-12T08:41:38.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 -- start start 2022-07-12T08:41:38.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd408061300 0x7fd40813e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd408061ac0 0x7fd408064f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd408144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080671f0 con 0x7fd408147eb0 2022-07-12T08:41:38.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408066d90 con 0x7fd408061ac0 2022-07-12T08:41:38.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd408145850 con 0x7fd408061300 2022-07-12T08:41:38.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd408144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd408144f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53628/0 (socket says 172.21.15.61:53628) 2022-07-12T08:41:38.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 -- 172.21.15.61:0/3140704911 learned_addr learned my addr 172.21.15.61:0/3140704911 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:38.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd406ffd700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd408061300 0x7fd40813e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4067fc700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd408061ac0 0x7fd408064f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 -- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd408061300 msgr2=0x7fd40813e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd408061300 0x7fd40813e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 -- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd408061ac0 msgr2=0x7fd408064f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd408061ac0 0x7fd408064f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 -- 172.21.15.61:0/3140704911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40000a610 con 0x7fd408147eb0 2022-07-12T08:41:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd4077fe700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd408144f60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fd4081459e0 tx=0x7fd40813f640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd40001f070 con 0x7fd408147eb0 2022-07-12T08:41:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd400010e80 con 0x7fd408147eb0 2022-07-12T08:41:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4081455b0 con 0x7fd408147eb0 2022-07-12T08:41:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd400019660 con 0x7fd408147eb0 2022-07-12T08:41:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.216+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd408064b60 con 0x7fd408147eb0 2022-07-12T08:41:38.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.220+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd408057460 con 0x7fd408147eb0 2022-07-12T08:41:38.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.220+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd40000e040 con 0x7fd408147eb0 2022-07-12T08:41:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.220+0000 7fd3f7fff700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3e4055b30 0x7fd3e4057ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.220+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd40006d0f0 con 0x7fd408147eb0 2022-07-12T08:41:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.220+0000 7fd406ffd700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3e4055b30 0x7fd3e4057ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.220+0000 7fd406ffd700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3e4055b30 0x7fd3e4057ff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd3f800d000 tx=0x7fd3f80078c0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.228+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd400043f40 con 0x7fd408147eb0 2022-07-12T08:41:38.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd40806ac10 con 0x7fd408147eb0 2022-07-12T08:41:38.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd3f7fff700 1 -- 172.21.15.61:0/3140704911 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd400047560 con 0x7fd408147eb0 2022-07-12T08:41:38.362 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:38.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3e4055b30 msgr2=0x7fd3e4057ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3e4055b30 0x7fd3e4057ff0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd3f800d000 tx=0x7fd3f80078c0).stop 2022-07-12T08:41:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 msgr2=0x7fd408144f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd408144f60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fd4081459e0 tx=0x7fd40813f640).stop 2022-07-12T08:41:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 shutdown_connections 2022-07-12T08:41:38.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd408147eb0 0x7fd408144f60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3e4055b30 0x7fd3e4057ff0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd408061ac0 0x7fd408064f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 --2- 172.21.15.61:0/3140704911 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd408061300 0x7fd40813e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 >> 172.21.15.61:0/3140704911 conn(0x7fd4080b5a40 msgr2=0x7fd408141500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 shutdown_connections 2022-07-12T08:41:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.360+0000 7fd40d960700 1 -- 172.21.15.61:0/3140704911 wait complete. 2022-07-12T08:41:38.375 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:38.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.480+0000 7f0042c9d700 1 Processor -- start 2022-07-12T08:41:38.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.480+0000 7f0042c9d700 1 -- start start 2022-07-12T08:41:38.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f003c0abc70 0x7f003c0ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c05c090 con 0x7f003c0b0160 2022-07-12T08:41:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003c141120 con 0x7f003c0aefb0 2022-07-12T08:41:38.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003c0b0790 con 0x7f003c0abc70 2022-07-12T08:41:38.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004249c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f003c0abc70 0x7f003c0ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004149a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f003c0b0160 0x7f003c0b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41852/0 (socket says 172.21.15.61:41852) 2022-07-12T08:41:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004149a700 1 -- 172.21.15.61:0/1118343781 learned_addr learned my addr 172.21.15.61:0/1118343781 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004249c700 1 -- 172.21.15.61:0/1118343781 >> v1:172.21.15.111:6789/0 conn(0x7f003c0aefb0 legacy=0x7f003c140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004249c700 1 -- 172.21.15.61:0/1118343781 >> v1:172.21.15.61:6789/0 conn(0x7f003c0b0160 legacy=0x7f003c0b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004249c700 1 -- 172.21.15.61:0/1118343781 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003c0b0a30 con 0x7f003c0abc70 2022-07-12T08:41:38.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f004249c700 1 --2- 172.21.15.61:0/1118343781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0abc70 0x7f003c0ac0e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f00340032f0 tx=0x7f00340091d0).ready entity=mon.0 client_cookie=57726ac3d3b688f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0040c99700 1 -- 172.21.15.61:0/1118343781 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0034009ea0 con 0x7f003c0abc70 2022-07-12T08:41:38.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0040c99700 1 -- 172.21.15.61:0/1118343781 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003400b8a0 con 0x7f003c0abc70 2022-07-12T08:41:38.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0040c99700 1 -- 172.21.15.61:0/1118343781 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f003400ba10 con 0x7f003c0abc70 2022-07-12T08:41:38.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- 172.21.15.61:0/1118343781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0abc70 msgr2=0x7f003c0ac0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1118343781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0abc70 0x7f003c0ac0e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f00340032f0 tx=0x7f00340091d0).stop 2022-07-12T08:41:38.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- 172.21.15.61:0/1118343781 shutdown_connections 2022-07-12T08:41:38.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1118343781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0abc70 0x7f003c0ac0e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- 172.21.15.61:0/1118343781 >> 172.21.15.61:0/1118343781 conn(0x7f003c0b57c0 msgr2=0x7f003c0b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- 172.21.15.61:0/1118343781 shutdown_connections 2022-07-12T08:41:38.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- 172.21.15.61:0/1118343781 wait complete. 2022-07-12T08:41:38.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 Processor -- start 2022-07-12T08:41:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.484+0000 7f0042c9d700 1 -- start start 2022-07-12T08:41:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f003c0abc70 0x7f003c0b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f003c0aefb0 0x7f003c0ad810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 0x7f003c0add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c09b240 con 0x7f003c0b0160 2022-07-12T08:41:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003c144920 con 0x7f003c0abc70 2022-07-12T08:41:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003c144b90 con 0x7f003c0aefb0 2022-07-12T08:41:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 0x7f003c0add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0041c9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f003c0abc70 0x7f003c0b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 0x7f003c0add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53642/0 (socket says 172.21.15.61:53642) 2022-07-12T08:41:38.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 -- 172.21.15.61:0/1504313838 learned_addr learned my addr 172.21.15.61:0/1504313838 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004149a700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f003c0aefb0 0x7f003c0ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 -- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f003c0aefb0 msgr2=0x7f003c0ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f003c0aefb0 0x7f003c0ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 -- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f003c0abc70 msgr2=0x7f003c0b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f003c0abc70 0x7f003c0b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 -- 172.21.15.61:0/1504313838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034009470 con 0x7f003c0b0160 2022-07-12T08:41:38.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f004249c700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 0x7f003c0add00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f003c144ab0 tx=0x7f0034016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0034009ea0 con 0x7f003c0b0160 2022-07-12T08:41:38.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003401f3f0 con 0x7f003c0b0160 2022-07-12T08:41:38.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003c145160 con 0x7f003c0b0160 2022-07-12T08:41:38.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f003401f750 con 0x7f003c0b0160 2022-07-12T08:41:38.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003c0ae1f0 con 0x7f003c0b0160 2022-07-12T08:41:38.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.488+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c144d90 con 0x7f003c0b0160 2022-07-12T08:41:38.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.492+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0034026030 con 0x7f003c0b0160 2022-07-12T08:41:38.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.492+0000 7f002effd700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f001c053ad0 0x7f001c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.492+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f003406cdc0 con 0x7f003c0b0160 2022-07-12T08:41:38.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.492+0000 7f0041c9b700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f001c053ad0 0x7f001c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.492+0000 7f0041c9b700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f001c053ad0 0x7f001c055f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f003000d000 tx=0x7f0030000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.496+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0034043c10 con 0x7f003c0b0160 2022-07-12T08:41:38.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.623+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f003c144f20 con 0x7f003c0b0160 2022-07-12T08:41:38.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.623+0000 7f002effd700 1 -- 172.21.15.61:0/1504313838 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0034015020 con 0x7f003c0b0160 2022-07-12T08:41:38.638 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:38.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f001c053ad0 msgr2=0x7f001c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f001c053ad0 0x7f001c055f90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f003000d000 tx=0x7f0030000bc0).stop 2022-07-12T08:41:38.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 msgr2=0x7f003c0add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 0x7f003c0add00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f003c144ab0 tx=0x7f0034016000).stop 2022-07-12T08:41:38.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 shutdown_connections 2022-07-12T08:41:38.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f003c0b0160 0x7f003c0add00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f003c0aefb0 0x7f003c0ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f001c053ad0 0x7f001c055f90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 --2- 172.21.15.61:0/1504313838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f003c0abc70 0x7f003c0b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 >> 172.21.15.61:0/1504313838 conn(0x7f003c0b57c0 msgr2=0x7f003c0b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 shutdown_connections 2022-07-12T08:41:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.627+0000 7f0042c9d700 1 -- 172.21.15.61:0/1504313838 wait complete. 2022-07-12T08:41:38.650 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:38.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 Processor -- start 2022-07-12T08:41:38.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 -- start start 2022-07-12T08:41:38.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f059009bce0 0x7f05900af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05900aecb0 con 0x7f05900af280 2022-07-12T08:41:38.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0590143b00 con 0x7f059009bce0 2022-07-12T08:41:38.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059704e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05901435c0 con 0x7f05900b3f60 2022-07-12T08:41:38.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f059009bce0 0x7f05900af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f059009bce0 0x7f05900af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53648/0 (socket says 172.21.15.61:53648) 2022-07-12T08:41:38.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 -- 172.21.15.61:0/1371571633 learned_addr learned my addr 172.21.15.61:0/1371571633 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:38.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059684d700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 -- 172.21.15.61:0/1371571633 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 msgr2=0x7f05900b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:38.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 -- 172.21.15.61:0/1371571633 >> v1:172.21.15.61:6789/0 conn(0x7f05900af280 legacy=0x7f0590060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059584b700 1 -- 172.21.15.61:0/1371571633 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0590143860 con 0x7f059009bce0 2022-07-12T08:41:38.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.759+0000 7f059684d700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:38.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059584b700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f05900af660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f058c002f70 tx=0x7f058c006eb0).ready entity=mon.0 client_cookie=ea8a1bb1781b9886 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059504a700 1 -- 172.21.15.61:0/1371571633 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f058c00d5c0 con 0x7f059009bce0 2022-07-12T08:41:38.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059504a700 1 -- 172.21.15.61:0/1371571633 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c000e90 con 0x7f059009bce0 2022-07-12T08:41:38.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059504a700 1 -- 172.21.15.61:0/1371571633 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f058c00d950 con 0x7f059009bce0 2022-07-12T08:41:38.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- 172.21.15.61:0/1371571633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 msgr2=0x7f05900af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f05900af660 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f058c002f70 tx=0x7f058c006eb0).stop 2022-07-12T08:41:38.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- 172.21.15.61:0/1371571633 shutdown_connections 2022-07-12T08:41:38.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 --2- 172.21.15.61:0/1371571633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f05900af660 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- 172.21.15.61:0/1371571633 >> 172.21.15.61:0/1371571633 conn(0x7f05900b92c0 msgr2=0x7f05900b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- 172.21.15.61:0/1371571633 shutdown_connections 2022-07-12T08:41:38.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- 172.21.15.61:0/1371571633 wait complete. 2022-07-12T08:41:38.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 Processor -- start 2022-07-12T08:41:38.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- start start 2022-07-12T08:41:38.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f0590142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900af280 0x7f059009c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05900b08b0 con 0x7f059009bce0 2022-07-12T08:41:38.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059009dc80 con 0x7f05900af280 2022-07-12T08:41:38.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.763+0000 7f059704e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059009df20 con 0x7f05900b3f60 2022-07-12T08:41:38.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f0590142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f0590142160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53656/0 (socket says 172.21.15.61:53656) 2022-07-12T08:41:38.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 -- 172.21.15.61:0/810236171 learned_addr learned my addr 172.21.15.61:0/810236171 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:38.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059584b700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900af280 0x7f059009c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059684d700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 -- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 msgr2=0x7f05900b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 -- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900af280 msgr2=0x7f059009c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900af280 0x7f059009c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 -- 172.21.15.61:0/810236171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c00b1e0 con 0x7f059009bce0 2022-07-12T08:41:38.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059684d700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:38.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059604c700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f0590142160 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f05900b70b0 tx=0x7f0580007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0580013450 con 0x7f059009bce0 2022-07-12T08:41:38.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800135b0 con 0x7f059009bce0 2022-07-12T08:41:38.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059009e1c0 con 0x7f059009bce0 2022-07-12T08:41:38.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0580013910 con 0x7f059009bce0 2022-07-12T08:41:38.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059009ee40 con 0x7f059009bce0 2022-07-12T08:41:38.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.767+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05900b0a20 con 0x7f059009bce0 2022-07-12T08:41:38.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.771+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0580010070 con 0x7f059009bce0 2022-07-12T08:41:38.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.771+0000 7f057f7fe700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0570053920 0x7f0570055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:38.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.771+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f058006b610 con 0x7f059009bce0 2022-07-12T08:41:38.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.771+0000 7f059584b700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0570053920 0x7f0570055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:38.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.771+0000 7f059584b700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0570053920 0x7f0570055de0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f058c01c000 tx=0x7f058c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:38.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.775+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05900b0a20 con 0x7f059009bce0 2022-07-12T08:41:38.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.903+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05900b09f0 con 0x7f059009bce0 2022-07-12T08:41:38.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.903+0000 7f057f7fe700 1 -- 172.21.15.61:0/810236171 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f0580018070 con 0x7f059009bce0 2022-07-12T08:41:38.919 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:38.920 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:38.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0570053920 msgr2=0x7f0570055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0570053920 0x7f0570055de0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f058c01c000 tx=0x7f058c015000).stop 2022-07-12T08:41:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 msgr2=0x7f0590142160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f0590142160 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f05900b70b0 tx=0x7f0580007bd0).stop 2022-07-12T08:41:38.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 shutdown_connections 2022-07-12T08:41:38.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05900b3f60 0x7f05900b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0570053920 0x7f0570055de0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05900af280 0x7f059009c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 --2- 172.21.15.61:0/810236171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059009bce0 0x7f0590142160 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 >> 172.21.15.61:0/810236171 conn(0x7f05900b92c0 msgr2=0x7f05900b8200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 shutdown_connections 2022-07-12T08:41:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:38.907+0000 7f059704e700 1 -- 172.21.15.61:0/810236171 wait complete. 2022-07-12T08:41:38.934 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:38.935 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:39.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 Processor -- start 2022-07-12T08:41:39.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 -- start start 2022-07-12T08:41:39.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb0060360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2cb0060f60 0x7f2cb013f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb01435e0 con 0x7f2cb0143110 2022-07-12T08:41:39.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb00b2910 con 0x7f2cb0060f60 2022-07-12T08:41:39.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caf59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00618c0 con 0x7f2cb00b2530 2022-07-12T08:41:39.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cadd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2cb0143110 0x7f2cb0060850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41880/0 (socket says 172.21.15.61:41880) 2022-07-12T08:41:39.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cadd9b700 1 -- 172.21.15.61:0/2834917415 learned_addr learned my addr 172.21.15.61:0/2834917415 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:39.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caed9d700 1 --2- 172.21.15.61:0/2834917415 >> v2:172.21.15.111:3300/0 conn(0x7f2cb0060f60 0x7f2cb013f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cae59c700 1 --2- 172.21.15.61:0/2834917415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb0060360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cad59a700 1 -- 172.21.15.61:0/2834917415 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403958625 0 0) 0x7f2cb01435e0 con 0x7f2cb0143110 2022-07-12T08:41:39.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cad59a700 1 -- 172.21.15.61:0/2834917415 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c94004370 con 0x7f2cb0143110 2022-07-12T08:41:39.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cae59c700 1 -- 172.21.15.61:0/2834917415 >> v2:172.21.15.111:3300/0 conn(0x7f2cb0060f60 msgr2=0x7f2cb013f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cae59c700 1 --2- 172.21.15.61:0/2834917415 >> v2:172.21.15.111:3300/0 conn(0x7f2cb0060f60 0x7f2cb013f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cae59c700 1 -- 172.21.15.61:0/2834917415 >> v1:172.21.15.61:6789/0 conn(0x7f2cb0143110 legacy=0x7f2cb0060850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cae59c700 1 -- 172.21.15.61:0/2834917415 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb0061b60 con 0x7f2cb00b2530 2022-07-12T08:41:39.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2caed9d700 1 --2- 172.21.15.61:0/2834917415 >> v2:172.21.15.111:3300/0 conn(0x7f2cb0060f60 0x7f2cb013f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:39.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cae59c700 1 --2- 172.21.15.61:0/2834917415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb0060360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2ca0002f70 tx=0x7f2ca0006df0).ready entity=mon.2 client_cookie=3250caf16334f538 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cad59a700 1 -- 172.21.15.61:0/2834917415 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca000d4a0 con 0x7f2cb00b2530 2022-07-12T08:41:39.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cad59a700 1 -- 172.21.15.61:0/2834917415 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca0000e90 con 0x7f2cb00b2530 2022-07-12T08:41:39.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.067+0000 7f2cad59a700 1 -- 172.21.15.61:0/2834917415 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca000d7f0 con 0x7f2cb00b2530 2022-07-12T08:41:39.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 -- 172.21.15.61:0/2834917415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 msgr2=0x7f2cb0060360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 --2- 172.21.15.61:0/2834917415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb0060360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f2ca0002f70 tx=0x7f2ca0006df0).stop 2022-07-12T08:41:39.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 -- 172.21.15.61:0/2834917415 shutdown_connections 2022-07-12T08:41:39.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 --2- 172.21.15.61:0/2834917415 >> v2:172.21.15.111:3300/0 conn(0x7f2cb0060f60 0x7f2cb013f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 --2- 172.21.15.61:0/2834917415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb0060360 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 -- 172.21.15.61:0/2834917415 >> 172.21.15.61:0/2834917415 conn(0x7f2cb005f140 msgr2=0x7f2cb005f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 -- 172.21.15.61:0/2834917415 shutdown_connections 2022-07-12T08:41:39.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 -- 172.21.15.61:0/2834917415 wait complete. 2022-07-12T08:41:39.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 Processor -- start 2022-07-12T08:41:39.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.071+0000 7f2caf59e700 1 -- start start 2022-07-12T08:41:39.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 0x7f2cb0066470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb013d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb0143110 0x7f2cb013d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0137650 con 0x7f2cb0060f60 2022-07-12T08:41:39.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00afb10 con 0x7f2cb0143110 2022-07-12T08:41:39.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0065dd0 con 0x7f2cb00b2530 2022-07-12T08:41:39.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 0x7f2cb0066470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 0x7f2cb0066470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53666/0 (socket says 172.21.15.61:53666) 2022-07-12T08:41:39.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 -- 172.21.15.61:0/3152824840 learned_addr learned my addr 172.21.15.61:0/3152824840 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:39.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cadd9b700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb013d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caed9d700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb0143110 0x7f2cb013d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 -- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 msgr2=0x7f2cb013d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb013d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 -- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb0143110 msgr2=0x7f2cb013d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb0143110 0x7f2cb013d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 -- 172.21.15.61:0/3152824840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca8009470 con 0x7f2cb0060f60 2022-07-12T08:41:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2cae59c700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 0x7f2cb0066470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2cb0065f60 tx=0x7f2ca0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca00139c0 con 0x7f2cb0060f60 2022-07-12T08:41:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca0013b20 con 0x7f2cb0060f60 2022-07-12T08:41:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca0013e80 con 0x7f2cb0060f60 2022-07-12T08:41:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca000b180 con 0x7f2cb0060f60 2022-07-12T08:41:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.075+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb0066960 con 0x7f2cb0060f60 2022-07-12T08:41:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.079+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb013bd50 con 0x7f2cb0060f60 2022-07-12T08:41:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.079+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2ca0025030 con 0x7f2cb0060f60 2022-07-12T08:41:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.079+0000 7f2c9f7fe700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c8c053920 0x7f2c8c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.079+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2ca006cb90 con 0x7f2cb0060f60 2022-07-12T08:41:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.079+0000 7f2cadd9b700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c8c053920 0x7f2c8c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.079+0000 7f2cadd9b700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c8c053920 0x7f2c8c055de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2ca400d000 tx=0x7f2ca4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.083+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2ca0043af0 con 0x7f2cb0060f60 2022-07-12T08:41:39.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f2cb0066bb0 con 0x7f2cb0060f60 2022-07-12T08:41:39.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2c9f7fe700 1 -- 172.21.15.61:0/3152824840 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f2ca0023070 con 0x7f2cb0060f60 2022-07-12T08:41:39.218 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:39.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c8c053920 msgr2=0x7f2c8c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c8c053920 0x7f2c8c055de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2ca400d000 tx=0x7f2ca4009040).stop 2022-07-12T08:41:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 msgr2=0x7f2cb0066470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 0x7f2cb0066470 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f2cb0065f60 tx=0x7f2ca0014000).stop 2022-07-12T08:41:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 shutdown_connections 2022-07-12T08:41:39.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c8c053920 0x7f2c8c055de0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb0143110 0x7f2cb013d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b2530 0x7f2cb013d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 --2- 172.21.15.61:0/3152824840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb0060f60 0x7f2cb0066470 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 >> 172.21.15.61:0/3152824840 conn(0x7f2cb005f140 msgr2=0x7f2cb0142860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 shutdown_connections 2022-07-12T08:41:39.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.215+0000 7f2caf59e700 1 -- 172.21.15.61:0/3152824840 wait complete. 2022-07-12T08:41:39.233 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:39.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 Processor -- start 2022-07-12T08:41:39.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 -- start start 2022-07-12T08:41:39.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efdb80b0280 0x7efdb814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80aa830 0x7efdb80aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb805c3e0 con 0x7efdb80b5920 2022-07-12T08:41:39.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80b5d00 con 0x7efdb80b0280 2022-07-12T08:41:39.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdbc9fd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb805c610 con 0x7efdb80aa830 2022-07-12T08:41:39.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80aa830 0x7efdb80aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efdb80b0280 0x7efdb814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb6d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7efdb80b5920 0x7efdb814a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:55494/0 (socket says 172.21.15.61:55494) 2022-07-12T08:41:39.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb6d9d700 1 -- 172.21.15.61:0/3501069837 learned_addr learned my addr 172.21.15.61:0/3501069837 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:39.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb5d9b700 1 -- 172.21.15.61:0/3501069837 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108322644 0 0) 0x7efdb805c3e0 con 0x7efdb80b5920 2022-07-12T08:41:39.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb759e700 1 -- 172.21.15.61:0/3501069837 >> v2:172.21.15.111:3300/0 conn(0x7efdb80b0280 msgr2=0x7efdb814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb759e700 1 --2- 172.21.15.61:0/3501069837 >> v2:172.21.15.111:3300/0 conn(0x7efdb80b0280 0x7efdb814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb759e700 1 -- 172.21.15.61:0/3501069837 >> v1:172.21.15.111:6789/0 conn(0x7efdb80b5920 legacy=0x7efdb814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb759e700 1 -- 172.21.15.61:0/3501069837 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb805c8b0 con 0x7efdb80aa830 2022-07-12T08:41:39.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb759e700 1 --2- 172.21.15.61:0/3501069837 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80aa830 0x7efdb80aac60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efda8002f70 tx=0x7efda8006df0).ready entity=mon.2 client_cookie=7560917058585caa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.339+0000 7efdb5d9b700 1 -- 172.21.15.61:0/3501069837 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efda800d570 con 0x7efdb80aa830 2022-07-12T08:41:39.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb5d9b700 1 -- 172.21.15.61:0/3501069837 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda8000e90 con 0x7efdb80aa830 2022-07-12T08:41:39.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb5d9b700 1 -- 172.21.15.61:0/3501069837 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efda800d8c0 con 0x7efdb80aa830 2022-07-12T08:41:39.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- 172.21.15.61:0/3501069837 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80aa830 msgr2=0x7efdb80aac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/3501069837 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80aa830 0x7efdb80aac60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efda8002f70 tx=0x7efda8006df0).stop 2022-07-12T08:41:39.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- 172.21.15.61:0/3501069837 shutdown_connections 2022-07-12T08:41:39.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/3501069837 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80aa830 0x7efdb80aac60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/3501069837 >> v2:172.21.15.111:3300/0 conn(0x7efdb80b0280 0x7efdb814b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- 172.21.15.61:0/3501069837 >> 172.21.15.61:0/3501069837 conn(0x7efdb8063340 msgr2=0x7efdb80b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- 172.21.15.61:0/3501069837 shutdown_connections 2022-07-12T08:41:39.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- 172.21.15.61:0/3501069837 wait complete. 2022-07-12T08:41:39.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 Processor -- start 2022-07-12T08:41:39.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- start start 2022-07-12T08:41:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 0x7efdb808ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb80b0280 0x7efdb8159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80b5920 0x7efdb8064b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb815a180 con 0x7efdb80aa830 2022-07-12T08:41:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81592b0 con 0x7efdb80b0280 2022-07-12T08:41:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdbc9fd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb8159550 con 0x7efdb80b5920 2022-07-12T08:41:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 0x7efdb808ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 0x7efdb808ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53680/0 (socket says 172.21.15.61:53680) 2022-07-12T08:41:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb6d9d700 1 -- 172.21.15.61:0/1888964490 learned_addr learned my addr 172.21.15.61:0/1888964490 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:39.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb659c700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb80b0280 0x7efdb8159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.343+0000 7efdb759e700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80b5920 0x7efdb8064b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb6d9d700 1 -- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80b5920 msgr2=0x7efdb8064b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb6d9d700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80b5920 0x7efdb8064b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb6d9d700 1 -- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb80b0280 msgr2=0x7efdb8159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb6d9d700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb80b0280 0x7efdb8159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb6d9d700 1 -- 172.21.15.61:0/1888964490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda800b180 con 0x7efdb80aa830 2022-07-12T08:41:39.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb759e700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80b5920 0x7efdb8064b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:39.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb6d9d700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 0x7efdb808ff20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7efdb0001380 tx=0x7efdb0001490).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdb0005b60 con 0x7efdb80aa830 2022-07-12T08:41:39.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb0005cc0 con 0x7efdb80aa830 2022-07-12T08:41:39.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb81597f0 con 0x7efdb80aa830 2022-07-12T08:41:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdb00072c0 con 0x7efdb80aa830 2022-07-12T08:41:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb8065080 con 0x7efdb80aa830 2022-07-12T08:41:39.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efdb001a060 con 0x7efdb80aa830 2022-07-12T08:41:39.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb8159980 con 0x7efdb80aa830 2022-07-12T08:41:39.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdaffff700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efd9c053a60 0x7efd9c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdb006b9e0 con 0x7efdb80aa830 2022-07-12T08:41:39.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb659c700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efd9c053a60 0x7efd9c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.347+0000 7efdb659c700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efd9c053a60 0x7efd9c055f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7efda000b000 tx=0x7efda0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.355+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efdb00428c0 con 0x7efdb80aa830 2022-07-12T08:41:39.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.475+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efdb806b2b0 con 0x7efdb80aa830 2022-07-12T08:41:39.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.475+0000 7efdaffff700 1 -- 172.21.15.61:0/1888964490 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7efdb0018030 con 0x7efdb80aa830 2022-07-12T08:41:39.484 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:39.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efd9c053a60 msgr2=0x7efd9c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efd9c053a60 0x7efd9c055f20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7efda000b000 tx=0x7efda0006df0).stop 2022-07-12T08:41:39.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 msgr2=0x7efdb808ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 0x7efdb808ff20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7efdb0001380 tx=0x7efdb0001490).stop 2022-07-12T08:41:39.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 shutdown_connections 2022-07-12T08:41:39.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efdb80b5920 0x7efdb8064b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efd9c053a60 0x7efd9c055f20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efdb80b0280 0x7efdb8159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 --2- 172.21.15.61:0/1888964490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdb80aa830 0x7efdb808ff20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 >> 172.21.15.61:0/1888964490 conn(0x7efdb8063340 msgr2=0x7efdb80b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 shutdown_connections 2022-07-12T08:41:39.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.479+0000 7efdbc9fd700 1 -- 172.21.15.61:0/1888964490 wait complete. 2022-07-12T08:41:39.497 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:39.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.615+0000 7f749c4b8700 1 Processor -- start 2022-07-12T08:41:39.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.615+0000 7f749c4b8700 1 -- start start 2022-07-12T08:41:39.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749c4b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f7494143120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749c4b8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f74940a9b40 0x7f74940b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749c4b8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749405f650 con 0x7f74940613f0 2022-07-12T08:41:39.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749c4b8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74940b1dc0 con 0x7f74940a9b40 2022-07-12T08:41:39.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749c4b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7494143610 con 0x7f74940b36c0 2022-07-12T08:41:39.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499a53700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f74940613f0 0x7f7494143ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41906/0 (socket says 172.21.15.61:41906) 2022-07-12T08:41:39.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499a53700 1 -- 172.21.15.61:0/3258575277 learned_addr learned my addr 172.21.15.61:0/3258575277 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:39.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f7494143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749aa55700 1 --2- 172.21.15.61:0/3258575277 >> v2:172.21.15.111:3300/0 conn(0x7f74940a9b40 0x7f74940b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860560732 0 0) 0x7f749405f650 con 0x7f74940613f0 2022-07-12T08:41:39.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f747c003260 con 0x7f74940613f0 2022-07-12T08:41:39.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 854873226 0 0) 0x7f747c003260 con 0x7f74940613f0 2022-07-12T08:41:39.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 msgr2=0x7f7494143120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f7494143120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 >> v2:172.21.15.111:3300/0 conn(0x7f74940a9b40 msgr2=0x7f74940b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 --2- 172.21.15.61:0/3258575277 >> v2:172.21.15.111:3300/0 conn(0x7f74940a9b40 0x7f74940b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749405f2d0 con 0x7f74940613f0 2022-07-12T08:41:39.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f7484001660 con 0x7f74940613f0 2022-07-12T08:41:39.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 >> v1:172.21.15.61:6789/0 conn(0x7f74940613f0 legacy=0x7f7494143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f747c003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c003c80 0x7f747c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0049f0 0x7f747c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74940b1dc0 con 0x7f747c0049f0 2022-07-12T08:41:39.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494143610 con 0x7f747c003c80 2022-07-12T08:41:39.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c00b750 con 0x7f74940b36c0 2022-07-12T08:41:39.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749aa55700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0049f0 0x7f747c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499a53700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c003c80 0x7f747c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f747c003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 -- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c003c80 msgr2=0x7f747c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c003c80 0x7f747c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 -- 172.21.15.61:0/3258575277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0049f0 msgr2=0x7f747c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0049f0 0x7f747c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 -- 172.21.15.61:0/3258575277 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c003260 con 0x7f74940b36c0 2022-07-12T08:41:39.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749aa55700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0049f0 0x7f747c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:39.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f749a254700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f747c003750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f749000c000 tx=0x7f7490006df0).ready entity=mon.2 client_cookie=450964dec356513a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f749000e830 con 0x7f74940b36c0 2022-07-12T08:41:39.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749000e990 con 0x7f74940b36c0 2022-07-12T08:41:39.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.619+0000 7f7499252700 1 -- 172.21.15.61:0/3258575277 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f749000ecf0 con 0x7f74940b36c0 2022-07-12T08:41:39.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 -- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 msgr2=0x7f747c003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3258575277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74940b36c0 0x7f747c003750 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f749000c000 tx=0x7f7490006df0).stop 2022-07-12T08:41:39.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749aa55700 1 -- 172.21.15.61:0/3258575277 reap_dead start 2022-07-12T08:41:39.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 -- 172.21.15.61:0/3258575277 shutdown_connections 2022-07-12T08:41:39.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 -- 172.21.15.61:0/3258575277 >> 172.21.15.61:0/3258575277 conn(0x7f74940b6e40 msgr2=0x7f74940b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 -- 172.21.15.61:0/3258575277 shutdown_connections 2022-07-12T08:41:39.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 -- 172.21.15.61:0/3258575277 wait complete. 2022-07-12T08:41:39.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 Processor -- start 2022-07-12T08:41:39.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.623+0000 7f749c4b8700 1 -- start start 2022-07-12T08:41:39.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74940b36c0 0x7f74940afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7494147080 0x7f74940b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 0x7f749409b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494152020 con 0x7f74941534b0 2022-07-12T08:41:39.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7494152250 con 0x7f74940b36c0 2022-07-12T08:41:39.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74940ae400 con 0x7f7494147080 2022-07-12T08:41:39.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 0x7f749409b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 0x7f749409b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53704/0 (socket says 172.21.15.61:53704) 2022-07-12T08:41:39.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 -- 172.21.15.61:0/3260527495 learned_addr learned my addr 172.21.15.61:0/3260527495 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:39.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f7499a53700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7494147080 0x7f74940b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749a254700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74940b36c0 0x7f74940afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 -- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7494147080 msgr2=0x7f74940b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7494147080 0x7f74940b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 -- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74940b36c0 msgr2=0x7f74940afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74940b36c0 0x7f74940afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 -- 172.21.15.61:0/3260527495 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74940ae6a0 con 0x7f74941534b0 2022-07-12T08:41:39.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749aa55700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 0x7f749409b910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f748c00b000 tx=0x7f748c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f748c019070 con 0x7f74941534b0 2022-07-12T08:41:39.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f748c00f910 con 0x7f74941534b0 2022-07-12T08:41:39.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74940ae0c0 con 0x7f74941534b0 2022-07-12T08:41:39.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f748c00fc70 con 0x7f74941534b0 2022-07-12T08:41:39.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.627+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749409c020 con 0x7f74941534b0 2022-07-12T08:41:39.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.631+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7494057460 con 0x7f74941534b0 2022-07-12T08:41:39.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.631+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f748c014440 con 0x7f74941534b0 2022-07-12T08:41:39.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.631+0000 7f74837fe700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7474053af0 0x7f7474055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.631+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f748c06c8f0 con 0x7f74941534b0 2022-07-12T08:41:39.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.631+0000 7f749a254700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7474053af0 0x7f7474055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.631+0000 7f749a254700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7474053af0 0x7f7474055fb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f749001e000 tx=0x7f7490019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.639+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f748c0436e0 con 0x7f74941534b0 2022-07-12T08:41:39.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.763+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f749406ac80 con 0x7f74941534b0 2022-07-12T08:41:39.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f74837fe700 1 -- 172.21.15.61:0/3260527495 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7f748c046d00 con 0x7f74941534b0 2022-07-12T08:41:39.776 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:39.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7474053af0 msgr2=0x7f7474055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7474053af0 0x7f7474055fb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f749001e000 tx=0x7f7490019000).stop 2022-07-12T08:41:39.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 msgr2=0x7f749409b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 0x7f749409b910 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f748c00b000 tx=0x7f748c0091d0).stop 2022-07-12T08:41:39.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 shutdown_connections 2022-07-12T08:41:39.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74941534b0 0x7f749409b910 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7494147080 0x7f74940b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7474053af0 0x7f7474055fb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 --2- 172.21.15.61:0/3260527495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74940b36c0 0x7f74940afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 >> 172.21.15.61:0/3260527495 conn(0x7f74940b6e40 msgr2=0x7f7494143920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 shutdown_connections 2022-07-12T08:41:39.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.767+0000 7f749c4b8700 1 -- 172.21.15.61:0/3260527495 wait complete. 2022-07-12T08:41:39.829 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:39.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.955+0000 7f87b759e700 1 Processor -- start 2022-07-12T08:41:39.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.955+0000 7f87b759e700 1 -- start start 2022-07-12T08:41:39.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87b80b0240 0x7f87b80b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f87b8142f70 0x7f87b81433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b80abfa0 con 0x7f87b80b5920 2022-07-12T08:41:39.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b805c870 con 0x7f87b8142f70 2022-07-12T08:41:39.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b80b0ba0 con 0x7f87b80b0240 2022-07-12T08:41:39.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f87b8142f70 0x7f87b81433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f87b8142f70 0x7f87b81433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53712/0 (socket says 172.21.15.61:53712) 2022-07-12T08:41:39.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 -- 172.21.15.61:0/1587593421 learned_addr learned my addr 172.21.15.61:0/1587593421 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:39.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b5d9b700 1 --2- 172.21.15.61:0/1587593421 >> v2:172.21.15.111:3300/0 conn(0x7f87b80b0240 0x7f87b80b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 -- 172.21.15.61:0/1587593421 >> v2:172.21.15.111:3300/0 conn(0x7f87b80b0240 msgr2=0x7f87b80b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:39.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 --2- 172.21.15.61:0/1587593421 >> v2:172.21.15.111:3300/0 conn(0x7f87b80b0240 0x7f87b80b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 -- 172.21.15.61:0/1587593421 >> v1:172.21.15.111:6789/0 conn(0x7f87b80b5920 legacy=0x7f87b80aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 -- 172.21.15.61:0/1587593421 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b805cd00 con 0x7f87b8142f70 2022-07-12T08:41:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b6d9d700 1 --2- 172.21.15.61:0/1587593421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b81433e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f87b80b0ce0 tx=0x7f87b0009290).ready entity=mon.0 client_cookie=5ff003430d0de505 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b559a700 1 -- 172.21.15.61:0/1587593421 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b000b830 con 0x7f87b8142f70 2022-07-12T08:41:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b559a700 1 -- 172.21.15.61:0/1587593421 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b000b990 con 0x7f87b8142f70 2022-07-12T08:41:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b559a700 1 -- 172.21.15.61:0/1587593421 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b000bb00 con 0x7f87b8142f70 2022-07-12T08:41:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- 172.21.15.61:0/1587593421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 msgr2=0x7f87b81433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 --2- 172.21.15.61:0/1587593421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b81433e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f87b80b0ce0 tx=0x7f87b0009290).stop 2022-07-12T08:41:39.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- 172.21.15.61:0/1587593421 shutdown_connections 2022-07-12T08:41:39.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 --2- 172.21.15.61:0/1587593421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b81433e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 --2- 172.21.15.61:0/1587593421 >> v2:172.21.15.111:3300/0 conn(0x7f87b80b0240 0x7f87b80b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- 172.21.15.61:0/1587593421 >> 172.21.15.61:0/1587593421 conn(0x7f87b8063340 msgr2=0x7f87b80b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- 172.21.15.61:0/1587593421 shutdown_connections 2022-07-12T08:41:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.959+0000 7f87b759e700 1 -- 172.21.15.61:0/1587593421 wait complete. 2022-07-12T08:41:39.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 Processor -- start 2022-07-12T08:41:39.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 -- start start 2022-07-12T08:41:39.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 0x7f87b80ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87b80b5920 0x7f87b8148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b8096e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b8156550 con 0x7f87b8142f70 2022-07-12T08:41:39.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b80ac740 con 0x7f87b80b5920 2022-07-12T08:41:39.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b80ad5f0 con 0x7f87b80b0240 2022-07-12T08:41:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b8096e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b659c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 0x7f87b80ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87b80b5920 0x7f87b8148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b659c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 0x7f87b80ae1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:35910/0 (socket says 172.21.15.61:35910) 2022-07-12T08:41:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b659c700 1 -- 172.21.15.61:0/3676991408 learned_addr learned my addr 172.21.15.61:0/3676991408 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b6d9d700 1 -- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 msgr2=0x7f87b80ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b6d9d700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 0x7f87b80ae1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b6d9d700 1 -- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87b80b5920 msgr2=0x7f87b8148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b6d9d700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87b80b5920 0x7f87b8148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b6d9d700 1 -- 172.21.15.61:0/3676991408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b0009530 con 0x7f87b8142f70 2022-07-12T08:41:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.963+0000 7f87b659c700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 0x7f87b80ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87b6d9d700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b8096e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f87b80997c0 tx=0x7f87b0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b00032f0 con 0x7f87b8142f70 2022-07-12T08:41:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b00203f0 con 0x7f87b8142f70 2022-07-12T08:41:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b805cad0 con 0x7f87b8142f70 2022-07-12T08:41:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b0020750 con 0x7f87b8142f70 2022-07-12T08:41:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b80af870 con 0x7f87b8142f70 2022-07-12T08:41:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f87b000f070 con 0x7f87b8142f70 2022-07-12T08:41:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b808fcb0 con 0x7f87b8142f70 2022-07-12T08:41:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87af7fe700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f879c053ad0 0x7f879c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f87b006bd40 con 0x7f87b8142f70 2022-07-12T08:41:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87b659c700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f879c053ad0 0x7f879c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.967+0000 7f87b659c700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f879c053ad0 0x7f879c055f90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f87a000d000 tx=0x7f87a0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:39.975+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f87b0042b90 con 0x7f87b8142f70 2022-07-12T08:41:40.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87b80ad3d0 con 0x7f87b8142f70 2022-07-12T08:41:40.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87af7fe700 1 -- 172.21.15.61:0/3676991408 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f87b0015030 con 0x7f87b8142f70 2022-07-12T08:41:40.117 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:40.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f879c053ad0 msgr2=0x7f879c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f879c053ad0 0x7f879c055f90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f87a000d000 tx=0x7f87a0009040).stop 2022-07-12T08:41:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 msgr2=0x7f87b8096e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b8096e70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f87b80997c0 tx=0x7f87b0016000).stop 2022-07-12T08:41:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 shutdown_connections 2022-07-12T08:41:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87b8142f70 0x7f87b8096e70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f879c053ad0 0x7f879c055f90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87b80b5920 0x7f87b8148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 --2- 172.21.15.61:0/3676991408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87b80b0240 0x7f87b80ae1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.107+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 >> 172.21.15.61:0/3676991408 conn(0x7f87b8063340 msgr2=0x7f87b8143380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.111+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 shutdown_connections 2022-07-12T08:41:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.111+0000 7f87b759e700 1 -- 172.21.15.61:0/3676991408 wait complete. 2022-07-12T08:41:40.123 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:40.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.219+0000 7fd70e352700 1 Processor -- start 2022-07-12T08:41:40.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.219+0000 7fd70e352700 1 -- start start 2022-07-12T08:41:40.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7080b1f80 0x7fd7080ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080afc70 0x7fd70805c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70805d480 con 0x7fd7080abdf0 2022-07-12T08:41:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7080b0fc0 con 0x7fd7080b1f80 2022-07-12T08:41:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080b2360 con 0x7fd7080afc70 2022-07-12T08:41:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd6fffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd7080abdf0 0x7fd70805bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41940/0 (socket says 172.21.15.61:41940) 2022-07-12T08:41:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd6fffff700 1 -- 172.21.15.61:0/3785243862 learned_addr learned my addr 172.21.15.61:0/3785243862 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd707fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7080b1f80 0x7fd7080ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70c8ef700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080afc70 0x7fd70805c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd7077fe700 1 -- 172.21.15.61:0/3785243862 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208787828 0 0) 0x7fd70805d480 con 0x7fd7080abdf0 2022-07-12T08:41:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd7077fe700 1 -- 172.21.15.61:0/3785243862 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e8003260 con 0x7fd7080abdf0 2022-07-12T08:41:40.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd707fff700 1 -- 172.21.15.61:0/3785243862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080afc70 msgr2=0x7fd70805c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:40.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd707fff700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080afc70 0x7fd70805c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd707fff700 1 -- 172.21.15.61:0/3785243862 >> v1:172.21.15.61:6789/0 conn(0x7fd7080abdf0 legacy=0x7fd70805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd707fff700 1 -- 172.21.15.61:0/3785243862 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7080b16d0 con 0x7fd7080b1f80 2022-07-12T08:41:40.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70c8ef700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080afc70 0x7fd70805c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:40.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd707fff700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080b1f80 0x7fd7080ab900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd6f8002f70 tx=0x7fd6f8006df0).ready entity=mon.0 client_cookie=df7f6860cd4144c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd7077fe700 1 -- 172.21.15.61:0/3785243862 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f800d570 con 0x7fd7080b1f80 2022-07-12T08:41:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd7077fe700 1 -- 172.21.15.61:0/3785243862 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f8000e90 con 0x7fd7080b1f80 2022-07-12T08:41:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd7077fe700 1 -- 172.21.15.61:0/3785243862 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f800d900 con 0x7fd7080b1f80 2022-07-12T08:41:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- 172.21.15.61:0/3785243862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080b1f80 msgr2=0x7fd7080ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080b1f80 0x7fd7080ab900 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd6f8002f70 tx=0x7fd6f8006df0).stop 2022-07-12T08:41:40.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- 172.21.15.61:0/3785243862 shutdown_connections 2022-07-12T08:41:40.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080afc70 0x7fd70805c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 --2- 172.21.15.61:0/3785243862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080b1f80 0x7fd7080ab900 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- 172.21.15.61:0/3785243862 >> 172.21.15.61:0/3785243862 conn(0x7fd708062c00 msgr2=0x7fd708062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:40.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- 172.21.15.61:0/3785243862 shutdown_connections 2022-07-12T08:41:40.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- 172.21.15.61:0/3785243862 wait complete. 2022-07-12T08:41:40.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 Processor -- start 2022-07-12T08:41:40.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.223+0000 7fd70e352700 1 -- start start 2022-07-12T08:41:40.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 0x7fd708147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080afc70 0x7fd7081470d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080b1f80 0x7fd708152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd70813f870 con 0x7fd7080abdf0 2022-07-12T08:41:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd708066f80 con 0x7fd7080afc70 2022-07-12T08:41:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7080676f0 con 0x7fd7080b1f80 2022-07-12T08:41:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 0x7fd708147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 0x7fd708147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53730/0 (socket says 172.21.15.61:53730) 2022-07-12T08:41:40.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 -- 172.21.15.61:0/3578708334 learned_addr learned my addr 172.21.15.61:0/3578708334 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:40.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd6fffff700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080afc70 0x7fd7081470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70c8ef700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080b1f80 0x7fd708152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 -- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080b1f80 msgr2=0x7fd708152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080b1f80 0x7fd708152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 -- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080afc70 msgr2=0x7fd7081470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080afc70 0x7fd7081470d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 -- 172.21.15.61:0/3578708334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f800b180 con 0x7fd7080abdf0 2022-07-12T08:41:40.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70c8ef700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080b1f80 0x7fd708152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:40.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd707fff700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 0x7fd708147b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fd6f8002f70 tx=0x7fd6f8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f8006b20 con 0x7fd7080abdf0 2022-07-12T08:41:40.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f8013e90 con 0x7fd7080abdf0 2022-07-12T08:41:40.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f801e5f0 con 0x7fd7080abdf0 2022-07-12T08:41:40.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd708066ae0 con 0x7fd7080abdf0 2022-07-12T08:41:40.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd708147760 con 0x7fd7080abdf0 2022-07-12T08:41:40.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd708147980 con 0x7fd7080abdf0 2022-07-12T08:41:40.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd6f801c070 con 0x7fd7080abdf0 2022-07-12T08:41:40.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd705ffb700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6ec053ad0 0x7fd6ec055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.227+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd6f806c890 con 0x7fd7080abdf0 2022-07-12T08:41:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.231+0000 7fd6fffff700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6ec053ad0 0x7fd6ec055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.231+0000 7fd6fffff700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6ec053ad0 0x7fd6ec055f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fd6f000d000 tx=0x7fd6f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.235+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd6f8043830 con 0x7fd7080abdf0 2022-07-12T08:41:40.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.367+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd708144770 con 0x7fd7080abdf0 2022-07-12T08:41:40.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.367+0000 7fd705ffb700 1 -- 172.21.15.61:0/3578708334 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fd6f8023020 con 0x7fd7080abdf0 2022-07-12T08:41:40.540 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:40.550 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:40.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6ec053ad0 msgr2=0x7fd6ec055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6ec053ad0 0x7fd6ec055f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fd6f000d000 tx=0x7fd6f0009040).stop 2022-07-12T08:41:40.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 msgr2=0x7fd708147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 0x7fd708147b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fd6f8002f70 tx=0x7fd6f8014000).stop 2022-07-12T08:41:40.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 shutdown_connections 2022-07-12T08:41:40.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7080b1f80 0x7fd708152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6ec053ad0 0x7fd6ec055f90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7080afc70 0x7fd7081470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 --2- 172.21.15.61:0/3578708334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7080abdf0 0x7fd708147b60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 >> 172.21.15.61:0/3578708334 conn(0x7fd708062c00 msgr2=0x7fd708062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:40.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 shutdown_connections 2022-07-12T08:41:40.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.371+0000 7fd70e352700 1 -- 172.21.15.61:0/3578708334 wait complete. 2022-07-12T08:41:40.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:40.567 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:40.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 Processor -- start 2022-07-12T08:41:40.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 -- start start 2022-07-12T08:41:40.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f79fc0b6ae0 0x7f79fc1460e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b25f0 0x7f79fc141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79fc0b1a20 con 0x7f79fc0b0f60 2022-07-12T08:41:40.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79fc061300 con 0x7f79fc0b6ae0 2022-07-12T08:41:40.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0315d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc0b6ec0 con 0x7f79fc0b25f0 2022-07-12T08:41:40.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0195a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f79fc0b0f60 0x7f79fc0b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41954/0 (socket says 172.21.15.61:41954) 2022-07-12T08:41:40.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0195a700 1 -- 172.21.15.61:0/2103589490 learned_addr learned my addr 172.21.15.61:0/2103589490 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:40.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0215b700 1 --2- 172.21.15.61:0/2103589490 >> v2:172.21.15.111:3300/0 conn(0x7f79fc0b6ae0 0x7f79fc1460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.686+0000 7f7a0295c700 1 --2- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b25f0 0x7f79fc141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a01159700 1 -- 172.21.15.61:0/2103589490 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672117403 0 0) 0x7f79fc0b1a20 con 0x7f79fc0b0f60 2022-07-12T08:41:40.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a01159700 1 -- 172.21.15.61:0/2103589490 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e4003260 con 0x7f79fc0b0f60 2022-07-12T08:41:40.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0215b700 1 -- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b25f0 msgr2=0x7f79fc141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:40.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0215b700 1 --2- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b25f0 0x7f79fc141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0215b700 1 -- 172.21.15.61:0/2103589490 >> v1:172.21.15.61:6789/0 conn(0x7f79fc0b0f60 legacy=0x7f79fc0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0215b700 1 -- 172.21.15.61:0/2103589490 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc0b14e0 con 0x7f79fc0b6ae0 2022-07-12T08:41:40.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0215b700 1 --2- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b6ae0 0x7f79fc1460e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f79f8002f70 tx=0x7f79f8006df0).ready entity=mon.1 client_cookie=ffe1186ea695b5b1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a01159700 1 -- 172.21.15.61:0/2103589490 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79f800d4a0 con 0x7f79fc0b6ae0 2022-07-12T08:41:40.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a01159700 1 -- 172.21.15.61:0/2103589490 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f8000e90 con 0x7f79fc0b6ae0 2022-07-12T08:41:40.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a01159700 1 -- 172.21.15.61:0/2103589490 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79f800d7f0 con 0x7f79fc0b6ae0 2022-07-12T08:41:40.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b6ae0 msgr2=0x7f79fc1460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 --2- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b6ae0 0x7f79fc1460e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f79f8002f70 tx=0x7f79f8006df0).stop 2022-07-12T08:41:40.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- 172.21.15.61:0/2103589490 shutdown_connections 2022-07-12T08:41:40.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 --2- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b25f0 0x7f79fc141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 --2- 172.21.15.61:0/2103589490 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b6ae0 0x7f79fc1460e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- 172.21.15.61:0/2103589490 >> 172.21.15.61:0/2103589490 conn(0x7f79fc064590 msgr2=0x7f79fc0b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:40.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- 172.21.15.61:0/2103589490 shutdown_connections 2022-07-12T08:41:40.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- 172.21.15.61:0/2103589490 wait complete. 2022-07-12T08:41:40.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 Processor -- start 2022-07-12T08:41:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- start start 2022-07-12T08:41:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 0x7f79fc0b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b25f0 0x7f79fc1471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b6ae0 0x7f79fc147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc065db0 con 0x7f79fc0b0f60 2022-07-12T08:41:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc155c90 con 0x7f79fc0b25f0 2022-07-12T08:41:40.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0315d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc155f30 con 0x7f79fc0b6ae0 2022-07-12T08:41:40.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.690+0000 7f7a0215b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 0x7f79fc0b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 0x7f79fc0b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53742/0 (socket says 172.21.15.61:53742) 2022-07-12T08:41:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 -- 172.21.15.61:0/50370944 learned_addr learned my addr 172.21.15.61:0/50370944 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:40.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0195a700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b25f0 0x7f79fc1471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0295c700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b6ae0 0x7f79fc147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 -- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b6ae0 msgr2=0x7f79fc147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b6ae0 0x7f79fc147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 -- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b25f0 msgr2=0x7f79fc1471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b25f0 0x7f79fc1471a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 -- 172.21.15.61:0/50370944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f800b180 con 0x7f79fc0b0f60 2022-07-12T08:41:40.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0195a700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b25f0 0x7f79fc1471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:40.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0215b700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 0x7f79fc0b4200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f79f8002f70 tx=0x7f79f8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79f8012930 con 0x7f79fc0b0f60 2022-07-12T08:41:40.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f8012a90 con 0x7f79fc0b0f60 2022-07-12T08:41:40.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79fc065a30 con 0x7f79fc0b0f60 2022-07-12T08:41:40.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79f8012df0 con 0x7f79fc0b0f60 2022-07-12T08:41:40.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79fc0657b0 con 0x7f79fc0b0f60 2022-07-12T08:41:40.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f79f801c070 con 0x7f79fc0b0f60 2022-07-12T08:41:40.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79fc065bc0 con 0x7f79fc0b0f60 2022-07-12T08:41:40.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.694+0000 7f79eb7fe700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f79dc0539b0 0x7f79dc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.698+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f79f806c950 con 0x7f79fc0b0f60 2022-07-12T08:41:40.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.698+0000 7f7a0195a700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f79dc0539b0 0x7f79dc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.698+0000 7f7a0195a700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f79dc0539b0 0x7f79dc055e70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f79ec00d000 tx=0x7f79ec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.702+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79f8043760 con 0x7f79fc0b0f60 2022-07-12T08:41:40.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.826+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f79fc0657b0 con 0x7f79fc0b0f60 2022-07-12T08:41:40.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.826+0000 7f79eb7fe700 1 -- 172.21.15.61:0/50370944 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f79fc0657b0 con 0x7f79fc0b0f60 2022-07-12T08:41:40.841 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:40.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f79dc0539b0 msgr2=0x7f79dc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f79dc0539b0 0x7f79dc055e70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f79ec00d000 tx=0x7f79ec009040).stop 2022-07-12T08:41:40.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 msgr2=0x7f79fc0b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 0x7f79fc0b4200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f79f8002f70 tx=0x7f79f8014000).stop 2022-07-12T08:41:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 shutdown_connections 2022-07-12T08:41:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79fc0b6ae0 0x7f79fc147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f79dc0539b0 0x7f79dc055e70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79fc0b25f0 0x7f79fc1471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 --2- 172.21.15.61:0/50370944 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79fc0b0f60 0x7f79fc0b4200 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 >> 172.21.15.61:0/50370944 conn(0x7f79fc064590 msgr2=0x7f79fc069e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:40.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 shutdown_connections 2022-07-12T08:41:40.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.830+0000 7f7a0315d700 1 -- 172.21.15.61:0/50370944 wait complete. 2022-07-12T08:41:40.861 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:40.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.970+0000 7fa223a5e700 1 Processor -- start 2022-07-12T08:41:40.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.970+0000 7fa223a5e700 1 -- start start 2022-07-12T08:41:40.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa21c0b3730 0x7fa21c1436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:40.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0b3560 con 0x7fa21c0b5480 2022-07-12T08:41:40.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0b5b80 con 0x7fa21c0b05f0 2022-07-12T08:41:40.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c0b5e20 con 0x7fa21c0b3730 2022-07-12T08:41:40.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa21c0b3730 0x7fa21c1436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:40.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa21c0b3730 0x7fa21c1436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53754/0 (socket says 172.21.15.61:53754) 2022-07-12T08:41:40.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 -- 172.21.15.61:0/109402521 learned_addr learned my addr 172.21.15.61:0/109402521 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:40.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 -- 172.21.15.61:0/109402521 >> v1:172.21.15.111:6789/0 conn(0x7fa21c0b05f0 legacy=0x7fa21c142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 -- 172.21.15.61:0/109402521 >> v1:172.21.15.61:6789/0 conn(0x7fa21c0b5480 legacy=0x7fa21c0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 -- 172.21.15.61:0/109402521 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0b5860 con 0x7fa21c0b3730 2022-07-12T08:41:40.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa221ffb700 1 --2- 172.21.15.61:0/109402521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b3730 0x7fa21c1436a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa2140032f0 tx=0x7fa214009290).ready entity=mon.0 client_cookie=7d596275f93aa0ee server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:40.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa213fff700 1 -- 172.21.15.61:0/109402521 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21400b8a0 con 0x7fa21c0b3730 2022-07-12T08:41:40.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa213fff700 1 -- 172.21.15.61:0/109402521 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21400ba00 con 0x7fa21c0b3730 2022-07-12T08:41:40.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa213fff700 1 -- 172.21.15.61:0/109402521 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21400bb70 con 0x7fa21c0b3730 2022-07-12T08:41:40.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- 172.21.15.61:0/109402521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b3730 msgr2=0x7fa21c1436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:40.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 --2- 172.21.15.61:0/109402521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b3730 0x7fa21c1436a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa2140032f0 tx=0x7fa214009290).stop 2022-07-12T08:41:40.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- 172.21.15.61:0/109402521 shutdown_connections 2022-07-12T08:41:40.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 --2- 172.21.15.61:0/109402521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b3730 0x7fa21c1436a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:40.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- 172.21.15.61:0/109402521 >> 172.21.15.61:0/109402521 conn(0x7fa21c0b96b0 msgr2=0x7fa21c0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:40.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- 172.21.15.61:0/109402521 shutdown_connections 2022-07-12T08:41:41.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.974+0000 7fa223a5e700 1 -- 172.21.15.61:0/109402521 wait complete. 2022-07-12T08:41:41.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 Processor -- start 2022-07-12T08:41:41.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 -- start start 2022-07-12T08:41:41.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa21c0b05f0 0x7fa21c147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa21c0b3730 0x7fa21c112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 0x7fa21c0acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c1533d0 con 0x7fa21c0b5480 2022-07-12T08:41:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c153130 con 0x7fa21c0b05f0 2022-07-12T08:41:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa223a5e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c1129b0 con 0x7fa21c0b3730 2022-07-12T08:41:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa221ffb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 0x7fa21c0acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa221ffb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 0x7fa21c0acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53758/0 (socket says 172.21.15.61:53758) 2022-07-12T08:41:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa221ffb700 1 -- 172.21.15.61:0/2949673101 learned_addr learned my addr 172.21.15.61:0/2949673101 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa2217fa700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa21c0b05f0 0x7fa21c147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.978+0000 7fa220ff9700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa21c0b3730 0x7fa21c112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa221ffb700 1 -- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa21c0b3730 msgr2=0x7fa21c112e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa221ffb700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa21c0b3730 0x7fa21c112e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa221ffb700 1 -- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa21c0b05f0 msgr2=0x7fa21c147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa221ffb700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa21c0b05f0 0x7fa21c147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa221ffb700 1 -- 172.21.15.61:0/2949673101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214009530 con 0x7fa21c0b5480 2022-07-12T08:41:41.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa221ffb700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 0x7fa21c0acb60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fa2140032f0 tx=0x7fa214016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21400b8a0 con 0x7fa21c0b5480 2022-07-12T08:41:41.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21401f3f0 con 0x7fa21c0b5480 2022-07-12T08:41:41.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c113340 con 0x7fa21c0b5480 2022-07-12T08:41:41.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21401f750 con 0x7fa21c0b5480 2022-07-12T08:41:41.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c152c90 con 0x7fa21c0b5480 2022-07-12T08:41:41.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c057460 con 0x7fa21c0b5480 2022-07-12T08:41:41.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa214026030 con 0x7fa21c0b5480 2022-07-12T08:41:41.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2127fc700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2040539b0 0x7fa204055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa21406cbd0 con 0x7fa21c0b5480 2022-07-12T08:41:41.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.982+0000 7fa2217fa700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2040539b0 0x7fa204055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.986+0000 7fa2217fa700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2040539b0 0x7fa204055e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa20c00d000 tx=0x7fa20c007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:40.986+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa214043a90 con 0x7fa21c0b5480 2022-07-12T08:41:41.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.110+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa21c112b70 con 0x7fa21c0b5480 2022-07-12T08:41:41.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.110+0000 7fa2127fc700 1 -- 172.21.15.61:0/2949673101 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa214024020 con 0x7fa21c0b5480 2022-07-12T08:41:41.127 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:41.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2040539b0 msgr2=0x7fa204055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2040539b0 0x7fa204055e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fa20c00d000 tx=0x7fa20c007b70).stop 2022-07-12T08:41:41.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 msgr2=0x7fa21c0acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 0x7fa21c0acb60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fa2140032f0 tx=0x7fa214016000).stop 2022-07-12T08:41:41.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 shutdown_connections 2022-07-12T08:41:41.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa21c0b5480 0x7fa21c0acb60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa21c0b3730 0x7fa21c112e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa2040539b0 0x7fa204055e70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 --2- 172.21.15.61:0/2949673101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa21c0b05f0 0x7fa21c147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 >> 172.21.15.61:0/2949673101 conn(0x7fa21c0b96b0 msgr2=0x7fa21c0b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:41.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 shutdown_connections 2022-07-12T08:41:41.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.114+0000 7fa223a5e700 1 -- 172.21.15.61:0/2949673101 wait complete. 2022-07-12T08:41:41.147 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:41.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 Processor -- start 2022-07-12T08:41:41.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- start start 2022-07-12T08:41:41.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8880ad4d0 0x7fe8881419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe888141ec0 0x7fe8881422e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88805bdb0 con 0x7fe8880b7bd0 2022-07-12T08:41:41.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8880b7fb0 con 0x7fe888141ec0 2022-07-12T08:41:41.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8880b2030 con 0x7fe8880ad4d0 2022-07-12T08:41:41.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe888141ec0 0x7fe8881422e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe888141ec0 0x7fe8881422e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53768/0 (socket says 172.21.15.61:53768) 2022-07-12T08:41:41.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 -- 172.21.15.61:0/3446143395 learned_addr learned my addr 172.21.15.61:0/3446143395 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:41.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8867fc700 1 --2- 172.21.15.61:0/3446143395 >> v2:172.21.15.111:3300/0 conn(0x7fe8880ad4d0 0x7fe8881419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 -- 172.21.15.61:0/3446143395 >> v2:172.21.15.111:3300/0 conn(0x7fe8880ad4d0 msgr2=0x7fe8881419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:41.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 --2- 172.21.15.61:0/3446143395 >> v2:172.21.15.111:3300/0 conn(0x7fe8880ad4d0 0x7fe8881419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 -- 172.21.15.61:0/3446143395 >> v1:172.21.15.111:6789/0 conn(0x7fe8880b7bd0 legacy=0x7fe8880acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 -- 172.21.15.61:0/3446143395 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8881427d0 con 0x7fe888141ec0 2022-07-12T08:41:41.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe8877fe700 1 --2- 172.21.15.61:0/3446143395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe888141ec0 0x7fe8881422e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe8800032f0 tx=0x7fe880009290).ready entity=mon.0 client_cookie=c3fc5ca2ca5d47fe server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe885ffb700 1 -- 172.21.15.61:0/3446143395 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe88000b870 con 0x7fe888141ec0 2022-07-12T08:41:41.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe885ffb700 1 -- 172.21.15.61:0/3446143395 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88000b9d0 con 0x7fe888141ec0 2022-07-12T08:41:41.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe885ffb700 1 -- 172.21.15.61:0/3446143395 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe88000bb40 con 0x7fe888141ec0 2022-07-12T08:41:41.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- 172.21.15.61:0/3446143395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe888141ec0 msgr2=0x7fe8881422e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 --2- 172.21.15.61:0/3446143395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe888141ec0 0x7fe8881422e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe8800032f0 tx=0x7fe880009290).stop 2022-07-12T08:41:41.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- 172.21.15.61:0/3446143395 shutdown_connections 2022-07-12T08:41:41.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 --2- 172.21.15.61:0/3446143395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe888141ec0 0x7fe8881422e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 --2- 172.21.15.61:0/3446143395 >> v2:172.21.15.111:3300/0 conn(0x7fe8880ad4d0 0x7fe8881419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- 172.21.15.61:0/3446143395 >> 172.21.15.61:0/3446143395 conn(0x7fe8880655e0 msgr2=0x7fe8880b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:41.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.270+0000 7fe887fff700 1 -- 172.21.15.61:0/3446143395 shutdown_connections 2022-07-12T08:41:41.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- 172.21.15.61:0/3446143395 wait complete. 2022-07-12T08:41:41.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 Processor -- start 2022-07-12T08:41:41.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- start start 2022-07-12T08:41:41.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 0x7fe888141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8880b7bd0 0x7fe888147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe888141ec0 0x7fe8880b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880b59e0 con 0x7fe8880ad4d0 2022-07-12T08:41:41.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880b0260 con 0x7fe8880b7bd0 2022-07-12T08:41:41.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880b0500 con 0x7fe888141ec0 2022-07-12T08:41:41.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 0x7fe888141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8880b7bd0 0x7fe888147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 0x7fe888141160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53770/0 (socket says 172.21.15.61:53770) 2022-07-12T08:41:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe888141ec0 0x7fe8880b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 -- 172.21.15.61:0/567833509 learned_addr learned my addr 172.21.15.61:0/567833509 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:41.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 -- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe888141ec0 msgr2=0x7fe8880b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe888141ec0 0x7fe8880b09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 -- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8880b7bd0 msgr2=0x7fe888147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8880b7bd0 0x7fe888147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 -- 172.21.15.61:0/567833509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe880009530 con 0x7fe8880ad4d0 2022-07-12T08:41:41.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe886ffd700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 0x7fe888141160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fe888064510 tx=0x7fe878007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe878012400 con 0x7fe8880ad4d0 2022-07-12T08:41:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe878012560 con 0x7fe8880ad4d0 2022-07-12T08:41:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8880b5660 con 0x7fe8880ad4d0 2022-07-12T08:41:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8780128c0 con 0x7fe8880ad4d0 2022-07-12T08:41:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.274+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8880b1240 con 0x7fe8880ad4d0 2022-07-12T08:41:41.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.278+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8880b06c0 con 0x7fe8880ad4d0 2022-07-12T08:41:41.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.278+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe878019030 con 0x7fe8880ad4d0 2022-07-12T08:41:41.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.278+0000 7fe877fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe864053af0 0x7fe864055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.278+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe87806b1f0 con 0x7fe8880ad4d0 2022-07-12T08:41:41.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.278+0000 7fe8867fc700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe864053af0 0x7fe864055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.278+0000 7fe8867fc700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe864053af0 0x7fe864055fb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe87c00b000 tx=0x7fe87c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.286+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8780420f0 con 0x7fe8880ad4d0 2022-07-12T08:41:41.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.414+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8880b57f0 con 0x7fe8880ad4d0 2022-07-12T08:41:41.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.414+0000 7fe877fff700 1 -- 172.21.15.61:0/567833509 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe878045710 con 0x7fe8880ad4d0 2022-07-12T08:41:41.436 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:41.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe864053af0 msgr2=0x7fe864055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe864053af0 0x7fe864055fb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe87c00b000 tx=0x7fe87c006df0).stop 2022-07-12T08:41:41.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 msgr2=0x7fe888141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 0x7fe888141160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fe888064510 tx=0x7fe878007c50).stop 2022-07-12T08:41:41.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 shutdown_connections 2022-07-12T08:41:41.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe888141ec0 0x7fe8880b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe864053af0 0x7fe864055fb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8880b7bd0 0x7fe888147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 --2- 172.21.15.61:0/567833509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8880ad4d0 0x7fe888141160 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 >> 172.21.15.61:0/567833509 conn(0x7fe8880655e0 msgr2=0x7fe8880acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:41.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 shutdown_connections 2022-07-12T08:41:41.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.418+0000 7fe887fff700 1 -- 172.21.15.61:0/567833509 wait complete. 2022-07-12T08:41:41.450 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:41.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.562+0000 7f99c3fff700 1 Processor -- start 2022-07-12T08:41:41.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.562+0000 7f99c3fff700 1 -- start start 2022-07-12T08:41:41.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99c40aad90 0x7f99c40ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c40ab170 con 0x7f99c40b2e70 2022-07-12T08:41:41.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c4061b90 con 0x7f99c405bd10 2022-07-12T08:41:41.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c41435c0 con 0x7f99c40aad90 2022-07-12T08:41:41.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99c40aad90 0x7f99c40ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99bbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f99c40b2e70 0x7f99c40603f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41998/0 (socket says 172.21.15.61:41998) 2022-07-12T08:41:41.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99bbfff700 1 -- 172.21.15.61:0/2758990425 learned_addr learned my addr 172.21.15.61:0/2758990425 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:41.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c27fc700 1 -- 172.21.15.61:0/2758990425 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444660301 0 0) 0x7f99c40ab170 con 0x7f99c40b2e70 2022-07-12T08:41:41.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c27fc700 1 -- 172.21.15.61:0/2758990425 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a4003260 con 0x7f99c40b2e70 2022-07-12T08:41:41.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c2ffd700 1 -- 172.21.15.61:0/2758990425 >> v1:172.21.15.111:6789/0 conn(0x7f99c405bd10 legacy=0x7f99c4141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c27fc700 1 -- 172.21.15.61:0/2758990425 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213557538 0 0) 0x7f99c4061b90 con 0x7f99c405bd10 2022-07-12T08:41:41.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c2ffd700 1 -- 172.21.15.61:0/2758990425 >> v1:172.21.15.61:6789/0 conn(0x7f99c40b2e70 legacy=0x7f99c40603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c2ffd700 1 -- 172.21.15.61:0/2758990425 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4143860 con 0x7f99c40aad90 2022-07-12T08:41:41.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c2ffd700 1 --2- 172.21.15.61:0/2758990425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c40aad90 0x7f99c40ab490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f99b4002f70 tx=0x7f99b4006df0).ready entity=mon.0 client_cookie=f67c51b8ddc5de7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c27fc700 1 -- 172.21.15.61:0/2758990425 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b400d610 con 0x7f99c40aad90 2022-07-12T08:41:41.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c27fc700 1 -- 172.21.15.61:0/2758990425 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b4000e90 con 0x7f99c40aad90 2022-07-12T08:41:41.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c27fc700 1 -- 172.21.15.61:0/2758990425 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b400d960 con 0x7f99c40aad90 2022-07-12T08:41:41.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- 172.21.15.61:0/2758990425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c40aad90 msgr2=0x7f99c40ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 --2- 172.21.15.61:0/2758990425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c40aad90 0x7f99c40ab490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f99b4002f70 tx=0x7f99b4006df0).stop 2022-07-12T08:41:41.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- 172.21.15.61:0/2758990425 shutdown_connections 2022-07-12T08:41:41.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 --2- 172.21.15.61:0/2758990425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c40aad90 0x7f99c40ab490 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- 172.21.15.61:0/2758990425 >> 172.21.15.61:0/2758990425 conn(0x7f99c40b57c0 msgr2=0x7f99c40b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:41.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- 172.21.15.61:0/2758990425 shutdown_connections 2022-07-12T08:41:41.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.566+0000 7f99c3fff700 1 -- 172.21.15.61:0/2758990425 wait complete. 2022-07-12T08:41:41.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 Processor -- start 2022-07-12T08:41:41.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 -- start start 2022-07-12T08:41:41.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 0x7f99c4146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c40aad90 0x7f99c413f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c40b2e70 0x7f99c4152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c41453a0 con 0x7f99c405bd10 2022-07-12T08:41:41.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c413f7a0 con 0x7f99c40b2e70 2022-07-12T08:41:41.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c3fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c41519f0 con 0x7f99c40aad90 2022-07-12T08:41:41.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 0x7f99c4146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 0x7f99c4146640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53784/0 (socket says 172.21.15.61:53784) 2022-07-12T08:41:41.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c2ffd700 1 -- 172.21.15.61:0/1558857107 learned_addr learned my addr 172.21.15.61:0/1558857107 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:41.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99bbfff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c40aad90 0x7f99c413f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.570+0000 7f99c37fe700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c40b2e70 0x7f99c4152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c2ffd700 1 -- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c40aad90 msgr2=0x7f99c413f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c2ffd700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c40aad90 0x7f99c413f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c2ffd700 1 -- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c40b2e70 msgr2=0x7f99c4152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c2ffd700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c40b2e70 0x7f99c4152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c2ffd700 1 -- 172.21.15.61:0/1558857107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b400b180 con 0x7f99c405bd10 2022-07-12T08:41:41.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c2ffd700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 0x7f99c4146640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f99b4002f70 tx=0x7f99b4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b4013de0 con 0x7f99c405bd10 2022-07-12T08:41:41.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b400d290 con 0x7f99c405bd10 2022-07-12T08:41:41.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c4151c90 con 0x7f99c405bd10 2022-07-12T08:41:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b401e5f0 con 0x7f99c405bd10 2022-07-12T08:41:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c4145f40 con 0x7f99c405bd10 2022-07-12T08:41:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c4151820 con 0x7f99c405bd10 2022-07-12T08:41:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f99b4025030 con 0x7f99c405bd10 2022-07-12T08:41:41.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c0ff9700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a8053ad0 0x7f99a8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f99b406ccc0 con 0x7f99c405bd10 2022-07-12T08:41:41.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.574+0000 7f99bbfff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a8053ad0 0x7f99a8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.578+0000 7f99bbfff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a8053ad0 0x7f99a8055f90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f99ac00d000 tx=0x7f99ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.582+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99b4043b80 con 0x7f99c405bd10 2022-07-12T08:41:41.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.710+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99c4151e20 con 0x7f99c405bd10 2022-07-12T08:41:41.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.710+0000 7f99c0ff9700 1 -- 172.21.15.61:0/1558857107 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f99b4023020 con 0x7f99c405bd10 2022-07-12T08:41:41.717 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:41.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.710+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a8053ad0 msgr2=0x7f99a8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.710+0000 7f99c3fff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a8053ad0 0x7f99a8055f90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f99ac00d000 tx=0x7f99ac009040).stop 2022-07-12T08:41:41.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.710+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 msgr2=0x7f99c4146640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.710+0000 7f99c3fff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 0x7f99c4146640 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f99b4002f70 tx=0x7f99b4014000).stop 2022-07-12T08:41:41.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 shutdown_connections 2022-07-12T08:41:41.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a8053ad0 0x7f99a8055f90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c40b2e70 0x7f99c4152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c40aad90 0x7f99c413f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 --2- 172.21.15.61:0/1558857107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c405bd10 0x7f99c4146640 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 >> 172.21.15.61:0/1558857107 conn(0x7f99c40b57c0 msgr2=0x7f99c40b1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:41.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 shutdown_connections 2022-07-12T08:41:41.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.714+0000 7f99c3fff700 1 -- 172.21.15.61:0/1558857107 wait complete. 2022-07-12T08:41:41.737 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:41.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437b59e700 1 Processor -- start 2022-07-12T08:41:41.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437b59e700 1 -- start start 2022-07-12T08:41:41.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f437c09bce0 0x7f437c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c0aecb0 con 0x7f437c0af280 2022-07-12T08:41:41.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c1435c0 con 0x7f437c0b3f60 2022-07-12T08:41:41.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c143b00 con 0x7f437c09bce0 2022-07-12T08:41:41.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f437c0af280 0x7f437c060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42010/0 (socket says 172.21.15.61:42010) 2022-07-12T08:41:41.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f437a59c700 1 -- 172.21.15.61:0/4087683234 learned_addr learned my addr 172.21.15.61:0/4087683234 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:41.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.846+0000 7f4379d9b700 1 --2- 172.21.15.61:0/4087683234 >> v2:172.21.15.111:3300/0 conn(0x7f437c09bce0 0x7f437c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437959a700 1 -- 172.21.15.61:0/4087683234 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376629225 0 0) 0x7f437c0aecb0 con 0x7f437c0af280 2022-07-12T08:41:41.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437959a700 1 -- 172.21.15.61:0/4087683234 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4360003260 con 0x7f437c0af280 2022-07-12T08:41:41.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f4379d9b700 1 -- 172.21.15.61:0/4087683234 >> v1:172.21.15.111:6789/0 conn(0x7f437c0b3f60 legacy=0x7f437c0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f4379d9b700 1 -- 172.21.15.61:0/4087683234 >> v1:172.21.15.61:6789/0 conn(0x7f437c0af280 legacy=0x7f437c060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f4379d9b700 1 -- 172.21.15.61:0/4087683234 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c143860 con 0x7f437c09bce0 2022-07-12T08:41:41.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f4379d9b700 1 --2- 172.21.15.61:0/4087683234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c09bce0 0x7f437c0af660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f436c002f70 tx=0x7f436c006df0).ready entity=mon.1 client_cookie=bf4b51aa5692ac32 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437959a700 1 -- 172.21.15.61:0/4087683234 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f436c00d610 con 0x7f437c09bce0 2022-07-12T08:41:41.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437959a700 1 -- 172.21.15.61:0/4087683234 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c000e90 con 0x7f437c09bce0 2022-07-12T08:41:41.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437959a700 1 -- 172.21.15.61:0/4087683234 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f436c00d960 con 0x7f437c09bce0 2022-07-12T08:41:41.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 -- 172.21.15.61:0/4087683234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c09bce0 msgr2=0x7f437c0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 --2- 172.21.15.61:0/4087683234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c09bce0 0x7f437c0af660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f436c002f70 tx=0x7f436c006df0).stop 2022-07-12T08:41:41.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 -- 172.21.15.61:0/4087683234 shutdown_connections 2022-07-12T08:41:41.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 --2- 172.21.15.61:0/4087683234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c09bce0 0x7f437c0af660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 -- 172.21.15.61:0/4087683234 >> 172.21.15.61:0/4087683234 conn(0x7f437c0b92c0 msgr2=0x7f437c0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:41.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 -- 172.21.15.61:0/4087683234 shutdown_connections 2022-07-12T08:41:41.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.850+0000 7f437b59e700 1 -- 172.21.15.61:0/4087683234 wait complete. 2022-07-12T08:41:41.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 Processor -- start 2022-07-12T08:41:41.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 -- start start 2022-07-12T08:41:41.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f437c09bce0 0x7f437c0b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 0x7f437c0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c0b3f60 0x7f437c0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c146d10 con 0x7f437c0af280 2022-07-12T08:41:41.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c152a40 con 0x7f437c0b3f60 2022-07-12T08:41:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c152ce0 con 0x7f437c09bce0 2022-07-12T08:41:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 0x7f437c0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 0x7f437c0b2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53800/0 (socket says 172.21.15.61:53800) 2022-07-12T08:41:41.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 -- 172.21.15.61:0/1698636878 learned_addr learned my addr 172.21.15.61:0/1698636878 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:41.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437ad9d700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c0b3f60 0x7f437c0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f437a59c700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f437c09bce0 0x7f437c0b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 -- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f437c09bce0 msgr2=0x7f437c0b3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f437c09bce0 0x7f437c0b3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 -- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c0b3f60 msgr2=0x7f437c0b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:41.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.854+0000 7f4379d9b700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c0b3f60 0x7f437c0b1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:41.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f4379d9b700 1 -- 172.21.15.61:0/1698636878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c00b180 con 0x7f437c0af280 2022-07-12T08:41:41.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f4379d9b700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 0x7f437c0b2b10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f436c002f70 tx=0x7f436c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f436c013de0 con 0x7f437c0af280 2022-07-12T08:41:41.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436c00d290 con 0x7f437c0af280 2022-07-12T08:41:41.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c146990 con 0x7f437c0af280 2022-07-12T08:41:41.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f436c01e5f0 con 0x7f437c0af280 2022-07-12T08:41:41.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c0b1740 con 0x7f437c0af280 2022-07-12T08:41:41.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c146b80 con 0x7f437c0af280 2022-07-12T08:41:41.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f436c025030 con 0x7f437c0af280 2022-07-12T08:41:41.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f436b7fe700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4358053a40 0x7f4358055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:41.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.858+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f436c06ccd0 con 0x7f437c0af280 2022-07-12T08:41:41.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.862+0000 7f437a59c700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4358053a40 0x7f4358055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:41.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.862+0000 7f437a59c700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4358053a40 0x7f4358055f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f437000d000 tx=0x7f4370007a40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:41.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.866+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f436c043b90 con 0x7f437c0af280 2022-07-12T08:41:42.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.998+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f437c057460 con 0x7f437c0af280 2022-07-12T08:41:42.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:41.998+0000 7f436b7fe700 1 -- 172.21.15.61:0/1698636878 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f436c023070 con 0x7f437c0af280 2022-07-12T08:41:42.011 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:42.012 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":22,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:15.583994+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:12.785924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:13.618519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:42.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4358053a40 msgr2=0x7f4358055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4358053a40 0x7f4358055f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f437000d000 tx=0x7f4370007a40).stop 2022-07-12T08:41:42.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 msgr2=0x7f437c0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 0x7f437c0b2b10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f436c002f70 tx=0x7f436c014000).stop 2022-07-12T08:41:42.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 shutdown_connections 2022-07-12T08:41:42.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4358053a40 0x7f4358055f00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f437c0b3f60 0x7f437c0b1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437c0af280 0x7f437c0b2b10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 --2- 172.21.15.61:0/1698636878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f437c09bce0 0x7f437c0b3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 >> 172.21.15.61:0/1698636878 conn(0x7f437c0b92c0 msgr2=0x7f437c0602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 shutdown_connections 2022-07-12T08:41:42.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.002+0000 7f437b59e700 1 -- 172.21.15.61:0/1698636878 wait complete. 2022-07-12T08:41:42.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:12.785924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:13.618519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:41:42.024 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:42.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.150+0000 7f73493da700 1 Processor -- start 2022-07-12T08:41:42.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- start start 2022-07-12T08:41:42.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f73440aef10 0x7f73440b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f734413e950 0x7f734413ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440b15d0 con 0x7f73440acda0 2022-07-12T08:41:42.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73440ac820 con 0x7f734413e950 2022-07-12T08:41:42.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73440b1800 con 0x7f73440aef10 2022-07-12T08:41:42.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f734413e950 0x7f734413ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f734413e950 0x7f734413ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53810/0 (socket says 172.21.15.61:53810) 2022-07-12T08:41:42.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 -- 172.21.15.61:0/3756120561 learned_addr learned my addr 172.21.15.61:0/3756120561 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f733bfff700 1 --2- 172.21.15.61:0/3756120561 >> v2:172.21.15.111:3300/0 conn(0x7f73440aef10 0x7f73440b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 -- 172.21.15.61:0/3756120561 >> v2:172.21.15.111:3300/0 conn(0x7f73440aef10 msgr2=0x7f73440b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 --2- 172.21.15.61:0/3756120561 >> v2:172.21.15.111:3300/0 conn(0x7f73440aef10 0x7f73440b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 -- 172.21.15.61:0/3756120561 >> v1:172.21.15.111:6789/0 conn(0x7f73440acda0 legacy=0x7f73440ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 -- 172.21.15.61:0/3756120561 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7344067430 con 0x7f734413e950 2022-07-12T08:41:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f7348bd9700 1 --2- 172.21.15.61:0/3756120561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f734413e950 0x7f734413ed80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f733c0032f0 tx=0x7f733c00b530).ready entity=mon.0 client_cookie=35cdfdbff66553fd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73437fe700 1 -- 172.21.15.61:0/3756120561 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f733c00be70 con 0x7f734413e950 2022-07-12T08:41:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73437fe700 1 -- 172.21.15.61:0/3756120561 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733c010470 con 0x7f734413e950 2022-07-12T08:41:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73437fe700 1 -- 172.21.15.61:0/3756120561 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f733c0107d0 con 0x7f734413e950 2022-07-12T08:41:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- 172.21.15.61:0/3756120561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f734413e950 msgr2=0x7f734413ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 --2- 172.21.15.61:0/3756120561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f734413e950 0x7f734413ed80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f733c0032f0 tx=0x7f733c00b530).stop 2022-07-12T08:41:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- 172.21.15.61:0/3756120561 shutdown_connections 2022-07-12T08:41:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 --2- 172.21.15.61:0/3756120561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f734413e950 0x7f734413ed80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 --2- 172.21.15.61:0/3756120561 >> v2:172.21.15.111:3300/0 conn(0x7f73440aef10 0x7f73440b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- 172.21.15.61:0/3756120561 >> 172.21.15.61:0/3756120561 conn(0x7f734405f140 msgr2=0x7f734405f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- 172.21.15.61:0/3756120561 shutdown_connections 2022-07-12T08:41:42.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.154+0000 7f73493da700 1 -- 172.21.15.61:0/3756120561 wait complete. 2022-07-12T08:41:42.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 Processor -- start 2022-07-12T08:41:42.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 -- start start 2022-07-12T08:41:42.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 0x7f7344144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73440aef10 0x7f73440641f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734413e950 0x7f7344141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73441417c0 con 0x7f73440acda0 2022-07-12T08:41:42.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7344141a60 con 0x7f734413e950 2022-07-12T08:41:42.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7344065a00 con 0x7f73440aef10 2022-07-12T08:41:42.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 0x7f7344144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 0x7f7344144540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53812/0 (socket says 172.21.15.61:53812) 2022-07-12T08:41:42.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 -- 172.21.15.61:0/781070138 learned_addr learned my addr 172.21.15.61:0/781070138 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:42.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7348bd9700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734413e950 0x7f7344141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f733bfff700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73440aef10 0x7f73440641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 -- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73440aef10 msgr2=0x7f73440641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73440aef10 0x7f73440641f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 -- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734413e950 msgr2=0x7f7344141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734413e950 0x7f7344141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 -- 172.21.15.61:0/781070138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c00b7d0 con 0x7f73440acda0 2022-07-12T08:41:42.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7343fff700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 0x7f7344144540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f734405c4d0 tx=0x7f7334007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f733400ce30 con 0x7f73440acda0 2022-07-12T08:41:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7334007240 con 0x7f73440acda0 2022-07-12T08:41:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73441414f0 con 0x7f73440acda0 2022-07-12T08:41:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7334006700 con 0x7f73440acda0 2022-07-12T08:41:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.158+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7344065f80 con 0x7f73440acda0 2022-07-12T08:41:42.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.162+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7344143e80 con 0x7f73440acda0 2022-07-12T08:41:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.162+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7334010030 con 0x7f73440acda0 2022-07-12T08:41:42.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.162+0000 7f7341ffb700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f731c053920 0x7f731c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.162+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f733406bc50 con 0x7f73440acda0 2022-07-12T08:41:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.162+0000 7f733bfff700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f731c053920 0x7f731c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.162+0000 7f733bfff700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f731c053920 0x7f731c055de0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f732c00c000 tx=0x7f732c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.166+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7334042aa0 con 0x7f73440acda0 2022-07-12T08:41:42.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7344065d20 con 0x7f73440acda0 2022-07-12T08:41:42.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f7341ffb700 1 -- 172.21.15.61:0/781070138 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v22) v1 ==== 110+0+10 (secure 0 0 0) 0x7f73340460c0 con 0x7f73440acda0 2022-07-12T08:41:42.307 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:42.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f731c053920 msgr2=0x7f731c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f731c053920 0x7f731c055de0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f732c00c000 tx=0x7f732c008040).stop 2022-07-12T08:41:42.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 msgr2=0x7f7344144540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 0x7f7344144540 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f734405c4d0 tx=0x7f7334007c10).stop 2022-07-12T08:41:42.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 shutdown_connections 2022-07-12T08:41:42.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f731c053920 0x7f731c055de0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f734413e950 0x7f7344141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73440aef10 0x7f73440641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.294+0000 7f73493da700 1 --2- 172.21.15.61:0/781070138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73440acda0 0x7f7344144540 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.298+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 >> 172.21.15.61:0/781070138 conn(0x7f734405f140 msgr2=0x7f73440b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.298+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 shutdown_connections 2022-07-12T08:41:42.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.298+0000 7f73493da700 1 -- 172.21.15.61:0/781070138 wait complete. 2022-07-12T08:41:42.320 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:42.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 Processor -- start 2022-07-12T08:41:42.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 -- start start 2022-07-12T08:41:42.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c05c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f955c05d130 0x7f955c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c143500 con 0x7f955c0b5f10 2022-07-12T08:41:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f955c143290 con 0x7f955c05d130 2022-07-12T08:41:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95634ef700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c143760 con 0x7f955c142eb0 2022-07-12T08:41:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f9561cec700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f955c05d130 0x7f955c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f9561cec700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f955c05d130 0x7f955c0b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53820/0 (socket says 172.21.15.61:53820) 2022-07-12T08:41:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f9561cec700 1 -- 172.21.15.61:0/2494928325 learned_addr learned my addr 172.21.15.61:0/2494928325 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f9561cec700 1 -- 172.21.15.61:0/2494928325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 msgr2=0x7f955c05c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.434+0000 7f95624ed700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c05c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f9561cec700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c05c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f9561cec700 1 -- 172.21.15.61:0/2494928325 >> v1:172.21.15.111:6789/0 conn(0x7f955c0b5f10 legacy=0x7f955c146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f9561cec700 1 -- 172.21.15.61:0/2494928325 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c0b4030 con 0x7f955c05d130 2022-07-12T08:41:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95624ed700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c05c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f9561cec700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c05d130 0x7f955c0b5a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9558002f70 tx=0x7f9558006eb0).ready entity=mon.0 client_cookie=a94c4cccfb5dd51b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95614eb700 1 -- 172.21.15.61:0/2494928325 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f955800d5c0 con 0x7f955c05d130 2022-07-12T08:41:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95614eb700 1 -- 172.21.15.61:0/2494928325 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558000e90 con 0x7f955c05d130 2022-07-12T08:41:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95614eb700 1 -- 172.21.15.61:0/2494928325 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f955800d950 con 0x7f955c05d130 2022-07-12T08:41:42.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 -- 172.21.15.61:0/2494928325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c05d130 msgr2=0x7f955c0b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c05d130 0x7f955c0b5a20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f9558002f70 tx=0x7f9558006eb0).stop 2022-07-12T08:41:42.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 -- 172.21.15.61:0/2494928325 shutdown_connections 2022-07-12T08:41:42.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c05d130 0x7f955c0b5a20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 --2- 172.21.15.61:0/2494928325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c05c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 -- 172.21.15.61:0/2494928325 >> 172.21.15.61:0/2494928325 conn(0x7f955c066d70 msgr2=0x7f955c067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 -- 172.21.15.61:0/2494928325 shutdown_connections 2022-07-12T08:41:42.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 -- 172.21.15.61:0/2494928325 wait complete. 2022-07-12T08:41:42.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.438+0000 7f95634ef700 1 Processor -- start 2022-07-12T08:41:42.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 -- start start 2022-07-12T08:41:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f955c05d130 0x7f955c0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 0x7f955c146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c13fa60 con 0x7f955c0b5f10 2022-07-12T08:41:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c13fd00 con 0x7f955c05d130 2022-07-12T08:41:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c144280 con 0x7f955c142eb0 2022-07-12T08:41:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 0x7f955c146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 0x7f955c146920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53828/0 (socket says 172.21.15.61:53828) 2022-07-12T08:41:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 -- 172.21.15.61:0/3336075120 learned_addr learned my addr 172.21.15.61:0/3336075120 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95624ed700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f955c05d130 0x7f955c0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9562cee700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 -- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 msgr2=0x7f955c151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 -- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f955c05d130 msgr2=0x7f955c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f955c05d130 0x7f955c0b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 -- 172.21.15.61:0/3336075120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955800b1e0 con 0x7f955c0b5f10 2022-07-12T08:41:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f9561cec700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 0x7f955c146920 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9558002f70 tx=0x7f9558015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9558011920 con 0x7f955c0b5f10 2022-07-12T08:41:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558011a80 con 0x7f955c0b5f10 2022-07-12T08:41:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c144fd0 con 0x7f955c0b5f10 2022-07-12T08:41:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9558011de0 con 0x7f955c0b5f10 2022-07-12T08:41:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.442+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c151fb0 con 0x7f955c0b5f10 2022-07-12T08:41:42.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.446+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c057460 con 0x7f955c0b5f10 2022-07-12T08:41:42.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.446+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9558025030 con 0x7f955c0b5f10 2022-07-12T08:41:42.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.446+0000 7f954b7fe700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f953c053af0 0x7f953c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.446+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f955806d1f0 con 0x7f955c0b5f10 2022-07-12T08:41:42.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.446+0000 7f95624ed700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f953c053af0 0x7f953c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.446+0000 7f95624ed700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f953c053af0 0x7f953c055fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f954c00d000 tx=0x7f954c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.450+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9558044150 con 0x7f955c0b5f10 2022-07-12T08:41:42.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.578+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f955c06b2b0 con 0x7f955c0b5f10 2022-07-12T08:41:42.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.578+0000 7f954b7fe700 1 -- 172.21.15.61:0/3336075120 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v22) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9558023020 con 0x7f955c0b5f10 2022-07-12T08:41:42.587 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f953c053af0 msgr2=0x7f953c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f953c053af0 0x7f953c055fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f954c00d000 tx=0x7f954c008040).stop 2022-07-12T08:41:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 msgr2=0x7f955c146920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 0x7f955c146920 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9558002f70 tx=0x7f9558015000).stop 2022-07-12T08:41:42.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 shutdown_connections 2022-07-12T08:41:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f955c142eb0 0x7f955c151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f955c0b5f10 0x7f955c146920 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f953c053af0 0x7f953c055fb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 --2- 172.21.15.61:0/3336075120 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f955c05d130 0x7f955c0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 >> 172.21.15.61:0/3336075120 conn(0x7f955c066d70 msgr2=0x7f955c0b8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 shutdown_connections 2022-07-12T08:41:42.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.582+0000 7f95634ef700 1 -- 172.21.15.61:0/3336075120 wait complete. 2022-07-12T08:41:42.656 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:41:42.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 Processor -- start 2022-07-12T08:41:42.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 -- start start 2022-07-12T08:41:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff0fc142eb0 0x7ff0fc05c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff0fc05d130 0x7ff0fc0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc0b5f10 0x7ff0fc146090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0fc143500 con 0x7ff0fc05d130 2022-07-12T08:41:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0fc143290 con 0x7ff0fc142eb0 2022-07-12T08:41:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff103039700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc143760 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff0fc05d130 0x7ff0fc0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff0fc05d130 0x7ff0fc0b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53834/0 (socket says 172.21.15.61:53834) 2022-07-12T08:41:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 -- 172.21.15.61:0/3562687780 learned_addr learned my addr 172.21.15.61:0/3562687780 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff102838700 1 --2- 172.21.15.61:0/3562687780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc0b5f10 0x7ff0fc146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff102037700 1 --2- 172.21.15.61:0/3562687780 >> v2:172.21.15.111:3300/0 conn(0x7ff0fc142eb0 0x7ff0fc05c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 -- 172.21.15.61:0/3562687780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc0b5f10 msgr2=0x7ff0fc146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:42.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 --2- 172.21.15.61:0/3562687780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc0b5f10 0x7ff0fc146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 -- 172.21.15.61:0/3562687780 >> v2:172.21.15.111:3300/0 conn(0x7ff0fc142eb0 msgr2=0x7ff0fc05c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:42.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 --2- 172.21.15.61:0/3562687780 >> v2:172.21.15.111:3300/0 conn(0x7ff0fc142eb0 0x7ff0fc05c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 -- 172.21.15.61:0/3562687780 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0fc0b4030 con 0x7ff0fc05d130 2022-07-12T08:41:42.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.769+0000 7ff101836700 1 --2- 172.21.15.61:0/3562687780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc05d130 0x7ff0fc0b5a20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff0f8002f70 tx=0x7ff0f8006eb0).ready entity=mon.0 client_cookie=b329ebd0bdcce262 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101035700 1 -- 172.21.15.61:0/3562687780 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f800d570 con 0x7ff0fc05d130 2022-07-12T08:41:42.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101035700 1 -- 172.21.15.61:0/3562687780 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f8000e90 con 0x7ff0fc05d130 2022-07-12T08:41:42.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101035700 1 -- 172.21.15.61:0/3562687780 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f800d900 con 0x7ff0fc05d130 2022-07-12T08:41:42.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/3562687780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc05d130 msgr2=0x7ff0fc0b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- 172.21.15.61:0/3562687780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc05d130 0x7ff0fc0b5a20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff0f8002f70 tx=0x7ff0f8006eb0).stop 2022-07-12T08:41:42.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/3562687780 shutdown_connections 2022-07-12T08:41:42.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- 172.21.15.61:0/3562687780 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc0b5f10 0x7ff0fc146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- 172.21.15.61:0/3562687780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc05d130 0x7ff0fc0b5a20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- 172.21.15.61:0/3562687780 >> v2:172.21.15.111:3300/0 conn(0x7ff0fc142eb0 0x7ff0fc05c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/3562687780 >> 172.21.15.61:0/3562687780 conn(0x7ff0fc066d70 msgr2=0x7ff0fc067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/3562687780 shutdown_connections 2022-07-12T08:41:42.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/3562687780 wait complete. 2022-07-12T08:41:42.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 Processor -- start 2022-07-12T08:41:42.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- start start 2022-07-12T08:41:42.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0fc05d130 0x7ff0fc13f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 0x7ff0fc154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc142eb0 0x7ff0fc13d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc13ec60 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc13e720 con 0x7ff0fc05d130 2022-07-12T08:41:42.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc13e9c0 con 0x7ff0fc142eb0 2022-07-12T08:41:42.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 0x7ff0fc154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 0x7ff0fc154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53842/0 (socket says 172.21.15.61:53842) 2022-07-12T08:41:42.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 -- 172.21.15.61:0/1760815892 learned_addr learned my addr 172.21.15.61:0/1760815892 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:42.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff102838700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc142eb0 0x7ff0fc13d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff102037700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0fc05d130 0x7ff0fc13f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 -- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc142eb0 msgr2=0x7ff0fc13d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc142eb0 0x7ff0fc13d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 -- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0fc05d130 msgr2=0x7ff0fc13f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0fc05d130 0x7ff0fc13f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 -- 172.21.15.61:0/1760815892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f800b1e0 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff102838700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc142eb0 0x7ff0fc13d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:42.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff102037700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0fc05d130 0x7ff0fc13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:42.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff101836700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 0x7ff0fc154f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff0f8002f70 tx=0x7ff0f8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f8011990 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f8011af0 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc0ad840 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc0af070 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.773+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f8011e50 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.777+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc057460 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.777+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff0f8025030 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.777+0000 7ff0eb7fe700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff0dc053ad0 0x7ff0dc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:42.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.777+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff0f806d1c0 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.777+0000 7ff102037700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff0dc053ad0 0x7ff0dc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:42.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.777+0000 7ff102037700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff0dc053ad0 0x7ff0dc055f90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff0ec008000 tx=0x7ff0ec008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:42.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.781+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0f8044120 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.905+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0fc06b2b0 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff0eb7fe700 1 -- 172.21.15.61:0/1760815892 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff0f8047740 con 0x7ff0fc0b5f10 2022-07-12T08:41:42.916 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:42.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff0dc053ad0 msgr2=0x7ff0dc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff0dc053ad0 0x7ff0dc055f90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7ff0ec008000 tx=0x7ff0ec008040).stop 2022-07-12T08:41:42.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 msgr2=0x7ff0fc154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:42.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 0x7ff0fc154f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff0f8002f70 tx=0x7ff0f8015000).stop 2022-07-12T08:41:42.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 shutdown_connections 2022-07-12T08:41:42.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff0fc142eb0 0x7ff0fc13d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0fc0b5f10 0x7ff0fc154f90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff0dc053ad0 0x7ff0dc055f90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 --2- 172.21.15.61:0/1760815892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0fc05d130 0x7ff0fc13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:42.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 >> 172.21.15.61:0/1760815892 conn(0x7ff0fc066d70 msgr2=0x7ff0fc141e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:42.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 shutdown_connections 2022-07-12T08:41:42.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:42.909+0000 7ff103039700 1 -- 172.21.15.61:0/1760815892 wait complete. 2022-07-12T08:41:42.936 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:41:43.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.041+0000 7f534759e700 1 Processor -- start 2022-07-12T08:41:43.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.041+0000 7f534759e700 1 -- start start 2022-07-12T08:41:43.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f534805c6d0 0x7f53480b6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348143500 con 0x7f5348142eb0 2022-07-12T08:41:43.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348143290 con 0x7f534805d130 2022-07-12T08:41:43.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5348143760 con 0x7f534805c6d0 2022-07-12T08:41:43.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f5346d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f534805c6d0 0x7f53480b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534659c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5348142eb0 0x7f53480b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42066/0 (socket says 172.21.15.61:42066) 2022-07-12T08:41:43.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534659c700 1 -- 172.21.15.61:0/2214806866 learned_addr learned my addr 172.21.15.61:0/2214806866 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:43.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f5346d9d700 1 -- 172.21.15.61:0/2214806866 >> v1:172.21.15.111:6789/0 conn(0x7f534805d130 legacy=0x7f53480b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f5346d9d700 1 -- 172.21.15.61:0/2214806866 >> v1:172.21.15.61:6789/0 conn(0x7f5348142eb0 legacy=0x7f53480b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534559a700 1 -- 172.21.15.61:0/2214806866 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269028711 0 0) 0x7f5348143500 con 0x7f5348142eb0 2022-07-12T08:41:43.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f5346d9d700 1 -- 172.21.15.61:0/2214806866 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53480b4030 con 0x7f534805c6d0 2022-07-12T08:41:43.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f5346d9d700 1 --2- 172.21.15.61:0/2214806866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805c6d0 0x7f53480b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5348143c30 tx=0x7f53400091d0).ready entity=mon.0 client_cookie=e3d6d809ccab0b16 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:43.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534559a700 1 -- 172.21.15.61:0/2214806866 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f534000b860 con 0x7f534805c6d0 2022-07-12T08:41:43.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534559a700 1 -- 172.21.15.61:0/2214806866 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534000b9c0 con 0x7f534805c6d0 2022-07-12T08:41:43.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534559a700 1 -- 172.21.15.61:0/2214806866 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f534000bb30 con 0x7f534805c6d0 2022-07-12T08:41:43.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- 172.21.15.61:0/2214806866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805c6d0 msgr2=0x7f53480b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 --2- 172.21.15.61:0/2214806866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805c6d0 0x7f53480b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f5348143c30 tx=0x7f53400091d0).stop 2022-07-12T08:41:43.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- 172.21.15.61:0/2214806866 shutdown_connections 2022-07-12T08:41:43.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 --2- 172.21.15.61:0/2214806866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805c6d0 0x7f53480b6130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- 172.21.15.61:0/2214806866 >> 172.21.15.61:0/2214806866 conn(0x7f5348066d70 msgr2=0x7f5348067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:43.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- 172.21.15.61:0/2214806866 shutdown_connections 2022-07-12T08:41:43.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.045+0000 7f534759e700 1 -- 172.21.15.61:0/2214806866 wait complete. 2022-07-12T08:41:43.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 Processor -- start 2022-07-12T08:41:43.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 -- start start 2022-07-12T08:41:43.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534805c6d0 0x7f53480b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 0x7f5348155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5348142eb0 0x7f5348155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5348145c00 con 0x7f534805d130 2022-07-12T08:41:43.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53481457d0 con 0x7f534805c6d0 2022-07-12T08:41:43.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5348143f90 con 0x7f5348142eb0 2022-07-12T08:41:43.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 0x7f5348155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 0x7f5348155230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53856/0 (socket says 172.21.15.61:53856) 2022-07-12T08:41:43.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f534659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534805c6d0 0x7f53480b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 -- 172.21.15.61:0/877925470 learned_addr learned my addr 172.21.15.61:0/877925470 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:43.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5346d9d700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5348142eb0 0x7f5348155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 -- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5348142eb0 msgr2=0x7f5348155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5348142eb0 0x7f5348155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 -- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534805c6d0 msgr2=0x7f53480b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534805c6d0 0x7f53480b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.049+0000 7f5345d9b700 1 -- 172.21.15.61:0/877925470 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340009470 con 0x7f534805d130 2022-07-12T08:41:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f5345d9b700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 0x7f5348155230 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f534805d5a0 tx=0x7f533c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f533c012400 con 0x7f534805d130 2022-07-12T08:41:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533c012560 con 0x7f534805d130 2022-07-12T08:41:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534813f2f0 con 0x7f534805d130 2022-07-12T08:41:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f533c0128c0 con 0x7f534805d130 2022-07-12T08:41:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53480af890 con 0x7f534805d130 2022-07-12T08:41:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5348144a90 con 0x7f534805d130 2022-07-12T08:41:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f533c019030 con 0x7f534805d130 2022-07-12T08:41:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f53377fe700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5324053a60 0x7f5324055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.053+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f533c06b240 con 0x7f534805d130 2022-07-12T08:41:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.057+0000 7f534659c700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5324053a60 0x7f5324055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.057+0000 7f534659c700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5324053a60 0x7f5324055f20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f533800d000 tx=0x7f5338009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:43.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.061+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5348144a90 con 0x7f534805d130 2022-07-12T08:41:43.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.185+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5348144a90 con 0x7f534805d130 2022-07-12T08:41:43.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.185+0000 7f53377fe700 1 -- 172.21.15.61:0/877925470 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5348144a90 con 0x7f534805d130 2022-07-12T08:41:43.196 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:43.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5324053a60 msgr2=0x7f5324055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5324053a60 0x7f5324055f20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f533800d000 tx=0x7f5338009040).stop 2022-07-12T08:41:43.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 msgr2=0x7f5348155230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 0x7f5348155230 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f534805d5a0 tx=0x7f533c007c50).stop 2022-07-12T08:41:43.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 shutdown_connections 2022-07-12T08:41:43.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5348142eb0 0x7f5348155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f534805d130 0x7f5348155230 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5324053a60 0x7f5324055f20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 --2- 172.21.15.61:0/877925470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f534805c6d0 0x7f53480b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 >> 172.21.15.61:0/877925470 conn(0x7f5348066d70 msgr2=0x7f5348141e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:43.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 shutdown_connections 2022-07-12T08:41:43.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.189+0000 7f534759e700 1 -- 172.21.15.61:0/877925470 wait complete. 2022-07-12T08:41:43.260 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-12T08:41:43.262 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-12T08:41:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.365+0000 7f83989b8700 1 Processor -- start 2022-07-12T08:41:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.365+0000 7f83989b8700 1 -- start start 2022-07-12T08:41:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f83940b4720 0x7f8394146570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8394146a60 0x7f839405bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83940b6b50 con 0x7f8394146a60 2022-07-12T08:41:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83940b4f10 con 0x7f83940b4720 2022-07-12T08:41:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83940b51b0 con 0x7f83940b6770 2022-07-12T08:41:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8394146a60 0x7f839405bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f83940b4720 0x7f8394146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8394146a60 0x7f839405bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53866/0 (socket says 172.21.15.61:53866) 2022-07-12T08:41:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f83940b4720 0x7f8394146570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:60668/0 (socket says 172.21.15.61:60668) 2022-07-12T08:41:43.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 -- 172.21.15.61:0/1359781628 learned_addr learned my addr 172.21.15.61:0/1359781628 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:43.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f8392d9d700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 -- 172.21.15.61:0/1359781628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 msgr2=0x7f8394146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 -- 172.21.15.61:0/1359781628 >> v2:172.21.15.111:3300/0 conn(0x7f83940b4720 msgr2=0x7f8394146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:43.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 --2- 172.21.15.61:0/1359781628 >> v2:172.21.15.111:3300/0 conn(0x7f83940b4720 0x7f8394146570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 -- 172.21.15.61:0/1359781628 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83940b5450 con 0x7f8394146a60 2022-07-12T08:41:43.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f8392d9d700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:43.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f839359e700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839405bd10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f838c002f70 tx=0x7f838c006eb0).ready entity=mon.0 client_cookie=bdba58ebb3614806 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:43.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f8391d9b700 1 -- 172.21.15.61:0/1359781628 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838c00d570 con 0x7f8394146a60 2022-07-12T08:41:43.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f8391d9b700 1 -- 172.21.15.61:0/1359781628 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c000e90 con 0x7f8394146a60 2022-07-12T08:41:43.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f8391d9b700 1 -- 172.21.15.61:0/1359781628 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838c00d900 con 0x7f8394146a60 2022-07-12T08:41:43.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- 172.21.15.61:0/1359781628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 msgr2=0x7f839405bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839405bd10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f838c002f70 tx=0x7f838c006eb0).stop 2022-07-12T08:41:43.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- 172.21.15.61:0/1359781628 shutdown_connections 2022-07-12T08:41:43.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839405bd10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- 172.21.15.61:0/1359781628 >> v2:172.21.15.111:3300/0 conn(0x7f83940b4720 0x7f8394146570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 --2- 172.21.15.61:0/1359781628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- 172.21.15.61:0/1359781628 >> 172.21.15.61:0/1359781628 conn(0x7f83940b9d50 msgr2=0x7f83940b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:43.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- 172.21.15.61:0/1359781628 shutdown_connections 2022-07-12T08:41:43.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.369+0000 7f83989b8700 1 -- 172.21.15.61:0/1359781628 wait complete. 2022-07-12T08:41:43.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 Processor -- start 2022-07-12T08:41:43.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 -- start start 2022-07-12T08:41:43.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83940b4720 0x7f839409d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839409c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940b3230 con 0x7f8394146a60 2022-07-12T08:41:43.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83940b34d0 con 0x7f83940b4720 2022-07-12T08:41:43.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839409ec90 con 0x7f83940b6770 2022-07-12T08:41:43.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839409c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839409c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53872/0 (socket says 172.21.15.61:53872) 2022-07-12T08:41:43.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 -- 172.21.15.61:0/1349942950 learned_addr learned my addr 172.21.15.61:0/1349942950 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:43.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f8392d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83940b4720 0x7f839409d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839259c700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 -- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 msgr2=0x7f8394143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394143340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 -- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83940b4720 msgr2=0x7f839409d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:43.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83940b4720 0x7f839409d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:43.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 -- 172.21.15.61:0/1349942950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c00b1e0 con 0x7f8394146a60 2022-07-12T08:41:43.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f839359e700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839409c340 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f838c002f70 tx=0x7f838c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:43.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838c006be0 con 0x7f8394146a60 2022-07-12T08:41:43.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c011c30 con 0x7f8394146a60 2022-07-12T08:41:43.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f838c01e460 con 0x7f8394146a60 2022-07-12T08:41:43.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839409c830 con 0x7f8394146a60 2022-07-12T08:41:43.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.373+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839409b1c0 con 0x7f8394146a60 2022-07-12T08:41:43.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.377+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839409ee50 con 0x7f8394146a60 2022-07-12T08:41:43.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.377+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f838c025030 con 0x7f8394146a60 2022-07-12T08:41:43.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.377+0000 7f838bfff700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8378053920 0x7f8378055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:43.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.377+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f838c06d000 con 0x7f8394146a60 2022-07-12T08:41:43.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.377+0000 7f8392d9d700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8378053920 0x7f8378055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:43.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.377+0000 7f8392d9d700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8378053920 0x7f8378055de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f837c00c000 tx=0x7f837c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:43.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.385+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f838c043f60 con 0x7f8394146a60 2022-07-12T08:41:43.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:43.549+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7f83940b2b80 con 0x7f8394146a60 2022-07-12T08:41:44.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.197+0000 7f838bfff700 1 -- 172.21.15.61:0/1349942950 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v46) v1 ==== 142+0+0 (secure 0 0 0) 0x7f838c023070 con 0x7f8394146a60 2022-07-12T08:41:44.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8378053920 msgr2=0x7f8378055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8378053920 0x7f8378055de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f837c00c000 tx=0x7f837c008040).stop 2022-07-12T08:41:44.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 msgr2=0x7f839409c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839409c340 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f838c002f70 tx=0x7f838c015000).stop 2022-07-12T08:41:44.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 shutdown_connections 2022-07-12T08:41:44.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8394146a60 0x7f839409c340 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83940b6770 0x7f8394143340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8378053920 0x7f8378055de0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 --2- 172.21.15.61:0/1349942950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83940b4720 0x7f839409d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 >> 172.21.15.61:0/1349942950 conn(0x7f83940b9d50 msgr2=0x7f83940b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:44.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 shutdown_connections 2022-07-12T08:41:44.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.201+0000 7f83989b8700 1 -- 172.21.15.61:0/1349942950 wait complete. 2022-07-12T08:41:44.224 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:41:44.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.341+0000 7f2faa1af700 1 Processor -- start 2022-07-12T08:41:44.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.341+0000 7f2faa1af700 1 -- start start 2022-07-12T08:41:44.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa40b9db0 0x7f2fa40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:44.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2fa40b3ea0 0x7f2fa405c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:44.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa40b3120 con 0x7f2fa405c5f0 2022-07-12T08:41:44.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa405bd10 con 0x7f2fa40b3ea0 2022-07-12T08:41:44.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40b33c0 con 0x7f2fa40b9db0 2022-07-12T08:41:44.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2fa405c5f0 0x7f2fa40b4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42098/0 (socket says 172.21.15.61:42098) 2022-07-12T08:41:44.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2fa40b3ea0 0x7f2fa405c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:44.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa3fff700 1 -- 172.21.15.61:0/427870111 learned_addr learned my addr 172.21.15.61:0/427870111 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:44.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa37fe700 1 --2- 172.21.15.61:0/427870111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa40b9db0 0x7f2fa40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:44.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa27fc700 1 -- 172.21.15.61:0/427870111 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601901533 0 0) 0x7f2fa40b3120 con 0x7f2fa405c5f0 2022-07-12T08:41:44.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa27fc700 1 -- 172.21.15.61:0/427870111 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f84003260 con 0x7f2fa405c5f0 2022-07-12T08:41:44.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa2ffd700 1 -- 172.21.15.61:0/427870111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa40b9db0 msgr2=0x7f2fa40b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:44.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa2ffd700 1 --2- 172.21.15.61:0/427870111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa40b9db0 0x7f2fa40b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa2ffd700 1 -- 172.21.15.61:0/427870111 >> v1:172.21.15.61:6789/0 conn(0x7f2fa405c5f0 legacy=0x7f2fa40b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa2ffd700 1 -- 172.21.15.61:0/427870111 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40ba190 con 0x7f2fa40b3ea0 2022-07-12T08:41:44.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa2ffd700 1 --2- 172.21.15.61:0/427870111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa405c100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2f8c002f70 tx=0x7f2f8c006df0).ready entity=mon.0 client_cookie=e8502296a8e0c36f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:44.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa27fc700 1 -- 172.21.15.61:0/427870111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f8c00d4a0 con 0x7f2fa40b3ea0 2022-07-12T08:41:44.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa27fc700 1 -- 172.21.15.61:0/427870111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8c000e90 con 0x7f2fa40b3ea0 2022-07-12T08:41:44.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2fa27fc700 1 -- 172.21.15.61:0/427870111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f8c00d7f0 con 0x7f2fa40b3ea0 2022-07-12T08:41:44.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- 172.21.15.61:0/427870111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b3ea0 msgr2=0x7f2fa405c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 --2- 172.21.15.61:0/427870111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa405c100 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2f8c002f70 tx=0x7f2f8c006df0).stop 2022-07-12T08:41:44.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- 172.21.15.61:0/427870111 shutdown_connections 2022-07-12T08:41:44.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 --2- 172.21.15.61:0/427870111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa405c100 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 --2- 172.21.15.61:0/427870111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa40b9db0 0x7f2fa40b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- 172.21.15.61:0/427870111 >> 172.21.15.61:0/427870111 conn(0x7f2fa4067960 msgr2=0x7f2fa40b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:44.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- 172.21.15.61:0/427870111 shutdown_connections 2022-07-12T08:41:44.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.345+0000 7f2faa1af700 1 -- 172.21.15.61:0/427870111 wait complete. 2022-07-12T08:41:44.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 Processor -- start 2022-07-12T08:41:44.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 -- start start 2022-07-12T08:41:44.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa405c5f0 0x7f2fa40b9260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:44.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa4155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:44.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 0x7f2fa41468c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:44.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40aed20 con 0x7f2fa40b9db0 2022-07-12T08:41:44.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40aeae0 con 0x7f2fa40b3ea0 2022-07-12T08:41:44.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2faa1af700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40aef50 con 0x7f2fa405c5f0 2022-07-12T08:41:44.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 0x7f2fa41468c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:44.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 0x7f2fa41468c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53884/0 (socket says 172.21.15.61:53884) 2022-07-12T08:41:44.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa37fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa405c5f0 0x7f2fa40b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:44.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 -- 172.21.15.61:0/1124693904 learned_addr learned my addr 172.21.15.61:0/1124693904 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:44.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa2ffd700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa4155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:44.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 -- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa405c5f0 msgr2=0x7f2fa40b9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa405c5f0 0x7f2fa40b9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 -- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa40b3ea0 msgr2=0x7f2fa4155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa4155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa3fff700 1 -- 172.21.15.61:0/1124693904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c00b180 con 0x7f2fa40b9db0 2022-07-12T08:41:44.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.349+0000 7f2fa37fe700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa405c5f0 0x7f2fa40b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:44.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa3fff700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 0x7f2fa41468c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f2f9c001490 tx=0x7f2f9c007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:44.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f9c005ba0 con 0x7f2fa40b9db0 2022-07-12T08:41:44.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c005d00 con 0x7f2fa40b9db0 2022-07-12T08:41:44.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa40af1f0 con 0x7f2fa40b9db0 2022-07-12T08:41:44.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f9c0072c0 con 0x7f2fa40b9db0 2022-07-12T08:41:44.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa4144e40 con 0x7f2fa40b9db0 2022-07-12T08:41:44.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa40b0000 con 0x7f2fa40b9db0 2022-07-12T08:41:44.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2f9c01a070 con 0x7f2fa40b9db0 2022-07-12T08:41:44.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa0ff9700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2f88053ad0 0x7f2f88055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:44.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2f9c06b1c0 con 0x7f2fa40b9db0 2022-07-12T08:41:44.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa37fe700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2f88053ad0 0x7f2f88055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:44.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.353+0000 7f2fa37fe700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2f88053ad0 0x7f2f88055f90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2f9400b000 tx=0x7f2f94006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:44.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.357+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2fa40b0000 con 0x7f2fa40b9db0 2022-07-12T08:41:44.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.481+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2fa40b0000 con 0x7f2fa40b9db0 2022-07-12T08:41:44.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.489+0000 7f2fa0ff9700 1 -- 172.21.15.61:0/1124693904 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2fa40b0000 con 0x7f2fa40b9db0 2022-07-12T08:41:44.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2f88053ad0 msgr2=0x7f2f88055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2f88053ad0 0x7f2f88055f90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2f9400b000 tx=0x7f2f94006df0).stop 2022-07-12T08:41:44.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 msgr2=0x7f2fa41468c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:44.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 0x7f2fa41468c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f2f9c001490 tx=0x7f2f9c007e70).stop 2022-07-12T08:41:44.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 shutdown_connections 2022-07-12T08:41:44.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fa40b9db0 0x7f2fa41468c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2f88053ad0 0x7f2f88055f90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa40b3ea0 0x7f2fa4155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 --2- 172.21.15.61:0/1124693904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fa405c5f0 0x7f2fa40b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:44.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 >> 172.21.15.61:0/1124693904 conn(0x7f2fa4067960 msgr2=0x7f2fa4143c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:44.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 shutdown_connections 2022-07-12T08:41:44.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:44.493+0000 7f2faa1af700 1 -- 172.21.15.61:0/1124693904 wait complete. 2022-07-12T08:41:44.515 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:41:44.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:41:44.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:41:44.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:41:44.632 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:41:44.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:41:44.689 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:41:44.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:44.691 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:44.692 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:44.710 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:41:44.712 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:41:44.713 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:44.714 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:44.715 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:44.754 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:41:44.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:41:44.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:44.761 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:44.761 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:41:44.816 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-12T08:41:44.817 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-12T08:41:44.818 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:41:44.819 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:41:44.820 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:44.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:44.917 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:44.918 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:44.926 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:41:44.928 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:41:44.929 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:41:44.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:41:44.930 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:41:45.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.044 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.044 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:45.068 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:41:45.070 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:41:45.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.071 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.072 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:45.133 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:41:45.134 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:41:45.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.136 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.136 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:41:45.200 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected 2022-07-12T08:41:45.201 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2022-07-12T08:41:45.201 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:41:45.207 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:41:45.210 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:45.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.279 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.280 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:45.362 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:41:45.363 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:41:45.364 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:41:45.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:41:45.365 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:41:45.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.436 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.436 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:45.448 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2022-07-12T08:41:45.448 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-07-12T08:41:45.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.450 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.451 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:45.471 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2022-07-12T08:41:45.471 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-07-12T08:41:45.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.475 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.475 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:41:45.543 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': Transport endpoint is not connected 2022-07-12T08:41:45.544 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.2 2022-07-12T08:41:45.550 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2022-07-12T08:41:45.551 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:41:45.552 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:45.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:41:45.641 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:41:45.642 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:45.756 INFO:teuthology.orchestra.run.smithi137.stdout:ext2/ext3 2022-07-12T08:41:45.757 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:41:45.767 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:41:45.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:41:45.769 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:41:45.797 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:41:45.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f43748f6700 1 Processor -- start 2022-07-12T08:41:45.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f43748f6700 1 -- start start 2022-07-12T08:41:45.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f43748f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f43700b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:45.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f43748f6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700b0980 con 0x7f4370142690 2022-07-12T08:41:45.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f43748f6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700af0d0 con 0x7f437005fb90 2022-07-12T08:41:45.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f43748f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437005ff70 con 0x7f43700ad000 2022-07-12T08:41:45.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4370142690 0x7f4370142b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42112/0 (socket says 172.21.15.61:42112) 2022-07-12T08:41:45.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436ffff700 1 -- 172.21.15.61:0/2457759936 learned_addr learned my addr 172.21.15.61:0/2457759936 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:45.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436effd700 1 --2- 172.21.15.61:0/2457759936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f43700b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:45.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790979000 0 0) 0x7f43700af0d0 con 0x7f437005fb90 2022-07-12T08:41:45.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c004370 con 0x7f437005fb90 2022-07-12T08:41:45.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521813915 0 0) 0x7f43700b0980 con 0x7f4370142690 2022-07-12T08:41:46.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43700af0d0 con 0x7f4370142690 2022-07-12T08:41:46.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436effd700 1 -- 172.21.15.61:0/2457759936 >> v1:172.21.15.111:6789/0 conn(0x7f437005fb90 legacy=0x7f43700b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436effd700 1 -- 172.21.15.61:0/2457759936 >> v1:172.21.15.61:6789/0 conn(0x7f4370142690 legacy=0x7f4370142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436effd700 1 -- 172.21.15.61:0/2457759936 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700af3b0 con 0x7f43700ad000 2022-07-12T08:41:46.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436effd700 1 --2- 172.21.15.61:0/2457759936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f43700b1290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4364002f70 tx=0x7f4364006df0).ready entity=mon.2 client_cookie=90bd21d79eff6ecd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:46.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f436400d5c0 con 0x7f43700ad000 2022-07-12T08:41:46.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364000e90 con 0x7f43700ad000 2022-07-12T08:41:46.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.928+0000 7f436e7fc700 1 -- 172.21.15.61:0/2457759936 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f436400d910 con 0x7f43700ad000 2022-07-12T08:41:46.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- 172.21.15.61:0/2457759936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 msgr2=0x7f43700b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 --2- 172.21.15.61:0/2457759936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f43700b1290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4364002f70 tx=0x7f4364006df0).stop 2022-07-12T08:41:46.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- 172.21.15.61:0/2457759936 shutdown_connections 2022-07-12T08:41:46.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 --2- 172.21.15.61:0/2457759936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f43700b1290 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- 172.21.15.61:0/2457759936 >> 172.21.15.61:0/2457759936 conn(0x7f4370063cd0 msgr2=0x7f437005f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- 172.21.15.61:0/2457759936 shutdown_connections 2022-07-12T08:41:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- 172.21.15.61:0/2457759936 wait complete. 2022-07-12T08:41:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 Processor -- start 2022-07-12T08:41:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- start start 2022-07-12T08:41:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 0x7f4370147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:46.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f4370139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:46.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370142690 0x7f4370139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:46.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4370065b90 con 0x7f437005fb90 2022-07-12T08:41:46.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437013abb0 con 0x7f4370142690 2022-07-12T08:41:46.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f43748f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437013ae50 con 0x7f43700ad000 2022-07-12T08:41:46.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f436f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 0x7f4370147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:46.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f436f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 0x7f4370147ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53896/0 (socket says 172.21.15.61:53896) 2022-07-12T08:41:46.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f436f7fe700 1 -- 172.21.15.61:0/3786595696 learned_addr learned my addr 172.21.15.61:0/3786595696 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:46.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f436effd700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f4370139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:46.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.932+0000 7f436ffff700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370142690 0x7f4370139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:46.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436f7fe700 1 -- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 msgr2=0x7f4370139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436f7fe700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f4370139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436f7fe700 1 -- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370142690 msgr2=0x7f4370139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436f7fe700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370142690 0x7f4370139b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436f7fe700 1 -- 172.21.15.61:0/3786595696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436400b180 con 0x7f437005fb90 2022-07-12T08:41:46.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436f7fe700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 0x7f4370147ad0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4360001110 tx=0x7f4360007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:46.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4360012400 con 0x7f437005fb90 2022-07-12T08:41:46.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360012560 con 0x7f437005fb90 2022-07-12T08:41:46.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4370065880 con 0x7f437005fb90 2022-07-12T08:41:46.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4370066090 con 0x7f437005fb90 2022-07-12T08:41:46.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43600128c0 con 0x7f437005fb90 2022-07-12T08:41:46.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4370141da0 con 0x7f437005fb90 2022-07-12T08:41:46.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4360019030 con 0x7f437005fb90 2022-07-12T08:41:46.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436cff9700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4350053920 0x7f4350055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:46.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.936+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f436006b360 con 0x7f437005fb90 2022-07-12T08:41:46.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.940+0000 7f436effd700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4350053920 0x7f4350055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:46.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.940+0000 7f436effd700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4350053920 0x7f4350055de0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f436401a000 tx=0x7f4364014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:46.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:45.944+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4370141da0 con 0x7f437005fb90 2022-07-12T08:41:46.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.072+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4370141da0 con 0x7f437005fb90 2022-07-12T08:41:46.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f436cff9700 1 -- 172.21.15.61:0/3786595696 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v22) v1 ==== 81+0+33693 (secure 0 0 0) 0x7f4370141da0 con 0x7f437005fb90 2022-07-12T08:41:46.121 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:46.121 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:41:46.124 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 22, 2022-07-12T08:41:46.124 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:41:46.125 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:41:46.125 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:41:15.583994+0000", 2022-07-12T08:41:46.126 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:41:46.126 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:41:46.127 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:41:46.127 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:41:46.128 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:41:46.128 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:41:46.129 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:41:46.130 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:41:46.130 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:41:46.131 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:41:46.131 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:41:46.131 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:41:46.137 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:41:46.138 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:41:46.138 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:41:46.138 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 4, 2022-07-12T08:41:46.138 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:41:46.139 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:41:46.139 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:41:46.139 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:41:46.139 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:41:46.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.140 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:41:46.140 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:41:46.140 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:41:46.140 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:41:46.141 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:41:46.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:41:46.141 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:41:46.141 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:41:46.142 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:41:46.142 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:41:46.142 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:41:46.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:41:46.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:41:46.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:41:46.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:41:46.144 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:41:46.144 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:41:46.144 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:41:46.144 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:41:46.145 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:41:46.158 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:41:46.158 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:41:46.160 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:41:46.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.160 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:41:46.161 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:41:46.161 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:41:46.162 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:41:46.162 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:41:46.162 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:41:46.163 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:41:46.163 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:41:46.164 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:41:46.164 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:41:46.164 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:41:46.180 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:41:46.180 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:41:46.181 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:41:46.182 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:41:46.182 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:41:46.182 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:41:46.183 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:41:46.183 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:41:46.184 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:41:46.184 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:41:46.185 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:41:46.185 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:41:46.187 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:41:46.187 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:41:46.188 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:41:46.188 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:41:46.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.189 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:41:46.189 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:41:46.190 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:41:46.190 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:41:46.191 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:41:46.192 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:41:46.192 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:41:46.193 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:41:46.193 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:41:46.193 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:41:46.194 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:41:46.194 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:41:46.195 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:41:46.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.196 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:41:46.196 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:41:46.196 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:41:46.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:41:46.199 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:41:46.199 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:41:46.199 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:41:46.200 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:41:46.200 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:41:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:41:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:41:46.220 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:41:46.225 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:41:46.225 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:41:46.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:41:46.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:41:46.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:41:46.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:41:46.227 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:41:46.263 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:41:46.275 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:41:46.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:41:46.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:41:46.276 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:41:46.276 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:41:46.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.277 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:41:46.277 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:41:46.278 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:41:46.278 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:41:46.279 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:41:46.279 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:41:46.280 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:41:46.280 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:41:46.280 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:41:46.281 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:41:46.281 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:41:46.282 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:41:46.282 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:41:46.283 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:41:46.283 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:41:46.284 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:41:46.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:41:46.284 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:41:46.285 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:41:46.285 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:41:46.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:41:46.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:41:46.289 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:41:46.289 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:41:46.289 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:41:46.290 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:41:46.290 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:41:46.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.291 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:41:46.292 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:41:46.292 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:41:46.292 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:41:46.293 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:41:46.293 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:41:46.296 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:41:46.298 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:41:46.298 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:41:46.300 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:41:46.308 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:41:46.309 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:41:46.309 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:41:46.310 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:41:46.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.311 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.311 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.322 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 3, 2022-07-12T08:41:46.323 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:41:46.323 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:12.785924+0000", 2022-07-12T08:41:46.324 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:41:46.324 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:41:46.324 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:41:46.325 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:41:46.325 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:41:46.328 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:41:46.328 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:41:46.329 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:41:46.329 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:41:46.335 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:41:46.336 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:41:46.336 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:41:46.337 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:41:46.337 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:41:46.337 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:41:46.338 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:41:46.338 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:41:46.339 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:41:46.339 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:41:46.340 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:41:46.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.340 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "21", 2022-07-12T08:41:46.341 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:41:46.341 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:41:46.348 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:41:46.349 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:41:46.349 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:41:46.350 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:41:46.350 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:41:46.351 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:41:46.351 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:41:46.352 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:41:46.352 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:41:46.352 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:41:46.353 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:41:46.353 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:41:46.354 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:41:46.354 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:41:46.355 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:41:46.355 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:41:46.355 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:41:46.356 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:41:46.356 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:41:46.357 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:41:46.357 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:41:46.358 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:41:46.358 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:41:46.359 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:41:46.359 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.360 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:41:46.361 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:41:46.361 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:41:46.361 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:41:46.362 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:41:46.386 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:41:46.387 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:41:46.387 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:41:46.388 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:41:46.388 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:41:46.389 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:41:46.389 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:41:46.389 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:41:46.390 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:41:46.390 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:41:46.391 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.391 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:41:46.392 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:41:46.392 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:41:46.393 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.393 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.394 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 4, 2022-07-12T08:41:46.395 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:41:46.395 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:13.618519+0000", 2022-07-12T08:41:46.396 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:41:46.397 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:41:46.397 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:41:46.397 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:41:46.398 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:41:46.399 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:41:46.399 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:41:46.399 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:41:46.401 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:41:46.402 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:41:46.402 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:41:46.403 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:41:46.403 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:41:46.404 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:41:46.404 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:41:46.405 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:41:46.405 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:41:46.406 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:41:46.407 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:41:46.407 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:41:46.407 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.408 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "22", 2022-07-12T08:41:46.408 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:41:46.409 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:41:46.409 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:41:46.410 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:41:46.411 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:41:46.411 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:41:46.412 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:41:46.412 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:41:46.413 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:41:46.413 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:41:46.414 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:41:46.414 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:41:46.415 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:41:46.415 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:41:46.415 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:41:46.416 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:41:46.417 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:41:46.417 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:41:46.418 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:41:46.418 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:41:46.419 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:41:46.419 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:41:46.419 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:41:46.420 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:41:46.420 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:41:46.421 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:41:46.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.422 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:41:46.423 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:41:46.423 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:41:46.423 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:41:46.424 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:41:46.424 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:41:46.425 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:41:46.427 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:41:46.427 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:41:46.427 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:41:46.428 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:41:46.428 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:41:46.429 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:41:46.429 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:41:46.430 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:41:46.431 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.431 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.432 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.432 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:41:46.433 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:41:46.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.434 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:41:46.434 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:41:46.435 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.435 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.436 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.436 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.437 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.437 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.437 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.438 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-12T08:41:46.439 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.440 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.444 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.444 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.454 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:41:46.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.457 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:41:46.458 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.458 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.459 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.459 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.460 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.461 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.461 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.461 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.462 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:41:46.462 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.463 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.463 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.464 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.464 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:41:46.466 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.466 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.467 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.467 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.468 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.468 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.468 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.468 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.469 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:41:46.469 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.469 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.469 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.470 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.470 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:41:46.471 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.471 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.471 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.472 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.472 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.472 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.473 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.473 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.473 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:41:46.474 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.474 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.475 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:41:46.486 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:41:46.486 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.486 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.487 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.487 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:41:46.488 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:41:46.488 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:41:46.489 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:41:46.489 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.490 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.490 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.491 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.491 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.492 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.492 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:41:46.493 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:41:46.493 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.494 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.494 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.495 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.495 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.496 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.497 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.497 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-12T08:41:46.498 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.499 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.499 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.500 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.501 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.510 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.511 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:41:46.511 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.511 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.512 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.512 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.512 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:41:46.512 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.513 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.513 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.513 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.513 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.513 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.516 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.516 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.516 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:41:46.516 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.517 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.526 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.526 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.527 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:41:46.527 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.527 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.528 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.528 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.528 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.528 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.529 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.529 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.529 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:41:46.529 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.529 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.530 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.530 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:41:46.530 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.531 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.531 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.531 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.531 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.532 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.532 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.532 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:41:46.533 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.533 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.533 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.533 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:41:46.534 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:41:46.534 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.534 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.535 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.535 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:41:46.535 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:41:46.535 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:41:46.536 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:41:46.536 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.536 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.536 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.537 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.537 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.537 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.537 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:41:46.537 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:41:46.538 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.538 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.538 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.538 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.539 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.539 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.539 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.539 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 16, 2022-07-12T08:41:46.540 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.540 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.540 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.540 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.541 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.541 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.541 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:41:46.541 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.542 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.542 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.542 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.542 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:41:46.543 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.543 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.543 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.543 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.544 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.544 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.544 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.544 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.545 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:41:46.545 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.545 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.546 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.546 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:41:46.546 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.546 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.547 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.547 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.547 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.548 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.548 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:41:46.549 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.550 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.550 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.550 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:41:46.550 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.551 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.551 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.552 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.552 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.553 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.553 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:41:46.553 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.554 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.555 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.555 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:41:46.555 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:41:46.555 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:41:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:41:46.557 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:41:46.557 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:41:46.557 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.557 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.559 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:41:46.559 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:41:46.560 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.560 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.560 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.560 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.560 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.561 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.561 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.561 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-12T08:41:46.561 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.562 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.562 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.562 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.562 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.563 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.563 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:41:46.563 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.564 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.564 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.565 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.565 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:41:46.565 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.565 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.566 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.566 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.566 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.566 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.567 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.567 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.567 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:41:46.568 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.568 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.568 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.569 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:41:46.569 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.569 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.569 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.570 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.570 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.570 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.570 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.571 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.571 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:41:46.571 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.571 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:41:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.574 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.574 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.574 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:41:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:41:46.576 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:41:46.576 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.576 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.576 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.577 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:41:46.577 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:41:46.580 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:41:46.580 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:41:46.581 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.582 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.582 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.583 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.583 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.583 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.583 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:41:46.584 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:41:46.584 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.584 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.584 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.585 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.585 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.585 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.585 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-12T08:41:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.588 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.588 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.588 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.588 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:41:46.589 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.589 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.589 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.589 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.590 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:41:46.590 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.590 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.590 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.591 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.591 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.591 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.591 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.592 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.592 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:41:46.592 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.593 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.606 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.606 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.607 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:41:46.607 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.607 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.607 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.608 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.608 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.608 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.608 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.609 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.609 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:41:46.609 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.614 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.615 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.615 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.615 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:41:46.616 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.616 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.616 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.616 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.625 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.626 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.626 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.626 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.626 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:41:46.627 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.627 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.627 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.628 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.629 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:41:46.629 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:41:46.629 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.629 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.630 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.630 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:41:46.630 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:41:46.630 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:41:46.631 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:41:46.631 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.631 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.632 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.632 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.632 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.632 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.648 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:41:46.648 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:41:46.648 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.648 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.649 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.649 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.649 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.649 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.670 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.671 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 19, 2022-07-12T08:41:46.671 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.672 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.672 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.673 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.673 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.673 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.682 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:41:46.683 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.683 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.684 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.684 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.685 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:41:46.685 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.685 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.689 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.699 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.699 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.700 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.700 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.701 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.701 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:41:46.709 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.710 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.710 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.711 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.711 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:41:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.713 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.713 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.714 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.714 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.715 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.715 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:41:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.717 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:41:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.719 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.719 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.721 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.722 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:41:46.722 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.723 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.723 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.724 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.724 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:41:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:41:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.726 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.726 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:41:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:41:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:41:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:41:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.730 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.730 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.732 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:41:46.732 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:41:46.733 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.733 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.734 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-12T08:41:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:41:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:41:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:41:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:41:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.753 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.753 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.753 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.755 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.755 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:41:46.755 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:41:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.758 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.758 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:41:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:41:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:41:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.762 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.762 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.762 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:41:46.763 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:41:46.763 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:41:46.763 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:41:46.763 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.764 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.764 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.764 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.764 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.765 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.765 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:41:46.765 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:41:46.765 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:41:46.766 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:41:46.766 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:41:46.766 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:41:46.767 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:41:46.767 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:41:46.767 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:41:46.768 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 16, 2022-07-12T08:41:46.768 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:41:46.768 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:41:46.769 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:41:46.769 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.769 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.769 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.770 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:41:46.770 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.770 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.770 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:41:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:41:46.773 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.778 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.778 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.779 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:41:46.779 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:41:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.782 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:41:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:41:46.786 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:41:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:41:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:41:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:41:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:41:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.793 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.793 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.794 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:41:46.794 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:41:46.795 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:41:46.795 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.803 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.804 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:41:46.804 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:41:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:41:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:41:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:41:46.806 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:41:46.806 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:41:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:41:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:41:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:41:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.811 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.811 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.812 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:41:46.812 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:41:46.814 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.814 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.815 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.815 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.816 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:41:46.816 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.817 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.817 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.818 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:41:46.818 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.818 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.819 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.819 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.820 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.820 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:41:46.821 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.821 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.822 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.822 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:41:46.824 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.827 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.838 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.838 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.839 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.839 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:41:46.839 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.840 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.840 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.841 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:41:46.841 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.842 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.842 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.843 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.843 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.843 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:41:46.844 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.844 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.844 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.845 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:41:46.845 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.846 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.846 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:41:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:41:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:41:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:41:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:41:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:41:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:41:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:41:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:41:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:41:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:41:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:41:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:41:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:41:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:41:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:41:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:41:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:41:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:41:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:41:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:41:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:41:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:41:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:41:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:41:46.864 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:46.864 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:46.865 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:41:46.865 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:41:46.865 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:41:46.866 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:41:46.866 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:41:46.867 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:41:46.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4350053920 msgr2=0x7f4350055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4350053920 0x7f4350055de0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f436401a000 tx=0x7f4364014000).stop 2022-07-12T08:41:46.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 msgr2=0x7f4370147ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:46.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 0x7f4370147ad0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f4360001110 tx=0x7f4360007b00).stop 2022-07-12T08:41:46.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 shutdown_connections 2022-07-12T08:41:46.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4350053920 0x7f4350055de0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4370142690 0x7f4370139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43700ad000 0x7f4370139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 --2- 172.21.15.61:0/3786595696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f437005fb90 0x7f4370147ad0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:46.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 >> 172.21.15.61:0/3786595696 conn(0x7f4370063cd0 msgr2=0x7f43700672f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:46.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 shutdown_connections 2022-07-12T08:41:46.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.076+0000 7f43748f6700 1 -- 172.21.15.61:0/3786595696 wait complete. 2022-07-12T08:41:46.877 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:47.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 Processor -- start 2022-07-12T08:41:47.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- start start 2022-07-12T08:41:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f5820151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58200abc70 con 0x7f58200ab670 2022-07-12T08:41:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582005c720 con 0x7f58200ab290 2022-07-12T08:41:47.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58200aba90 con 0x7f582005f140 2022-07-12T08:41:47.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581e7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f58200ab670 0x7f58200afa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42122/0 (socket says 172.21.15.61:42122) 2022-07-12T08:41:47.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581e7fc700 1 -- 172.21.15.61:0/153959702 learned_addr learned my addr 172.21.15.61:0/153959702 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:47.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581f7fe700 1 --2- 172.21.15.61:0/153959702 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f5820151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581dffb700 1 -- 172.21.15.61:0/153959702 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466389838 0 0) 0x7f582005c720 con 0x7f58200ab290 2022-07-12T08:41:47.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581dffb700 1 -- 172.21.15.61:0/153959702 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5804003260 con 0x7f58200ab290 2022-07-12T08:41:47.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581dffb700 1 -- 172.21.15.61:0/153959702 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53876055 0 0) 0x7f58200abc70 con 0x7f58200ab670 2022-07-12T08:41:47.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581f7fe700 1 -- 172.21.15.61:0/153959702 >> v1:172.21.15.111:6789/0 conn(0x7f58200ab290 legacy=0x7f582005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581f7fe700 1 -- 172.21.15.61:0/153959702 >> v1:172.21.15.61:6789/0 conn(0x7f58200ab670 legacy=0x7f58200afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581f7fe700 1 -- 172.21.15.61:0/153959702 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582005c9c0 con 0x7f582005f140 2022-07-12T08:41:47.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581f7fe700 1 --2- 172.21.15.61:0/153959702 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f5820151bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5810002f70 tx=0x7f5810006df0).ready entity=mon.2 client_cookie=dd6f0e0ad64d23ce server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:47.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581dffb700 1 -- 172.21.15.61:0/153959702 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f581000d570 con 0x7f582005f140 2022-07-12T08:41:47.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581dffb700 1 -- 172.21.15.61:0/153959702 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5810000e90 con 0x7f582005f140 2022-07-12T08:41:47.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581dffb700 1 -- 172.21.15.61:0/153959702 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f581000d8c0 con 0x7f582005f140 2022-07-12T08:41:47.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- 172.21.15.61:0/153959702 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 msgr2=0x7f5820151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 --2- 172.21.15.61:0/153959702 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f5820151bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5810002f70 tx=0x7f5810006df0).stop 2022-07-12T08:41:47.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- 172.21.15.61:0/153959702 shutdown_connections 2022-07-12T08:41:47.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 --2- 172.21.15.61:0/153959702 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f5820151bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- 172.21.15.61:0/153959702 >> 172.21.15.61:0/153959702 conn(0x7f58200b1950 msgr2=0x7f58200b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:47.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:46.996+0000 7f581ffff700 1 -- 172.21.15.61:0/153959702 shutdown_connections 2022-07-12T08:41:47.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- 172.21.15.61:0/153959702 wait complete. 2022-07-12T08:41:47.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 Processor -- start 2022-07-12T08:41:47.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- start start 2022-07-12T08:41:47.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f582014d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 0x7f58200aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58200ab670 0x7f5820159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582005f5b0 con 0x7f58200ab290 2022-07-12T08:41:47.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582014a6a0 con 0x7f58200ab670 2022-07-12T08:41:47.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5820159480 con 0x7f582005f140 2022-07-12T08:41:47.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 0x7f58200aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 0x7f58200aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53914/0 (socket says 172.21.15.61:53914) 2022-07-12T08:41:47.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 -- 172.21.15.61:0/1955793680 learned_addr learned my addr 172.21.15.61:0/1955793680 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:47.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581f7fe700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58200ab670 0x7f5820159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581effd700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f582014d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 -- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 msgr2=0x7f582014d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f582014d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 -- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58200ab670 msgr2=0x7f5820159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58200ab670 0x7f5820159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 -- 172.21.15.61:0/1955793680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581000b180 con 0x7f58200ab290 2022-07-12T08:41:47.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581e7fc700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 0x7f58200aea00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5818000f40 tx=0x7f5818007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:47.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5818005ba0 con 0x7f58200ab290 2022-07-12T08:41:47.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582014a940 con 0x7f58200ab290 2022-07-12T08:41:47.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5818005d00 con 0x7f58200ab290 2022-07-12T08:41:47.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5818005e70 con 0x7f58200ab290 2022-07-12T08:41:47.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.000+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5820065c40 con 0x7f58200ab290 2022-07-12T08:41:47.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.004+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582014a500 con 0x7f58200ab290 2022-07-12T08:41:47.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.004+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5818010070 con 0x7f58200ab290 2022-07-12T08:41:47.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.004+0000 7f5817fff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57f80539d0 0x7f57f8055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.004+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f581806b6e0 con 0x7f58200ab290 2022-07-12T08:41:47.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.004+0000 7f581effd700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57f80539d0 0x7f57f8055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.004+0000 7f581effd700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57f80539d0 0x7f57f8055e90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f580800d000 tx=0x7f5808007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:47.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.008+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f582014a500 con 0x7f58200ab290 2022-07-12T08:41:47.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.168+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5820159640 con 0x7f58200ab290 2022-07-12T08:41:47.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.168+0000 7f5817fff700 1 -- 172.21.15.61:0/1955793680 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+2137 (secure 0 0 0) 0x7f5818042540 con 0x7f58200ab290 2022-07-12T08:41:47.186 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:47.187 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:15.584363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.111:6835/1570404457","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1570404457},{"type":"v1","addr":"172.21.15.111:6835","nonce":1570404457}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:47.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57f80539d0 msgr2=0x7f57f8055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57f80539d0 0x7f57f8055e90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f580800d000 tx=0x7f5808007ad0).stop 2022-07-12T08:41:47.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 msgr2=0x7f58200aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 0x7f58200aea00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f5818000f40 tx=0x7f5818007f70).stop 2022-07-12T08:41:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 shutdown_connections 2022-07-12T08:41:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57f80539d0 0x7f57f8055e90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f58200ab670 0x7f5820159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58200ab290 0x7f58200aea00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 --2- 172.21.15.61:0/1955793680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f582005f140 0x7f582014d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 >> 172.21.15.61:0/1955793680 conn(0x7f58200b1950 msgr2=0x7f58200b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:47.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 shutdown_connections 2022-07-12T08:41:47.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.172+0000 7f581ffff700 1 -- 172.21.15.61:0/1955793680 wait complete. 2022-07-12T08:41:47.195 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 5 2022-07-12T08:41:47.204 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:41:47.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f84572f9700 1 Processor -- start 2022-07-12T08:41:47.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f84572f9700 1 -- start start 2022-07-12T08:41:47.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f84572f9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8450140150 0x7f84501405e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f84572f9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8450141f90 con 0x7f845005c940 2022-07-12T08:41:47.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f84572f9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8450064150 con 0x7f84500b06b0 2022-07-12T08:41:47.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f84572f9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845005c580 con 0x7f8450140150 2022-07-12T08:41:47.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f8454894700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f845005c940 0x7f8450144320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42136/0 (socket says 172.21.15.61:42136) 2022-07-12T08:41:47.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.304+0000 7f8454894700 1 -- 172.21.15.61:0/2036396190 learned_addr learned my addr 172.21.15.61:0/2036396190 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:47.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8455896700 1 --2- 172.21.15.61:0/2036396190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8450140150 0x7f84501405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8447fff700 1 -- 172.21.15.61:0/2036396190 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888757470 0 0) 0x7f8450141f90 con 0x7f845005c940 2022-07-12T08:41:47.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8447fff700 1 -- 172.21.15.61:0/2036396190 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8434003260 con 0x7f845005c940 2022-07-12T08:41:47.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8447fff700 1 -- 172.21.15.61:0/2036396190 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177983604 0 0) 0x7f8450064150 con 0x7f84500b06b0 2022-07-12T08:41:47.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8455896700 1 -- 172.21.15.61:0/2036396190 >> v1:172.21.15.111:6789/0 conn(0x7f84500b06b0 legacy=0x7f8450064850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8455896700 1 -- 172.21.15.61:0/2036396190 >> v1:172.21.15.61:6789/0 conn(0x7f845005c940 legacy=0x7f8450144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8455896700 1 -- 172.21.15.61:0/2036396190 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500643b0 con 0x7f8450140150 2022-07-12T08:41:47.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8455896700 1 --2- 172.21.15.61:0/2036396190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8450140150 0x7f84501405e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f84480032f0 tx=0x7f84480091d0).ready entity=mon.2 client_cookie=524858b14e4dd2cf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:47.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8447fff700 1 -- 172.21.15.61:0/2036396190 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f844800b6c0 con 0x7f8450140150 2022-07-12T08:41:47.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8447fff700 1 -- 172.21.15.61:0/2036396190 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844800b820 con 0x7f8450140150 2022-07-12T08:41:47.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f8447fff700 1 -- 172.21.15.61:0/2036396190 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f844800b990 con 0x7f8450140150 2022-07-12T08:41:47.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 -- 172.21.15.61:0/2036396190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8450140150 msgr2=0x7f84501405e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 --2- 172.21.15.61:0/2036396190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8450140150 0x7f84501405e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f84480032f0 tx=0x7f84480091d0).stop 2022-07-12T08:41:47.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 -- 172.21.15.61:0/2036396190 shutdown_connections 2022-07-12T08:41:47.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 --2- 172.21.15.61:0/2036396190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8450140150 0x7f84501405e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 -- 172.21.15.61:0/2036396190 >> 172.21.15.61:0/2036396190 conn(0x7f8450062c00 msgr2=0x7f8450062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:47.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 -- 172.21.15.61:0/2036396190 shutdown_connections 2022-07-12T08:41:47.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.308+0000 7f84572f9700 1 -- 172.21.15.61:0/2036396190 wait complete. 2022-07-12T08:41:47.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 Processor -- start 2022-07-12T08:41:47.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 -- start start 2022-07-12T08:41:47.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 0x7f8450155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84500b06b0 0x7f84501467d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8450140150 0x7f845009e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845013ea00 con 0x7f845005c940 2022-07-12T08:41:47.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450155540 con 0x7f8450140150 2022-07-12T08:41:47.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84501557e0 con 0x7f84500b06b0 2022-07-12T08:41:47.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 0x7f8450155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8454894700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84500b06b0 0x7f84501467d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 0x7f8450155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53924/0 (socket says 172.21.15.61:53924) 2022-07-12T08:41:47.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 -- 172.21.15.61:0/2040172946 learned_addr learned my addr 172.21.15.61:0/2040172946 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:47.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455896700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8450140150 0x7f845009e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 -- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84500b06b0 msgr2=0x7f84501467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84500b06b0 0x7f84501467d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 -- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8450140150 msgr2=0x7f845009e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:47.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8450140150 0x7f845009e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:47.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 -- 172.21.15.61:0/2040172946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8448009470 con 0x7f845005c940 2022-07-12T08:41:47.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f8455095700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 0x7f8450155f20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8440000bc0 tx=0x7f8440007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:47.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8440012400 con 0x7f845005c940 2022-07-12T08:41:47.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8440012560 con 0x7f845005c940 2022-07-12T08:41:47.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8450155a80 con 0x7f845005c940 2022-07-12T08:41:47.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84400128c0 con 0x7f845005c940 2022-07-12T08:41:47.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.312+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8450146cc0 con 0x7f845005c940 2022-07-12T08:41:47.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.316+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8450057460 con 0x7f845005c940 2022-07-12T08:41:47.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.316+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8440019030 con 0x7f845005c940 2022-07-12T08:41:47.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.316+0000 7f84467fc700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8438053ad0 0x7f8438055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:47.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.316+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f844006b510 con 0x7f845005c940 2022-07-12T08:41:47.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.316+0000 7f8454894700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8438053ad0 0x7f8438055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:47.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.316+0000 7f8454894700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8438053ad0 0x7f8438055f90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f844c00d000 tx=0x7f844c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:47.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.320+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8440042520 con 0x7f845005c940 2022-07-12T08:41:47.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:47.480+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f84500447a0 con 0x7f845005c940 2022-07-12T08:41:48.225 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:41:48.208+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:41:48.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.212+0000 7f84467fc700 1 -- 172.21.15.61:0/2040172946 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v6) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8440021070 con 0x7f845005c940 2022-07-12T08:41:48.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8438053ad0 msgr2=0x7f8438055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8438053ad0 0x7f8438055f90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f844c00d000 tx=0x7f844c009040).stop 2022-07-12T08:41:48.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 msgr2=0x7f8450155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 0x7f8450155f20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8440000bc0 tx=0x7f8440007af0).stop 2022-07-12T08:41:48.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 shutdown_connections 2022-07-12T08:41:48.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8438053ad0 0x7f8438055f90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8450140150 0x7f845009e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84500b06b0 0x7f84501467d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 --2- 172.21.15.61:0/2040172946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f845005c940 0x7f8450155f20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 >> 172.21.15.61:0/2040172946 conn(0x7f8450062c00 msgr2=0x7f84501419b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:48.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 shutdown_connections 2022-07-12T08:41:48.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.216+0000 7f84572f9700 1 -- 172.21.15.61:0/2040172946 wait complete. 2022-07-12T08:41:48.236 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:41:48.244 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:41:48.255 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:41:48.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.368+0000 7f0f2d934700 1 Processor -- start 2022-07-12T08:41:48.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.368+0000 7f0f2d934700 1 -- start start 2022-07-12T08:41:48.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.368+0000 7f0f2d934700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f280b6ac0 0x7f0f28143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0f2805caf0 0x7f0f280b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0f2805d040 0x7f0f280b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f28060340 con 0x7f0f2805d040 2022-07-12T08:41:48.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f280607e0 con 0x7f0f2805caf0 2022-07-12T08:41:48.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280b6ea0 con 0x7f0f280b6ac0 2022-07-12T08:41:48.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d133700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0f2805d040 0x7f0f280b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d133700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0f2805d040 0x7f0f280b2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53936/0 (socket says 172.21.15.61:53936) 2022-07-12T08:41:48.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d133700 1 -- 172.21.15.61:0/2550372069 learned_addr learned my addr 172.21.15.61:0/2550372069 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:48.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 --2- 172.21.15.61:0/2550372069 >> v2:172.21.15.111:3300/0 conn(0x7f0f2805caf0 0x7f0f280b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2c932700 1 --2- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f280b6ac0 0x7f0f28143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 -- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f280b6ac0 msgr2=0x7f0f28143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:48.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 --2- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f280b6ac0 0x7f0f28143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 -- 172.21.15.61:0/2550372069 >> v2:172.21.15.61:3300/0 conn(0x7f0f2805d040 msgr2=0x7f0f280b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:48.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 --2- 172.21.15.61:0/2550372069 >> v2:172.21.15.61:3300/0 conn(0x7f0f2805d040 0x7f0f280b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 -- 172.21.15.61:0/2550372069 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f28060540 con 0x7f0f2805caf0 2022-07-12T08:41:48.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f27fff700 1 --2- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f280b1e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f0f14002f70 tx=0x7f0f14006df0).ready entity=mon.1 client_cookie=cedf6cf9215c4faa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:48.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f277fe700 1 -- 172.21.15.61:0/2550372069 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f1400d4a0 con 0x7f0f2805caf0 2022-07-12T08:41:48.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f277fe700 1 -- 172.21.15.61:0/2550372069 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f14000e90 con 0x7f0f2805caf0 2022-07-12T08:41:48.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f277fe700 1 -- 172.21.15.61:0/2550372069 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f1400d7f0 con 0x7f0f2805caf0 2022-07-12T08:41:48.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 msgr2=0x7f0f280b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 --2- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f280b1e70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f0f14002f70 tx=0x7f0f14006df0).stop 2022-07-12T08:41:48.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- 172.21.15.61:0/2550372069 shutdown_connections 2022-07-12T08:41:48.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 --2- 172.21.15.61:0/2550372069 >> v2:172.21.15.61:3300/0 conn(0x7f0f2805d040 0x7f0f280b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 --2- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f280b1e70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 --2- 172.21.15.61:0/2550372069 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f280b6ac0 0x7f0f28143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- 172.21.15.61:0/2550372069 >> 172.21.15.61:0/2550372069 conn(0x7f0f28064570 msgr2=0x7f0f280b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:48.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- 172.21.15.61:0/2550372069 shutdown_connections 2022-07-12T08:41:48.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.372+0000 7f0f2d934700 1 -- 172.21.15.61:0/2550372069 wait complete. 2022-07-12T08:41:48.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 Processor -- start 2022-07-12T08:41:48.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 -- start start 2022-07-12T08:41:48.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f28140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f2805d040 0x7f0f28147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 0x7f0f28152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280b0100 con 0x7f0f280b6ac0 2022-07-12T08:41:48.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280b03a0 con 0x7f0f2805caf0 2022-07-12T08:41:48.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280b0640 con 0x7f0f2805d040 2022-07-12T08:41:48.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 0x7f0f28152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 0x7f0f28152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53940/0 (socket says 172.21.15.61:53940) 2022-07-12T08:41:48.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 -- 172.21.15.61:0/1251509796 learned_addr learned my addr 172.21.15.61:0/1251509796 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:48.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2c932700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f28140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f27fff700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f2805d040 0x7f0f28147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 -- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f2805d040 msgr2=0x7f0f28147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f2805d040 0x7f0f28147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 -- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 msgr2=0x7f0f28140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f28140420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 -- 172.21.15.61:0/1251509796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1400b180 con 0x7f0f280b6ac0 2022-07-12T08:41:48.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d133700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 0x7f0f28152c60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0f2000c000 tx=0x7f0f20009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:48.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f20012600 con 0x7f0f280b6ac0 2022-07-12T08:41:48.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f20012760 con 0x7f0f280b6ac0 2022-07-12T08:41:48.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2810f260 con 0x7f0f280b6ac0 2022-07-12T08:41:48.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f20012ac0 con 0x7f0f280b6ac0 2022-07-12T08:41:48.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.376+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f28147400 con 0x7f0f280b6ac0 2022-07-12T08:41:48.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.380+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f28057460 con 0x7f0f280b6ac0 2022-07-12T08:41:48.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.380+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0f20022070 con 0x7f0f280b6ac0 2022-07-12T08:41:48.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.380+0000 7f0f25ffb700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0f08053920 0x7f0f08055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.380+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0f2006c620 con 0x7f0f280b6ac0 2022-07-12T08:41:48.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.380+0000 7f0f2c932700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0f08053920 0x7f0f08055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.380+0000 7f0f2c932700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0f08053920 0x7f0f08055de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0f1c00b000 tx=0x7f0f1c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:48.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.392+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0f200433a0 con 0x7f0f280b6ac0 2022-07-12T08:41:48.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.560+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f2810f3f0 con 0x7f0f280b6ac0 2022-07-12T08:41:48.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.560+0000 7f0f25ffb700 1 -- 172.21.15.61:0/1251509796 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+1774 (secure 0 0 0) 0x7f0f20016090 con 0x7f0f280b6ac0 2022-07-12T08:41:48.569 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:48.569 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4236,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2521801743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2521801743},{"type":"v1","addr":"172.21.15.61:6833","nonce":2521801743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:14.580428+0000","modified":"2022-07-12T08:41:48.212496+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":23,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-12T08:41:48.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0f08053920 msgr2=0x7f0f08055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0f08053920 0x7f0f08055de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0f1c00b000 tx=0x7f0f1c006df0).stop 2022-07-12T08:41:48.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 msgr2=0x7f0f28152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 0x7f0f28152c60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f0f2000c000 tx=0x7f0f20009290).stop 2022-07-12T08:41:48.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 shutdown_connections 2022-07-12T08:41:48.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f280b6ac0 0x7f0f28152c60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0f2805d040 0x7f0f28147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0f08053920 0x7f0f08055de0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 --2- 172.21.15.61:0/1251509796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0f2805caf0 0x7f0f28140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 >> 172.21.15.61:0/1251509796 conn(0x7f0f28064570 msgr2=0x7f0f28141ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:48.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 shutdown_connections 2022-07-12T08:41:48.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.564+0000 7f0f2d934700 1 -- 172.21.15.61:0/1251509796 wait complete. 2022-07-12T08:41:48.589 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 6 2022-07-12T08:41:48.591 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:41:48.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.700+0000 7fe40ab6c700 1 Processor -- start 2022-07-12T08:41:48.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.700+0000 7fe40ab6c700 1 -- start start 2022-07-12T08:41:48.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe40405cd20 0x7fe40405d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4041465a0 0x7fe4041469e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40414a560 con 0x7fe404060e80 2022-07-12T08:41:48.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe404097d40 con 0x7fe4041465a0 2022-07-12T08:41:48.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40414b340 con 0x7fe40405cd20 2022-07-12T08:41:48.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40a36b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4041465a0 0x7fe4041469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe409369700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe404060e80 0x7fe40414ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42166/0 (socket says 172.21.15.61:42166) 2022-07-12T08:41:48.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe409369700 1 -- 172.21.15.61:0/4223191561 learned_addr learned my addr 172.21.15.61:0/4223191561 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe409b6a700 1 --2- 172.21.15.61:0/4223191561 >> v2:172.21.15.111:3300/0 conn(0x7fe40405cd20 0x7fe40405d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40a36b700 1 -- 172.21.15.61:0/4223191561 >> v2:172.21.15.111:3300/0 conn(0x7fe40405cd20 msgr2=0x7fe40405d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:48.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40a36b700 1 --2- 172.21.15.61:0/4223191561 >> v2:172.21.15.111:3300/0 conn(0x7fe40405cd20 0x7fe40405d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40a36b700 1 -- 172.21.15.61:0/4223191561 >> v1:172.21.15.61:6789/0 conn(0x7fe404060e80 legacy=0x7fe40414ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe408b68700 1 -- 172.21.15.61:0/4223191561 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283204606 0 0) 0x7fe40414a560 con 0x7fe404060e80 2022-07-12T08:41:48.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40a36b700 1 -- 172.21.15.61:0/4223191561 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4040aafa0 con 0x7fe4041465a0 2022-07-12T08:41:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40a36b700 1 --2- 172.21.15.61:0/4223191561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4041469e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe3fc0032f0 tx=0x7fe3fc0091d0).ready entity=mon.0 client_cookie=abc2ee63e658d248 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:48.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe408b68700 1 -- 172.21.15.61:0/4223191561 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3fc009ea0 con 0x7fe4041465a0 2022-07-12T08:41:48.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe408b68700 1 -- 172.21.15.61:0/4223191561 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc00b870 con 0x7fe4041465a0 2022-07-12T08:41:48.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe408b68700 1 -- 172.21.15.61:0/4223191561 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3fc00b9e0 con 0x7fe4041465a0 2022-07-12T08:41:48.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- 172.21.15.61:0/4223191561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 msgr2=0x7fe4041469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/4223191561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4041469e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fe3fc0032f0 tx=0x7fe3fc0091d0).stop 2022-07-12T08:41:48.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- 172.21.15.61:0/4223191561 shutdown_connections 2022-07-12T08:41:48.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/4223191561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4041469e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/4223191561 >> v2:172.21.15.111:3300/0 conn(0x7fe40405cd20 0x7fe40405d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- 172.21.15.61:0/4223191561 >> 172.21.15.61:0/4223191561 conn(0x7fe4040b1a50 msgr2=0x7fe4040b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:48.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- 172.21.15.61:0/4223191561 shutdown_connections 2022-07-12T08:41:48.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.704+0000 7fe40ab6c700 1 -- 172.21.15.61:0/4223191561 wait complete. 2022-07-12T08:41:48.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 Processor -- start 2022-07-12T08:41:48.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 -- start start 2022-07-12T08:41:48.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe40405cd20 0x7fe40409a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404060e80 0x7fe404098940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4040ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe404099de0 con 0x7fe4041465a0 2022-07-12T08:41:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040ae110 con 0x7fe404060e80 2022-07-12T08:41:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40ab6c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4040ae3b0 con 0x7fe40405cd20 2022-07-12T08:41:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.708+0000 7fe40a36b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4040ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404060e80 0x7fe404098940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe40a36b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4040ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53956/0 (socket says 172.21.15.61:53956) 2022-07-12T08:41:48.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe40a36b700 1 -- 172.21.15.61:0/109661619 learned_addr learned my addr 172.21.15.61:0/109661619 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:48.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409b6a700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe40405cd20 0x7fe40409a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 -- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe40405cd20 msgr2=0x7fe40409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe40405cd20 0x7fe40409a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 -- 172.21.15.61:0/109661619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 msgr2=0x7fe4040ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:48.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4040ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:48.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 -- 172.21.15.61:0/109661619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc009470 con 0x7fe404060e80 2022-07-12T08:41:48.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe40a36b700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4040ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:48.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe409369700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404060e80 0x7fe404098940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe4040ad490 tx=0x7fe400007a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:48.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe40000ce40 con 0x7fe404060e80 2022-07-12T08:41:48.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4000070d0 con 0x7fe404060e80 2022-07-12T08:41:48.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40414e3f0 con 0x7fe404060e80 2022-07-12T08:41:48.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe400007430 con 0x7fe404060e80 2022-07-12T08:41:48.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40409afb0 con 0x7fe404060e80 2022-07-12T08:41:48.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4040aef30 con 0x7fe404060e80 2022-07-12T08:41:48.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.712+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe40001a060 con 0x7fe404060e80 2022-07-12T08:41:48.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.716+0000 7fe3f2ffd700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe3e4053a40 0x7fe3e4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:48.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.716+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe40006b5c0 con 0x7fe404060e80 2022-07-12T08:41:48.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.716+0000 7fe409b6a700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe3e4053a40 0x7fe3e4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:48.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.716+0000 7fe409b6a700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe3e4053a40 0x7fe3e4055f00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe3f400c000 tx=0x7fe3f4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:48.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.724+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4000424d0 con 0x7fe404060e80 2022-07-12T08:41:48.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:48.888+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe404057460 con 0x7fe404060e80 2022-07-12T08:41:49.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.232+0000 7fe3f2ffd700 1 -- 172.21.15.61:0/109661619 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe400018020 con 0x7fe404060e80 2022-07-12T08:41:49.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe3e4053a40 msgr2=0x7fe3e4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:49.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe3e4053a40 0x7fe3e4055f00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe3f400c000 tx=0x7fe3f4008040).stop 2022-07-12T08:41:49.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404060e80 msgr2=0x7fe404098940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:49.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404060e80 0x7fe404098940 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe4040ad490 tx=0x7fe400007a80).stop 2022-07-12T08:41:49.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 shutdown_connections 2022-07-12T08:41:49.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4041465a0 0x7fe4040ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe3e4053a40 0x7fe3e4055f00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe404060e80 0x7fe404098940 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 --2- 172.21.15.61:0/109661619 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe40405cd20 0x7fe40409a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 >> 172.21.15.61:0/109661619 conn(0x7fe4040b1a50 msgr2=0x7fe4040b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:49.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 shutdown_connections 2022-07-12T08:41:49.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.236+0000 7fe40ab6c700 1 -- 172.21.15.61:0/109661619 wait complete. 2022-07-12T08:41:49.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:41:49.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.388+0000 7f28f7fff700 1 Processor -- start 2022-07-12T08:41:49.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.388+0000 7f28f7fff700 1 -- start start 2022-07-12T08:41:49.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f8146cb0 0x7f28f80af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:49.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f28f81430f0 0x7f28f805c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:49.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f81460a0 con 0x7f28f80b65d0 2022-07-12T08:41:49.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f8066670 con 0x7f28f81430f0 2022-07-12T08:41:49.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f81462d0 con 0x7f28f8146cb0 2022-07-12T08:41:49.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f28f81430f0 0x7f28f805c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:49.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f28f81430f0 0x7f28f805c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53964/0 (socket says 172.21.15.61:53964) 2022-07-12T08:41:49.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 -- 172.21.15.61:0/14710945 learned_addr learned my addr 172.21.15.61:0/14710945 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:49.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f67fc700 1 --2- 172.21.15.61:0/14710945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f8146cb0 0x7f28f80af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:49.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 -- 172.21.15.61:0/14710945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f8146cb0 msgr2=0x7f28f80af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:49.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 --2- 172.21.15.61:0/14710945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f8146cb0 0x7f28f80af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 -- 172.21.15.61:0/14710945 >> v1:172.21.15.111:6789/0 conn(0x7f28f80b65d0 legacy=0x7f28f805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:49.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 -- 172.21.15.61:0/14710945 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8146570 con 0x7f28f81430f0 2022-07-12T08:41:49.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f77fe700 1 --2- 172.21.15.61:0/14710945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f81430f0 0x7f28f805c420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f28f00032f0 tx=0x7f28f0009290).ready entity=mon.0 client_cookie=3abf68db6dfd6182 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:49.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f5ffb700 1 -- 172.21.15.61:0/14710945 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f000b8a0 con 0x7f28f81430f0 2022-07-12T08:41:49.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f5ffb700 1 -- 172.21.15.61:0/14710945 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f000ba00 con 0x7f28f81430f0 2022-07-12T08:41:49.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f5ffb700 1 -- 172.21.15.61:0/14710945 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28f000bb70 con 0x7f28f81430f0 2022-07-12T08:41:49.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- 172.21.15.61:0/14710945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f81430f0 msgr2=0x7f28f805c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:49.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 --2- 172.21.15.61:0/14710945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f81430f0 0x7f28f805c420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f28f00032f0 tx=0x7f28f0009290).stop 2022-07-12T08:41:49.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- 172.21.15.61:0/14710945 shutdown_connections 2022-07-12T08:41:49.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 --2- 172.21.15.61:0/14710945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f81430f0 0x7f28f805c420 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 --2- 172.21.15.61:0/14710945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f8146cb0 0x7f28f80af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- 172.21.15.61:0/14710945 >> 172.21.15.61:0/14710945 conn(0x7f28f80b9d50 msgr2=0x7f28f80b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:49.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- 172.21.15.61:0/14710945 shutdown_connections 2022-07-12T08:41:49.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.392+0000 7f28f7fff700 1 -- 172.21.15.61:0/14710945 wait complete. 2022-07-12T08:41:49.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 Processor -- start 2022-07-12T08:41:49.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 -- start start 2022-07-12T08:41:49.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 0x7f28f8140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f81430f0 0x7f28f80b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f8146cb0 0x7f28f80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:49.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f80b3590 con 0x7f28f80b65d0 2022-07-12T08:41:49.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f81457c0 con 0x7f28f8146cb0 2022-07-12T08:41:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f8145a60 con 0x7f28f81430f0 2022-07-12T08:41:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 0x7f28f8140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:49.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 0x7f28f8140f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53966/0 (socket says 172.21.15.61:53966) 2022-07-12T08:41:49.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f6ffd700 1 -- 172.21.15.61:0/2368787831 learned_addr learned my addr 172.21.15.61:0/2368787831 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f67fc700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f81430f0 0x7f28f80b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.396+0000 7f28f77fe700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f8146cb0 0x7f28f80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f6ffd700 1 -- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f81430f0 msgr2=0x7f28f80b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:49.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f6ffd700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f81430f0 0x7f28f80b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f6ffd700 1 -- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f8146cb0 msgr2=0x7f28f80b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:49.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f6ffd700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f8146cb0 0x7f28f80b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:49.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f6ffd700 1 -- 172.21.15.61:0/2368787831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f0009530 con 0x7f28f80b65d0 2022-07-12T08:41:49.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f67fc700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f81430f0 0x7f28f80b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:49.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f6ffd700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 0x7f28f8140f40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f28f8143e90 tx=0x7f28e8007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:49.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28e8007030 con 0x7f28f80b65d0 2022-07-12T08:41:49.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e8007190 con 0x7f28f80b65d0 2022-07-12T08:41:49.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f8145d00 con 0x7f28f80b65d0 2022-07-12T08:41:49.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28e8006510 con 0x7f28f80b65d0 2022-07-12T08:41:49.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f80b1090 con 0x7f28f80b65d0 2022-07-12T08:41:49.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.400+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f8057460 con 0x7f28f80b65d0 2022-07-12T08:41:49.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.404+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f28e8010030 con 0x7f28f80b65d0 2022-07-12T08:41:49.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.404+0000 7f28effff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28dc053920 0x7f28dc055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:49.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.404+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f28e806b9d0 con 0x7f28f80b65d0 2022-07-12T08:41:49.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.404+0000 7f28f67fc700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28dc053920 0x7f28dc055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:49.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.404+0000 7f28f67fc700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28dc053920 0x7f28dc055de0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f28e0006000 tx=0x7f28e0006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:49.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.412+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f28e80428e0 con 0x7f28f80b65d0 2022-07-12T08:41:49.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:49.536+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f28f809ba00 con 0x7f28f80b65d0 2022-07-12T08:41:50.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.240+0000 7f28effff700 1 -- 172.21.15.61:0/2368787831 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 185+0+0 (secure 0 0 0) 0x7f28e8045f00 con 0x7f28f80b65d0 2022-07-12T08:41:50.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28dc053920 msgr2=0x7f28dc055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:50.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28dc053920 0x7f28dc055de0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f28e0006000 tx=0x7f28e0006d90).stop 2022-07-12T08:41:50.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 msgr2=0x7f28f8140f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:50.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 0x7f28f8140f40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f28f8143e90 tx=0x7f28e8007c10).stop 2022-07-12T08:41:50.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 shutdown_connections 2022-07-12T08:41:50.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28dc053920 0x7f28dc055de0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f28f8146cb0 0x7f28f80b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f28f81430f0 0x7f28f80b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 --2- 172.21.15.61:0/2368787831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28f80b65d0 0x7f28f8140f40 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0x7f28f8143e90 tx=0x7f28e8007c10).stop 2022-07-12T08:41:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 >> 172.21.15.61:0/2368787831 conn(0x7f28f80b9d50 msgr2=0x7f28f80af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 shutdown_connections 2022-07-12T08:41:50.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.244+0000 7f28f7fff700 1 -- 172.21.15.61:0/2368787831 wait complete. 2022-07-12T08:41:50.263 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:41:50.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:41:50.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.384+0000 7f7c8cb79700 1 Processor -- start 2022-07-12T08:41:50.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.384+0000 7f7c8cb79700 1 -- start start 2022-07-12T08:41:50.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c8cb79700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c880ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c8cb79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7c880ad8b0 0x7f7c880aed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c8cb79700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7c880af240 0x7f7c88142410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c8cb79700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c880b1c60 con 0x7f7c880ad8b0 2022-07-12T08:41:50.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c8cb79700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8813f350 con 0x7f7c880af240 2022-07-12T08:41:50.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c8cb79700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8813f5f0 con 0x7f7c8805c8e0 2022-07-12T08:41:50.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7c880ad8b0 0x7f7c880aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7c880ad8b0 0x7f7c880aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53976/0 (socket says 172.21.15.61:53976) 2022-07-12T08:41:50.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 -- 172.21.15.61:0/922673643 learned_addr learned my addr 172.21.15.61:0/922673643 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:50.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c87fff700 1 --2- 172.21.15.61:0/922673643 >> v2:172.21.15.111:3300/0 conn(0x7f7c880af240 0x7f7c88142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c877fe700 1 --2- 172.21.15.61:0/922673643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c880ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 -- 172.21.15.61:0/922673643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 msgr2=0x7f7c880ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:50.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 --2- 172.21.15.61:0/922673643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c880ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 -- 172.21.15.61:0/922673643 >> v2:172.21.15.111:3300/0 conn(0x7f7c880af240 msgr2=0x7f7c88142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:50.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 --2- 172.21.15.61:0/922673643 >> v2:172.21.15.111:3300/0 conn(0x7f7c880af240 0x7f7c88142410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 -- 172.21.15.61:0/922673643 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8813f890 con 0x7f7c880ad8b0 2022-07-12T08:41:50.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c7effd700 1 --2- 172.21.15.61:0/922673643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880ad8b0 0x7f7c880aed50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7c70002f70 tx=0x7f7c70006eb0).ready entity=mon.0 client_cookie=9e34f11b52b13246 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:50.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c86ffd700 1 -- 172.21.15.61:0/922673643 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c7000d570 con 0x7f7c880ad8b0 2022-07-12T08:41:50.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c86ffd700 1 -- 172.21.15.61:0/922673643 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c70000e90 con 0x7f7c880ad8b0 2022-07-12T08:41:50.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.388+0000 7f7c86ffd700 1 -- 172.21.15.61:0/922673643 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c7000d900 con 0x7f7c880ad8b0 2022-07-12T08:41:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 -- 172.21.15.61:0/922673643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880ad8b0 msgr2=0x7f7c880aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/922673643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880ad8b0 0x7f7c880aed50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f7c70002f70 tx=0x7f7c70006eb0).stop 2022-07-12T08:41:50.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 -- 172.21.15.61:0/922673643 shutdown_connections 2022-07-12T08:41:50.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/922673643 >> v2:172.21.15.111:3300/0 conn(0x7f7c880af240 0x7f7c88142410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/922673643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880ad8b0 0x7f7c880aed50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/922673643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c880ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 -- 172.21.15.61:0/922673643 >> 172.21.15.61:0/922673643 conn(0x7f7c880b5780 msgr2=0x7f7c880b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:50.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 -- 172.21.15.61:0/922673643 shutdown_connections 2022-07-12T08:41:50.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 -- 172.21.15.61:0/922673643 wait complete. 2022-07-12T08:41:50.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 Processor -- start 2022-07-12T08:41:50.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.392+0000 7f7c8cb79700 1 -- start start 2022-07-12T08:41:50.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c88142e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c880ad8b0 0x7f7c8813c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 0x7f7c88150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88140310 con 0x7f7c880af240 2022-07-12T08:41:50.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88141840 con 0x7f7c880ad8b0 2022-07-12T08:41:50.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88141ac0 con 0x7f7c8805c8e0 2022-07-12T08:41:50.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 0x7f7c88150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 0x7f7c88150b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:53984/0 (socket says 172.21.15.61:53984) 2022-07-12T08:41:50.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 -- 172.21.15.61:0/1024373060 learned_addr learned my addr 172.21.15.61:0/1024373060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:50.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c7effd700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c880ad8b0 0x7f7c8813c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c877fe700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c88142e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 -- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 msgr2=0x7f7c88142e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:50.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c88142e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 -- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c880ad8b0 msgr2=0x7f7c8813c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:50.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c880ad8b0 0x7f7c8813c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:50.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 -- 172.21.15.61:0/1024373060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7000b1e0 con 0x7f7c880af240 2022-07-12T08:41:50.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c87fff700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 0x7f7c88150b40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7c80006e00 tx=0x7f7c800091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:50.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80012600 con 0x7f7c880af240 2022-07-12T08:41:50.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80012760 con 0x7f7c880af240 2022-07-12T08:41:50.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c88141d20 con 0x7f7c880af240 2022-07-12T08:41:50.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80012ac0 con 0x7f7c880af240 2022-07-12T08:41:50.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c881511d0 con 0x7f7c880af240 2022-07-12T08:41:50.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.396+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88057460 con 0x7f7c880af240 2022-07-12T08:41:50.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.400+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7c80022070 con 0x7f7c880af240 2022-07-12T08:41:50.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.400+0000 7f7c857fa700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7c6c053a40 0x7f7c6c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:50.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.400+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f7c8006c3a0 con 0x7f7c880af240 2022-07-12T08:41:50.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.400+0000 7f7c877fe700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7c6c053a40 0x7f7c6c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:50.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.400+0000 7f7c877fe700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7c6c053a40 0x7f7c6c055f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7c78009000 tx=0x7f7c78006e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:50.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.404+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7c800432d0 con 0x7f7c880af240 2022-07-12T08:41:50.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:50.536+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7c88068400 con 0x7f7c880af240 2022-07-12T08:41:51.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.248+0000 7f7c857fa700 1 -- 172.21.15.61:0/1024373060 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7c80016020 con 0x7f7c880af240 2022-07-12T08:41:51.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.248+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7c6c053a40 msgr2=0x7f7c6c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.248+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7c6c053a40 0x7f7c6c055f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f7c78009000 tx=0x7f7c78006e80).stop 2022-07-12T08:41:51.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.248+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 msgr2=0x7f7c88150b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.248+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 0x7f7c88150b40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f7c80006e00 tx=0x7f7c800091d0).stop 2022-07-12T08:41:51.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 shutdown_connections 2022-07-12T08:41:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c880af240 0x7f7c88150b40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7c6c053a40 0x7f7c6c055f00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7c880ad8b0 0x7f7c8813c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 --2- 172.21.15.61:0/1024373060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7c8805c8e0 0x7f7c88142e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 >> 172.21.15.61:0/1024373060 conn(0x7f7c880b5780 msgr2=0x7f7c880aeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:51.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 shutdown_connections 2022-07-12T08:41:51.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.252+0000 7f7c8cb79700 1 -- 172.21.15.61:0/1024373060 wait complete. 2022-07-12T08:41:51.274 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:41:51.276 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:41:51.277 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:41:51.278 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:41:51.278 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:41:51.279 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:41:51.279 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:41:51.281 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:41:51.283 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:41:51.300 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:41:51.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:41:51.305 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:41:51.306 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:41:51.331 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:41:51.332 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:41:51.333 DEBUG:teuthology.parallel:result is None 2022-07-12T08:41:51.334 DEBUG:teuthology.parallel:result is None 2022-07-12T08:41:51.335 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:41:51.407 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:41:51.420 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:41:51.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.452+0000 7f2b1a2ad700 1 Processor -- start 2022-07-12T08:41:51.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.452+0000 7f2b1a2ad700 1 -- start start 2022-07-12T08:41:51.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1a2ad700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 0x7f2b140b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1a2ad700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140b5740 con 0x7f2b140b1080 2022-07-12T08:41:51.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1a2ad700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1405cc00 con 0x7f2b14142f30 2022-07-12T08:41:51.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1a2ad700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140b1460 con 0x7f2b140b4090 2022-07-12T08:41:51.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b13fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2b140b1080 0x7f2b1405c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42218/0 (socket says 172.21.15.61:42218) 2022-07-12T08:41:51.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b13fff700 1 -- 172.21.15.61:0/2905275774 learned_addr learned my addr 172.21.15.61:0/2905275774 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:51.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b137fe700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 0x7f2b140b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515835721 0 0) 0x7f2b140b5740 con 0x7f2b140b1080 2022-07-12T08:41:51.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b00003260 con 0x7f2b140b1080 2022-07-12T08:41:51.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3000184207 0 0) 0x7f2b00003260 con 0x7f2b140b1080 2022-07-12T08:41:51.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 msgr2=0x7f2b140b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:51.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 0x7f2b140b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 >> v1:172.21.15.111:6789/0 conn(0x7f2b14142f30 legacy=0x7f2b140afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140b6880 con 0x7f2b140b1080 2022-07-12T08:41:51.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f2b04001660 con 0x7f2b140b1080 2022-07-12T08:41:51.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 >> v1:172.21.15.61:6789/0 conn(0x7f2b140b1080 legacy=0x7f2b1405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b00003780 0x7f2b00003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 0x7f2b00004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b000049f0 0x7f2b00004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b140b1460 con 0x7f2b000049f0 2022-07-12T08:41:51.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0000b750 con 0x7f2b00003780 2022-07-12T08:41:51.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0000b910 con 0x7f2b140b4090 2022-07-12T08:41:51.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b000049f0 0x7f2b00004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b137fe700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 0x7f2b00004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b13fff700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b00003780 0x7f2b00003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 -- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 msgr2=0x7f2b00004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:51.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b140b4090 0x7f2b00004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 -- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b00003780 msgr2=0x7f2b00003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:51.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b00003780 0x7f2b00003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 -- 172.21.15.61:0/2905275774 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b00003260 con 0x7f2b000049f0 2022-07-12T08:41:51.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b1884a700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b000049f0 0x7f2b00004e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f2b0c0013a0 tx=0x7f2b0c007d60).ready entity=mon.0 client_cookie=cda6185c707dc3ac server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:51.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0c0016c0 con 0x7f2b000049f0 2022-07-12T08:41:51.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0c006890 con 0x7f2b000049f0 2022-07-12T08:41:51.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.456+0000 7f2b12ffd700 1 -- 172.21.15.61:0/2905275774 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0c006bf0 con 0x7f2b000049f0 2022-07-12T08:41:51.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/2905275774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b000049f0 msgr2=0x7f2b00004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/2905275774 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b000049f0 0x7f2b00004e70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f2b0c0013a0 tx=0x7f2b0c007d60).stop 2022-07-12T08:41:51.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1884a700 1 -- 172.21.15.61:0/2905275774 reap_dead start 2022-07-12T08:41:51.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/2905275774 shutdown_connections 2022-07-12T08:41:51.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/2905275774 >> 172.21.15.61:0/2905275774 conn(0x7f2b14067020 msgr2=0x7f2b14067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:51.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/2905275774 shutdown_connections 2022-07-12T08:41:51.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/2905275774 wait complete. 2022-07-12T08:41:51.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 Processor -- start 2022-07-12T08:41:51.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.460+0000 7f2b1a2ad700 1 -- start start 2022-07-12T08:41:51.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b14112e10 0x7f2b141141f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 0x7f2b141166e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b14114b60 0x7f2b14116bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14140d50 con 0x7f2b141146e0 2022-07-12T08:41:51.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14152990 con 0x7f2b14114b60 2022-07-12T08:41:51.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14152c30 con 0x7f2b14112e10 2022-07-12T08:41:51.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 0x7f2b141166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 0x7f2b141166e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54014/0 (socket says 172.21.15.61:54014) 2022-07-12T08:41:51.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 -- 172.21.15.61:0/3243142457 learned_addr learned my addr 172.21.15.61:0/3243142457 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:51.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b13fff700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b14112e10 0x7f2b141141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1884a700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b14114b60 0x7f2b14116bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 -- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b14112e10 msgr2=0x7f2b141141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b14112e10 0x7f2b141141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 -- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b14114b60 msgr2=0x7f2b14116bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:51.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b14114b60 0x7f2b14116bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:51.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 -- 172.21.15.61:0/3243142457 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14152ed0 con 0x7f2b141146e0 2022-07-12T08:41:51.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b137fe700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 0x7f2b141166e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2b0800b000 tx=0x7f2b08006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:51.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0800f760 con 0x7f2b141146e0 2022-07-12T08:41:51.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0800f8c0 con 0x7f2b141146e0 2022-07-12T08:41:51.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b141136b0 con 0x7f2b141146e0 2022-07-12T08:41:51.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b0800fc20 con 0x7f2b141146e0 2022-07-12T08:41:51.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14152740 con 0x7f2b141146e0 2022-07-12T08:41:51.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.464+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b14153060 con 0x7f2b141146e0 2022-07-12T08:41:51.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.468+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b08014440 con 0x7f2b141146e0 2022-07-12T08:41:51.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.468+0000 7f2b117fa700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2af4053a60 0x7f2af4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:51.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.468+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f2b0806c530 con 0x7f2b141146e0 2022-07-12T08:41:51.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.468+0000 7f2b13fff700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2af4053a60 0x7f2af4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:51.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.468+0000 7f2b13fff700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2af4053a60 0x7f2af4055f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f2b04005000 tx=0x7f2b140b6a10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:51.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.472+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b08043670 con 0x7f2b141146e0 2022-07-12T08:41:51.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:51.608+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f2b14057460 con 0x7f2b141146e0 2022-07-12T08:41:52.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.436+0000 7f2b117fa700 1 -- 172.21.15.61:0/3243142457 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v26) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2b08018020 con 0x7f2b141146e0 2022-07-12T08:41:52.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2af4053a60 msgr2=0x7f2af4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2af4053a60 0x7f2af4055f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f2b04005000 tx=0x7f2b140b6a10).stop 2022-07-12T08:41:52.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 msgr2=0x7f2b141166e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 0x7f2b141166e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f2b0800b000 tx=0x7f2b08006df0).stop 2022-07-12T08:41:52.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 shutdown_connections 2022-07-12T08:41:52.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2af4053a60 0x7f2af4055f20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b14114b60 0x7f2b14116bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b141146e0 0x7f2b141166e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 --2- 172.21.15.61:0/3243142457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b14112e10 0x7f2b141141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 >> 172.21.15.61:0/3243142457 conn(0x7f2b14067020 msgr2=0x7f2b14141760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:52.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 shutdown_connections 2022-07-12T08:41:52.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.440+0000 7f2b1a2ad700 1 -- 172.21.15.61:0/3243142457 wait complete. 2022-07-12T08:41:52.457 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:41:52.466 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:41:52.468 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:41:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 Processor -- start 2022-07-12T08:41:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 -- start start 2022-07-12T08:41:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9280b5780 0x7fa928146540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9280abdf0 0x7fa928146a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:52.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280ab3f0 con 0x7fa92805c730 2022-07-12T08:41:52.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280ab950 con 0x7fa9280abdf0 2022-07-12T08:41:52.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa92c934700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280ab690 con 0x7fa9280b5780 2022-07-12T08:41:52.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9280abdf0 0x7fa928146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:52.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9280abdf0 0x7fa928146a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54020/0 (socket says 172.21.15.61:54020) 2022-07-12T08:41:52.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 -- 172.21.15.61:0/4181215329 learned_addr learned my addr 172.21.15.61:0/4181215329 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa9277fe700 1 --2- 172.21.15.61:0/4181215329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9280b5780 0x7fa928146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 -- 172.21.15.61:0/4181215329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9280b5780 msgr2=0x7fa928146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 --2- 172.21.15.61:0/4181215329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9280b5780 0x7fa928146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 -- 172.21.15.61:0/4181215329 >> v1:172.21.15.111:6789/0 conn(0x7fa92805c730 legacy=0x7fa92805faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 -- 172.21.15.61:0/4181215329 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9280b5b60 con 0x7fa9280abdf0 2022-07-12T08:41:52.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa926ffd700 1 --2- 172.21.15.61:0/4181215329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa928146a30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa9280abcd0 tx=0x7fa918009290).ready entity=mon.0 client_cookie=e5440ff719dbbfb4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:52.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.576+0000 7fa9267fc700 1 -- 172.21.15.61:0/4181215329 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa91800b860 con 0x7fa9280abdf0 2022-07-12T08:41:52.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa9267fc700 1 -- 172.21.15.61:0/4181215329 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91800b9c0 con 0x7fa9280abdf0 2022-07-12T08:41:52.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa9267fc700 1 -- 172.21.15.61:0/4181215329 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa91800bb30 con 0x7fa9280abdf0 2022-07-12T08:41:52.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- 172.21.15.61:0/4181215329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 msgr2=0x7fa928146a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 --2- 172.21.15.61:0/4181215329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa928146a30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa9280abcd0 tx=0x7fa918009290).stop 2022-07-12T08:41:52.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- 172.21.15.61:0/4181215329 shutdown_connections 2022-07-12T08:41:52.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 --2- 172.21.15.61:0/4181215329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa928146a30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 --2- 172.21.15.61:0/4181215329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9280b5780 0x7fa928146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- 172.21.15.61:0/4181215329 >> 172.21.15.61:0/4181215329 conn(0x7fa928063380 msgr2=0x7fa9280b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:52.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- 172.21.15.61:0/4181215329 shutdown_connections 2022-07-12T08:41:52.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- 172.21.15.61:0/4181215329 wait complete. 2022-07-12T08:41:52.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 Processor -- start 2022-07-12T08:41:52.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- start start 2022-07-12T08:41:52.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa92805c730 0x7fa9280ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:52.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa9280649a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:52.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9280b5780 0x7fa9280aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:52.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280ad520 con 0x7fa9280abdf0 2022-07-12T08:41:52.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92815a440 con 0x7fa9280b5780 2022-07-12T08:41:52.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa92c934700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa928065320 con 0x7fa92805c730 2022-07-12T08:41:52.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa9280649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:52.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa9280649a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54026/0 (socket says 172.21.15.61:54026) 2022-07-12T08:41:52.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 -- 172.21.15.61:0/3894323232 learned_addr learned my addr 172.21.15.61:0/3894323232 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:52.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa9277fe700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa92805c730 0x7fa9280ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:52.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa927fff700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9280b5780 0x7fa9280aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:52.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 -- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa92805c730 msgr2=0x7fa9280ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa92805c730 0x7fa9280ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 -- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9280b5780 msgr2=0x7fa9280aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9280b5780 0x7fa9280aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 -- 172.21.15.61:0/3894323232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918009530 con 0x7fa9280abdf0 2022-07-12T08:41:52.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa927fff700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9280b5780 0x7fa9280aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:52.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.580+0000 7fa926ffd700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa9280649a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa9280abcd0 tx=0x7fa918016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:52.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa918021630 con 0x7fa9280abdf0 2022-07-12T08:41:52.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918021790 con 0x7fa9280abdf0 2022-07-12T08:41:52.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa918021af0 con 0x7fa9280abdf0 2022-07-12T08:41:52.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9280acab0 con 0x7fa9280abdf0 2022-07-12T08:41:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa928064e90 con 0x7fa9280abdf0 2022-07-12T08:41:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92815a2a0 con 0x7fa9280abdf0 2022-07-12T08:41:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa918027020 con 0x7fa9280abdf0 2022-07-12T08:41:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa924ff9700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa90c053ad0 0x7fa90c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:52.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa91806d580 con 0x7fa9280abdf0 2022-07-12T08:41:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa9277fe700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa90c053ad0 0x7fa90c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.584+0000 7fa9277fe700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa90c053ad0 0x7fa90c055f90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa91000c000 tx=0x7fa910008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:52.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.592+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa92815a2a0 con 0x7fa9280abdf0 2022-07-12T08:41:52.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.760+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa92815a2a0 con 0x7fa9280abdf0 2022-07-12T08:41:52.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa924ff9700 1 -- 172.21.15.61:0/3894323232 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+5627 (secure 0 0 0) 0x7fa92815a2a0 con 0x7fa9280abdf0 2022-07-12T08:41:52.780 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:52.781 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:41:52.781 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:41:52.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.782 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:41:52.783 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:41:52.783 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.784 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:41:52.784 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.784 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.799 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.800 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.800 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.801 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:41:52.801 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:41:52.813 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.813 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:41:52.814 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.814 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.815 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.815 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.816 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.816 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.817 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:41:52.817 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:41:52.817 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.822 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.827 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.827 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.828 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.829 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.829 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:41:52.829 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:41:52.832 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.861 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.866 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.867 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.867 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.867 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.867 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.868 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:41:52.868 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:41:52.868 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.868 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.869 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.869 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.869 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.869 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.870 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.870 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:41:52.870 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:41:52.870 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.871 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.871 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.871 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.871 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.871 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.872 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.872 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:41:52.872 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:41:52.872 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.873 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.873 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.873 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.873 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.874 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.874 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:41:52.874 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:41:52.874 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.875 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.875 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.881 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.881 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.882 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.883 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.883 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:41:52.883 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:41:52.884 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.884 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.884 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.885 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.885 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.886 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.886 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.887 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:41:52.887 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:41:52.887 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.888 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.888 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.888 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.889 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.889 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.890 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.890 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:41:52.890 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:41:52.891 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.891 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:41:52.891 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow r", 2022-07-12T08:41:52.892 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow rw", 2022-07-12T08:41:52.892 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rwx" 2022-07-12T08:41:52.893 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.893 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.893 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.894 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:41:52.894 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:41:52.895 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.895 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:41:52.896 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow r", 2022-07-12T08:41:52.896 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow rw", 2022-07-12T08:41:52.897 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rwx" 2022-07-12T08:41:52.897 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.897 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.898 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.899 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:41:52.899 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:41:52.900 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.900 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:41:52.901 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow r", 2022-07-12T08:41:52.901 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow rw", 2022-07-12T08:41:52.901 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rwx" 2022-07-12T08:41:52.902 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.903 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.903 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:41:52.904 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:41:52.904 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.905 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:41:52.905 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:41:52.906 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:41:52.906 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.907 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.907 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.907 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.908 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-mds", 2022-07-12T08:41:52.908 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCWM81i14mQEhAAGnYnHWnOiIWl5gjkeups8A==", 2022-07-12T08:41:52.909 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.909 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-mds" 2022-07-12T08:41:52.920 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.920 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.921 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.921 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-mgr", 2022-07-12T08:41:52.922 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCWM81iUqeQEhAAs5sDN/P1GURzPMXwbJAKFQ==", 2022-07-12T08:41:52.922 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.923 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-mgr" 2022-07-12T08:41:52.923 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.924 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.924 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.924 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-osd", 2022-07-12T08:41:52.925 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCWM81isb6QEhAA/vKA8MbtNsw6HqZMh9Vi5w==", 2022-07-12T08:41:52.925 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.926 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-osd" 2022-07-12T08:41:52.926 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.927 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.927 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.928 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-rbd", 2022-07-12T08:41:52.928 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCWM81il+aQEhAAbjgmGKs+de9AvajCyq1b4Q==", 2022-07-12T08:41:52.929 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.929 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-rbd" 2022-07-12T08:41:52.929 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.930 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.930 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.931 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-rbd-mirror", 2022-07-12T08:41:52.931 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCWM81iCQeREhAAFafzHx1qORtSi1kyicFH9g==", 2022-07-12T08:41:52.932 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.932 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2022-07-12T08:41:52.933 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.933 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.946 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.947 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-rgw", 2022-07-12T08:41:52.948 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCWM81imx+REhAAfIjmDqn2OQOjTL1sv7UA/A==", 2022-07-12T08:41:52.948 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.948 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-rgw" 2022-07-12T08:41:52.949 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.949 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.956 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.956 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:41:52.956 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:41:52.956 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.957 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:41:52.957 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:41:52.957 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.957 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.958 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:52.958 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:52.958 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:41:52.958 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:41:52.959 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:41:52.959 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:41:52.959 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:41:52.959 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:41:52.959 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.960 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:52.960 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:41:52.960 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:41:52.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa90c053ad0 msgr2=0x7fa90c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa90c053ad0 0x7fa90c055f90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa91000c000 tx=0x7fa910008040).stop 2022-07-12T08:41:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 msgr2=0x7fa9280649a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa9280649a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fa9280abcd0 tx=0x7fa918016000).stop 2022-07-12T08:41:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 shutdown_connections 2022-07-12T08:41:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa90c053ad0 0x7fa90c055f90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9280b5780 0x7fa9280aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9280abdf0 0x7fa9280649a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 --2- 172.21.15.61:0/3894323232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa92805c730 0x7fa9280ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:52.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 >> 172.21.15.61:0/3894323232 conn(0x7fa928063380 msgr2=0x7fa9280b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:52.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 shutdown_connections 2022-07-12T08:41:52.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:52.764+0000 7fa92c934700 1 -- 172.21.15.61:0/3894323232 wait complete. 2022-07-12T08:41:52.967 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2022-07-12T08:41:53.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.088+0000 7fcad9e90700 1 Processor -- start 2022-07-12T08:41:53.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.088+0000 7fcad9e90700 1 -- start start 2022-07-12T08:41:53.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcad405fb90 0x7fcad40ad000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad40b0980 con 0x7fcad40b0b80 2022-07-12T08:41:53.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad40af0d0 con 0x7fcad4142690 2022-07-12T08:41:53.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad405ff70 con 0x7fcad405fb90 2022-07-12T08:41:53.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcad40b0b80 0x7fcad40b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42254/0 (socket says 172.21.15.61:42254) 2022-07-12T08:41:53.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacbfff700 1 -- 172.21.15.61:0/4153236222 learned_addr learned my addr 172.21.15.61:0/4153236222 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:53.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad8e8e700 1 --2- 172.21.15.61:0/4153236222 >> v2:172.21.15.111:3300/0 conn(0x7fcad405fb90 0x7fcad40ad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacb7fe700 1 -- 172.21.15.61:0/4153236222 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715131234 0 0) 0x7fcad40b0980 con 0x7fcad40b0b80 2022-07-12T08:41:53.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacb7fe700 1 -- 172.21.15.61:0/4153236222 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac0003260 con 0x7fcad40b0b80 2022-07-12T08:41:53.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad8e8e700 1 -- 172.21.15.61:0/4153236222 >> v1:172.21.15.111:6789/0 conn(0x7fcad4142690 legacy=0x7fcad4142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad8e8e700 1 -- 172.21.15.61:0/4153236222 >> v1:172.21.15.61:6789/0 conn(0x7fcad40b0b80 legacy=0x7fcad40b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad8e8e700 1 -- 172.21.15.61:0/4153236222 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad40af3b0 con 0x7fcad405fb90 2022-07-12T08:41:53.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad8e8e700 1 --2- 172.21.15.61:0/4153236222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad405fb90 0x7fcad40ad000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fcad0002f70 tx=0x7fcad0006df0).ready entity=mon.1 client_cookie=37b4e0982937b206 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacb7fe700 1 -- 172.21.15.61:0/4153236222 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad000d610 con 0x7fcad405fb90 2022-07-12T08:41:53.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacb7fe700 1 -- 172.21.15.61:0/4153236222 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad0000e90 con 0x7fcad405fb90 2022-07-12T08:41:53.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcacb7fe700 1 -- 172.21.15.61:0/4153236222 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad000d960 con 0x7fcad405fb90 2022-07-12T08:41:53.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- 172.21.15.61:0/4153236222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad405fb90 msgr2=0x7fcad40ad000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 --2- 172.21.15.61:0/4153236222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad405fb90 0x7fcad40ad000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fcad0002f70 tx=0x7fcad0006df0).stop 2022-07-12T08:41:53.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- 172.21.15.61:0/4153236222 shutdown_connections 2022-07-12T08:41:53.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 --2- 172.21.15.61:0/4153236222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad405fb90 0x7fcad40ad000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- 172.21.15.61:0/4153236222 >> 172.21.15.61:0/4153236222 conn(0x7fcad4063cd0 msgr2=0x7fcad405f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:53.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- 172.21.15.61:0/4153236222 shutdown_connections 2022-07-12T08:41:53.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.092+0000 7fcad9e90700 1 -- 172.21.15.61:0/4153236222 wait complete. 2022-07-12T08:41:53.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 Processor -- start 2022-07-12T08:41:53.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 -- start start 2022-07-12T08:41:53.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 0x7fcad410ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcad40b0b80 0x7fcad413deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad4142690 0x7fcad413e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad4111740 con 0x7fcad40b0b80 2022-07-12T08:41:53.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad40665d0 con 0x7fcad4142690 2022-07-12T08:41:53.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad9e90700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad4066870 con 0x7fcad405fb90 2022-07-12T08:41:53.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcacbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcad40b0b80 0x7fcad413deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad8e8e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 0x7fcad410ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad8e8e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 0x7fcad410ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36236/0 (socket says 172.21.15.61:36236) 2022-07-12T08:41:53.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad8e8e700 1 -- 172.21.15.61:0/3027042664 learned_addr learned my addr 172.21.15.61:0/3027042664 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:53.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad968f700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad4142690 0x7fcad413e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcacbfff700 1 -- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 msgr2=0x7fcad410ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcacbfff700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 0x7fcad410ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcacbfff700 1 -- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad4142690 msgr2=0x7fcad413e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcacbfff700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad4142690 0x7fcad413e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcacbfff700 1 -- 172.21.15.61:0/3027042664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad000b180 con 0x7fcad40b0b80 2022-07-12T08:41:53.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.096+0000 7fcad8e8e700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 0x7fcad410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:53.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcacbfff700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcad40b0b80 0x7fcad413deb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fcac4001110 tx=0x7fcac4007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcac40070d0 con 0x7fcad40b0b80 2022-07-12T08:41:53.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac4007230 con 0x7fcad40b0b80 2022-07-12T08:41:53.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcac4005610 con 0x7fcad40b0b80 2022-07-12T08:41:53.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad4111390 con 0x7fcad40b0b80 2022-07-12T08:41:53.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad410f580 con 0x7fcad40b0b80 2022-07-12T08:41:53.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad4057460 con 0x7fcad40b0b80 2022-07-12T08:41:53.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcac4010070 con 0x7fcad40b0b80 2022-07-12T08:41:53.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcac9ffb700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcab4053920 0x7fcab4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcac406afb0 con 0x7fcad40b0b80 2022-07-12T08:41:53.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.100+0000 7fcad8e8e700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcab4053920 0x7fcab4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.104+0000 7fcad8e8e700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcab4053920 0x7fcab4055de0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fcad001a000 tx=0x7fcad0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.108+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcac4042240 con 0x7fcad40b0b80 2022-07-12T08:41:53.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.284+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fcad410f580 con 0x7fcad40b0b80 2022-07-12T08:41:53.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.292+0000 7fcac9ffb700 1 -- 172.21.15.61:0/3027042664 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v5) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcad410f580 con 0x7fcad40b0b80 2022-07-12T08:41:53.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcab4053920 msgr2=0x7fcab4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcab4053920 0x7fcab4055de0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fcad001a000 tx=0x7fcad0014000).stop 2022-07-12T08:41:53.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcad40b0b80 msgr2=0x7fcad413deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcad40b0b80 0x7fcad413deb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fcac4001110 tx=0x7fcac4007ac0).stop 2022-07-12T08:41:53.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 shutdown_connections 2022-07-12T08:41:53.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcab4053920 0x7fcab4055de0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcad4142690 0x7fcad413e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcad40b0b80 0x7fcad413deb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 --2- 172.21.15.61:0/3027042664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcad405fb90 0x7fcad410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 >> 172.21.15.61:0/3027042664 conn(0x7fcad4063cd0 msgr2=0x7fcad413f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:53.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 shutdown_connections 2022-07-12T08:41:53.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.296+0000 7fcad9e90700 1 -- 172.21.15.61:0/3027042664 wait complete. 2022-07-12T08:41:53.311 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-12T08:41:53.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2022-07-12T08:41:53.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.420+0000 7ff360abb700 1 Processor -- start 2022-07-12T08:41:53.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.420+0000 7ff360abb700 1 -- start start 2022-07-12T08:41:53.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c0b2960 0x7ff35c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff35c0ae510 0x7ff35c143720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff35c13f7e0 0x7ff35c13fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0b3230 con 0x7ff35c0ae510 2022-07-12T08:41:53.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c140140 con 0x7ff35c13f7e0 2022-07-12T08:41:53.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0af3b0 con 0x7ff35c0b2960 2022-07-12T08:41:53.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff35c0ae510 0x7ff35c143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff35c0ae510 0x7ff35c143720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54048/0 (socket says 172.21.15.61:54048) 2022-07-12T08:41:53.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 -- 172.21.15.61:0/3829987147 learned_addr learned my addr 172.21.15.61:0/3829987147 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:53.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35b7fe700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c0b2960 0x7ff35c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35bfff700 1 --2- 172.21.15.61:0/3829987147 >> v2:172.21.15.111:3300/0 conn(0x7ff35c13f7e0 0x7ff35c13fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 -- 172.21.15.61:0/3829987147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c0b2960 msgr2=0x7ff35c0b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c0b2960 0x7ff35c0b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 -- 172.21.15.61:0/3829987147 >> v2:172.21.15.111:3300/0 conn(0x7ff35c13f7e0 msgr2=0x7ff35c13fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 --2- 172.21.15.61:0/3829987147 >> v2:172.21.15.111:3300/0 conn(0x7ff35c13f7e0 0x7ff35c13fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 -- 172.21.15.61:0/3829987147 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c142db0 con 0x7ff35c0ae510 2022-07-12T08:41:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35bfff700 1 --2- 172.21.15.61:0/3829987147 >> v2:172.21.15.111:3300/0 conn(0x7ff35c13f7e0 0x7ff35c13fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35b7fe700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c0b2960 0x7ff35c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:41:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35affd700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c143720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ff350002f70 tx=0x7ff350006eb0).ready entity=mon.0 client_cookie=eebffe0dc2841f6f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35a7fc700 1 -- 172.21.15.61:0/3829987147 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff35000d570 con 0x7ff35c0ae510 2022-07-12T08:41:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35a7fc700 1 -- 172.21.15.61:0/3829987147 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff350000e90 con 0x7ff35c0ae510 2022-07-12T08:41:53.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff35a7fc700 1 -- 172.21.15.61:0/3829987147 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff35000d900 con 0x7ff35c0ae510 2022-07-12T08:41:53.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- 172.21.15.61:0/3829987147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 msgr2=0x7ff35c143720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c143720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ff350002f70 tx=0x7ff350006eb0).stop 2022-07-12T08:41:53.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- 172.21.15.61:0/3829987147 shutdown_connections 2022-07-12T08:41:53.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- 172.21.15.61:0/3829987147 >> v2:172.21.15.111:3300/0 conn(0x7ff35c13f7e0 0x7ff35c13fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c143720 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 --2- 172.21.15.61:0/3829987147 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c0b2960 0x7ff35c0b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- 172.21.15.61:0/3829987147 >> 172.21.15.61:0/3829987147 conn(0x7ff35c063a00 msgr2=0x7ff35c063de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:53.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- 172.21.15.61:0/3829987147 shutdown_connections 2022-07-12T08:41:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.424+0000 7ff360abb700 1 -- 172.21.15.61:0/3829987147 wait complete. 2022-07-12T08:41:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.428+0000 7ff360abb700 1 Processor -- start 2022-07-12T08:41:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.428+0000 7ff360abb700 1 -- start start 2022-07-12T08:41:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.428+0000 7ff360abb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.428+0000 7ff360abb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c0b2960 0x7ff35c066da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c13f7e0 0x7ff35c067290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c152f80 con 0x7ff35c0ae510 2022-07-12T08:41:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c1278f0 con 0x7ff35c0b2960 2022-07-12T08:41:53.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c127b90 con 0x7ff35c13f7e0 2022-07-12T08:41:53.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c0b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54054/0 (socket says 172.21.15.61:54054) 2022-07-12T08:41:53.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 -- 172.21.15.61:0/310326895 learned_addr learned my addr 172.21.15.61:0/310326895 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:53.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35affd700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c0b2960 0x7ff35c066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35bfff700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c13f7e0 0x7ff35c067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 -- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c13f7e0 msgr2=0x7ff35c067290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c13f7e0 0x7ff35c067290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 -- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c0b2960 msgr2=0x7ff35c066da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c0b2960 0x7ff35c066da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 -- 172.21.15.61:0/310326895 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35000b1e0 con 0x7ff35c0ae510 2022-07-12T08:41:53.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff35b7fe700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c0b5790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7ff34c007000 tx=0x7ff34c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff34c010580 con 0x7ff35c0ae510 2022-07-12T08:41:53.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c0106e0 con 0x7ff35c0ae510 2022-07-12T08:41:53.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c127e30 con 0x7ff35c0ae510 2022-07-12T08:41:53.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff34c010a40 con 0x7ff35c0ae510 2022-07-12T08:41:53.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c141980 con 0x7ff35c0ae510 2022-07-12T08:41:53.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.432+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c057460 con 0x7ff35c0ae510 2022-07-12T08:41:53.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.436+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff34c010ba0 con 0x7ff35c0ae510 2022-07-12T08:41:53.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.436+0000 7ff358ff9700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff33c0539b0 0x7ff33c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.436+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff34c06bc50 con 0x7ff35c0ae510 2022-07-12T08:41:53.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.436+0000 7ff35affd700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff33c0539b0 0x7ff33c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.436+0000 7ff35affd700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff33c0539b0 0x7ff33c055e70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff35001c000 tx=0x7ff350015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.444+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff34c042e80 con 0x7ff35c0ae510 2022-07-12T08:41:53.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.608+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7ff35c060820 con 0x7ff35c0ae510 2022-07-12T08:41:53.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff358ff9700 1 -- 172.21.15.61:0/310326895 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v6) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff34c0464a0 con 0x7ff35c0ae510 2022-07-12T08:41:53.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff33c0539b0 msgr2=0x7ff33c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff33c0539b0 0x7ff33c055e70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ff35001c000 tx=0x7ff350015000).stop 2022-07-12T08:41:53.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 msgr2=0x7ff35c0b5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c0b5790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7ff34c007000 tx=0x7ff34c008040).stop 2022-07-12T08:41:53.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 shutdown_connections 2022-07-12T08:41:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff35c13f7e0 0x7ff35c067290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff33c0539b0 0x7ff33c055e70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff35c0b2960 0x7ff35c066da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 --2- 172.21.15.61:0/310326895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35c0ae510 0x7ff35c0b5790 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 >> 172.21.15.61:0/310326895 conn(0x7ff35c063a00 msgr2=0x7ff35c0b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:53.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 shutdown_connections 2022-07-12T08:41:53.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.616+0000 7ff360abb700 1 -- 172.21.15.61:0/310326895 wait complete. 2022-07-12T08:41:53.649 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-12T08:41:53.653 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2022-07-12T08:41:53.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 Processor -- start 2022-07-12T08:41:53.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 -- start start 2022-07-12T08:41:53.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b005fb90 0x7f47b014a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47b005cb70 0x7f47b014a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b005ff70 con 0x7f47b014add0 2022-07-12T08:41:53.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b0147370 con 0x7f47b005cb70 2022-07-12T08:41:53.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b5515700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0147610 con 0x7f47b005fb90 2022-07-12T08:41:53.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47b005cb70 0x7f47b014a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47b005cb70 0x7f47b014a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54064/0 (socket says 172.21.15.61:54064) 2022-07-12T08:41:53.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 -- 172.21.15.61:0/2226468858 learned_addr learned my addr 172.21.15.61:0/2226468858 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:53.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47b4d14700 1 --1- 172.21.15.61:0/2226468858 >> v1:172.21.15.61:6789/0 conn(0x7f47b014add0 0x7f47b00b21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42284/0 (socket says 172.21.15.61:42284) 2022-07-12T08:41:53.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47affff700 1 --2- 172.21.15.61:0/2226468858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b005fb90 0x7f47b014a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 -- 172.21.15.61:0/2226468858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b005fb90 msgr2=0x7f47b014a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:53.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 --2- 172.21.15.61:0/2226468858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b005fb90 0x7f47b014a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 -- 172.21.15.61:0/2226468858 >> v1:172.21.15.61:6789/0 conn(0x7f47b014add0 legacy=0x7f47b00b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 -- 172.21.15.61:0/2226468858 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b01478b0 con 0x7f47b005cb70 2022-07-12T08:41:53.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47af7fe700 1 --2- 172.21.15.61:0/2226468858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b014a8e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f47a00032f0 tx=0x7f47a0009290).ready entity=mon.0 client_cookie=58214943021dd75d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47aeffd700 1 -- 172.21.15.61:0/2226468858 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a000b8a0 con 0x7f47b005cb70 2022-07-12T08:41:53.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47aeffd700 1 -- 172.21.15.61:0/2226468858 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a000ba00 con 0x7f47b005cb70 2022-07-12T08:41:53.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.756+0000 7f47aeffd700 1 -- 172.21.15.61:0/2226468858 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a000bb70 con 0x7f47b005cb70 2022-07-12T08:41:53.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 -- 172.21.15.61:0/2226468858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 msgr2=0x7f47b014a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 --2- 172.21.15.61:0/2226468858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b014a8e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f47a00032f0 tx=0x7f47a0009290).stop 2022-07-12T08:41:53.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 -- 172.21.15.61:0/2226468858 shutdown_connections 2022-07-12T08:41:53.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 --2- 172.21.15.61:0/2226468858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b014a8e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 --2- 172.21.15.61:0/2226468858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b005fb90 0x7f47b014a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 -- 172.21.15.61:0/2226468858 >> 172.21.15.61:0/2226468858 conn(0x7f47b0063cd0 msgr2=0x7f47b005f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:53.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 -- 172.21.15.61:0/2226468858 shutdown_connections 2022-07-12T08:41:53.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 -- 172.21.15.61:0/2226468858 wait complete. 2022-07-12T08:41:53.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 Processor -- start 2022-07-12T08:41:53.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.760+0000 7f47b5515700 1 -- start start 2022-07-12T08:41:53.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b0142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47b005fb90 0x7f47b0155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b014add0 0x7f47b0159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0156330 con 0x7f47b005cb70 2022-07-12T08:41:53.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00adab0 con 0x7f47b005fb90 2022-07-12T08:41:53.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00add50 con 0x7f47b014add0 2022-07-12T08:41:53.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b0142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b0142d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54068/0 (socket says 172.21.15.61:54068) 2022-07-12T08:41:53.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 -- 172.21.15.61:0/1110378826 learned_addr learned my addr 172.21.15.61:0/1110378826 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:53.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b4d14700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b014add0 0x7f47b0159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47af7fe700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47b005fb90 0x7f47b0155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 -- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b014add0 msgr2=0x7f47b0159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b014add0 0x7f47b0159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 -- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47b005fb90 msgr2=0x7f47b0155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47b005fb90 0x7f47b0155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 -- 172.21.15.61:0/1110378826 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a0009530 con 0x7f47b005cb70 2022-07-12T08:41:53.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47affff700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b0142d70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f47980030b0 tx=0x7f4798008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4798011780 con 0x7f47b005cb70 2022-07-12T08:41:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47980118e0 con 0x7f47b005cb70 2022-07-12T08:41:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b01594b0 con 0x7f47b005cb70 2022-07-12T08:41:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4798011c40 con 0x7f47b005cb70 2022-07-12T08:41:53.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b00af3d0 con 0x7f47b005cb70 2022-07-12T08:41:53.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.764+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0057460 con 0x7f47b005cb70 2022-07-12T08:41:53.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.768+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f479801c460 con 0x7f47b005cb70 2022-07-12T08:41:53.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.768+0000 7f47ad7fa700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4794053a40 0x7f4794055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:53.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.768+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f479806c6a0 con 0x7f47b005cb70 2022-07-12T08:41:53.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.768+0000 7f47af7fe700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4794053a40 0x7f4794055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:53.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.768+0000 7f47af7fe700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4794053a40 0x7f4794055f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f47a001b000 tx=0x7f47a0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:53.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.772+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47980438d0 con 0x7f47b005cb70 2022-07-12T08:41:53.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.940+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f47b0159640 con 0x7f47b005cb70 2022-07-12T08:41:53.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.944+0000 7f47ad7fa700 1 -- 172.21.15.61:0/1110378826 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4798046ef0 con 0x7f47b005cb70 2022-07-12T08:41:53.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4794053a40 msgr2=0x7f4794055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4794053a40 0x7f4794055f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f47a001b000 tx=0x7f47a0016000).stop 2022-07-12T08:41:53.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 msgr2=0x7f47b0142d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:53.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b0142d70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f47980030b0 tx=0x7f4798008040).stop 2022-07-12T08:41:53.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 shutdown_connections 2022-07-12T08:41:53.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47b014add0 0x7f47b0159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4794053a40 0x7f4794055f00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47b005fb90 0x7f47b0155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 --2- 172.21.15.61:0/1110378826 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47b005cb70 0x7f47b0142d70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:53.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 >> 172.21.15.61:0/1110378826 conn(0x7f47b0063cd0 msgr2=0x7f47b01460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:53.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 shutdown_connections 2022-07-12T08:41:53.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:53.948+0000 7f47b5515700 1 -- 172.21.15.61:0/1110378826 wait complete. 2022-07-12T08:41:53.975 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-12T08:41:53.975 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2022-07-12T08:41:54.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.080+0000 7f39dbef9700 1 Processor -- start 2022-07-12T08:41:54.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.080+0000 7f39dbef9700 1 -- start start 2022-07-12T08:41:54.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39dbef9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39d40b4390 0x7f39d4140570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39dbef9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d41431e0 0x7f39d4143650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39dbef9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d40b3e80 con 0x7f39d40b6250 2022-07-12T08:41:54.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39dbef9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d4147390 con 0x7f39d40b4390 2022-07-12T08:41:54.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39dbef9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d40b6630 con 0x7f39d41431e0 2022-07-12T08:41:54.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d9c95700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f39d40b6250 0x7f39d413fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42294/0 (socket says 172.21.15.61:42294) 2022-07-12T08:41:54.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d9c95700 1 -- 172.21.15.61:0/3403835943 learned_addr learned my addr 172.21.15.61:0/3403835943 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:54.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39da496700 1 --2- 172.21.15.61:0/3403835943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d41431e0 0x7f39d4143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d9494700 1 --2- 172.21.15.61:0/3403835943 >> v2:172.21.15.111:3300/0 conn(0x7f39d40b4390 0x7f39d4140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d8c93700 1 -- 172.21.15.61:0/3403835943 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121600411 0 0) 0x7f39d40b3e80 con 0x7f39d40b6250 2022-07-12T08:41:54.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d8c93700 1 -- 172.21.15.61:0/3403835943 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39bc003260 con 0x7f39d40b6250 2022-07-12T08:41:54.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39da496700 1 -- 172.21.15.61:0/3403835943 >> v2:172.21.15.111:3300/0 conn(0x7f39d40b4390 msgr2=0x7f39d4140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:54.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39da496700 1 --2- 172.21.15.61:0/3403835943 >> v2:172.21.15.111:3300/0 conn(0x7f39d40b4390 0x7f39d4140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39da496700 1 -- 172.21.15.61:0/3403835943 >> v1:172.21.15.61:6789/0 conn(0x7f39d40b6250 legacy=0x7f39d413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39da496700 1 -- 172.21.15.61:0/3403835943 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d41467b0 con 0x7f39d41431e0 2022-07-12T08:41:54.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39da496700 1 --2- 172.21.15.61:0/3403835943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d41431e0 0x7f39d4143650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f39cc003340 tx=0x7f39cc0091d0).ready entity=mon.2 client_cookie=2081a29cbbafbb61 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d8c93700 1 -- 172.21.15.61:0/3403835943 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39cc00b600 con 0x7f39d41431e0 2022-07-12T08:41:54.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d8c93700 1 -- 172.21.15.61:0/3403835943 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39cc00b760 con 0x7f39d41431e0 2022-07-12T08:41:54.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.084+0000 7f39d8c93700 1 -- 172.21.15.61:0/3403835943 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39cc00b8d0 con 0x7f39d41431e0 2022-07-12T08:41:54.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 -- 172.21.15.61:0/3403835943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d41431e0 msgr2=0x7f39d4143650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 --2- 172.21.15.61:0/3403835943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d41431e0 0x7f39d4143650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f39cc003340 tx=0x7f39cc0091d0).stop 2022-07-12T08:41:54.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 -- 172.21.15.61:0/3403835943 shutdown_connections 2022-07-12T08:41:54.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 --2- 172.21.15.61:0/3403835943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d41431e0 0x7f39d4143650 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 --2- 172.21.15.61:0/3403835943 >> v2:172.21.15.111:3300/0 conn(0x7f39d40b4390 0x7f39d4140570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 -- 172.21.15.61:0/3403835943 >> 172.21.15.61:0/3403835943 conn(0x7f39d4066f00 msgr2=0x7f39d40672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:54.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 -- 172.21.15.61:0/3403835943 shutdown_connections 2022-07-12T08:41:54.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 -- 172.21.15.61:0/3403835943 wait complete. 2022-07-12T08:41:54.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 Processor -- start 2022-07-12T08:41:54.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.088+0000 7f39dbef9700 1 -- start start 2022-07-12T08:41:54.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d40b4390 0x7f39d4148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 0x7f39d40acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d41431e0 0x7f39d41444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d413a070 con 0x7f39d40b6250 2022-07-12T08:41:54.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d4146010 con 0x7f39d41431e0 2022-07-12T08:41:54.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d4148630 con 0x7f39d40b4390 2022-07-12T08:41:54.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 0x7f39d40acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 0x7f39d40acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54084/0 (socket says 172.21.15.61:54084) 2022-07-12T08:41:54.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 -- 172.21.15.61:0/1068619168 learned_addr learned my addr 172.21.15.61:0/1068619168 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:54.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9c95700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d40b4390 0x7f39d4148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39da496700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d41431e0 0x7f39d41444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 -- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d40b4390 msgr2=0x7f39d4148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d40b4390 0x7f39d4148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 -- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d41431e0 msgr2=0x7f39d41444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d41431e0 0x7f39d41444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 -- 172.21.15.61:0/1068619168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc009470 con 0x7f39d40b6250 2022-07-12T08:41:54.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39d9494700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 0x7f39d40acb60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f39c800c000 tx=0x7f39c8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39c8010590 con 0x7f39d40b6250 2022-07-12T08:41:54.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c80106f0 con 0x7f39d40b6250 2022-07-12T08:41:54.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d4148fc0 con 0x7f39d40b6250 2022-07-12T08:41:54.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39c8010a50 con 0x7f39d40b6250 2022-07-12T08:41:54.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.092+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d4147a50 con 0x7f39d40b6250 2022-07-12T08:41:54.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.096+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f39c8010bb0 con 0x7f39d40b6250 2022-07-12T08:41:54.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.096+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4142dc0 con 0x7f39d40b6250 2022-07-12T08:41:54.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.096+0000 7f39c6ffd700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39b4053a40 0x7f39b4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.096+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f39c806bb60 con 0x7f39d40b6250 2022-07-12T08:41:54.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.096+0000 7f39d9c95700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39b4053a40 0x7f39b4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.096+0000 7f39d9c95700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39b4053a40 0x7f39b4055f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f39d000d000 tx=0x7f39d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.104+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f39c8042d50 con 0x7f39d40b6250 2022-07-12T08:41:54.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.272+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f39d4057460 con 0x7f39d40b6250 2022-07-12T08:41:54.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39c6ffd700 1 -- 172.21.15.61:0/1068619168 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f39c8046370 con 0x7f39d40b6250 2022-07-12T08:41:54.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39b4053a40 msgr2=0x7f39b4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39b4053a40 0x7f39b4055f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f39d000d000 tx=0x7f39d0009040).stop 2022-07-12T08:41:54.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 msgr2=0x7f39d40acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 0x7f39d40acb60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f39c800c000 tx=0x7f39c8006df0).stop 2022-07-12T08:41:54.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 shutdown_connections 2022-07-12T08:41:54.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39b4053a40 0x7f39b4055f00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39d41431e0 0x7f39d41444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39d40b6250 0x7f39d40acb60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 --2- 172.21.15.61:0/1068619168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39d40b4390 0x7f39d4148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 >> 172.21.15.61:0/1068619168 conn(0x7f39d4066f00 msgr2=0x7f39d40b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:54.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 shutdown_connections 2022-07-12T08:41:54.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.280+0000 7f39dbef9700 1 -- 172.21.15.61:0/1068619168 wait complete. 2022-07-12T08:41:54.293 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-12T08:41:54.308 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2022-07-12T08:41:54.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.416+0000 7fc417fff700 1 Processor -- start 2022-07-12T08:41:54.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.416+0000 7fc417fff700 1 -- start start 2022-07-12T08:41:54.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4180b0f60 0x7fc41814a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4180b0b30 con 0x7fc41814a520 2022-07-12T08:41:54.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41814af10 con 0x7fc418060ca0 2022-07-12T08:41:54.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4180b1660 con 0x7fc4180b0f60 2022-07-12T08:41:54.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4180b0f60 0x7fc41814a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4180b0f60 0x7fc41814a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54090/0 (socket says 172.21.15.61:54090) 2022-07-12T08:41:54.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 -- 172.21.15.61:0/4124023054 learned_addr learned my addr 172.21.15.61:0/4124023054 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:54.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc416ffd700 1 --1- 172.21.15.61:0/4124023054 >> v1:172.21.15.61:6789/0 conn(0x7fc41814a520 0x7fc4181407e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42310/0 (socket says 172.21.15.61:42310) 2022-07-12T08:41:54.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 -- 172.21.15.61:0/4124023054 >> v1:172.21.15.111:6789/0 conn(0x7fc418060ca0 legacy=0x7fc41805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 -- 172.21.15.61:0/4124023054 >> v1:172.21.15.61:6789/0 conn(0x7fc41814a520 legacy=0x7fc4181407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 -- 172.21.15.61:0/4124023054 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41814b1b0 con 0x7fc4180b0f60 2022-07-12T08:41:54.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc4167fc700 1 --2- 172.21.15.61:0/4124023054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4180b0f60 0x7fc41814a900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc400002f70 tx=0x7fc400006eb0).ready entity=mon.0 client_cookie=726069ebe6141fc9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc415ffb700 1 -- 172.21.15.61:0/4124023054 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc40000d660 con 0x7fc4180b0f60 2022-07-12T08:41:54.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc415ffb700 1 -- 172.21.15.61:0/4124023054 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc400000e90 con 0x7fc4180b0f60 2022-07-12T08:41:54.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc415ffb700 1 -- 172.21.15.61:0/4124023054 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc40000d9f0 con 0x7fc4180b0f60 2022-07-12T08:41:54.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- 172.21.15.61:0/4124023054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4180b0f60 msgr2=0x7fc41814a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 --2- 172.21.15.61:0/4124023054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4180b0f60 0x7fc41814a900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc400002f70 tx=0x7fc400006eb0).stop 2022-07-12T08:41:54.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- 172.21.15.61:0/4124023054 shutdown_connections 2022-07-12T08:41:54.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 --2- 172.21.15.61:0/4124023054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4180b0f60 0x7fc41814a900 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- 172.21.15.61:0/4124023054 >> 172.21.15.61:0/4124023054 conn(0x7fc4180b5780 msgr2=0x7fc4180b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:54.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- 172.21.15.61:0/4124023054 shutdown_connections 2022-07-12T08:41:54.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.420+0000 7fc417fff700 1 -- 172.21.15.61:0/4124023054 wait complete. 2022-07-12T08:41:54.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 Processor -- start 2022-07-12T08:41:54.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 -- start start 2022-07-12T08:41:54.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 0x7fc418065290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180b0f60 0x7fc4180ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 0x7fc41813fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc418065fb0 con 0x7fc41814a520 2022-07-12T08:41:54.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4180661e0 con 0x7fc4180b0f60 2022-07-12T08:41:54.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc418066480 con 0x7fc418060ca0 2022-07-12T08:41:54.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc4177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 0x7fc41813fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 0x7fc418065290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc4167fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180b0f60 0x7fc4180ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc4177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 0x7fc41813fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54100/0 (socket says 172.21.15.61:54100) 2022-07-12T08:41:54.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc4167fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180b0f60 0x7fc4180ae440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:60904/0 (socket says 172.21.15.61:60904) 2022-07-12T08:41:54.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc4167fc700 1 -- 172.21.15.61:0/313009428 learned_addr learned my addr 172.21.15.61:0/313009428 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:54.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 0x7fc418065290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36292/0 (socket says 172.21.15.61:36292) 2022-07-12T08:41:54.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 -- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180b0f60 msgr2=0x7fc4180ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180b0f60 0x7fc4180ae440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 -- 172.21.15.61:0/313009428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 msgr2=0x7fc41813fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 0x7fc41813fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 -- 172.21.15.61:0/313009428 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40000b1e0 con 0x7fc418060ca0 2022-07-12T08:41:54.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc4177fe700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 0x7fc41813fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:54.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc416ffd700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 0x7fc418065290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc418062eb0 tx=0x7fc408007b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4080070d0 con 0x7fc418060ca0 2022-07-12T08:41:54.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc408007230 con 0x7fc418060ca0 2022-07-12T08:41:54.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc418066720 con 0x7fc418060ca0 2022-07-12T08:41:54.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc408005550 con 0x7fc418060ca0 2022-07-12T08:41:54.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.424+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc418066f40 con 0x7fc418060ca0 2022-07-12T08:41:54.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.428+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc418057460 con 0x7fc418060ca0 2022-07-12T08:41:54.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.428+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc408010070 con 0x7fc418060ca0 2022-07-12T08:41:54.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.428+0000 7fc40ffff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc3fc053a60 0x7fc3fc055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.428+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc40806b210 con 0x7fc418060ca0 2022-07-12T08:41:54.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.428+0000 7fc4167fc700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc3fc053a60 0x7fc3fc055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.428+0000 7fc4167fc700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc3fc053a60 0x7fc3fc055f20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fc40001c000 tx=0x7fc400015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.440+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc408042310 con 0x7fc418060ca0 2022-07-12T08:41:54.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.604+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fc418068400 con 0x7fc418060ca0 2022-07-12T08:41:54.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.612+0000 7fc40ffff700 1 -- 172.21.15.61:0/313009428 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc408045930 con 0x7fc418060ca0 2022-07-12T08:41:54.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc3fc053a60 msgr2=0x7fc3fc055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc3fc053a60 0x7fc3fc055f20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fc40001c000 tx=0x7fc400015000).stop 2022-07-12T08:41:54.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 msgr2=0x7fc418065290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 0x7fc418065290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc418062eb0 tx=0x7fc408007b80).stop 2022-07-12T08:41:54.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 shutdown_connections 2022-07-12T08:41:54.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc41814a520 0x7fc41813fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc3fc053a60 0x7fc3fc055f20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4180b0f60 0x7fc4180ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 --2- 172.21.15.61:0/313009428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc418060ca0 0x7fc418065290 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 >> 172.21.15.61:0/313009428 conn(0x7fc4180b5780 msgr2=0x7fc4181409f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:54.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 shutdown_connections 2022-07-12T08:41:54.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.616+0000 7fc417fff700 1 -- 172.21.15.61:0/313009428 wait complete. 2022-07-12T08:41:54.640 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-12T08:41:54.643 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2022-07-12T08:41:54.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 Processor -- start 2022-07-12T08:41:54.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 -- start start 2022-07-12T08:41:54.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 0x7f0280146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f028013fcb0 0x7f0280140140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028005f1e0 con 0x7f0280065490 2022-07-12T08:41:54.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02800b11d0 con 0x7f028013fcb0 2022-07-12T08:41:54.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0286390700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0280060170 con 0x7f02800b0df0 2022-07-12T08:41:54.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0285b8f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f028013fcb0 0x7f0280140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f028538e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 0x7f0280146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f028538e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 0x7f0280146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36300/0 (socket says 172.21.15.61:36300) 2022-07-12T08:41:54.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f028538e700 1 -- 172.21.15.61:0/1028424251 learned_addr learned my addr 172.21.15.61:0/1028424251 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:54.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.740+0000 7f0285b8f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f028013fcb0 0x7f0280140140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54108/0 (socket says 172.21.15.61:54108) 2022-07-12T08:41:54.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f028538e700 1 -- 172.21.15.61:0/1028424251 >> v2:172.21.15.61:3300/0 conn(0x7f028013fcb0 msgr2=0x7f0280140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:54.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f028538e700 1 --2- 172.21.15.61:0/1028424251 >> v2:172.21.15.61:3300/0 conn(0x7f028013fcb0 0x7f0280140140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f028538e700 1 -- 172.21.15.61:0/1028424251 >> v1:172.21.15.111:6789/0 conn(0x7f0280065490 legacy=0x7f02800b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f028538e700 1 -- 172.21.15.61:0/1028424251 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028005f410 con 0x7f02800b0df0 2022-07-12T08:41:54.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f028538e700 1 --2- 172.21.15.61:0/1028424251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 0x7f0280146140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f027c002f70 tx=0x7f027c006eb0).ready entity=mon.2 client_cookie=e6cdd13509905734 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0277fff700 1 -- 172.21.15.61:0/1028424251 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f027c00d570 con 0x7f02800b0df0 2022-07-12T08:41:54.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0277fff700 1 -- 172.21.15.61:0/1028424251 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c000e90 con 0x7f02800b0df0 2022-07-12T08:41:54.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0277fff700 1 -- 172.21.15.61:0/1028424251 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f027c00d900 con 0x7f02800b0df0 2022-07-12T08:41:54.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 -- 172.21.15.61:0/1028424251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 msgr2=0x7f0280146140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 --2- 172.21.15.61:0/1028424251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 0x7f0280146140 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f027c002f70 tx=0x7f027c006eb0).stop 2022-07-12T08:41:54.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 -- 172.21.15.61:0/1028424251 shutdown_connections 2022-07-12T08:41:54.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 --2- 172.21.15.61:0/1028424251 >> v2:172.21.15.61:3300/0 conn(0x7f028013fcb0 0x7f0280140140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 --2- 172.21.15.61:0/1028424251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02800b0df0 0x7f0280146140 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 -- 172.21.15.61:0/1028424251 >> 172.21.15.61:0/1028424251 conn(0x7f0280063ad0 msgr2=0x7f0280063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 -- 172.21.15.61:0/1028424251 shutdown_connections 2022-07-12T08:41:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.744+0000 7f0286390700 1 -- 172.21.15.61:0/1028424251 wait complete. 2022-07-12T08:41:54.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 Processor -- start 2022-07-12T08:41:54.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 -- start start 2022-07-12T08:41:54.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 0x7f0280155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f028013fcb0 0x7f028013d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800adc10 0x7f028013c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b5510 con 0x7f02800b0df0 2022-07-12T08:41:54.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b5740 con 0x7f02800adc10 2022-07-12T08:41:54.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02800b59e0 con 0x7f028013fcb0 2022-07-12T08:41:54.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 0x7f0280155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 0x7f0280155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54110/0 (socket says 172.21.15.61:54110) 2022-07-12T08:41:54.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 -- 172.21.15.61:0/2091622244 learned_addr learned my addr 172.21.15.61:0/2091622244 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:54.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0284b8d700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f028013fcb0 0x7f028013d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0285b8f700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800adc10 0x7f028013c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 -- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f028013fcb0 msgr2=0x7f028013d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f028013fcb0 0x7f028013d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 -- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800adc10 msgr2=0x7f028013c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800adc10 0x7f028013c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 -- 172.21.15.61:0/2091622244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c00b1e0 con 0x7f02800b0df0 2022-07-12T08:41:54.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f028538e700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 0x7f0280155270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f027c002f70 tx=0x7f027c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f027c011950 con 0x7f02800b0df0 2022-07-12T08:41:54.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027c011ab0 con 0x7f02800b0df0 2022-07-12T08:41:54.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f027c011e10 con 0x7f02800b0df0 2022-07-12T08:41:54.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02800ae0c0 con 0x7f02800b0df0 2022-07-12T08:41:54.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.748+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028013ce10 con 0x7f02800b0df0 2022-07-12T08:41:54.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.752+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f027c01a040 con 0x7f02800b0df0 2022-07-12T08:41:54.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.752+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0280057460 con 0x7f02800b0df0 2022-07-12T08:41:54.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.752+0000 7f02767fc700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0268053af0 0x7f0268055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:54.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.752+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f027c06ca80 con 0x7f02800b0df0 2022-07-12T08:41:54.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.752+0000 7f0284b8d700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0268053af0 0x7f0268055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:54.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.752+0000 7f0284b8d700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0268053af0 0x7f0268055fb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f027000b000 tx=0x7f0270009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:54.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.760+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f027c043d10 con 0x7f02800b0df0 2022-07-12T08:41:54.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.924+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f0280060a50 con 0x7f02800b0df0 2022-07-12T08:41:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f02767fc700 1 -- 172.21.15.61:0/2091622244 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f027c023020 con 0x7f02800b0df0 2022-07-12T08:41:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0268053af0 msgr2=0x7f0268055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0268053af0 0x7f0268055fb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f027000b000 tx=0x7f0270009040).stop 2022-07-12T08:41:54.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 msgr2=0x7f0280155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 0x7f0280155270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f027c002f70 tx=0x7f027c015000).stop 2022-07-12T08:41:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 shutdown_connections 2022-07-12T08:41:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0268053af0 0x7f0268055fb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02800adc10 0x7f028013c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f028013fcb0 0x7f028013d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 --2- 172.21.15.61:0/2091622244 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02800b0df0 0x7f0280155270 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 >> 172.21.15.61:0/2091622244 conn(0x7f0280063ad0 msgr2=0x7f0280066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 shutdown_connections 2022-07-12T08:41:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:54.932+0000 7f0286390700 1 -- 172.21.15.61:0/2091622244 wait complete. 2022-07-12T08:41:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-12T08:41:54.968 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:55.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 Processor -- start 2022-07-12T08:41:55.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 -- start start 2022-07-12T08:41:55.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f81c00ab290 0x7f81c00ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00afa60 0x7f81c00aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c00abc70 con 0x7f81c005f140 2022-07-12T08:41:55.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c005f580 con 0x7f81c00ab290 2022-07-12T08:41:55.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c005c720 con 0x7f81c00afa60 2022-07-12T08:41:55.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81be7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f81c005f140 0x7f81c005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42338/0 (socket says 172.21.15.61:42338) 2022-07-12T08:41:55.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81be7fc700 1 -- 172.21.15.61:0/3201521747 learned_addr learned my addr 172.21.15.61:0/3201521747 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:55.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/3201521747 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00afa60 0x7f81c00aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81beffd700 1 --2- 172.21.15.61:0/3201521747 >> v2:172.21.15.111:3300/0 conn(0x7f81c00ab290 0x7f81c00ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bdffb700 1 -- 172.21.15.61:0/3201521747 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1179305708 0 0) 0x7f81c00abc70 con 0x7f81c005f140 2022-07-12T08:41:55.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bdffb700 1 -- 172.21.15.61:0/3201521747 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a4003260 con 0x7f81c005f140 2022-07-12T08:41:55.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bdffb700 1 -- 172.21.15.61:0/3201521747 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3307606036 0 0) 0x7f81a4003260 con 0x7f81c005f140 2022-07-12T08:41:55.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3201521747 >> v2:172.21.15.111:3300/0 conn(0x7f81c00ab290 msgr2=0x7f81c00ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:55.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/3201521747 >> v2:172.21.15.111:3300/0 conn(0x7f81c00ab290 0x7f81c00ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3201521747 >> v1:172.21.15.61:6789/0 conn(0x7f81c005f140 legacy=0x7f81c005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3201521747 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c005c9c0 con 0x7f81c00afa60 2022-07-12T08:41:55.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81beffd700 1 --2- 172.21.15.61:0/3201521747 >> v2:172.21.15.111:3300/0 conn(0x7f81c00ab290 0x7f81c00ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:55.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/3201521747 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00afa60 0x7f81c00aff10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f81b0001b80 tx=0x7f81b8006df0).ready entity=mon.2 client_cookie=4f080170f10e9554 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bdffb700 1 -- 172.21.15.61:0/3201521747 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b00019e0 con 0x7f81c00afa60 2022-07-12T08:41:55.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bdffb700 1 -- 172.21.15.61:0/3201521747 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b8000e90 con 0x7f81c00afa60 2022-07-12T08:41:55.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.080+0000 7f81bdffb700 1 -- 172.21.15.61:0/3201521747 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b8012460 con 0x7f81c00afa60 2022-07-12T08:41:55.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- 172.21.15.61:0/3201521747 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00afa60 msgr2=0x7f81c00aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 --2- 172.21.15.61:0/3201521747 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00afa60 0x7f81c00aff10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f81b0001b80 tx=0x7f81b8006df0).stop 2022-07-12T08:41:55.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- 172.21.15.61:0/3201521747 shutdown_connections 2022-07-12T08:41:55.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 --2- 172.21.15.61:0/3201521747 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00afa60 0x7f81c00aff10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 --2- 172.21.15.61:0/3201521747 >> v2:172.21.15.111:3300/0 conn(0x7f81c00ab290 0x7f81c00ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- 172.21.15.61:0/3201521747 >> 172.21.15.61:0/3201521747 conn(0x7f81c00b1950 msgr2=0x7f81c00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- 172.21.15.61:0/3201521747 shutdown_connections 2022-07-12T08:41:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- 172.21.15.61:0/3201521747 wait complete. 2022-07-12T08:41:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 Processor -- start 2022-07-12T08:41:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- start start 2022-07-12T08:41:55.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 0x7f81c0159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00ab290 0x7f81c0063ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00afa60 0x7f81c014ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00667c0 con 0x7f81c005f140 2022-07-12T08:41:55.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c0066a60 con 0x7f81c00afa60 2022-07-12T08:41:55.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c0066400 con 0x7f81c00ab290 2022-07-12T08:41:55.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 0x7f81c0159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 0x7f81c0159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54126/0 (socket says 172.21.15.61:54126) 2022-07-12T08:41:55.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 -- 172.21.15.61:0/2386923946 learned_addr learned my addr 172.21.15.61:0/2386923946 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:55.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81be7fc700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00ab290 0x7f81c0063ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00afa60 0x7f81c014ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 -- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00ab290 msgr2=0x7f81c0063ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00ab290 0x7f81c0063ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 -- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00afa60 msgr2=0x7f81c014ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00afa60 0x7f81c014ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.084+0000 7f81beffd700 1 -- 172.21.15.61:0/2386923946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b400b180 con 0x7f81c005f140 2022-07-12T08:41:55.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81beffd700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 0x7f81c0159210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f81b400c000 tx=0x7f81b4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b400c7e0 con 0x7f81c005f140 2022-07-12T08:41:55.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b400c940 con 0x7f81c005f140 2022-07-12T08:41:55.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b800b180 con 0x7f81c005f140 2022-07-12T08:41:55.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b400cca0 con 0x7f81c005f140 2022-07-12T08:41:55.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c0149190 con 0x7f81c005f140 2022-07-12T08:41:55.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c0067830 con 0x7f81c005f140 2022-07-12T08:41:55.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f81b401c460 con 0x7f81c005f140 2022-07-12T08:41:55.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81affff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f819c053a60 0x7f819c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f81b406c0a0 con 0x7f81c005f140 2022-07-12T08:41:55.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.088+0000 7f81be7fc700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f819c053a60 0x7f819c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.092+0000 7f81be7fc700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f819c053a60 0x7f819c055f20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f81b000d000 tx=0x7f81b0001ca0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.092+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f81c0067830 con 0x7f81c005f140 2022-07-12T08:41:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.220+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81c0067830 con 0x7f81c005f140 2022-07-12T08:41:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.220+0000 7f81affff700 1 -- 172.21.15.61:0/2386923946 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+13229 (secure 0 0 0) 0x7f81c0067830 con 0x7f81c005f140 2022-07-12T08:41:55.231 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:55.232 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":26,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:52.433714+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:55.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f819c053a60 msgr2=0x7f819c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f819c053a60 0x7f819c055f20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f81b000d000 tx=0x7f81b0001ca0).stop 2022-07-12T08:41:55.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 msgr2=0x7f81c0159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 0x7f81c0159210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f81b400c000 tx=0x7f81b4006df0).stop 2022-07-12T08:41:55.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 shutdown_connections 2022-07-12T08:41:55.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f819c053a60 0x7f819c055f20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c00afa60 0x7f81c014ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c00ab290 0x7f81c0063ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 --2- 172.21.15.61:0/2386923946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c005f140 0x7f81c0159210 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 >> 172.21.15.61:0/2386923946 conn(0x7f81c00b1950 msgr2=0x7f81c00b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:55.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 shutdown_connections 2022-07-12T08:41:55.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.224+0000 7f81bffff700 1 -- 172.21.15.61:0/2386923946 wait complete. 2022-07-12T08:41:55.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:41:55.246 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:41:55.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e5241700 1 Processor -- start 2022-07-12T08:41:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e5241700 1 -- start start 2022-07-12T08:41:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e5241700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e013ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e5241700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e00b14e0 con 0x7fe7e0140490 2022-07-12T08:41:55.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e5241700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e00b1780 con 0x7fe7e0060a70 2022-07-12T08:41:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e5241700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e009a380 con 0x7fe7e005cf30 2022-07-12T08:41:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e4a40700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe7e0140490 0x7fe7e01408e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42354/0 (socket says 172.21.15.61:42354) 2022-07-12T08:41:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7e4a40700 1 -- 172.21.15.61:0/2445755414 learned_addr learned my addr 172.21.15.61:0/2445755414 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7df7fe700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e013ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822844099 0 0) 0x7fe7e00b14e0 con 0x7fe7e0140490 2022-07-12T08:41:55.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7cc003260 con 0x7fe7e0140490 2022-07-12T08:41:55.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173745114 0 0) 0x7fe7cc003260 con 0x7fe7e0140490 2022-07-12T08:41:55.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 msgr2=0x7fe7e013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:55.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e013ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 >> v1:172.21.15.111:6789/0 conn(0x7fe7e0060a70 legacy=0x7fe7e009a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.348+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0140ff0 con 0x7fe7e0140490 2022-07-12T08:41:55.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fe7d80019e0 con 0x7fe7e0140490 2022-07-12T08:41:55.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 >> v1:172.21.15.61:6789/0 conn(0x7fe7e0140490 legacy=0x7fe7e01408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7cc003780 0x7fe7cc003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7cc004500 0x7fe7cc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7cc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e009a380 con 0x7fe7cc004500 2022-07-12T08:41:55.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc00b750 con 0x7fe7cc003780 2022-07-12T08:41:55.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc00b8e0 con 0x7fe7e005cf30 2022-07-12T08:41:55.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7df7fe700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7cc004500 0x7fe7cc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7cc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7dffff700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7cc003780 0x7fe7cc003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 -- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7cc003780 msgr2=0x7fe7cc003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:55.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7cc003780 0x7fe7cc003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 -- 172.21.15.61:0/2445755414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7cc004500 msgr2=0x7fe7cc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:55.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7cc004500 0x7fe7cc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 -- 172.21.15.61:0/2445755414 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc003260 con 0x7fe7e005cf30 2022-07-12T08:41:55.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7df7fe700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7cc004500 0x7fe7cc004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:55.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7cc004e70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe7d80016f0 tx=0x7fe7d80021b0).ready entity=mon.2 client_cookie=a4ea3ed409df5ba2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7d800cca0 con 0x7fe7e005cf30 2022-07-12T08:41:55.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d800ce00 con 0x7fe7e005cf30 2022-07-12T08:41:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7deffd700 1 -- 172.21.15.61:0/2445755414 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7d8005b80 con 0x7fe7e005cf30 2022-07-12T08:41:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e5241700 1 -- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 msgr2=0x7fe7cc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e5241700 1 --2- 172.21.15.61:0/2445755414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7cc004e70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fe7d80016f0 tx=0x7fe7d80021b0).stop 2022-07-12T08:41:55.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e4a40700 1 -- 172.21.15.61:0/2445755414 reap_dead start 2022-07-12T08:41:55.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e5241700 1 -- 172.21.15.61:0/2445755414 shutdown_connections 2022-07-12T08:41:55.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e5241700 1 -- 172.21.15.61:0/2445755414 >> 172.21.15.61:0/2445755414 conn(0x7fe7e005bd10 msgr2=0x7fe7e005c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:55.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e5241700 1 -- 172.21.15.61:0/2445755414 shutdown_connections 2022-07-12T08:41:55.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.352+0000 7fe7e5241700 1 -- 172.21.15.61:0/2445755414 wait complete. 2022-07-12T08:41:55.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 Processor -- start 2022-07-12T08:41:55.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 -- start start 2022-07-12T08:41:55.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e01445c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e0063d60 0x7fe7e013dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 0x7fe7e013e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e0153140 con 0x7fe7e00ac730 2022-07-12T08:41:55.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e01533e0 con 0x7fe7e0063d60 2022-07-12T08:41:55.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e5241700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e0153680 con 0x7fe7e005cf30 2022-07-12T08:41:55.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e4a40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 0x7fe7e013e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7dffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e01445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e4a40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 0x7fe7e013e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54148/0 (socket says 172.21.15.61:54148) 2022-07-12T08:41:55.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7dffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e01445c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36340/0 (socket says 172.21.15.61:36340) 2022-07-12T08:41:55.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7df7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e0063d60 0x7fe7e013dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.356+0000 7fe7e4a40700 1 -- 172.21.15.61:0/727784177 learned_addr learned my addr 172.21.15.61:0/727784177 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:55.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e4a40700 1 -- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 msgr2=0x7fe7e01445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e01445c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e4a40700 1 -- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e0063d60 msgr2=0x7fe7e013dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e0063d60 0x7fe7e013dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e4a40700 1 -- 172.21.15.61:0/727784177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0144ab0 con 0x7fe7e00ac730 2022-07-12T08:41:55.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7df7fe700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e0063d60 0x7fe7e013dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:55.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e4a40700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 0x7fe7e013e240 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe7d8001490 tx=0x7fe7d8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7d80074d0 con 0x7fe7e00ac730 2022-07-12T08:41:55.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d8007630 con 0x7fe7e00ac730 2022-07-12T08:41:55.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e0152dc0 con 0x7fe7e00ac730 2022-07-12T08:41:55.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7d8007b50 con 0x7fe7e00ac730 2022-07-12T08:41:55.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e014f830 con 0x7fe7e00ac730 2022-07-12T08:41:55.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e0057460 con 0x7fe7e00ac730 2022-07-12T08:41:55.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.360+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe7d8010070 con 0x7fe7e00ac730 2022-07-12T08:41:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.364+0000 7fe7dd7fa700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c00539b0 0x7fe7c0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.364+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe7d806c740 con 0x7fe7e00ac730 2022-07-12T08:41:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.364+0000 7fe7dffff700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c00539b0 0x7fe7c0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.364+0000 7fe7dffff700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c00539b0 0x7fe7c0055e70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fe7d4003000 tx=0x7fe7d40079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.368+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe7d8043950 con 0x7fe7e00ac730 2022-07-12T08:41:55.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.496+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe7e006b2b0 con 0x7fe7e00ac730 2022-07-12T08:41:55.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.496+0000 7fe7dd7fa700 1 -- 172.21.15.61:0/727784177 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v27) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe7d8046f70 con 0x7fe7e00ac730 2022-07-12T08:41:55.511 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:41:55.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c00539b0 msgr2=0x7fe7c0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c00539b0 0x7fe7c0055e70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fe7d4003000 tx=0x7fe7d40079e0).stop 2022-07-12T08:41:55.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 msgr2=0x7fe7e013e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 0x7fe7e013e240 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fe7d8001490 tx=0x7fe7d8015000).stop 2022-07-12T08:41:55.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 shutdown_connections 2022-07-12T08:41:55.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e00ac730 0x7fe7e013e240 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c00539b0 0x7fe7c0055e70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e0063d60 0x7fe7e013dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 --2- 172.21.15.61:0/727784177 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e005cf30 0x7fe7e01445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 >> 172.21.15.61:0/727784177 conn(0x7fe7e005bd10 msgr2=0x7fe7e0141530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:55.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 shutdown_connections 2022-07-12T08:41:55.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.500+0000 7fe7e5241700 1 -- 172.21.15.61:0/727784177 wait complete. 2022-07-12T08:41:55.521 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:41:55.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 Processor -- start 2022-07-12T08:41:55.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 -- start start 2022-07-12T08:41:55.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1914142eb0 0x7f191405c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f191405d130 0x7f19140b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1914143500 con 0x7f19140b5f10 2022-07-12T08:41:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1914143290 con 0x7f1914142eb0 2022-07-12T08:41:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1918a77700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1914143760 con 0x7f191405d130 2022-07-12T08:41:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1913fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f19140b5f10 0x7f1914146090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42374/0 (socket says 172.21.15.61:42374) 2022-07-12T08:41:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f1913fff700 1 -- 172.21.15.61:0/4241834460 learned_addr learned my addr 172.21.15.61:0/4241834460 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.624+0000 7f19137fe700 1 --2- 172.21.15.61:0/4241834460 >> v2:172.21.15.61:3300/0 conn(0x7f1914142eb0 0x7f191405c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1912ffd700 1 --2- 172.21.15.61:0/4241834460 >> v2:172.21.15.111:3300/0 conn(0x7f191405d130 0x7f19140b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19127fc700 1 -- 172.21.15.61:0/4241834460 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870805205 0 0) 0x7f1914143500 con 0x7f19140b5f10 2022-07-12T08:41:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19127fc700 1 -- 172.21.15.61:0/4241834460 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f4003260 con 0x7f19140b5f10 2022-07-12T08:41:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19137fe700 1 -- 172.21.15.61:0/4241834460 >> v2:172.21.15.111:3300/0 conn(0x7f191405d130 msgr2=0x7f19140b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:55.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19137fe700 1 --2- 172.21.15.61:0/4241834460 >> v2:172.21.15.111:3300/0 conn(0x7f191405d130 0x7f19140b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19137fe700 1 -- 172.21.15.61:0/4241834460 >> v1:172.21.15.61:6789/0 conn(0x7f19140b5f10 legacy=0x7f1914146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19137fe700 1 -- 172.21.15.61:0/4241834460 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19140b4030 con 0x7f1914142eb0 2022-07-12T08:41:55.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19137fe700 1 --2- 172.21.15.61:0/4241834460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f191405c6d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1914143c30 tx=0x7f1904006df0).ready entity=mon.0 client_cookie=c3252d1eeed8fef9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19127fc700 1 -- 172.21.15.61:0/4241834460 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f190400d4f0 con 0x7f1914142eb0 2022-07-12T08:41:55.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19127fc700 1 -- 172.21.15.61:0/4241834460 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1904000e90 con 0x7f1914142eb0 2022-07-12T08:41:55.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f19127fc700 1 -- 172.21.15.61:0/4241834460 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f190400d840 con 0x7f1914142eb0 2022-07-12T08:41:55.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 -- 172.21.15.61:0/4241834460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 msgr2=0x7f191405c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 --2- 172.21.15.61:0/4241834460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f191405c6d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f1914143c30 tx=0x7f1904006df0).stop 2022-07-12T08:41:55.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 -- 172.21.15.61:0/4241834460 shutdown_connections 2022-07-12T08:41:55.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 --2- 172.21.15.61:0/4241834460 >> v2:172.21.15.111:3300/0 conn(0x7f191405d130 0x7f19140b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 --2- 172.21.15.61:0/4241834460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f191405c6d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 -- 172.21.15.61:0/4241834460 >> 172.21.15.61:0/4241834460 conn(0x7f1914066d70 msgr2=0x7f1914067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:55.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 -- 172.21.15.61:0/4241834460 shutdown_connections 2022-07-12T08:41:55.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 -- 172.21.15.61:0/4241834460 wait complete. 2022-07-12T08:41:55.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 Processor -- start 2022-07-12T08:41:55.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.628+0000 7f1918a77700 1 -- start start 2022-07-12T08:41:55.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f191405d130 0x7f19141495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19140b5f10 0x7f1914149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f1914140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1914112b10 con 0x7f1914142eb0 2022-07-12T08:41:55.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1914114100 con 0x7f19140b5f10 2022-07-12T08:41:55.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19141143a0 con 0x7f191405d130 2022-07-12T08:41:55.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f1914140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f1914140ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54162/0 (socket says 172.21.15.61:54162) 2022-07-12T08:41:55.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 -- 172.21.15.61:0/3916738507 learned_addr learned my addr 172.21.15.61:0/3916738507 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:55.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1912ffd700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19140b5f10 0x7f1914149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f19137fe700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f191405d130 0x7f19141495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 -- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f191405d130 msgr2=0x7f19141495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f191405d130 0x7f19141495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 -- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19140b5f10 msgr2=0x7f1914149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19140b5f10 0x7f1914149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 -- 172.21.15.61:0/3916738507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190400b180 con 0x7f1914142eb0 2022-07-12T08:41:55.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f19137fe700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f191405d130 0x7f19141495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:41:55.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1913fff700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f1914140ca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f190c001490 tx=0x7f190c007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f190c005ba0 con 0x7f1914142eb0 2022-07-12T08:41:55.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c005d00 con 0x7f1914142eb0 2022-07-12T08:41:55.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1914148a50 con 0x7f1914142eb0 2022-07-12T08:41:55.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f190c007300 con 0x7f1914142eb0 2022-07-12T08:41:55.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1914148ef0 con 0x7f1914142eb0 2022-07-12T08:41:55.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.632+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1914148be0 con 0x7f1914142eb0 2022-07-12T08:41:55.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.636+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f190c010030 con 0x7f1914142eb0 2022-07-12T08:41:55.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.636+0000 7f1910ff9700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f18f8053a40 0x7f18f8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.636+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f190c06bc00 con 0x7f1914142eb0 2022-07-12T08:41:55.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.636+0000 7f19137fe700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f18f8053a40 0x7f18f8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.636+0000 7f19137fe700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f18f8053a40 0x7f18f8055f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1904018000 tx=0x7f1904014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.640+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f190c071020 con 0x7f1914142eb0 2022-07-12T08:41:55.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.768+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1914057460 con 0x7f1914142eb0 2022-07-12T08:41:55.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.768+0000 7f1910ff9700 1 -- 172.21.15.61:0/3916738507 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f190c042df0 con 0x7f1914142eb0 2022-07-12T08:41:55.778 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:41:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f18f8053a40 msgr2=0x7f18f8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f18f8053a40 0x7f18f8055f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1904018000 tx=0x7f1904014000).stop 2022-07-12T08:41:55.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 msgr2=0x7f1914140ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f1914140ca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f190c001490 tx=0x7f190c007e70).stop 2022-07-12T08:41:55.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 shutdown_connections 2022-07-12T08:41:55.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1914142eb0 0x7f1914140ca0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f18f8053a40 0x7f18f8055f00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19140b5f10 0x7f1914149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 --2- 172.21.15.61:0/3916738507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f191405d130 0x7f19141495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 >> 172.21.15.61:0/3916738507 conn(0x7f1914066d70 msgr2=0x7f19140b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:55.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 shutdown_connections 2022-07-12T08:41:55.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.772+0000 7f1918a77700 1 -- 172.21.15.61:0/3916738507 wait complete. 2022-07-12T08:41:55.826 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:41:55.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 Processor -- start 2022-07-12T08:41:55.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 -- start start 2022-07-12T08:41:55.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa54c0ab290 0x7fa54c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c0afa60 0x7fa54c0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:55.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0abc70 con 0x7fa54c05f140 2022-07-12T08:41:55.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c05f580 con 0x7fa54c0ab290 2022-07-12T08:41:55.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55256e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c05c720 con 0x7fa54c0afa60 2022-07-12T08:41:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa551d6d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c0afa60 0x7fa54c0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.936+0000 7fa55156c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa54c0ab290 0x7fa54c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa550d6b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa54c05f140 0x7fa54c05ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:55988/0 (socket says 172.21.15.61:55988) 2022-07-12T08:41:55.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa550d6b700 1 -- 172.21.15.61:0/3722561212 learned_addr learned my addr 172.21.15.61:0/3722561212 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:55.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55156c700 1 -- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c0afa60 msgr2=0x7fa54c0aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:55.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55156c700 1 --2- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c0afa60 0x7fa54c0aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55156c700 1 -- 172.21.15.61:0/3722561212 >> v1:172.21.15.111:6789/0 conn(0x7fa54c05f140 legacy=0x7fa54c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55156c700 1 -- 172.21.15.61:0/3722561212 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c05c9c0 con 0x7fa54c0ab290 2022-07-12T08:41:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55156c700 1 --2- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c0ab670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa5440032f0 tx=0x7fa5440091d0).ready entity=mon.1 client_cookie=a3c114b6a36c156a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:55.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa54bfff700 1 -- 172.21.15.61:0/3722561212 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54400b6c0 con 0x7fa54c0ab290 2022-07-12T08:41:55.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa54bfff700 1 -- 172.21.15.61:0/3722561212 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54400b820 con 0x7fa54c0ab290 2022-07-12T08:41:56.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa54bfff700 1 -- 172.21.15.61:0/3722561212 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54400b990 con 0x7fa54c0ab290 2022-07-12T08:41:56.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 -- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 msgr2=0x7fa54c0ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 --2- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c0ab670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fa5440032f0 tx=0x7fa5440091d0).stop 2022-07-12T08:41:56.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 -- 172.21.15.61:0/3722561212 shutdown_connections 2022-07-12T08:41:56.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 --2- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c0afa60 0x7fa54c0aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 --2- 172.21.15.61:0/3722561212 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c0ab670 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 -- 172.21.15.61:0/3722561212 >> 172.21.15.61:0/3722561212 conn(0x7fa54c0b1950 msgr2=0x7fa54c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:56.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 -- 172.21.15.61:0/3722561212 shutdown_connections 2022-07-12T08:41:56.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 -- 172.21.15.61:0/3722561212 wait complete. 2022-07-12T08:41:56.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 Processor -- start 2022-07-12T08:41:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.940+0000 7fa55256e700 1 -- start start 2022-07-12T08:41:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c05f140 0x7fa54c0adba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c063860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 0x7fa54c156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c064f80 con 0x7fa54c0afa60 2022-07-12T08:41:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c149c30 con 0x7fa54c0ab290 2022-07-12T08:41:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c1562e0 con 0x7fa54c05f140 2022-07-12T08:41:56.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 0x7fa54c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 0x7fa54c156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54176/0 (socket says 172.21.15.61:54176) 2022-07-12T08:41:56.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55156c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c05f140 0x7fa54c0adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 -- 172.21.15.61:0/1616622471 learned_addr learned my addr 172.21.15.61:0/1616622471 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:56.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa550d6b700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c063860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 -- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c05f140 msgr2=0x7fa54c0adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c05f140 0x7fa54c0adba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 -- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 msgr2=0x7fa54c063860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c063860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 -- 172.21.15.61:0/1616622471 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa544009470 con 0x7fa54c0afa60 2022-07-12T08:41:56.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa550d6b700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c063860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:56.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa551d6d700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 0x7fa54c156780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa53c00c000 tx=0x7fa53c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:56.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa53c011750 con 0x7fa54c0afa60 2022-07-12T08:41:56.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c0118b0 con 0x7fa54c0afa60 2022-07-12T08:41:56.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa53c011c10 con 0x7fa54c0afa60 2022-07-12T08:41:56.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c155ff0 con 0x7fa54c0afa60 2022-07-12T08:41:56.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c063d50 con 0x7fa54c0afa60 2022-07-12T08:41:56.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.944+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c149a50 con 0x7fa54c0afa60 2022-07-12T08:41:56.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.948+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa53c011d70 con 0x7fa54c0afa60 2022-07-12T08:41:56.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.948+0000 7fa54a7fc700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa52c0539d0 0x7fa52c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.948+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa53c06be30 con 0x7fa54c0afa60 2022-07-12T08:41:56.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.948+0000 7fa55156c700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa52c0539d0 0x7fa52c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.948+0000 7fa55156c700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa52c0539d0 0x7fa52c055e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa54401b000 tx=0x7fa544016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:56.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:55.952+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa53c0430c0 con 0x7fa54c0afa60 2022-07-12T08:41:56.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa54c057460 con 0x7fa54c0afa60 2022-07-12T08:41:56.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa54a7fc700 1 -- 172.21.15.61:0/1616622471 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v27) v1 ==== 84+0+143 (secure 0 0 0) 0x7fa53c0466e0 con 0x7fa54c0afa60 2022-07-12T08:41:56.091 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:56.092 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:41:56.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:56.092 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:41:56.092 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:41:56.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:41:56.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:41:56.093 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:41:56.093 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:41:56.099 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:41:56.099 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:41:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa52c0539d0 msgr2=0x7fa52c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa52c0539d0 0x7fa52c055e90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fa54401b000 tx=0x7fa544016000).stop 2022-07-12T08:41:56.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 msgr2=0x7fa54c156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 0x7fa54c156780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fa53c00c000 tx=0x7fa53c006df0).stop 2022-07-12T08:41:56.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 shutdown_connections 2022-07-12T08:41:56.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa54c0afa60 0x7fa54c156780 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa52c0539d0 0x7fa52c055e90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa54c0ab290 0x7fa54c063860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 --2- 172.21.15.61:0/1616622471 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa54c05f140 0x7fa54c0adba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 >> 172.21.15.61:0/1616622471 conn(0x7fa54c0b1950 msgr2=0x7fa54c0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 shutdown_connections 2022-07-12T08:41:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.080+0000 7fa55256e700 1 -- 172.21.15.61:0/1616622471 wait complete. 2022-07-12T08:41:56.105 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:41:56.106 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:41:56.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6d03700 1 Processor -- start 2022-07-12T08:41:56.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6d03700 1 -- start start 2022-07-12T08:41:56.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6d03700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ac0065990 0x7f0ac00b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6d03700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00b6710 con 0x7f0ac013fba0 2022-07-12T08:41:56.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6d03700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac01431d0 con 0x7f0ac00b6330 2022-07-12T08:41:56.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6d03700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac00655a0 con 0x7f0ac0065990 2022-07-12T08:41:56.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac5500700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ac0065990 0x7f0ac00b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6502700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0ac013fba0 0x7f0ac013fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42402/0 (socket says 172.21.15.61:42402) 2022-07-12T08:41:56.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac6502700 1 -- 172.21.15.61:0/3038192008 learned_addr learned my addr 172.21.15.61:0/3038192008 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:56.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac5500700 1 -- 172.21.15.61:0/3038192008 >> v1:172.21.15.111:6789/0 conn(0x7f0ac00b6330 legacy=0x7f0ac00b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac5500700 1 -- 172.21.15.61:0/3038192008 >> v1:172.21.15.61:6789/0 conn(0x7f0ac013fba0 legacy=0x7f0ac013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac5500700 1 -- 172.21.15.61:0/3038192008 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0143470 con 0x7f0ac0065990 2022-07-12T08:41:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac4cff700 1 -- 172.21.15.61:0/3038192008 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216086245 0 0) 0x7f0ac01431d0 con 0x7f0ac00b6330 2022-07-12T08:41:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac5500700 1 --2- 172.21.15.61:0/3038192008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac00b1b30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0ab0002f70 tx=0x7f0ab0006df0).ready entity=mon.0 client_cookie=e33d09e56bb8c66b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac4cff700 1 -- 172.21.15.61:0/3038192008 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ab000d610 con 0x7f0ac0065990 2022-07-12T08:41:56.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac4cff700 1 -- 172.21.15.61:0/3038192008 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab0000e90 con 0x7f0ac0065990 2022-07-12T08:41:56.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.212+0000 7f0ac4cff700 1 -- 172.21.15.61:0/3038192008 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ab000d960 con 0x7f0ac0065990 2022-07-12T08:41:56.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/3038192008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 msgr2=0x7f0ac00b1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/3038192008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac00b1b30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f0ab0002f70 tx=0x7f0ab0006df0).stop 2022-07-12T08:41:56.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/3038192008 shutdown_connections 2022-07-12T08:41:56.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/3038192008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac00b1b30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/3038192008 >> 172.21.15.61:0/3038192008 conn(0x7f0ac0063ad0 msgr2=0x7f0ac0063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:56.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/3038192008 shutdown_connections 2022-07-12T08:41:56.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/3038192008 wait complete. 2022-07-12T08:41:56.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 Processor -- start 2022-07-12T08:41:56.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- start start 2022-07-12T08:41:56.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac005f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ac00b6330 0x7f0ac0144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ac013fba0 0x7f0ac0141a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01428f0 con 0x7f0ac0065990 2022-07-12T08:41:56.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0141070 con 0x7f0ac013fba0 2022-07-12T08:41:56.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0141310 con 0x7f0ac00b6330 2022-07-12T08:41:56.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac005f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac005f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54186/0 (socket says 172.21.15.61:54186) 2022-07-12T08:41:56.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 -- 172.21.15.61:0/4229364933 learned_addr learned my addr 172.21.15.61:0/4229364933 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:56.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5500700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ac00b6330 0x7f0ac0144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6502700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ac013fba0 0x7f0ac0141a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 -- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ac00b6330 msgr2=0x7f0ac0144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ac00b6330 0x7f0ac0144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 -- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ac013fba0 msgr2=0x7f0ac0141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:56.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ac013fba0 0x7f0ac0141a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:56.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 -- 172.21.15.61:0/4229364933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab000b180 con 0x7f0ac0065990 2022-07-12T08:41:56.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac5d01700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac005f480 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f0abc000fe0 tx=0x7f0abc007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:56.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc012400 con 0x7f0ac0065990 2022-07-12T08:41:56.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc012560 con 0x7f0ac0065990 2022-07-12T08:41:56.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac01415b0 con 0x7f0ac0065990 2022-07-12T08:41:56.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc0128c0 con 0x7f0ac0065990 2022-07-12T08:41:56.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.216+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00b59f0 con 0x7f0ac0065990 2022-07-12T08:41:56.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.220+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0abc019030 con 0x7f0ac0065990 2022-07-12T08:41:56.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.220+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0057460 con 0x7f0ac0065990 2022-07-12T08:41:56.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.220+0000 7f0aaeffd700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0aa0053a40 0x7f0aa0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:56.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.220+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0abc06af50 con 0x7f0ac0065990 2022-07-12T08:41:56.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.220+0000 7f0ac5500700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0aa0053a40 0x7f0aa0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:56.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.220+0000 7f0ac5500700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0aa0053a40 0x7f0aa0055f00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0ab001c000 tx=0x7f0ab0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:56.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.228+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0abc045b30 con 0x7f0ac0065990 2022-07-12T08:41:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:56.356+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f0ac0068350 con 0x7f0ac0065990 2022-07-12T08:41:57.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.288+0000 7f0aaeffd700 1 -- 172.21.15.61:0/4229364933 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v28) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0abc045720 con 0x7f0ac0065990 2022-07-12T08:41:57.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0aa0053a40 msgr2=0x7f0aa0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:57.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0aa0053a40 0x7f0aa0055f00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0ab001c000 tx=0x7f0ab0015000).stop 2022-07-12T08:41:57.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 msgr2=0x7f0ac005f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:57.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac005f480 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f0abc000fe0 tx=0x7f0abc007b50).stop 2022-07-12T08:41:57.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 shutdown_connections 2022-07-12T08:41:57.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0aa0053a40 0x7f0aa0055f00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ac013fba0 0x7f0ac0141a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ac00b6330 0x7f0ac0144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 --2- 172.21.15.61:0/4229364933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ac0065990 0x7f0ac005f480 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 >> 172.21.15.61:0/4229364933 conn(0x7f0ac0063ad0 msgr2=0x7f0ac00b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:57.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 shutdown_connections 2022-07-12T08:41:57.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.292+0000 7f0ac6d03700 1 -- 172.21.15.61:0/4229364933 wait complete. 2022-07-12T08:41:57.313 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:41:57.325 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:41:57.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 Processor -- start 2022-07-12T08:41:57.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 -- start start 2022-07-12T08:41:57.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f55cc0b5920 0x7f55cc0aa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:57.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 0x7f55cc0b0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:57.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc0abfa0 con 0x7f55cc05faf0 2022-07-12T08:41:57.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc05c870 con 0x7f55cc0b5920 2022-07-12T08:41:57.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.432+0000 7f55cb59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0b0b70 con 0x7f55cc0b0240 2022-07-12T08:41:57.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c9d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 0x7f55cc0b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:57.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f55cc0b5920 0x7f55cc0aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:57.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f55cc0b5920 0x7f55cc0aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32770/0 (socket says 172.21.15.61:32770) 2022-07-12T08:41:57.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cad9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f55cc05faf0 0x7f55cc05ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56020/0 (socket says 172.21.15.61:56020) 2022-07-12T08:41:57.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c9d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 0x7f55cc0b0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36394/0 (socket says 172.21.15.61:36394) 2022-07-12T08:41:57.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 -- 172.21.15.61:0/3793702080 learned_addr learned my addr 172.21.15.61:0/3793702080 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:57.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 -- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 msgr2=0x7f55cc0b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:57.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 --2- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 0x7f55cc0b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 -- 172.21.15.61:0/3793702080 >> v1:172.21.15.111:6789/0 conn(0x7f55cc05faf0 legacy=0x7f55cc05ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:57.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c959a700 1 -- 172.21.15.61:0/3793702080 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120834765 0 0) 0x7f55cc0abfa0 con 0x7f55cc05faf0 2022-07-12T08:41:57.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 -- 172.21.15.61:0/3793702080 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc05cd00 con 0x7f55cc0b5920 2022-07-12T08:41:57.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c9d9b700 1 --2- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 0x7f55cc0b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:57.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55ca59c700 1 --2- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b5920 0x7f55cc0aa500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f55b4002f70 tx=0x7f55b4006eb0).ready entity=mon.1 client_cookie=8309895bc20e5c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:57.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c959a700 1 -- 172.21.15.61:0/3793702080 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55b400d570 con 0x7f55cc0b5920 2022-07-12T08:41:57.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c959a700 1 -- 172.21.15.61:0/3793702080 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b4000e90 con 0x7f55cc0b5920 2022-07-12T08:41:57.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55c959a700 1 -- 172.21.15.61:0/3793702080 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55b400d8c0 con 0x7f55cc0b5920 2022-07-12T08:41:57.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 -- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b5920 msgr2=0x7f55cc0aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:57.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b5920 0x7f55cc0aa500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f55b4002f70 tx=0x7f55b4006eb0).stop 2022-07-12T08:41:57.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 -- 172.21.15.61:0/3793702080 shutdown_connections 2022-07-12T08:41:57.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b0240 0x7f55cc0b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3793702080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b5920 0x7f55cc0aa500 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 -- 172.21.15.61:0/3793702080 >> 172.21.15.61:0/3793702080 conn(0x7f55cc063340 msgr2=0x7f55cc0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:57.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 -- 172.21.15.61:0/3793702080 shutdown_connections 2022-07-12T08:41:57.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.436+0000 7f55cb59e700 1 -- 172.21.15.61:0/3793702080 wait complete. 2022-07-12T08:41:57.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 Processor -- start 2022-07-12T08:41:57.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 -- start start 2022-07-12T08:41:57.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 0x7f55cc13ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:57.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b0240 0x7f55cc147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:57.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b5920 0x7f55cc13f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:57.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0ae2f0 con 0x7f55cc05faf0 2022-07-12T08:41:57.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc1478f0 con 0x7f55cc0b0240 2022-07-12T08:41:57.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0aeef0 con 0x7f55cc0b5920 2022-07-12T08:41:57.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55ca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 0x7f55cc13ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:57.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b0240 0x7f55cc147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:57.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55ca59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 0x7f55cc13ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54204/0 (socket says 172.21.15.61:54204) 2022-07-12T08:41:57.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55ca59c700 1 -- 172.21.15.61:0/3425947765 learned_addr learned my addr 172.21.15.61:0/3425947765 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:57.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cad9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b5920 0x7f55cc13f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:57.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 -- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b5920 msgr2=0x7f55cc13f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:57.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b5920 0x7f55cc13f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 -- 172.21.15.61:0/3425947765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 msgr2=0x7f55cc13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:57.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 0x7f55cc13ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:57.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 -- 172.21.15.61:0/3425947765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55bc009470 con 0x7f55cc0b0240 2022-07-12T08:41:57.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55ca59c700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 0x7f55cc13ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c9d9b700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b0240 0x7f55cc147400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f55cc0ae430 tx=0x7f55bc0091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc00c850 con 0x7f55cc0b0240 2022-07-12T08:41:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc00c9b0 con 0x7f55cc0b0240 2022-07-12T08:41:57.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b400b1e0 con 0x7f55cc0b0240 2022-07-12T08:41:57.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55bc00cd10 con 0x7f55cc0b0240 2022-07-12T08:41:57.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.440+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc05cad0 con 0x7f55cc0b0240 2022-07-12T08:41:57.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.444+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc0af870 con 0x7f55cc0b0240 2022-07-12T08:41:57.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.444+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f55bc022070 con 0x7f55cc0b0240 2022-07-12T08:41:57.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.444+0000 7f55c37fe700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55b0053a60 0x7f55b0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:57.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.444+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f55bc06c450 con 0x7f55cc0b0240 2022-07-12T08:41:57.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.444+0000 7f55ca59c700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55b0053a60 0x7f55b0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:57.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.444+0000 7f55ca59c700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55b0053a60 0x7f55b0055f20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f55b401c000 tx=0x7f55b4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:57.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.452+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55bc0434e0 con 0x7f55cc0b0240 2022-07-12T08:41:57.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:57.584+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f55cc057460 con 0x7f55cc0b0240 2022-07-12T08:41:58.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.300+0000 7f55c37fe700 1 -- 172.21.15.61:0/3425947765 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v29) v1 ==== 125+0+0 (secure 0 0 0) 0x7f55bc046b00 con 0x7f55cc0b0240 2022-07-12T08:41:58.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55b0053a60 msgr2=0x7f55b0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55b0053a60 0x7f55b0055f20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f55b401c000 tx=0x7f55b4015000).stop 2022-07-12T08:41:58.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b0240 msgr2=0x7f55cc147400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b0240 0x7f55cc147400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f55cc0ae430 tx=0x7f55bc0091d0).stop 2022-07-12T08:41:58.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 shutdown_connections 2022-07-12T08:41:58.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55cc0b5920 0x7f55cc13f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55b0053a60 0x7f55b0055f20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55cc0b0240 0x7f55cc147400 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 --2- 172.21.15.61:0/3425947765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55cc05faf0 0x7f55cc13ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 >> 172.21.15.61:0/3425947765 conn(0x7f55cc063340 msgr2=0x7f55cc0b4e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 shutdown_connections 2022-07-12T08:41:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.308+0000 7f55cb59e700 1 -- 172.21.15.61:0/3425947765 wait complete. 2022-07-12T08:41:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:41:58.332 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:41:58.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.440+0000 7fb2c5289700 1 Processor -- start 2022-07-12T08:41:58.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.440+0000 7fb2c5289700 1 -- start start 2022-07-12T08:41:58.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.440+0000 7fb2c5289700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2c00b16a0 0x7fb2c0065b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.440+0000 7fb2c5289700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00aff70 con 0x7fb2c00673f0 2022-07-12T08:41:58.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.440+0000 7fb2c5289700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c00b1a80 con 0x7fb2c00b0c50 2022-07-12T08:41:58.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.440+0000 7fb2c5289700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c0064d60 con 0x7fb2c00b16a0 2022-07-12T08:41:58.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb2c00673f0 0x7fb2c0066010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42430/0 (socket says 172.21.15.61:42430) 2022-07-12T08:41:58.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bf7fe700 1 -- 172.21.15.61:0/1181637091 learned_addr learned my addr 172.21.15.61:0/1181637091 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:41:58.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bffff700 1 --2- 172.21.15.61:0/1181637091 >> v2:172.21.15.111:3300/0 conn(0x7fb2c00b16a0 0x7fb2c0065b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250601507 0 0) 0x7fb2c00aff70 con 0x7fb2c00673f0 2022-07-12T08:41:58.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8003260 con 0x7fb2c00673f0 2022-07-12T08:41:58.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797918415 0 0) 0x7fb2c00b1a80 con 0x7fb2c00b0c50 2022-07-12T08:41:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c00aff70 con 0x7fb2c00b0c50 2022-07-12T08:41:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bffff700 1 -- 172.21.15.61:0/1181637091 >> v1:172.21.15.111:6789/0 conn(0x7fb2c00b0c50 legacy=0x7fb2c013fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bffff700 1 -- 172.21.15.61:0/1181637091 >> v1:172.21.15.61:6789/0 conn(0x7fb2c00673f0 legacy=0x7fb2c0066010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bffff700 1 -- 172.21.15.61:0/1181637091 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c0066cb0 con 0x7fb2c00b16a0 2022-07-12T08:41:58.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2bffff700 1 --2- 172.21.15.61:0/1181637091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0065b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb2b0002f70 tx=0x7fb2b0006df0).ready entity=mon.1 client_cookie=ed9f0708909527d5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:58.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2b000d540 con 0x7fb2c00b16a0 2022-07-12T08:41:58.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b0000e90 con 0x7fb2c00b16a0 2022-07-12T08:41:58.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2beffd700 1 -- 172.21.15.61:0/1181637091 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2b000d890 con 0x7fb2c00b16a0 2022-07-12T08:41:58.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 -- 172.21.15.61:0/1181637091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b16a0 msgr2=0x7fb2c0065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 --2- 172.21.15.61:0/1181637091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0065b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb2b0002f70 tx=0x7fb2b0006df0).stop 2022-07-12T08:41:58.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 -- 172.21.15.61:0/1181637091 shutdown_connections 2022-07-12T08:41:58.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 --2- 172.21.15.61:0/1181637091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0065b20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 -- 172.21.15.61:0/1181637091 >> 172.21.15.61:0/1181637091 conn(0x7fb2c00b5860 msgr2=0x7fb2c00b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 -- 172.21.15.61:0/1181637091 shutdown_connections 2022-07-12T08:41:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.444+0000 7fb2c5289700 1 -- 172.21.15.61:0/1181637091 wait complete. 2022-07-12T08:41:58.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 Processor -- start 2022-07-12T08:41:58.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 -- start start 2022-07-12T08:41:58.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b0c50 0x7fb2c0143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c0141f80 0x7fb2c014f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0142cf0 con 0x7fb2c00b16a0 2022-07-12T08:41:58.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0142f30 con 0x7fb2c00b0c50 2022-07-12T08:41:58.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2c5289700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c01431d0 con 0x7fb2c0141f80 2022-07-12T08:41:58.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2bf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.448+0000 7fb2bf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54218/0 (socket says 172.21.15.61:54218) 2022-07-12T08:41:58.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b0c50 0x7fb2c0143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2c4a88700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c0141f80 0x7fb2c014f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 -- 172.21.15.61:0/229891306 learned_addr learned my addr 172.21.15.61:0/229891306 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:58.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 -- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c0141f80 msgr2=0x7fb2c014f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c0141f80 0x7fb2c014f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 -- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b0c50 msgr2=0x7fb2c0143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b0c50 0x7fb2c0143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 -- 172.21.15.61:0/229891306 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b000b180 con 0x7fb2c00b16a0 2022-07-12T08:41:58.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bf7fe700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0142800 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb2ac0011d0 tx=0x7fb2ac007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:58.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2ac0070d0 con 0x7fb2c00b16a0 2022-07-12T08:41:58.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac007230 con 0x7fb2c00b16a0 2022-07-12T08:41:58.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c0140d00 con 0x7fb2c00b16a0 2022-07-12T08:41:58.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2ac005610 con 0x7fb2c00b16a0 2022-07-12T08:41:58.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c014f720 con 0x7fb2c00b16a0 2022-07-12T08:41:58.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.452+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c0066f50 con 0x7fb2c00b16a0 2022-07-12T08:41:58.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.456+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb2ac01a060 con 0x7fb2c00b16a0 2022-07-12T08:41:58.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.456+0000 7fb2bd7fa700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb29c053ad0 0x7fb29c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.456+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fb2ac06b560 con 0x7fb2c00b16a0 2022-07-12T08:41:58.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.456+0000 7fb2bffff700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb29c053ad0 0x7fb29c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.456+0000 7fb2bffff700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb29c053ad0 0x7fb29c055f90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb2b001a000 tx=0x7fb2b0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:58.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.464+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb2ac042530 con 0x7fb2c00b16a0 2022-07-12T08:41:58.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.628+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fb2c0057460 con 0x7fb2c00b16a0 2022-07-12T08:41:58.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.632+0000 7fb2bd7fa700 1 -- 172.21.15.61:0/229891306 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 5 and data pool 6 v10) v1 ==== 187+0+0 (secure 0 0 0) 0x7fb2ac018020 con 0x7fb2c00b16a0 2022-07-12T08:41:58.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb29c053ad0 msgr2=0x7fb29c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb29c053ad0 0x7fb29c055f90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fb2b001a000 tx=0x7fb2b0014000).stop 2022-07-12T08:41:58.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 msgr2=0x7fb2c0142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0142800 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fb2ac0011d0 tx=0x7fb2ac007b50).stop 2022-07-12T08:41:58.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 shutdown_connections 2022-07-12T08:41:58.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c0141f80 0x7fb2c014f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c00b16a0 0x7fb2c0142800 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb29c053ad0 0x7fb29c055f90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 --2- 172.21.15.61:0/229891306 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c00b0c50 0x7fb2c0143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 >> 172.21.15.61:0/229891306 conn(0x7fb2c00b5860 msgr2=0x7fb2c01401b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:58.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 shutdown_connections 2022-07-12T08:41:58.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.640+0000 7fb2c5289700 1 -- 172.21.15.61:0/229891306 wait complete. 2022-07-12T08:41:58.656 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 5 and data pool 6 2022-07-12T08:41:58.671 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:58.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 Processor -- start 2022-07-12T08:41:58.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 -- start start 2022-07-12T08:41:58.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe600b6780 0x7fbe600b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 0x7fbe60065a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe60140010 con 0x7fbe600b25c0 2022-07-12T08:41:58.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe600668f0 con 0x7fbe600b6780 2022-07-12T08:41:58.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe664af700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60142f70 con 0x7fbe601436b0 2022-07-12T08:41:58.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe65cae700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 0x7fbe60065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe600b6780 0x7fbe600b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbe600b6780 0x7fbe600b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32810/0 (socket says 172.21.15.61:32810) 2022-07-12T08:41:58.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe65cae700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 0x7fbe60065a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36436/0 (socket says 172.21.15.61:36436) 2022-07-12T08:41:58.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 -- 172.21.15.61:0/4209786457 learned_addr learned my addr 172.21.15.61:0/4209786457 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:58.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 -- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 msgr2=0x7fbe60065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:58.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 --2- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 0x7fbe60065a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 -- 172.21.15.61:0/4209786457 >> v1:172.21.15.111:6789/0 conn(0x7fbe600b25c0 legacy=0x7fbe6013f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 -- 172.21.15.61:0/4209786457 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe60143210 con 0x7fbe600b6780 2022-07-12T08:41:58.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe65cae700 1 --2- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 0x7fbe60065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:58.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.768+0000 7fbe654ad700 1 --2- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe600b6780 0x7fbe600b6b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fbe60143590 tx=0x7fbe58009290).ready entity=mon.1 client_cookie=f923e92064b0d696 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:58.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe4ffff700 1 -- 172.21.15.61:0/4209786457 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe5800b860 con 0x7fbe600b6780 2022-07-12T08:41:58.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe4ffff700 1 -- 172.21.15.61:0/4209786457 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5800b9c0 con 0x7fbe600b6780 2022-07-12T08:41:58.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe4ffff700 1 -- 172.21.15.61:0/4209786457 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe5800bb30 con 0x7fbe600b6780 2022-07-12T08:41:58.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe600b6780 msgr2=0x7fbe600b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 --2- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe600b6780 0x7fbe600b6b60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fbe60143590 tx=0x7fbe58009290).stop 2022-07-12T08:41:58.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- 172.21.15.61:0/4209786457 shutdown_connections 2022-07-12T08:41:58.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 --2- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe601436b0 0x7fbe60065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 --2- 172.21.15.61:0/4209786457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe600b6780 0x7fbe600b6b60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- 172.21.15.61:0/4209786457 >> 172.21.15.61:0/4209786457 conn(0x7fbe60064570 msgr2=0x7fbe600b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:58.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- 172.21.15.61:0/4209786457 shutdown_connections 2022-07-12T08:41:58.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- 172.21.15.61:0/4209786457 wait complete. 2022-07-12T08:41:58.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 Processor -- start 2022-07-12T08:41:58.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- start start 2022-07-12T08:41:58.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 0x7fbe6013d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe600b6780 0x7fbe6013b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe601436b0 0x7fbe6013b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6014e7f0 con 0x7fbe600b25c0 2022-07-12T08:41:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6013dbd0 con 0x7fbe601436b0 2022-07-12T08:41:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe664af700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6013de70 con 0x7fbe600b6780 2022-07-12T08:41:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe654ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 0x7fbe6013d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe654ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 0x7fbe6013d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54242/0 (socket says 172.21.15.61:54242) 2022-07-12T08:41:58.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.772+0000 7fbe654ad700 1 -- 172.21.15.61:0/410923376 learned_addr learned my addr 172.21.15.61:0/410923376 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:58.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe65cae700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe601436b0 0x7fbe6013b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe64cac700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe600b6780 0x7fbe6013b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe654ad700 1 -- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe600b6780 msgr2=0x7fbe6013b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe654ad700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe600b6780 0x7fbe6013b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe654ad700 1 -- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe601436b0 msgr2=0x7fbe6013b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe654ad700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe601436b0 0x7fbe6013b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe654ad700 1 -- 172.21.15.61:0/410923376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c00b180 con 0x7fbe600b25c0 2022-07-12T08:41:58.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe654ad700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 0x7fbe6013d510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fbe60143590 tx=0x7fbe58016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:58.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe5800b860 con 0x7fbe600b25c0 2022-07-12T08:41:58.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe580032d0 con 0x7fbe600b25c0 2022-07-12T08:41:58.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe58009530 con 0x7fbe600b25c0 2022-07-12T08:41:58.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe58021630 con 0x7fbe600b25c0 2022-07-12T08:41:58.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6013bc30 con 0x7fbe600b25c0 2022-07-12T08:41:58.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6013be80 con 0x7fbe600b25c0 2022-07-12T08:41:58.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbe58027020 con 0x7fbe600b25c0 2022-07-12T08:41:58.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe4e7fc700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbe40053ad0 0x7fbe40055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:58.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbe5806d440 con 0x7fbe600b25c0 2022-07-12T08:41:58.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.776+0000 7fbe64cac700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbe40053ad0 0x7fbe40055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:58.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.780+0000 7fbe64cac700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbe40053ad0 0x7fbe40055f90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbe5000d000 tx=0x7fbe50007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:58.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.784+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbe58044340 con 0x7fbe600b25c0 2022-07-12T08:41:58.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.908+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe6014e650 con 0x7fbe600b25c0 2022-07-12T08:41:58.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.908+0000 7fbe4e7fc700 1 -- 172.21.15.61:0/410923376 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+16112 (secure 0 0 0) 0x7fbe6014e650 con 0x7fbe600b25c0 2022-07-12T08:41:58.929 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:58.935 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":30,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:58.633940+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:56.415996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:57.640916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbe40053ad0 msgr2=0x7fbe40055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbe40053ad0 0x7fbe40055f90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbe5000d000 tx=0x7fbe50007b90).stop 2022-07-12T08:41:58.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 msgr2=0x7fbe6013d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 0x7fbe6013d510 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fbe60143590 tx=0x7fbe58016000).stop 2022-07-12T08:41:58.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 shutdown_connections 2022-07-12T08:41:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbe40053ad0 0x7fbe40055f90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbe601436b0 0x7fbe6013b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbe600b6780 0x7fbe6013b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 --2- 172.21.15.61:0/410923376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbe600b25c0 0x7fbe6013d510 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:58.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 >> 172.21.15.61:0/410923376 conn(0x7fbe60064570 msgr2=0x7fbe600b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:58.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 shutdown_connections 2022-07-12T08:41:58.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:58.912+0000 7fbe664af700 1 -- 172.21.15.61:0/410923376 wait complete. 2022-07-12T08:41:58.958 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:41:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.060+0000 7efefcb24700 1 Processor -- start 2022-07-12T08:41:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.060+0000 7efefcb24700 1 -- start start 2022-07-12T08:41:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 0x7efef80b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efef80b4090 0x7efef805c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef80b5740 con 0x7efef8142f30 2022-07-12T08:41:59.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef80b1460 con 0x7efef80b4090 2022-07-12T08:41:59.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef80b6880 con 0x7efef80b1080 2022-07-12T08:41:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 0x7efef80b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 0x7efef80b62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36446/0 (socket says 172.21.15.61:36446) 2022-07-12T08:41:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 -- 172.21.15.61:0/2031124882 learned_addr learned my addr 172.21.15.61:0/2031124882 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efeeffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efef80b4090 0x7efef805c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 -- 172.21.15.61:0/2031124882 >> v2:172.21.15.111:3300/0 conn(0x7efef80b4090 msgr2=0x7efef805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 --2- 172.21.15.61:0/2031124882 >> v2:172.21.15.111:3300/0 conn(0x7efef80b4090 0x7efef805c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 -- 172.21.15.61:0/2031124882 >> v1:172.21.15.111:6789/0 conn(0x7efef8142f30 legacy=0x7efef80afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 -- 172.21.15.61:0/2031124882 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef805bdb0 con 0x7efef80b1080 2022-07-12T08:41:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef659c700 1 --2- 172.21.15.61:0/2031124882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 0x7efef80b62a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7efee8002f70 tx=0x7efee8006eb0).ready entity=mon.2 client_cookie=16f1ddb8665fda05 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:59.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef5d9b700 1 -- 172.21.15.61:0/2031124882 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efee800d570 con 0x7efef80b1080 2022-07-12T08:41:59.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef5d9b700 1 -- 172.21.15.61:0/2031124882 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee8000e90 con 0x7efef80b1080 2022-07-12T08:41:59.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efef5d9b700 1 -- 172.21.15.61:0/2031124882 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efee800d900 con 0x7efef80b1080 2022-07-12T08:41:59.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- 172.21.15.61:0/2031124882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 msgr2=0x7efef80b62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 --2- 172.21.15.61:0/2031124882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 0x7efef80b62a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7efee8002f70 tx=0x7efee8006eb0).stop 2022-07-12T08:41:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- 172.21.15.61:0/2031124882 shutdown_connections 2022-07-12T08:41:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 --2- 172.21.15.61:0/2031124882 >> v2:172.21.15.111:3300/0 conn(0x7efef80b4090 0x7efef805c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 --2- 172.21.15.61:0/2031124882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b1080 0x7efef80b62a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- 172.21.15.61:0/2031124882 >> 172.21.15.61:0/2031124882 conn(0x7efef8067020 msgr2=0x7efef8067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- 172.21.15.61:0/2031124882 shutdown_connections 2022-07-12T08:41:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.064+0000 7efefcb24700 1 -- 172.21.15.61:0/2031124882 wait complete. 2022-07-12T08:41:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 Processor -- start 2022-07-12T08:41:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 -- start start 2022-07-12T08:41:59.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 0x7efef81533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b4090 0x7efef8114b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef8142f30 0x7efef80adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef8113330 con 0x7efef80b1080 2022-07-12T08:41:59.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef81136d0 con 0x7efef8142f30 2022-07-12T08:41:59.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef8112df0 con 0x7efef80b4090 2022-07-12T08:41:59.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efef659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 0x7efef81533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efef659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 0x7efef81533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54256/0 (socket says 172.21.15.61:54256) 2022-07-12T08:41:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efef659c700 1 -- 172.21.15.61:0/3372194218 learned_addr learned my addr 172.21.15.61:0/3372194218 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:59.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b4090 0x7efef8114b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efef6d9d700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef8142f30 0x7efef80adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 -- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef8142f30 msgr2=0x7efef80adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef8142f30 0x7efef80adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 -- 172.21.15.61:0/3372194218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 msgr2=0x7efef81533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 0x7efef81533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 -- 172.21.15.61:0/3372194218 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee800b1e0 con 0x7efef80b4090 2022-07-12T08:41:59.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efef659c700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 0x7efef81533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:41:59.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeeffff700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b4090 0x7efef8114b70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7efee000c000 tx=0x7efee0006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:59.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efee0010780 con 0x7efef80b4090 2022-07-12T08:41:59.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee00108e0 con 0x7efef80b4090 2022-07-12T08:41:59.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef8113090 con 0x7efef80b4090 2022-07-12T08:41:59.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efee0010c40 con 0x7efef80b4090 2022-07-12T08:41:59.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.068+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef8115200 con 0x7efef80b4090 2022-07-12T08:41:59.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.072+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef81143d0 con 0x7efef80b4090 2022-07-12T08:41:59.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.072+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efee001b410 con 0x7efef80b4090 2022-07-12T08:41:59.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.072+0000 7efeef7fe700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efedc0539d0 0x7efedc055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.072+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efee006bdc0 con 0x7efef80b4090 2022-07-12T08:41:59.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.072+0000 7efef659c700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efedc0539d0 0x7efedc055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.072+0000 7efef659c700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efedc0539d0 0x7efedc055e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7efee801c000 tx=0x7efee8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:59.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.076+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efef81143d0 con 0x7efef80b4090 2022-07-12T08:41:59.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.204+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efef81143d0 con 0x7efef80b4090 2022-07-12T08:41:59.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efeef7fe700 1 -- 172.21.15.61:0/3372194218 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+16112 (secure 0 0 0) 0x7efef81143d0 con 0x7efef80b4090 2022-07-12T08:41:59.222 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:41:59.223 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":30,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:58.633940+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:56.415996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:57.640916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:41:59.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efedc0539d0 msgr2=0x7efedc055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efedc0539d0 0x7efedc055e90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7efee801c000 tx=0x7efee8015000).stop 2022-07-12T08:41:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b4090 msgr2=0x7efef8114b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b4090 0x7efef8114b70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7efee000c000 tx=0x7efee0006df0).stop 2022-07-12T08:41:59.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 shutdown_connections 2022-07-12T08:41:59.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efedc0539d0 0x7efedc055e90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efef8142f30 0x7efef80adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efef80b4090 0x7efef8114b70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 --2- 172.21.15.61:0/3372194218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efef80b1080 0x7efef81533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 >> 172.21.15.61:0/3372194218 conn(0x7efef8067020 msgr2=0x7efef8141760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:59.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 shutdown_connections 2022-07-12T08:41:59.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.208+0000 7efefcb24700 1 -- 172.21.15.61:0/3372194218 wait complete. 2022-07-12T08:41:59.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:41:59.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 Processor -- start 2022-07-12T08:41:59.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 -- start start 2022-07-12T08:41:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8040b3100 0x7fe8040b4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe804141d80 0x7fe8041421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8040b3940 con 0x7fe8040b34e0 2022-07-12T08:41:59.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8040afda0 con 0x7fe8040b3100 2022-07-12T08:41:59.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.360+0000 7fe80a304700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040b0040 con 0x7fe804141d80 2022-07-12T08:41:59.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8040b3100 0x7fe8040b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8040b3100 0x7fe8040b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54264/0 (socket says 172.21.15.61:54264) 2022-07-12T08:41:59.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 -- 172.21.15.61:0/261611962 learned_addr learned my addr 172.21.15.61:0/261611962 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:59.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe8088a1700 1 --2- 172.21.15.61:0/261611962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe804141d80 0x7fe8041421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 -- 172.21.15.61:0/261611962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe804141d80 msgr2=0x7fe8041421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:41:59.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 --2- 172.21.15.61:0/261611962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe804141d80 0x7fe8041421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 -- 172.21.15.61:0/261611962 >> v1:172.21.15.111:6789/0 conn(0x7fe8040b34e0 legacy=0x7fe804146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 -- 172.21.15.61:0/261611962 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8040b27c0 con 0x7fe8040b3100 2022-07-12T08:41:59.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe803fff700 1 --2- 172.21.15.61:0/261611962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b3100 0x7fe8040b4560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fe7f8002f70 tx=0x7fe7f8006eb0).ready entity=mon.0 client_cookie=acd085f1a8d40ac3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:59.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe802ffd700 1 -- 172.21.15.61:0/261611962 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f800d5c0 con 0x7fe8040b3100 2022-07-12T08:41:59.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe802ffd700 1 -- 172.21.15.61:0/261611962 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f8000e90 con 0x7fe8040b3100 2022-07-12T08:41:59.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe802ffd700 1 -- 172.21.15.61:0/261611962 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f800d950 con 0x7fe8040b3100 2022-07-12T08:41:59.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 -- 172.21.15.61:0/261611962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b3100 msgr2=0x7fe8040b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 --2- 172.21.15.61:0/261611962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b3100 0x7fe8040b4560 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fe7f8002f70 tx=0x7fe7f8006eb0).stop 2022-07-12T08:41:59.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 -- 172.21.15.61:0/261611962 shutdown_connections 2022-07-12T08:41:59.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 --2- 172.21.15.61:0/261611962 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe804141d80 0x7fe8041421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 --2- 172.21.15.61:0/261611962 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b3100 0x7fe8040b4560 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 -- 172.21.15.61:0/261611962 >> 172.21.15.61:0/261611962 conn(0x7fe804066e90 msgr2=0x7fe804067270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:41:59.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 -- 172.21.15.61:0/261611962 shutdown_connections 2022-07-12T08:41:59.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 -- 172.21.15.61:0/261611962 wait complete. 2022-07-12T08:41:59.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 Processor -- start 2022-07-12T08:41:59.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.364+0000 7fe80a304700 1 -- start start 2022-07-12T08:41:59.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8040b3100 0x7fe804147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 0x7fe8040b8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804141d80 0x7fe8041523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040aee10 con 0x7fe8040b34e0 2022-07-12T08:41:59.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040ae8d0 con 0x7fe804141d80 2022-07-12T08:41:59.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8040aeb70 con 0x7fe8040b3100 2022-07-12T08:41:59.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 0x7fe8040b8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 0x7fe8040b8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54272/0 (socket says 172.21.15.61:54272) 2022-07-12T08:41:59.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8088a1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804141d80 0x7fe8041523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 -- 172.21.15.61:0/4136068939 learned_addr learned my addr 172.21.15.61:0/4136068939 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:41:59.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe803fff700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8040b3100 0x7fe804147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 -- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8040b3100 msgr2=0x7fe804147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8040b3100 0x7fe804147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 -- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804141d80 msgr2=0x7fe8041523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:41:59.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804141d80 0x7fe8041523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:41:59.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 -- 172.21.15.61:0/4136068939 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f800b1e0 con 0x7fe8040b34e0 2022-07-12T08:41:59.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8037fe700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 0x7fe8040b8670 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fe8040b3d30 tx=0x7fe7f4007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:59.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f4007030 con 0x7fe8040b34e0 2022-07-12T08:41:59.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4007190 con 0x7fe8040b34e0 2022-07-12T08:41:59.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8040b8b60 con 0x7fe8040b34e0 2022-07-12T08:41:59.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7f4006510 con 0x7fe8040b34e0 2022-07-12T08:41:59.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe804151e90 con 0x7fe8040b34e0 2022-07-12T08:41:59.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.368+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8040af5b0 con 0x7fe8040b34e0 2022-07-12T08:41:59.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.372+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe7f4006670 con 0x7fe8040b34e0 2022-07-12T08:41:59.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.372+0000 7fe8017fa700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7e4053a60 0x7fe7e4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:41:59.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.372+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe7f406b6e0 con 0x7fe8040b34e0 2022-07-12T08:41:59.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.372+0000 7fe803fff700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7e4053a60 0x7fe7e4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:41:59.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.372+0000 7fe803fff700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7e4053a60 0x7fe7e4055f20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe7f801a000 tx=0x7fe7f8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:41:59.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.376+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8040af5b0 con 0x7fe8040b34e0 2022-07-12T08:41:59.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:41:59.540+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe8040af5b0 con 0x7fe8040b34e0 2022-07-12T08:42:00.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.312+0000 7fe8017fa700 1 -- 172.21.15.61:0/4136068939 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe8040af5b0 con 0x7fe8040b34e0 2022-07-12T08:42:00.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7e4053a60 msgr2=0x7fe7e4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7e4053a60 0x7fe7e4055f20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe7f801a000 tx=0x7fe7f8014000).stop 2022-07-12T08:42:00.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 msgr2=0x7fe8040b8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 0x7fe8040b8670 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fe8040b3d30 tx=0x7fe7f4007be0).stop 2022-07-12T08:42:00.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 shutdown_connections 2022-07-12T08:42:00.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7e4053a60 0x7fe7e4055f20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe804141d80 0x7fe8041523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8040b34e0 0x7fe8040b8670 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 --2- 172.21.15.61:0/4136068939 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8040b3100 0x7fe804147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 >> 172.21.15.61:0/4136068939 conn(0x7fe804066e90 msgr2=0x7fe8040b5250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:00.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 shutdown_connections 2022-07-12T08:42:00.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.316+0000 7fe80a304700 1 -- 172.21.15.61:0/4136068939 wait complete. 2022-07-12T08:42:00.357 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:42:00.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 Processor -- start 2022-07-12T08:42:00.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 -- start start 2022-07-12T08:42:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f34e00ab290 0x7f34e00ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f34e00afa60 0x7f34e00afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e00abc70 con 0x7f34e005f140 2022-07-12T08:42:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e005f580 con 0x7f34e00afa60 2022-07-12T08:42:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6d73700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e005c720 con 0x7f34e00ab290 2022-07-12T08:42:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f34e00afa60 0x7f34e00afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f34e00afa60 0x7f34e00afed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54282/0 (socket says 172.21.15.61:54282) 2022-07-12T08:42:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e5570700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f34e005f140 0x7f34e005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42496/0 (socket says 172.21.15.61:42496) 2022-07-12T08:42:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 -- 172.21.15.61:0/1416982954 learned_addr learned my addr 172.21.15.61:0/1416982954 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e5d71700 1 --2- 172.21.15.61:0/1416982954 >> v2:172.21.15.111:3300/0 conn(0x7f34e00ab290 0x7f34e00ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 -- 172.21.15.61:0/1416982954 >> v2:172.21.15.111:3300/0 conn(0x7f34e00ab290 msgr2=0x7f34e00ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 --2- 172.21.15.61:0/1416982954 >> v2:172.21.15.111:3300/0 conn(0x7f34e00ab290 0x7f34e00ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 -- 172.21.15.61:0/1416982954 >> v1:172.21.15.61:6789/0 conn(0x7f34e005f140 legacy=0x7f34e005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 -- 172.21.15.61:0/1416982954 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e005c9c0 con 0x7f34e00afa60 2022-07-12T08:42:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e5d71700 1 --2- 172.21.15.61:0/1416982954 >> v2:172.21.15.111:3300/0 conn(0x7f34e00ab290 0x7f34e00ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.468+0000 7f34e6572700 1 --2- 172.21.15.61:0/1416982954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00afa60 0x7f34e00afed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f34d80032f0 tx=0x7f34d8009290).ready entity=mon.0 client_cookie=e16cddc15ef59627 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e4d6f700 1 -- 172.21.15.61:0/1416982954 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d800b870 con 0x7f34e00afa60 2022-07-12T08:42:00.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e4d6f700 1 -- 172.21.15.61:0/1416982954 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d800b9d0 con 0x7f34e00afa60 2022-07-12T08:42:00.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e4d6f700 1 -- 172.21.15.61:0/1416982954 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d800bb40 con 0x7f34e00afa60 2022-07-12T08:42:00.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 -- 172.21.15.61:0/1416982954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00afa60 msgr2=0x7f34e00afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1416982954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00afa60 0x7f34e00afed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f34d80032f0 tx=0x7f34d8009290).stop 2022-07-12T08:42:00.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 -- 172.21.15.61:0/1416982954 shutdown_connections 2022-07-12T08:42:00.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1416982954 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00afa60 0x7f34e00afed0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1416982954 >> v2:172.21.15.111:3300/0 conn(0x7f34e00ab290 0x7f34e00ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 -- 172.21.15.61:0/1416982954 >> 172.21.15.61:0/1416982954 conn(0x7f34e00b1950 msgr2=0x7f34e00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:00.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 -- 172.21.15.61:0/1416982954 shutdown_connections 2022-07-12T08:42:00.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 -- 172.21.15.61:0/1416982954 wait complete. 2022-07-12T08:42:00.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 Processor -- start 2022-07-12T08:42:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.472+0000 7f34e6d73700 1 -- start start 2022-07-12T08:42:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34e005f140 0x7f34e00adb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 0x7f34e0063f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e00afa60 0x7f34e0064420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e0155da0 con 0x7f34e00ab290 2022-07-12T08:42:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e01497c0 con 0x7f34e00afa60 2022-07-12T08:42:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e01499f0 con 0x7f34e005f140 2022-07-12T08:42:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 0x7f34e0063f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 0x7f34e0063f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54286/0 (socket says 172.21.15.61:54286) 2022-07-12T08:42:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 -- 172.21.15.61:0/1983341676 learned_addr learned my addr 172.21.15.61:0/1983341676 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5d71700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34e005f140 0x7f34e00adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6572700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e00afa60 0x7f34e0064420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 -- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34e005f140 msgr2=0x7f34e00adb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34e005f140 0x7f34e00adb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 -- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e00afa60 msgr2=0x7f34e0064420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e00afa60 0x7f34e0064420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 -- 172.21.15.61:0/1983341676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8009530 con 0x7f34e00ab290 2022-07-12T08:42:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e5570700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 0x7f34e0063f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f34e00615d0 tx=0x7f34dc007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34dc0070d0 con 0x7f34e00ab290 2022-07-12T08:42:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc007230 con 0x7f34e00ab290 2022-07-12T08:42:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e0155f90 con 0x7f34e00ab290 2022-07-12T08:42:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34dc005740 con 0x7f34e00ab290 2022-07-12T08:42:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.476+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e014d190 con 0x7f34e00ab290 2022-07-12T08:42:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.480+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e0156120 con 0x7f34e00ab290 2022-07-12T08:42:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.480+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f34dc010070 con 0x7f34e00ab290 2022-07-12T08:42:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.480+0000 7f34ceffd700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34c0053ad0 0x7f34c0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.480+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f34dc06b2b0 con 0x7f34e00ab290 2022-07-12T08:42:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.480+0000 7f34e5d71700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34c0053ad0 0x7f34c0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.480+0000 7f34e5d71700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34c0053ad0 0x7f34c0055f90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f34d000c000 tx=0x7f34d0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.484+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34dc042210 con 0x7f34e00ab290 2022-07-12T08:42:00.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34e0057460 con 0x7f34e00ab290 2022-07-12T08:42:00.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34ceffd700 1 -- 172.21.15.61:0/1983341676 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f34dc018070 con 0x7f34e00ab290 2022-07-12T08:42:00.668 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:00.669 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:42:00.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34c0053ad0 msgr2=0x7f34c0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34c0053ad0 0x7f34c0055f90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f34d000c000 tx=0x7f34d0008040).stop 2022-07-12T08:42:00.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 msgr2=0x7f34e0063f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 0x7f34e0063f30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f34e00615d0 tx=0x7f34dc007c00).stop 2022-07-12T08:42:00.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 shutdown_connections 2022-07-12T08:42:00.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34c0053ad0 0x7f34c0055f90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34e00afa60 0x7f34e0064420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34e00ab290 0x7f34e0063f30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 --2- 172.21.15.61:0/1983341676 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34e005f140 0x7f34e00adb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 >> 172.21.15.61:0/1983341676 conn(0x7f34e00b1950 msgr2=0x7f34e00b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:00.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 shutdown_connections 2022-07-12T08:42:00.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.652+0000 7f34e6d73700 1 -- 172.21.15.61:0/1983341676 wait complete. 2022-07-12T08:42:00.679 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:42:00.682 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:42:00.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.788+0000 7f16ce60b700 1 Processor -- start 2022-07-12T08:42:00.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.788+0000 7f16ce60b700 1 -- start start 2022-07-12T08:42:00.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16ce60b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16c80af780 0x7f16c813c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16ce60b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 0x7f16c813c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16ce60b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c80afb60 con 0x7f16c8145f70 2022-07-12T08:42:00.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16ce60b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c81435a0 con 0x7f16c80af780 2022-07-12T08:42:00.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16ce60b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8143840 con 0x7f16c80613a0 2022-07-12T08:42:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cde0a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f16c8145f70 0x7f16c81463b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42514/0 (socket says 172.21.15.61:42514) 2022-07-12T08:42:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cde0a700 1 -- 172.21.15.61:0/557063067 learned_addr learned my addr 172.21.15.61:0/557063067 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:42:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cce08700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 0x7f16c813c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 --2- 172.21.15.61:0/557063067 >> v2:172.21.15.111:3300/0 conn(0x7f16c80af780 0x7f16c813c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885468597 0 0) 0x7f16c80afb60 con 0x7f16c8145f70 2022-07-12T08:42:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac003260 con 0x7f16c8145f70 2022-07-12T08:42:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328318230 0 0) 0x7f16ac003260 con 0x7f16c8145f70 2022-07-12T08:42:00.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 msgr2=0x7f16c813c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:00.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 0x7f16c813c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 >> v2:172.21.15.111:3300/0 conn(0x7f16c80af780 msgr2=0x7f16c813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:00.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 --2- 172.21.15.61:0/557063067 >> v2:172.21.15.111:3300/0 conn(0x7f16c80af780 0x7f16c813c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c813cd50 con 0x7f16c8145f70 2022-07-12T08:42:00.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f16c00019e0 con 0x7f16c8145f70 2022-07-12T08:42:00.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 >> v1:172.21.15.61:6789/0 conn(0x7f16c8145f70 legacy=0x7f16c81463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16ac003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 0x7f16ac004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac0049f0 0x7f16ac004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c81435a0 con 0x7f16ac003750 2022-07-12T08:42:00.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8143840 con 0x7f16ac0049f0 2022-07-12T08:42:00.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac00b750 con 0x7f16c80613a0 2022-07-12T08:42:00.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16ac003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cde0a700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac0049f0 0x7f16ac004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cce08700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 0x7f16ac004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 -- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 msgr2=0x7f16ac004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:00.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c80613a0 0x7f16ac004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 -- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac0049f0 msgr2=0x7f16ac004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:00.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16ac0049f0 0x7f16ac004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.792+0000 7f16cd609700 1 -- 172.21.15.61:0/557063067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac003260 con 0x7f16ac003750 2022-07-12T08:42:00.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16cd609700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16ac003b70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f16bc00c000 tx=0x7f16bc006df0).ready entity=mon.0 client_cookie=2a1e383ba3836233 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:00.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16bc00e830 con 0x7f16ac003750 2022-07-12T08:42:00.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc00e990 con 0x7f16ac003750 2022-07-12T08:42:00.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16b3fff700 1 -- 172.21.15.61:0/557063067 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16bc00ecf0 con 0x7f16ac003750 2022-07-12T08:42:00.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 -- 172.21.15.61:0/557063067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 msgr2=0x7f16ac003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 --2- 172.21.15.61:0/557063067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16ac003b70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f16bc00c000 tx=0x7f16bc006df0).stop 2022-07-12T08:42:00.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16cde0a700 1 -- 172.21.15.61:0/557063067 reap_dead start 2022-07-12T08:42:00.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 -- 172.21.15.61:0/557063067 shutdown_connections 2022-07-12T08:42:00.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 -- 172.21.15.61:0/557063067 >> 172.21.15.61:0/557063067 conn(0x7f16c80b5780 msgr2=0x7f16c80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:00.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 -- 172.21.15.61:0/557063067 shutdown_connections 2022-07-12T08:42:00.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 -- 172.21.15.61:0/557063067 wait complete. 2022-07-12T08:42:00.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 Processor -- start 2022-07-12T08:42:00.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.796+0000 7f16ce60b700 1 -- start start 2022-07-12T08:42:00.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16c8145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16c80651d0 0x7f16c8151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c8155580 0x7f16c8155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8149180 con 0x7f16ac003750 2022-07-12T08:42:00.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8152030 con 0x7f16c80651d0 2022-07-12T08:42:00.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c8155150 con 0x7f16c8155580 2022-07-12T08:42:00.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16c8145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cce08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16c80651d0 0x7f16c8151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cde0a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c8155580 0x7f16c8155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16c8145950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54308/0 (socket says 172.21.15.61:54308) 2022-07-12T08:42:00.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cce08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16c80651d0 0x7f16c8151b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32878/0 (socket says 172.21.15.61:32878) 2022-07-12T08:42:00.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cce08700 1 -- 172.21.15.61:0/3152254882 learned_addr learned my addr 172.21.15.61:0/3152254882 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:00.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cde0a700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c8155580 0x7f16c8155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36502/0 (socket says 172.21.15.61:36502) 2022-07-12T08:42:00.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 -- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c8155580 msgr2=0x7f16c8155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c8155580 0x7f16c8155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 -- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16c80651d0 msgr2=0x7f16c8151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16c80651d0 0x7f16c8151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 -- 172.21.15.61:0/3152254882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c8064270 con 0x7f16ac003750 2022-07-12T08:42:00.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16cd609700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16c8145950 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f16bc00c000 tx=0x7f16bc019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:00.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16bc01b820 con 0x7f16ac003750 2022-07-12T08:42:00.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc000e90 con 0x7f16ac003750 2022-07-12T08:42:00.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c8155ef0 con 0x7f16ac003750 2022-07-12T08:42:00.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16bc01bb70 con 0x7f16ac003750 2022-07-12T08:42:00.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c8148550 con 0x7f16ac003750 2022-07-12T08:42:00.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.800+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c81552c0 con 0x7f16ac003750 2022-07-12T08:42:00.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.804+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f16bc01bcd0 con 0x7f16ac003750 2022-07-12T08:42:00.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.804+0000 7f16baffd700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16a4053ad0 0x7f16a4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:00.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.804+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16bc06d160 con 0x7f16ac003750 2022-07-12T08:42:00.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.804+0000 7f16cce08700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16a4053ad0 0x7f16a4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:00.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.804+0000 7f16cce08700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16a4053ad0 0x7f16a4055f90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f16c400b000 tx=0x7f16c4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:00.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.808+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16c81552c0 con 0x7f16ac003750 2022-07-12T08:42:00.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16c81552c0 con 0x7f16ac003750 2022-07-12T08:42:00.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16baffd700 1 -- 172.21.15.61:0/3152254882 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f16c81552c0 con 0x7f16ac003750 2022-07-12T08:42:00.948 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:00.952 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:41:59.637221+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:56.415996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:57.640916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:42:00.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16a4053ad0 msgr2=0x7f16a4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16a4053ad0 0x7f16a4055f90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f16c400b000 tx=0x7f16c4006df0).stop 2022-07-12T08:42:00.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 msgr2=0x7f16c8145950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:00.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16c8145950 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f16bc00c000 tx=0x7f16bc019000).stop 2022-07-12T08:42:00.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 shutdown_connections 2022-07-12T08:42:00.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16c8155580 0x7f16c8155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16a4053ad0 0x7f16a4055f90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16c80651d0 0x7f16c8151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 --2- 172.21.15.61:0/3152254882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16ac003750 0x7f16c8145950 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 >> 172.21.15.61:0/3152254882 conn(0x7f16c80b5780 msgr2=0x7f16c80b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:00.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 shutdown_connections 2022-07-12T08:42:00.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:00.936+0000 7f16ce60b700 1 -- 172.21.15.61:0/3152254882 wait complete. 2022-07-12T08:42:00.973 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:42:01.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.080+0000 7f00bbfff700 1 Processor -- start 2022-07-12T08:42:01.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.080+0000 7f00bbfff700 1 -- start start 2022-07-12T08:42:01.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.080+0000 7f00bbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f00bc05cd20 0x7f00bc146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.080+0000 7f00bbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00bc05d100 0x7f00bc1431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.080+0000 7f00bbfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc146200 con 0x7f00bc0af490 2022-07-12T08:42:01.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc1464a0 con 0x7f00bc05d100 2022-07-12T08:42:01.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc146740 con 0x7f00bc05cd20 2022-07-12T08:42:01.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00bc05d100 0x7f00bc1431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00bc05d100 0x7f00bc1431c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54316/0 (socket says 172.21.15.61:54316) 2022-07-12T08:42:01.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 -- 172.21.15.61:0/397636406 learned_addr learned my addr 172.21.15.61:0/397636406 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:01.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00ba7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f00bc0af490 0x7f00bc05f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42532/0 (socket says 172.21.15.61:42532) 2022-07-12T08:42:01.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00baffd700 1 --2- 172.21.15.61:0/397636406 >> v2:172.21.15.111:3300/0 conn(0x7f00bc05cd20 0x7f00bc146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 -- 172.21.15.61:0/397636406 >> v2:172.21.15.111:3300/0 conn(0x7f00bc05cd20 msgr2=0x7f00bc146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:01.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 --2- 172.21.15.61:0/397636406 >> v2:172.21.15.111:3300/0 conn(0x7f00bc05cd20 0x7f00bc146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 -- 172.21.15.61:0/397636406 >> v1:172.21.15.61:6789/0 conn(0x7f00bc0af490 legacy=0x7f00bc05f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00b9ffb700 1 -- 172.21.15.61:0/397636406 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971067129 0 0) 0x7f00bc146200 con 0x7f00bc0af490 2022-07-12T08:42:01.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 -- 172.21.15.61:0/397636406 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc060870 con 0x7f00bc05d100 2022-07-12T08:42:01.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00baffd700 1 --2- 172.21.15.61:0/397636406 >> v2:172.21.15.111:3300/0 conn(0x7f00bc05cd20 0x7f00bc146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:01.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bb7fe700 1 --2- 172.21.15.61:0/397636406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc05d100 0x7f00bc1431c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f00b4002f70 tx=0x7f00b4006eb0).ready entity=mon.0 client_cookie=82ebf9ff1926128c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00b9ffb700 1 -- 172.21.15.61:0/397636406 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00b400d610 con 0x7f00bc05d100 2022-07-12T08:42:01.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00b9ffb700 1 -- 172.21.15.61:0/397636406 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b4000e90 con 0x7f00bc05d100 2022-07-12T08:42:01.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00b9ffb700 1 -- 172.21.15.61:0/397636406 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00b400d960 con 0x7f00bc05d100 2022-07-12T08:42:01.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- 172.21.15.61:0/397636406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc05d100 msgr2=0x7f00bc1431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 --2- 172.21.15.61:0/397636406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc05d100 0x7f00bc1431c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f00b4002f70 tx=0x7f00b4006eb0).stop 2022-07-12T08:42:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- 172.21.15.61:0/397636406 shutdown_connections 2022-07-12T08:42:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 --2- 172.21.15.61:0/397636406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc05d100 0x7f00bc1431c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 --2- 172.21.15.61:0/397636406 >> v2:172.21.15.111:3300/0 conn(0x7f00bc05cd20 0x7f00bc146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- 172.21.15.61:0/397636406 >> 172.21.15.61:0/397636406 conn(0x7f00bc0b9e20 msgr2=0x7f00bc0b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:01.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- 172.21.15.61:0/397636406 shutdown_connections 2022-07-12T08:42:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.084+0000 7f00bbfff700 1 -- 172.21.15.61:0/397636406 wait complete. 2022-07-12T08:42:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 Processor -- start 2022-07-12T08:42:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 -- start start 2022-07-12T08:42:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00bc05cd20 0x7f00bc0b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00bc05d100 0x7f00bc155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 0x7f00bc09c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc09d440 con 0x7f00bc0af490 2022-07-12T08:42:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc155500 con 0x7f00bc05d100 2022-07-12T08:42:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bbfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc155760 con 0x7f00bc05cd20 2022-07-12T08:42:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 0x7f00bc09c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00baffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00bc05cd20 0x7f00bc0b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 0x7f00bc09c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54322/0 (socket says 172.21.15.61:54322) 2022-07-12T08:42:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00bb7fe700 1 -- 172.21.15.61:0/201070048 learned_addr learned my addr 172.21.15.61:0/201070048 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:01.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.088+0000 7f00ba7fc700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00bc05d100 0x7f00bc155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bb7fe700 1 -- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00bc05cd20 msgr2=0x7f00bc0b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bb7fe700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00bc05cd20 0x7f00bc0b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bb7fe700 1 -- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00bc05d100 msgr2=0x7f00bc155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bb7fe700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00bc05d100 0x7f00bc155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bb7fe700 1 -- 172.21.15.61:0/201070048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b400b1e0 con 0x7f00bc0af490 2022-07-12T08:42:01.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bb7fe700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 0x7f00bc09c4a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f00b4002f70 tx=0x7f00b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00b4006be0 con 0x7f00bc0af490 2022-07-12T08:42:01.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b400d290 con 0x7f00bc0af490 2022-07-12T08:42:01.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc155a00 con 0x7f00bc0af490 2022-07-12T08:42:01.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00b40067e0 con 0x7f00bc0af490 2022-07-12T08:42:01.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc09cb30 con 0x7f00bc0af490 2022-07-12T08:42:01.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00bc155bc0 con 0x7f00bc0af490 2022-07-12T08:42:01.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f00b4025030 con 0x7f00bc0af490 2022-07-12T08:42:01.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00b3fff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f00a0053a60 0x7f00a0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f00b406cf00 con 0x7f00bc0af490 2022-07-12T08:42:01.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.092+0000 7f00baffd700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f00a0053a60 0x7f00a0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.096+0000 7f00baffd700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f00a0053a60 0x7f00a0055f20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f00ac00c000 tx=0x7f00ac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.100+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00b4043d50 con 0x7f00bc0af490 2022-07-12T08:42:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.268+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f00bc057460 con 0x7f00bc0af490 2022-07-12T08:42:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00b3fff700 1 -- 172.21.15.61:0/201070048 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v11) v1 ==== 184+0+0 (secure 0 0 0) 0x7f00b4023020 con 0x7f00bc0af490 2022-07-12T08:42:01.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f00a0053a60 msgr2=0x7f00a0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f00a0053a60 0x7f00a0055f20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f00ac00c000 tx=0x7f00ac008040).stop 2022-07-12T08:42:01.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 msgr2=0x7f00bc09c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 0x7f00bc09c4a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f00b4002f70 tx=0x7f00b4015000).stop 2022-07-12T08:42:01.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 shutdown_connections 2022-07-12T08:42:01.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00bc0af490 0x7f00bc09c4a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f00a0053a60 0x7f00a0055f20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00bc05d100 0x7f00bc155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 --2- 172.21.15.61:0/201070048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00bc05cd20 0x7f00bc0b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 >> 172.21.15.61:0/201070048 conn(0x7f00bc0b9e20 msgr2=0x7f00bc0b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:01.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 shutdown_connections 2022-07-12T08:42:01.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.276+0000 7f00bbfff700 1 -- 172.21.15.61:0/201070048 wait complete. 2022-07-12T08:42:01.298 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:42:01.301 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:42:01.350 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:01.363 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:01.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:01.369 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:01.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:01.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 Processor -- start 2022-07-12T08:42:01.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 -- start start 2022-07-12T08:42:01.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbc5405c590 0x7fbc540b9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b3d50 0x7fbc54142170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc540b5f60 con 0x7fbc540b9a40 2022-07-12T08:42:01.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc54142d30 con 0x7fbc5405c590 2022-07-12T08:42:01.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5ad2f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc540b4200 con 0x7fbc540b3d50 2022-07-12T08:42:01.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5952c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbc5405c590 0x7fbc540b9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5a52e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b3d50 0x7fbc54142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5952c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbc5405c590 0x7fbc540b9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32902/0 (socket says 172.21.15.61:32902) 2022-07-12T08:42:01.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc5952c700 1 -- 172.21.15.61:0/2698482275 learned_addr learned my addr 172.21.15.61:0/2698482275 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:01.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.416+0000 7fbc59d2d700 1 --1- 172.21.15.61:0/2698482275 >> v1:172.21.15.111:6789/0 conn(0x7fbc540b9a40 0x7fbc541431d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56146/0 (socket says 172.21.15.61:56146) 2022-07-12T08:42:01.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5a52e700 1 -- 172.21.15.61:0/2698482275 >> v2:172.21.15.111:3300/0 conn(0x7fbc5405c590 msgr2=0x7fbc540b9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5a52e700 1 --2- 172.21.15.61:0/2698482275 >> v2:172.21.15.111:3300/0 conn(0x7fbc5405c590 0x7fbc540b9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5a52e700 1 -- 172.21.15.61:0/2698482275 >> v1:172.21.15.111:6789/0 conn(0x7fbc540b9a40 legacy=0x7fbc541431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5a52e700 1 -- 172.21.15.61:0/2698482275 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5405c270 con 0x7fbc540b3d50 2022-07-12T08:42:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5a52e700 1 --2- 172.21.15.61:0/2698482275 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b3d50 0x7fbc54142170 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbc4c0032f0 tx=0x7fbc4c0091d0).ready entity=mon.2 client_cookie=de26932d747a4803 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc58d2b700 1 -- 172.21.15.61:0/2698482275 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4c00b6c0 con 0x7fbc540b3d50 2022-07-12T08:42:01.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc58d2b700 1 -- 172.21.15.61:0/2698482275 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c00b820 con 0x7fbc540b3d50 2022-07-12T08:42:01.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc58d2b700 1 -- 172.21.15.61:0/2698482275 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4c00b990 con 0x7fbc540b3d50 2022-07-12T08:42:01.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/2698482275 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b3d50 msgr2=0x7fbc54142170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/2698482275 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b3d50 0x7fbc54142170 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fbc4c0032f0 tx=0x7fbc4c0091d0).stop 2022-07-12T08:42:01.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/2698482275 shutdown_connections 2022-07-12T08:42:01.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/2698482275 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b3d50 0x7fbc54142170 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/2698482275 >> v2:172.21.15.111:3300/0 conn(0x7fbc5405c590 0x7fbc540b9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/2698482275 >> 172.21.15.61:0/2698482275 conn(0x7fbc540677d0 msgr2=0x7fbc540b9330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:01.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/2698482275 shutdown_connections 2022-07-12T08:42:01.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/2698482275 wait complete. 2022-07-12T08:42:01.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 Processor -- start 2022-07-12T08:42:01.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- start start 2022-07-12T08:42:01.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc5405c590 0x7fbc54145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 0x7fbc54155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b9a40 0x7fbc5413cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc540b1a90 con 0x7fbc540b3d50 2022-07-12T08:42:01.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5413c880 con 0x7fbc5405c590 2022-07-12T08:42:01.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5ad2f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc54155c60 con 0x7fbc540b9a40 2022-07-12T08:42:01.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5952c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 0x7fbc54155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5952c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 0x7fbc54155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54334/0 (socket says 172.21.15.61:54334) 2022-07-12T08:42:01.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.420+0000 7fbc5952c700 1 -- 172.21.15.61:0/607540381 learned_addr learned my addr 172.21.15.61:0/607540381 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:01.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc59d2d700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc5405c590 0x7fbc54145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5a52e700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b9a40 0x7fbc5413cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5952c700 1 -- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b9a40 msgr2=0x7fbc5413cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5952c700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b9a40 0x7fbc5413cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5952c700 1 -- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc5405c590 msgr2=0x7fbc54145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5952c700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc5405c590 0x7fbc54145660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5952c700 1 -- 172.21.15.61:0/607540381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c009470 con 0x7fbc540b3d50 2022-07-12T08:42:01.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5952c700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 0x7fbc54155770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fbc4800c000 tx=0x7fbc48008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4800fe30 con 0x7fbc540b3d50 2022-07-12T08:42:01.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc48013610 con 0x7fbc540b3d50 2022-07-12T08:42:01.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc540b2250 con 0x7fbc540b3d50 2022-07-12T08:42:01.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc48013970 con 0x7fbc540b3d50 2022-07-12T08:42:01.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc540b2760 con 0x7fbc540b3d50 2022-07-12T08:42:01.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5413d550 con 0x7fbc540b3d50 2022-07-12T08:42:01.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.424+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbc48023070 con 0x7fbc540b3d50 2022-07-12T08:42:01.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.428+0000 7fbc46ffd700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc340539b0 0x7fbc34055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.428+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc4806bfb0 con 0x7fbc540b3d50 2022-07-12T08:42:01.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.428+0000 7fbc59d2d700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc340539b0 0x7fbc34055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.428+0000 7fbc59d2d700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc340539b0 0x7fbc34055e70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fbc5000d000 tx=0x7fbc50007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.436+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc48042f10 con 0x7fbc540b3d50 2022-07-12T08:42:01.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.600+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fbc54057460 con 0x7fbc540b3d50 2022-07-12T08:42:01.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.608+0000 7fbc46ffd700 1 -- 172.21.15.61:0/607540381 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v12) v1 ==== 184+0+0 (secure 0 0 0) 0x7fbc48046530 con 0x7fbc540b3d50 2022-07-12T08:42:01.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc340539b0 msgr2=0x7fbc34055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc340539b0 0x7fbc34055e70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fbc5000d000 tx=0x7fbc50007b80).stop 2022-07-12T08:42:01.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 msgr2=0x7fbc54155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 0x7fbc54155770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fbc4800c000 tx=0x7fbc48008040).stop 2022-07-12T08:42:01.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 shutdown_connections 2022-07-12T08:42:01.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc540b9a40 0x7fbc5413cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc540b3d50 0x7fbc54155770 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc340539b0 0x7fbc34055e70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 --2- 172.21.15.61:0/607540381 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc5405c590 0x7fbc54145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 >> 172.21.15.61:0/607540381 conn(0x7fbc540677d0 msgr2=0x7fbc540ae6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:01.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 shutdown_connections 2022-07-12T08:42:01.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.612+0000 7fbc5ad2f700 1 -- 172.21.15.61:0/607540381 wait complete. 2022-07-12T08:42:01.630 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:42:01.638 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:42:01.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b7fff700 1 Processor -- start 2022-07-12T08:42:01.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b7fff700 1 -- start start 2022-07-12T08:42:01.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b9d80 0x7fa5b814add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b80ba160 con 0x7fa5b805d0f0 2022-07-12T08:42:01.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b814a690 con 0x7fa5b80b5a10 2022-07-12T08:42:01.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b814a930 con 0x7fa5b80b9d80 2022-07-12T08:42:01.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa5b805d0f0 0x7fa5b81461b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42562/0 (socket says 172.21.15.61:42562) 2022-07-12T08:42:01.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b77fe700 1 -- 172.21.15.61:0/1565982680 learned_addr learned my addr 172.21.15.61:0/1565982680 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:42:01.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b6ffd700 1 --2- 172.21.15.61:0/1565982680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b9d80 0x7fa5b814add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587457503 0 0) 0x7fa5b80ba160 con 0x7fa5b805d0f0 2022-07-12T08:42:01.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59c003260 con 0x7fa5b805d0f0 2022-07-12T08:42:01.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379553787 0 0) 0x7fa5b814a690 con 0x7fa5b80b5a10 2022-07-12T08:42:01.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b80ba160 con 0x7fa5b80b5a10 2022-07-12T08:42:01.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.756+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1189089228 0 0) 0x7fa59c003260 con 0x7fa5b805d0f0 2022-07-12T08:42:01.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b6ffd700 1 -- 172.21.15.61:0/1565982680 >> v1:172.21.15.111:6789/0 conn(0x7fa5b80b5a10 legacy=0x7fa5b805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b6ffd700 1 -- 172.21.15.61:0/1565982680 >> v1:172.21.15.61:6789/0 conn(0x7fa5b805d0f0 legacy=0x7fa5b81461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b6ffd700 1 -- 172.21.15.61:0/1565982680 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b814b2c0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b6ffd700 1 --2- 172.21.15.61:0/1565982680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b9d80 0x7fa5b814add0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa5b809f550 tx=0x7fa5a8006df0).ready entity=mon.2 client_cookie=5ebd08f29e3a5705 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5b00019e0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a8000e90 con 0x7fa5b80b9d80 2022-07-12T08:42:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b5ffb700 1 -- 172.21.15.61:0/1565982680 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5a8012460 con 0x7fa5b80b9d80 2022-07-12T08:42:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 -- 172.21.15.61:0/1565982680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b9d80 msgr2=0x7fa5b814add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/1565982680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b9d80 0x7fa5b814add0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fa5b809f550 tx=0x7fa5a8006df0).stop 2022-07-12T08:42:01.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 -- 172.21.15.61:0/1565982680 shutdown_connections 2022-07-12T08:42:01.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/1565982680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b9d80 0x7fa5b814add0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 -- 172.21.15.61:0/1565982680 >> 172.21.15.61:0/1565982680 conn(0x7fa5b80677d0 msgr2=0x7fa5b80b9330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:01.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 -- 172.21.15.61:0/1565982680 shutdown_connections 2022-07-12T08:42:01.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.760+0000 7fa5b7fff700 1 -- 172.21.15.61:0/1565982680 wait complete. 2022-07-12T08:42:01.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 Processor -- start 2022-07-12T08:42:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 -- start start 2022-07-12T08:42:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5b805d0f0 0x7fa5b80b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b5a10 0x7fa5b809e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 0x7fa5b8156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b809a6d0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b809a970 con 0x7fa5b805d0f0 2022-07-12T08:42:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b809ac10 con 0x7fa5b80b5a10 2022-07-12T08:42:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 0x7fa5b8156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 0x7fa5b8156700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54350/0 (socket says 172.21.15.61:54350) 2022-07-12T08:42:01.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b67fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b5a10 0x7fa5b809e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5b805d0f0 0x7fa5b80b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 -- 172.21.15.61:0/298847667 learned_addr learned my addr 172.21.15.61:0/298847667 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:01.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 -- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b5a10 msgr2=0x7fa5b809e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b5a10 0x7fa5b809e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 -- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5b805d0f0 msgr2=0x7fa5b80b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5b805d0f0 0x7fa5b80b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 -- 172.21.15.61:0/298847667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a800b180 con 0x7fa5b80b9d80 2022-07-12T08:42:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b77fe700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 0x7fa5b8156700 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fa5b0001360 tx=0x7fa5b0001b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5b0004ab0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b0004c10 con 0x7fa5b80b9d80 2022-07-12T08:42:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b8099c00 con 0x7fa5b80b9d80 2022-07-12T08:42:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5b0007170 con 0x7fa5b80b9d80 2022-07-12T08:42:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.764+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b809e6e0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.768+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b8099810 con 0x7fa5b80b9d80 2022-07-12T08:42:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.768+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa5b001b070 con 0x7fa5b80b9d80 2022-07-12T08:42:01.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.768+0000 7fa5a7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5940539d0 0x7fa594055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:01.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.768+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5b006b720 con 0x7fa5b80b9d80 2022-07-12T08:42:01.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.768+0000 7fa5b6ffd700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5940539d0 0x7fa594055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.768+0000 7fa5b6ffd700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5940539d0 0x7fa594055e90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fa5a801a000 tx=0x7fa5a8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.772+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5b0042680 con 0x7fa5b80b9d80 2022-07-12T08:42:01.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.940+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa5b809add0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.948+0000 7fa5a7fff700 1 -- 172.21.15.61:0/298847667 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7fa5b0045ca0 con 0x7fa5b80b9d80 2022-07-12T08:42:01.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5940539d0 msgr2=0x7fa594055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5940539d0 0x7fa594055e90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fa5a801a000 tx=0x7fa5a8014000).stop 2022-07-12T08:42:01.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 msgr2=0x7fa5b8156700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:01.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 0x7fa5b8156700 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fa5b0001360 tx=0x7fa5b0001b20).stop 2022-07-12T08:42:01.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 shutdown_connections 2022-07-12T08:42:01.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5b80b9d80 0x7fa5b8156700 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5b80b5a10 0x7fa5b809e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5940539d0 0x7fa594055e90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 --2- 172.21.15.61:0/298847667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5b805d0f0 0x7fa5b80b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:01.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 >> 172.21.15.61:0/298847667 conn(0x7fa5b80677d0 msgr2=0x7fa5b8097ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:01.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 shutdown_connections 2022-07-12T08:42:01.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:01.952+0000 7fa5b7fff700 1 -- 172.21.15.61:0/298847667 wait complete. 2022-07-12T08:42:01.971 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:42:01.973 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:42:02.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.080+0000 7f3ba316b700 1 Processor -- start 2022-07-12T08:42:02.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.080+0000 7f3ba316b700 1 -- start start 2022-07-12T08:42:02.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.080+0000 7f3ba316b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c0b29d0 0x7f3b9c062820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.080+0000 7f3ba316b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c0b7ee0 con 0x7f3b9c142f40 2022-07-12T08:42:02.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.080+0000 7f3ba316b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b9c062d10 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.080+0000 7f3ba316b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c060e80 con 0x7f3b9c0b29d0 2022-07-12T08:42:02.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba296a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3b9c142f40 0x7f3b9c1433d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42576/0 (socket says 172.21.15.61:42576) 2022-07-12T08:42:02.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba296a700 1 -- 172.21.15.61:0/1425332370 learned_addr learned my addr 172.21.15.61:0/1425332370 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:42:02.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1968700 1 --2- 172.21.15.61:0/1425332370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c0b29d0 0x7f3b9c062820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1167700 1 -- 172.21.15.61:0/1425332370 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74764949 0 0) 0x7f3b9c0b7ee0 con 0x7f3b9c142f40 2022-07-12T08:42:02.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1968700 1 -- 172.21.15.61:0/1425332370 >> v1:172.21.15.111:6789/0 conn(0x7f3b9c0b7b00 legacy=0x7f3b9c05fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1968700 1 -- 172.21.15.61:0/1425332370 >> v1:172.21.15.61:6789/0 conn(0x7f3b9c142f40 legacy=0x7f3b9c1433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1968700 1 -- 172.21.15.61:0/1425332370 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c061120 con 0x7f3b9c0b29d0 2022-07-12T08:42:02.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1968700 1 --2- 172.21.15.61:0/1425332370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c0b29d0 0x7f3b9c062820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3b98002f70 tx=0x7f3b98006df0).ready entity=mon.2 client_cookie=d924d2621c50a733 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:02.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1167700 1 -- 172.21.15.61:0/1425332370 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b9800d5c0 con 0x7f3b9c0b29d0 2022-07-12T08:42:02.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1167700 1 -- 172.21.15.61:0/1425332370 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b98000e90 con 0x7f3b9c0b29d0 2022-07-12T08:42:02.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba1167700 1 -- 172.21.15.61:0/1425332370 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b9800d910 con 0x7f3b9c0b29d0 2022-07-12T08:42:02.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 -- 172.21.15.61:0/1425332370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c0b29d0 msgr2=0x7f3b9c062820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 --2- 172.21.15.61:0/1425332370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c0b29d0 0x7f3b9c062820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3b98002f70 tx=0x7f3b98006df0).stop 2022-07-12T08:42:02.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 -- 172.21.15.61:0/1425332370 shutdown_connections 2022-07-12T08:42:02.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 --2- 172.21.15.61:0/1425332370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c0b29d0 0x7f3b9c062820 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 -- 172.21.15.61:0/1425332370 >> 172.21.15.61:0/1425332370 conn(0x7f3b9c065550 msgr2=0x7f3b9c0b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:02.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 -- 172.21.15.61:0/1425332370 shutdown_connections 2022-07-12T08:42:02.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 -- 172.21.15.61:0/1425332370 wait complete. 2022-07-12T08:42:02.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 Processor -- start 2022-07-12T08:42:02.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.084+0000 7f3ba316b700 1 -- start start 2022-07-12T08:42:02.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 0x7f3b9c141380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c0b7b00 0x7f3b9c1482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c142f40 0x7f3b9c148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0ad450 con 0x7f3b9c0b29d0 2022-07-12T08:42:02.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c09b7a0 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c0ac310 con 0x7f3b9c142f40 2022-07-12T08:42:02.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba2169700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 0x7f3b9c141380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba2169700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 0x7f3b9c141380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54362/0 (socket says 172.21.15.61:54362) 2022-07-12T08:42:02.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c0b7b00 0x7f3b9c1482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba2169700 1 -- 172.21.15.61:0/192223113 learned_addr learned my addr 172.21.15.61:0/192223113 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:02.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba296a700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c142f40 0x7f3b9c148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 -- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c142f40 msgr2=0x7f3b9c148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c142f40 0x7f3b9c148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 -- 172.21.15.61:0/192223113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 msgr2=0x7f3b9c141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 0x7f3b9c141380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 -- 172.21.15.61:0/192223113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9800b180 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba2169700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 0x7f3b9c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:42:02.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba1968700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c0b7b00 0x7f3b9c1482a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3b98002f70 tx=0x7f3b98015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:02.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b9800de60 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b98011b80 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b9c09ad10 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b9801e460 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b9c0acd10 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.088+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9c0ad150 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.092+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3b9801a040 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.092+0000 7f3b8b7fe700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b7c053a60 0x7f3b7c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.092+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b9806ca10 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.092+0000 7f3ba2169700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b7c053a60 0x7f3b7c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.092+0000 7f3ba2169700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b7c053a60 0x7f3b7c055f20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3b8c00d000 tx=0x7f3b8c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:02.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.100+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b98043970 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.264+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b9c099db0 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.268+0000 7f3b8b7fe700 1 -- 172.21.15.61:0/192223113 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3b98046f90 con 0x7f3b9c0b7b00 2022-07-12T08:42:02.281 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:02.281 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:42:02.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.268+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b7c053a60 msgr2=0x7f3b7c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.268+0000 7f3ba316b700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b7c053a60 0x7f3b7c055f20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3b8c00d000 tx=0x7f3b8c009040).stop 2022-07-12T08:42:02.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c0b7b00 msgr2=0x7f3b9c1482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c0b7b00 0x7f3b9c1482a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f3b98002f70 tx=0x7f3b98015000).stop 2022-07-12T08:42:02.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 shutdown_connections 2022-07-12T08:42:02.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b9c142f40 0x7f3b9c148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b7c053a60 0x7f3b7c055f20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b9c0b7b00 0x7f3b9c1482a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 --2- 172.21.15.61:0/192223113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b9c0b29d0 0x7f3b9c141380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 >> 172.21.15.61:0/192223113 conn(0x7f3b9c065550 msgr2=0x7f3b9c0b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 shutdown_connections 2022-07-12T08:42:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.272+0000 7f3ba316b700 1 -- 172.21.15.61:0/192223113 wait complete. 2022-07-12T08:42:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:42:02.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:41:58.634283+0000', 'modified': '2022-07-12T08:42:00.304215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4905}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4905': {'gid': 4905, 'name': 'a', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3202017529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3202017529}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3202017529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:42:02.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:42:02.502 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:42:02.642 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":4905,"want_state":"up:active","state":"up:active","rank_uptime":3.9805663089999999,"mdsmap_epoch":12,"osdmap_epoch":32,"osdmap_epoch_barrier":31,"uptime":11.210595039999999} 2022-07-12T08:42:02.643 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 4905, "mdsmap_epoch": 12, "osdmap_epoch": 32, "osdmap_epoch_barrier": 31, "rank_uptime": 3.980566309, "state": "up:active", "uptime": 11.21059504, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:42:02.645 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:42:02.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.756+0000 7f0644da0700 1 Processor -- start 2022-07-12T08:42:02.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.756+0000 7f0644da0700 1 -- start start 2022-07-12T08:42:02.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 0x7f064005f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f06400b2470 0x7f064013a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f064005d2f0 con 0x7f06400af9a0 2022-07-12T08:42:02.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0640140a80 con 0x7f06400b2470 2022-07-12T08:42:02.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0640140c70 con 0x7f0640141820 2022-07-12T08:42:02.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 0x7f064005f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f06400b2470 0x7f064013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 0x7f064005f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36564/0 (socket says 172.21.15.61:36564) 2022-07-12T08:42:02.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063ed9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f06400b2470 0x7f064013a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32946/0 (socket says 172.21.15.61:32946) 2022-07-12T08:42:02.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063e59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f06400af9a0 0x7f0640141110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56188/0 (socket says 172.21.15.61:56188) 2022-07-12T08:42:02.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 -- 172.21.15.61:0/4187219451 learned_addr learned my addr 172.21.15.61:0/4187219451 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:02.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 -- 172.21.15.61:0/4187219451 >> v2:172.21.15.111:3300/0 conn(0x7f06400b2470 msgr2=0x7f064013a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:02.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 --2- 172.21.15.61:0/4187219451 >> v2:172.21.15.111:3300/0 conn(0x7f06400b2470 0x7f064013a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 -- 172.21.15.61:0/4187219451 >> v1:172.21.15.111:6789/0 conn(0x7f06400af9a0 legacy=0x7f0640141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 -- 172.21.15.61:0/4187219451 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064005cf70 con 0x7f0640141820 2022-07-12T08:42:02.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063dd9b700 1 --2- 172.21.15.61:0/4187219451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 0x7f064005f440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0634002f70 tx=0x7f0634006eb0).ready entity=mon.2 client_cookie=20cf82fbac033427 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:02.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063d59a700 1 -- 172.21.15.61:0/4187219451 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f063400d570 con 0x7f0640141820 2022-07-12T08:42:02.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063d59a700 1 -- 172.21.15.61:0/4187219451 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634000e90 con 0x7f0640141820 2022-07-12T08:42:02.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f063d59a700 1 -- 172.21.15.61:0/4187219451 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f063400d900 con 0x7f0640141820 2022-07-12T08:42:02.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- 172.21.15.61:0/4187219451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 msgr2=0x7f064005f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 --2- 172.21.15.61:0/4187219451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 0x7f064005f440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f0634002f70 tx=0x7f0634006eb0).stop 2022-07-12T08:42:02.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- 172.21.15.61:0/4187219451 shutdown_connections 2022-07-12T08:42:02.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 --2- 172.21.15.61:0/4187219451 >> v2:172.21.15.111:3300/0 conn(0x7f06400b2470 0x7f064013a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 --2- 172.21.15.61:0/4187219451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0640141820 0x7f064005f440 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- 172.21.15.61:0/4187219451 >> 172.21.15.61:0/4187219451 conn(0x7f0640063bd0 msgr2=0x7f0640063fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:02.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.760+0000 7f0644da0700 1 -- 172.21.15.61:0/4187219451 shutdown_connections 2022-07-12T08:42:02.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 -- 172.21.15.61:0/4187219451 wait complete. 2022-07-12T08:42:02.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 Processor -- start 2022-07-12T08:42:02.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 -- start start 2022-07-12T08:42:02.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 0x7f064013ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b2470 0x7f064013da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0640141820 0x7f064013df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400aea10 con 0x7f0640141820 2022-07-12T08:42:02.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400aecb0 con 0x7f06400b2470 2022-07-12T08:42:02.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f0644da0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400aef50 con 0x7f06400af9a0 2022-07-12T08:42:02.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f063e59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 0x7f064013ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f063ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0640141820 0x7f064013df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f063dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b2470 0x7f064013da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f063e59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 0x7f064013ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36570/0 (socket says 172.21.15.61:36570) 2022-07-12T08:42:02.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f063dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b2470 0x7f064013da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32950/0 (socket says 172.21.15.61:32950) 2022-07-12T08:42:02.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.764+0000 7f063ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0640141820 0x7f064013df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54378/0 (socket says 172.21.15.61:54378) 2022-07-12T08:42:02.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 -- 172.21.15.61:0/4000664339 learned_addr learned my addr 172.21.15.61:0/4000664339 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:02.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 -- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b2470 msgr2=0x7f064013da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b2470 0x7f064013da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 -- 172.21.15.61:0/4000664339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0640141820 msgr2=0x7f064013df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0640141820 0x7f064013df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 -- 172.21.15.61:0/4000664339 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f063400b1e0 con 0x7f06400af9a0 2022-07-12T08:42:02.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f063e59c700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 0x7f064013ff20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0640140ff0 tx=0x7f0630007bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:02.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06300070d0 con 0x7f06400af9a0 2022-07-12T08:42:02.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0630007230 con 0x7f06400af9a0 2022-07-12T08:42:02.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064013e430 con 0x7f06400af9a0 2022-07-12T08:42:02.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06300055a0 con 0x7f06400af9a0 2022-07-12T08:42:02.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0640155710 con 0x7f06400af9a0 2022-07-12T08:42:02.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0640066860 con 0x7f06400af9a0 2022-07-12T08:42:02.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.768+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0630010070 con 0x7f06400af9a0 2022-07-12T08:42:02.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.772+0000 7f062f7fe700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c053a20 0x7f061c055ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:02.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.772+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f063006b4c0 con 0x7f06400af9a0 2022-07-12T08:42:02.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.772+0000 7f063dd9b700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c053a20 0x7f061c055ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:02.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.772+0000 7f063dd9b700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c053a20 0x7f061c055ee0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f063401c000 tx=0x7f0634015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:02.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.784+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0630042310 con 0x7f06400af9a0 2022-07-12T08:42:02.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.908+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0640057460 con 0x7f06400af9a0 2022-07-12T08:42:02.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f062f7fe700 1 -- 172.21.15.61:0/4000664339 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f0630045930 con 0x7f06400af9a0 2022-07-12T08:42:02.922 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:02.923 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":32,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:42:01.061999+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:41:56.415996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-12T08:41:57.640916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:42:02.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c053a20 msgr2=0x7f061c055ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c053a20 0x7f061c055ee0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f063401c000 tx=0x7f0634015000).stop 2022-07-12T08:42:02.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 msgr2=0x7f064013ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:02.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 0x7f064013ff20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0640140ff0 tx=0x7f0630007bf0).stop 2022-07-12T08:42:02.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 shutdown_connections 2022-07-12T08:42:02.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0640141820 0x7f064013df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c053a20 0x7f061c055ee0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b2470 0x7f064013da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 --2- 172.21.15.61:0/4000664339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400af9a0 0x7f064013ff20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:02.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 >> 172.21.15.61:0/4000664339 conn(0x7f0640063bd0 msgr2=0x7f0640143820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:02.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 shutdown_connections 2022-07-12T08:42:02.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:02.916+0000 7f0644da0700 1 -- 172.21.15.61:0/4000664339 wait complete. 2022-07-12T08:42:02.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:41:56.415996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:41:57.640916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:42:02.941 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:42:03.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebe0f1700 1 Processor -- start 2022-07-12T08:42:03.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebe0f1700 1 -- start start 2022-07-12T08:42:03.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebe0f1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faeb80b1000 0x7faeb80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebe0f1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb8099010 con 0x7faeb8060c10 2022-07-12T08:42:03.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebe0f1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb814b780 con 0x7faeb80afdc0 2022-07-12T08:42:03.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebe0f1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb8060ff0 con 0x7faeb80b1000 2022-07-12T08:42:03.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faeb80b1000 0x7faeb80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faeb80b1000 0x7faeb80af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54382/0 (socket says 172.21.15.61:54382) 2022-07-12T08:42:03.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 -- 172.21.15.61:0/1213559197 learned_addr learned my addr 172.21.15.61:0/1213559197 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 -- 172.21.15.61:0/1213559197 >> v1:172.21.15.111:6789/0 conn(0x7faeb80afdc0 legacy=0x7faeb809a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 -- 172.21.15.61:0/1213559197 >> v1:172.21.15.61:6789/0 conn(0x7faeb8060c10 legacy=0x7faeb814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 -- 172.21.15.61:0/1213559197 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb814a930 con 0x7faeb80b1000 2022-07-12T08:42:03.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faebc8ee700 1 --2- 172.21.15.61:0/1213559197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb80af8d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7faeb40032f0 tx=0x7faeb4009290).ready entity=mon.0 client_cookie=12ed47ede465e9dd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faeaffff700 1 -- 172.21.15.61:0/1213559197 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faeb400b8a0 con 0x7faeb80b1000 2022-07-12T08:42:03.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faeaffff700 1 -- 172.21.15.61:0/1213559197 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb400ba00 con 0x7faeb80b1000 2022-07-12T08:42:03.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.080+0000 7faeaffff700 1 -- 172.21.15.61:0/1213559197 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faeb400bb70 con 0x7faeb80b1000 2022-07-12T08:42:03.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- 172.21.15.61:0/1213559197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 msgr2=0x7faeb80af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1213559197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb80af8d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7faeb40032f0 tx=0x7faeb4009290).stop 2022-07-12T08:42:03.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- 172.21.15.61:0/1213559197 shutdown_connections 2022-07-12T08:42:03.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1213559197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb80af8d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- 172.21.15.61:0/1213559197 >> 172.21.15.61:0/1213559197 conn(0x7faeb805bd10 msgr2=0x7faeb805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- 172.21.15.61:0/1213559197 shutdown_connections 2022-07-12T08:42:03.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- 172.21.15.61:0/1213559197 wait complete. 2022-07-12T08:42:03.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 Processor -- start 2022-07-12T08:42:03.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- start start 2022-07-12T08:42:03.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb8060c10 0x7faeb8156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faeb80afdc0 0x7faeb80ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb8063770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8157490 con 0x7faeb80b1000 2022-07-12T08:42:03.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb8062d80 con 0x7faeb8060c10 2022-07-12T08:42:03.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebe0f1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb80661a0 con 0x7faeb80afdc0 2022-07-12T08:42:03.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebd8f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb8063770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebd8f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb8063770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54390/0 (socket says 172.21.15.61:54390) 2022-07-12T08:42:03.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebd8f0700 1 -- 172.21.15.61:0/1064888997 learned_addr learned my addr 172.21.15.61:0/1064888997 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebc8ee700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faeb80afdc0 0x7faeb80ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.084+0000 7faebd0ef700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb8060c10 0x7faeb8156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd8f0700 1 -- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faeb80afdc0 msgr2=0x7faeb80ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd8f0700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faeb80afdc0 0x7faeb80ae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd8f0700 1 -- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb8060c10 msgr2=0x7faeb8156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd8f0700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb8060c10 0x7faeb8156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd8f0700 1 -- 172.21.15.61:0/1064888997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb4009530 con 0x7faeb80b1000 2022-07-12T08:42:03.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd8f0700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb8063770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7faeb805dad0 tx=0x7faeb0007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faeb0012400 con 0x7faeb80b1000 2022-07-12T08:42:03.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb0012560 con 0x7faeb80b1000 2022-07-12T08:42:03.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb8156910 con 0x7faeb80b1000 2022-07-12T08:42:03.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faeb00128c0 con 0x7faeb80b1000 2022-07-12T08:42:03.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb8063c60 con 0x7faeb80b1000 2022-07-12T08:42:03.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb8062be0 con 0x7faeb80b1000 2022-07-12T08:42:03.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faeb0019030 con 0x7faeb80b1000 2022-07-12T08:42:03.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faeae7fc700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faea0053920 0x7faea0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faeb006b040 con 0x7faeb80b1000 2022-07-12T08:42:03.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.088+0000 7faebd0ef700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faea0053920 0x7faea0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.092+0000 7faebd0ef700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faea0053920 0x7faea0055de0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7faea800b000 tx=0x7faea8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.096+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faeb8062be0 con 0x7faeb80b1000 2022-07-12T08:42:03.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7faeb814b780 con 0x7faeb80b1000 2022-07-12T08:42:03.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faeae7fc700 1 -- 172.21.15.61:0/1064888997 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v32) v1 ==== 110+0+10 (secure 0 0 0) 0x7faeb814b780 con 0x7faeb80b1000 2022-07-12T08:42:03.237 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:42:03.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faea0053920 msgr2=0x7faea0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faea0053920 0x7faea0055de0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7faea800b000 tx=0x7faea8009040).stop 2022-07-12T08:42:03.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 msgr2=0x7faeb8063770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb8063770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7faeb805dad0 tx=0x7faeb0007b50).stop 2022-07-12T08:42:03.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 shutdown_connections 2022-07-12T08:42:03.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faeb80b1000 0x7faeb8063770 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faeb80afdc0 0x7faeb80ae560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faea0053920 0x7faea0055de0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 --2- 172.21.15.61:0/1064888997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faeb8060c10 0x7faeb8156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 >> 172.21.15.61:0/1064888997 conn(0x7faeb805bd10 msgr2=0x7faeb805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 shutdown_connections 2022-07-12T08:42:03.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.220+0000 7faebe0f1700 1 -- 172.21.15.61:0/1064888997 wait complete. 2022-07-12T08:42:03.257 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:42:03.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 Processor -- start 2022-07-12T08:42:03.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 -- start start 2022-07-12T08:42:03.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f15500b7bd0 0x7f15500ace00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f155005bd10 0x7f15500ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500ad7e0 0x7f155014a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1550148200 con 0x7f155005bd10 2022-07-12T08:42:03.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15500ae0c0 con 0x7f15500b7bd0 2022-07-12T08:42:03.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556a79700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15500b7fb0 con 0x7f15500ad7e0 2022-07-12T08:42:03.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f155005bd10 0x7f15500ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f155005bd10 0x7f15500ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54396/0 (socket says 172.21.15.61:54396) 2022-07-12T08:42:03.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 -- 172.21.15.61:0/230432068 learned_addr learned my addr 172.21.15.61:0/230432068 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1556278700 1 --2- 172.21.15.61:0/230432068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500ad7e0 0x7f155014a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555a77700 1 --2- 172.21.15.61:0/230432068 >> v2:172.21.15.111:3300/0 conn(0x7f15500b7bd0 0x7f15500ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 -- 172.21.15.61:0/230432068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500ad7e0 msgr2=0x7f155014a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:03.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 --2- 172.21.15.61:0/230432068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500ad7e0 0x7f155014a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 -- 172.21.15.61:0/230432068 >> v2:172.21.15.111:3300/0 conn(0x7f15500b7bd0 msgr2=0x7f15500ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:03.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 --2- 172.21.15.61:0/230432068 >> v2:172.21.15.111:3300/0 conn(0x7f15500b7bd0 0x7f15500ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 -- 172.21.15.61:0/230432068 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15500b1eb0 con 0x7f155005bd10 2022-07-12T08:42:03.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.364+0000 7f1555276700 1 --2- 172.21.15.61:0/230432068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f155005bd10 0x7f15500ad2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1540002f70 tx=0x7f1540006eb0).ready entity=mon.0 client_cookie=1d888b96e8e6fe98 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1554a75700 1 -- 172.21.15.61:0/230432068 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154000d570 con 0x7f155005bd10 2022-07-12T08:42:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1554a75700 1 -- 172.21.15.61:0/230432068 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1540000e90 con 0x7f155005bd10 2022-07-12T08:42:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1554a75700 1 -- 172.21.15.61:0/230432068 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154000d900 con 0x7f155005bd10 2022-07-12T08:42:03.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 -- 172.21.15.61:0/230432068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f155005bd10 msgr2=0x7f15500ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 --2- 172.21.15.61:0/230432068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f155005bd10 0x7f15500ad2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f1540002f70 tx=0x7f1540006eb0).stop 2022-07-12T08:42:03.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 -- 172.21.15.61:0/230432068 shutdown_connections 2022-07-12T08:42:03.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 --2- 172.21.15.61:0/230432068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500ad7e0 0x7f155014a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 --2- 172.21.15.61:0/230432068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f155005bd10 0x7f15500ad2f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 --2- 172.21.15.61:0/230432068 >> v2:172.21.15.111:3300/0 conn(0x7f15500b7bd0 0x7f15500ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 -- 172.21.15.61:0/230432068 >> 172.21.15.61:0/230432068 conn(0x7f15500655e0 msgr2=0x7f15500b7180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 -- 172.21.15.61:0/230432068 shutdown_connections 2022-07-12T08:42:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 -- 172.21.15.61:0/230432068 wait complete. 2022-07-12T08:42:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 Processor -- start 2022-07-12T08:42:03.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.368+0000 7f1556a79700 1 -- start start 2022-07-12T08:42:03.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f155005bd10 0x7f1550155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 0x7f15500b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500b7bd0 0x7f15500b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15500af3b0 con 0x7f15500ad7e0 2022-07-12T08:42:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15500b6f00 con 0x7f155005bd10 2022-07-12T08:42:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15500aeb90 con 0x7f15500b7bd0 2022-07-12T08:42:03.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 0x7f15500b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555a77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f155005bd10 0x7f1550155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 0x7f15500b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54402/0 (socket says 172.21.15.61:54402) 2022-07-12T08:42:03.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555a77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f155005bd10 0x7f1550155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:32974/0 (socket says 172.21.15.61:32974) 2022-07-12T08:42:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 -- 172.21.15.61:0/3069086108 learned_addr learned my addr 172.21.15.61:0/3069086108 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556278700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500b7bd0 0x7f15500b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 -- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500b7bd0 msgr2=0x7f15500b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500b7bd0 0x7f15500b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 -- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f155005bd10 msgr2=0x7f1550155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f155005bd10 0x7f1550155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 -- 172.21.15.61:0/3069086108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154000b1e0 con 0x7f15500ad7e0 2022-07-12T08:42:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1555276700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 0x7f15500b0890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1540002f70 tx=0x7f1540015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f154000d910 con 0x7f15500ad7e0 2022-07-12T08:42:03.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15400068b0 con 0x7f15500ad7e0 2022-07-12T08:42:03.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1540011e20 con 0x7f15500ad7e0 2022-07-12T08:42:03.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15500b5540 con 0x7f15500ad7e0 2022-07-12T08:42:03.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15500b0d80 con 0x7f15500ad7e0 2022-07-12T08:42:03.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.372+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15500aed50 con 0x7f15500ad7e0 2022-07-12T08:42:03.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.376+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f154001a040 con 0x7f15500ad7e0 2022-07-12T08:42:03.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.376+0000 7f153effd700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1530053a60 0x7f1530055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.376+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f154006cce0 con 0x7f15500ad7e0 2022-07-12T08:42:03.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.376+0000 7f1555a77700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1530053a60 0x7f1530055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.376+0000 7f1555a77700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1530053a60 0x7f1530055f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f154c00b000 tx=0x7f154c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.384+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1540043bf0 con 0x7f15500ad7e0 2022-07-12T08:42:03.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.512+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1550057460 con 0x7f15500ad7e0 2022-07-12T08:42:03.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f153effd700 1 -- 172.21.15.61:0/3069086108 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v32) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1540047150 con 0x7f15500ad7e0 2022-07-12T08:42:03.523 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:42:03.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1530053a60 msgr2=0x7f1530055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1530053a60 0x7f1530055f20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f154c00b000 tx=0x7f154c0091d0).stop 2022-07-12T08:42:03.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 msgr2=0x7f15500b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 0x7f15500b0890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f1540002f70 tx=0x7f1540015000).stop 2022-07-12T08:42:03.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 shutdown_connections 2022-07-12T08:42:03.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15500b7bd0 0x7f15500b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15500ad7e0 0x7f15500b0890 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1530053a60 0x7f1530055f20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 --2- 172.21.15.61:0/3069086108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f155005bd10 0x7f1550155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 >> 172.21.15.61:0/3069086108 conn(0x7f15500655e0 msgr2=0x7f15500602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 shutdown_connections 2022-07-12T08:42:03.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.516+0000 7f1556a79700 1 -- 172.21.15.61:0/3069086108 wait complete. 2022-07-12T08:42:03.542 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:42:03.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 Processor -- start 2022-07-12T08:42:03.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 -- start start 2022-07-12T08:42:03.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310146710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43100b2660 0x7f4310146c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f431005c5a0 0x7f431005ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431005fa70 con 0x7f43100b2660 2022-07-12T08:42:03.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43100b3130 con 0x7f431005c5a0 2022-07-12T08:42:03.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4316bfd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4310146270 con 0x7f431005f690 2022-07-12T08:42:03.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f43153fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43100b2660 0x7f4310146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4315bfb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4315bfb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310146710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36604/0 (socket says 172.21.15.61:36604) 2022-07-12T08:42:03.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f4315bfb700 1 -- 172.21.15.61:0/684525016 learned_addr learned my addr 172.21.15.61:0/684525016 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f43153fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43100b2660 0x7f4310146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54410/0 (socket says 172.21.15.61:54410) 2022-07-12T08:42:03.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.648+0000 7f43163fc700 1 --2- 172.21.15.61:0/684525016 >> v2:172.21.15.111:3300/0 conn(0x7f431005c5a0 0x7f431005ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 -- 172.21.15.61:0/684525016 >> v2:172.21.15.111:3300/0 conn(0x7f431005c5a0 msgr2=0x7f431005ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:03.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 --2- 172.21.15.61:0/684525016 >> v2:172.21.15.111:3300/0 conn(0x7f431005c5a0 0x7f431005ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 -- 172.21.15.61:0/684525016 >> v2:172.21.15.61:3300/0 conn(0x7f43100b2660 msgr2=0x7f4310146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:03.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 --2- 172.21.15.61:0/684525016 >> v2:172.21.15.61:3300/0 conn(0x7f43100b2660 0x7f4310146c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 -- 172.21.15.61:0/684525016 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43100b2d50 con 0x7f431005f690 2022-07-12T08:42:03.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 --2- 172.21.15.61:0/684525016 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310146710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c006eb0).ready entity=mon.2 client_cookie=3b588aa7768c8b17 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4314bf9700 1 -- 172.21.15.61:0/684525016 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00d570 con 0x7f431005f690 2022-07-12T08:42:03.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4314bf9700 1 -- 172.21.15.61:0/684525016 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c000e90 con 0x7f431005f690 2022-07-12T08:42:03.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4314bf9700 1 -- 172.21.15.61:0/684525016 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00d900 con 0x7f431005f690 2022-07-12T08:42:03.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- 172.21.15.61:0/684525016 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 msgr2=0x7f4310146710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- 172.21.15.61:0/684525016 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310146710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c006eb0).stop 2022-07-12T08:42:03.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- 172.21.15.61:0/684525016 shutdown_connections 2022-07-12T08:42:03.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- 172.21.15.61:0/684525016 >> v2:172.21.15.111:3300/0 conn(0x7f431005c5a0 0x7f431005ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- 172.21.15.61:0/684525016 >> v2:172.21.15.61:3300/0 conn(0x7f43100b2660 0x7f4310146c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- 172.21.15.61:0/684525016 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310146710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- 172.21.15.61:0/684525016 >> 172.21.15.61:0/684525016 conn(0x7f4310063ab0 msgr2=0x7f4310063ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- 172.21.15.61:0/684525016 shutdown_connections 2022-07-12T08:42:03.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- 172.21.15.61:0/684525016 wait complete. 2022-07-12T08:42:03.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 Processor -- start 2022-07-12T08:42:03.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- start start 2022-07-12T08:42:03.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 0x7f4310066dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b2660 0x7f4310141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43100ae080 con 0x7f431005c5a0 2022-07-12T08:42:03.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43100ad670 con 0x7f43100b2660 2022-07-12T08:42:03.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4316bfd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43100ae2b0 con 0x7f431005f690 2022-07-12T08:42:03.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 0x7f4310066dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 0x7f4310066dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54414/0 (socket says 172.21.15.61:54414) 2022-07-12T08:42:03.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f4315bfb700 1 -- 172.21.15.61:0/1999651828 learned_addr learned my addr 172.21.15.61:0/1999651828 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f43153fa700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.652+0000 7f43163fc700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b2660 0x7f4310141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4315bfb700 1 -- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 msgr2=0x7f4310140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4315bfb700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4315bfb700 1 -- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b2660 msgr2=0x7f4310141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4315bfb700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b2660 0x7f4310141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4315bfb700 1 -- 172.21.15.61:0/1999651828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c00b1e0 con 0x7f431005c5a0 2022-07-12T08:42:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4315bfb700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 0x7f4310066dd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c0116e0 con 0x7f431005c5a0 2022-07-12T08:42:03.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c011840 con 0x7f431005c5a0 2022-07-12T08:42:03.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c011ba0 con 0x7f431005c5a0 2022-07-12T08:42:03.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4310141850 con 0x7f431005c5a0 2022-07-12T08:42:03.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4310151cb0 con 0x7f431005c5a0 2022-07-12T08:42:03.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310057460 con 0x7f431005c5a0 2022-07-12T08:42:03.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f430c01a040 con 0x7f431005c5a0 2022-07-12T08:42:03.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.656+0000 7f42feffd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f42f00539b0 0x7f42f0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.660+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f430c06ccb0 con 0x7f431005c5a0 2022-07-12T08:42:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.660+0000 7f43153fa700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f42f00539b0 0x7f42f0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.660+0000 7f43153fa700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f42f00539b0 0x7f42f0055e70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f430000b000 tx=0x7f4300006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.664+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f430c043c90 con 0x7f431005c5a0 2022-07-12T08:42:03.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.792+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f431005d120 con 0x7f431005c5a0 2022-07-12T08:42:03.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.792+0000 7f42feffd700 1 -- 172.21.15.61:0/1999651828 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v32) v1 ==== 104+0+10 (secure 0 0 0) 0x7f430c0230a0 con 0x7f431005c5a0 2022-07-12T08:42:03.803 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:42:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f42f00539b0 msgr2=0x7f42f0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f42f00539b0 0x7f42f0055e70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f430000b000 tx=0x7f4300006df0).stop 2022-07-12T08:42:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 msgr2=0x7f4310066dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 0x7f4310066dd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f430c002f70 tx=0x7f430c015000).stop 2022-07-12T08:42:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 shutdown_connections 2022-07-12T08:42:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f42f00539b0 0x7f42f0055e70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43100b2660 0x7f4310141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f431005f690 0x7f4310140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 --2- 172.21.15.61:0/1999651828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431005c5a0 0x7f4310066dd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 >> 172.21.15.61:0/1999651828 conn(0x7f4310063ab0 msgr2=0x7f431005ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 shutdown_connections 2022-07-12T08:42:03.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.796+0000 7f4316bfd700 1 -- 172.21.15.61:0/1999651828 wait complete. 2022-07-12T08:42:03.816 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:42:03.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.920+0000 7efc7b59e700 1 Processor -- start 2022-07-12T08:42:03.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.920+0000 7efc7b59e700 1 -- start start 2022-07-12T08:42:03.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.920+0000 7efc7b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05c6d0 0x7efc7c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.920+0000 7efc7b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c143500 con 0x7efc7c142eb0 2022-07-12T08:42:03.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.920+0000 7efc7b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c143290 con 0x7efc7c05d130 2022-07-12T08:42:03.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.920+0000 7efc7b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c143760 con 0x7efc7c05c6d0 2022-07-12T08:42:03.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efc7c142eb0 0x7efc7c0b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42640/0 (socket says 172.21.15.61:42640) 2022-07-12T08:42:03.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7a59c700 1 -- 172.21.15.61:0/3171829104 learned_addr learned my addr 172.21.15.61:0/3171829104 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:42:03.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7ad9d700 1 --2- 172.21.15.61:0/3171829104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05c6d0 0x7efc7c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7959a700 1 -- 172.21.15.61:0/3171829104 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361284661 0 0) 0x7efc7c143500 con 0x7efc7c142eb0 2022-07-12T08:42:03.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7959a700 1 -- 172.21.15.61:0/3171829104 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc5c003260 con 0x7efc7c142eb0 2022-07-12T08:42:03.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7ad9d700 1 -- 172.21.15.61:0/3171829104 >> v1:172.21.15.111:6789/0 conn(0x7efc7c05d130 legacy=0x7efc7c0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7959a700 1 -- 172.21.15.61:0/3171829104 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962081456 0 0) 0x7efc7c143290 con 0x7efc7c05d130 2022-07-12T08:42:03.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7ad9d700 1 -- 172.21.15.61:0/3171829104 >> v1:172.21.15.61:6789/0 conn(0x7efc7c142eb0 legacy=0x7efc7c0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7ad9d700 1 -- 172.21.15.61:0/3171829104 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c0b4030 con 0x7efc7c05c6d0 2022-07-12T08:42:03.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7ad9d700 1 --2- 172.21.15.61:0/3171829104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05c6d0 0x7efc7c0b6130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7efc6c001800 tx=0x7efc740091d0).ready entity=mon.2 client_cookie=6da7d31b938bd12c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7959a700 1 -- 172.21.15.61:0/3171829104 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc6c001660 con 0x7efc7c05c6d0 2022-07-12T08:42:03.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7959a700 1 -- 172.21.15.61:0/3171829104 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7400b580 con 0x7efc7c05c6d0 2022-07-12T08:42:03.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7959a700 1 -- 172.21.15.61:0/3171829104 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc7400b6f0 con 0x7efc7c05c6d0 2022-07-12T08:42:03.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 -- 172.21.15.61:0/3171829104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05c6d0 msgr2=0x7efc7c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 --2- 172.21.15.61:0/3171829104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05c6d0 0x7efc7c0b6130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7efc6c001800 tx=0x7efc740091d0).stop 2022-07-12T08:42:03.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 -- 172.21.15.61:0/3171829104 shutdown_connections 2022-07-12T08:42:03.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 --2- 172.21.15.61:0/3171829104 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05c6d0 0x7efc7c0b6130 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 -- 172.21.15.61:0/3171829104 >> 172.21.15.61:0/3171829104 conn(0x7efc7c066d70 msgr2=0x7efc7c067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:03.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 -- 172.21.15.61:0/3171829104 shutdown_connections 2022-07-12T08:42:03.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.924+0000 7efc7b59e700 1 -- 172.21.15.61:0/3171829104 wait complete. 2022-07-12T08:42:03.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 Processor -- start 2022-07-12T08:42:03.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 -- start start 2022-07-12T08:42:03.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc7c05c6d0 0x7efc7c145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05d130 0x7efc7c154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 0x7efc7c0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b2c10 con 0x7efc7c142eb0 2022-07-12T08:42:03.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b2430 con 0x7efc7c05c6d0 2022-07-12T08:42:03.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b26d0 con 0x7efc7c05d130 2022-07-12T08:42:03.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 0x7efc7c0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 0x7efc7c0af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54432/0 (socket says 172.21.15.61:54432) 2022-07-12T08:42:03.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 -- 172.21.15.61:0/4055284785 learned_addr learned my addr 172.21.15.61:0/4055284785 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:03.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7a59c700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc7c05c6d0 0x7efc7c145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc79d9b700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05d130 0x7efc7c154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 -- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05d130 msgr2=0x7efc7c154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05d130 0x7efc7c154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 -- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc7c05c6d0 msgr2=0x7efc7c145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:03.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc7c05c6d0 0x7efc7c145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:03.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 -- 172.21.15.61:0/4055284785 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74009470 con 0x7efc7c142eb0 2022-07-12T08:42:03.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7ad9d700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 0x7efc7c0af3a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7efc6c001800 tx=0x7efc7c143c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc740032e0 con 0x7efc7c142eb0 2022-07-12T08:42:03.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74011b20 con 0x7efc7c142eb0 2022-07-12T08:42:03.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.928+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c0b2970 con 0x7efc7c142eb0 2022-07-12T08:42:03.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc74011e80 con 0x7efc7c142eb0 2022-07-12T08:42:03.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c146f00 con 0x7efc7c142eb0 2022-07-12T08:42:03.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efc74024030 con 0x7efc7c142eb0 2022-07-12T08:42:03.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c0ada60 con 0x7efc7c142eb0 2022-07-12T08:42:03.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc737fe700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc60053a60 0x7efc60055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:03.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efc7406d230 con 0x7efc7c142eb0 2022-07-12T08:42:03.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc7a59c700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc60053a60 0x7efc60055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:03.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.932+0000 7efc7a59c700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc60053a60 0x7efc60055f20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efc6c00d000 tx=0x7efc6c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:03.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:03.940+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc7c0ada60 con 0x7efc7c142eb0 2022-07-12T08:42:04.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.060+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc7c146f00 con 0x7efc7c142eb0 2022-07-12T08:42:04.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc737fe700 1 -- 172.21.15.61:0/4055284785 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v32) v1 ==== 100+0+10 (secure 0 0 0) 0x7efc7c146f00 con 0x7efc7c142eb0 2022-07-12T08:42:04.073 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:42:04.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc60053a60 msgr2=0x7efc60055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc60053a60 0x7efc60055f20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efc6c00d000 tx=0x7efc6c000bc0).stop 2022-07-12T08:42:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 msgr2=0x7efc7c0af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 0x7efc7c0af3a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7efc6c001800 tx=0x7efc7c143c00).stop 2022-07-12T08:42:04.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 shutdown_connections 2022-07-12T08:42:04.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc7c142eb0 0x7efc7c0af3a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc7c05d130 0x7efc7c154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc60053a60 0x7efc60055f20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 --2- 172.21.15.61:0/4055284785 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc7c05c6d0 0x7efc7c145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 >> 172.21.15.61:0/4055284785 conn(0x7efc7c066d70 msgr2=0x7efc7c0b8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:04.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 shutdown_connections 2022-07-12T08:42:04.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.064+0000 7efc7b59e700 1 -- 172.21.15.61:0/4055284785 wait complete. 2022-07-12T08:42:04.085 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:42:04.087 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:42:04.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 Processor -- start 2022-07-12T08:42:04.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 -- start start 2022-07-12T08:42:04.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100ab670 0x7f741005f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:04.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f74100afa60 0x7f74100afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:04.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74100abc70 con 0x7f74100ab290 2022-07-12T08:42:04.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74100aba90 con 0x7f74100afa60 2022-07-12T08:42:04.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f74153c9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741005c720 con 0x7f74100ab670 2022-07-12T08:42:04.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f74100afa60 0x7f74100afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:04.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f74100afa60 0x7f74100afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54438/0 (socket says 172.21.15.61:54438) 2022-07-12T08:42:04.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 -- 172.21.15.61:0/2334291824 learned_addr learned my addr 172.21.15.61:0/2334291824 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:04.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f740f7fe700 1 --2- 172.21.15.61:0/2334291824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100ab670 0x7f741005f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:04.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 -- 172.21.15.61:0/2334291824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100ab670 msgr2=0x7f741005f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:04.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 --2- 172.21.15.61:0/2334291824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100ab670 0x7f741005f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 -- 172.21.15.61:0/2334291824 >> v1:172.21.15.111:6789/0 conn(0x7f74100ab290 legacy=0x7f741005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 -- 172.21.15.61:0/2334291824 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741005c9c0 con 0x7f74100afa60 2022-07-12T08:42:04.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.188+0000 7f7414bc8700 1 --2- 172.21.15.61:0/2334291824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100afa60 0x7f74100afef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7400002f70 tx=0x7f7400006eb0).ready entity=mon.0 client_cookie=7f6adfea857bc5e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:04.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740effd700 1 -- 172.21.15.61:0/2334291824 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f740000d5c0 con 0x7f74100afa60 2022-07-12T08:42:04.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740effd700 1 -- 172.21.15.61:0/2334291824 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7400000e90 con 0x7f74100afa60 2022-07-12T08:42:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740effd700 1 -- 172.21.15.61:0/2334291824 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f740000d950 con 0x7f74100afa60 2022-07-12T08:42:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- 172.21.15.61:0/2334291824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100afa60 msgr2=0x7f74100afef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 --2- 172.21.15.61:0/2334291824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100afa60 0x7f74100afef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7400002f70 tx=0x7f7400006eb0).stop 2022-07-12T08:42:04.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- 172.21.15.61:0/2334291824 shutdown_connections 2022-07-12T08:42:04.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 --2- 172.21.15.61:0/2334291824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100afa60 0x7f74100afef0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 --2- 172.21.15.61:0/2334291824 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100ab670 0x7f741005f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- 172.21.15.61:0/2334291824 >> 172.21.15.61:0/2334291824 conn(0x7f74100b1950 msgr2=0x7f74100b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:04.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- 172.21.15.61:0/2334291824 shutdown_connections 2022-07-12T08:42:04.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- 172.21.15.61:0/2334291824 wait complete. 2022-07-12T08:42:04.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 Processor -- start 2022-07-12T08:42:04.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- start start 2022-07-12T08:42:04.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 0x7f7410156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:04.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74100ab670 0x7f74100ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:04.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100afa60 0x7f7410063860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:04.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7410156250 con 0x7f74100ab290 2022-07-12T08:42:04.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74101564f0 con 0x7f74100ab670 2022-07-12T08:42:04.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f74153c9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7410156790 con 0x7f74100afa60 2022-07-12T08:42:04.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 0x7f7410156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:04.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 0x7f7410156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54442/0 (socket says 172.21.15.61:54442) 2022-07-12T08:42:04.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 -- 172.21.15.61:0/2608545338 learned_addr learned my addr 172.21.15.61:0/2608545338 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:04.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f7414bc8700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100afa60 0x7f7410063860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:04.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740f7fe700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74100ab670 0x7f74100ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:04.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 -- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100afa60 msgr2=0x7f7410063860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100afa60 0x7f7410063860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 -- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74100ab670 msgr2=0x7f74100ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74100ab670 0x7f74100ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 -- 172.21.15.61:0/2608545338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740000b1e0 con 0x7f74100ab290 2022-07-12T08:42:04.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740f7fe700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74100ab670 0x7f74100ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:04.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.192+0000 7f740ffff700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 0x7f7410156c30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f741005cd40 tx=0x7f74080014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:04.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f740800e4a0 con 0x7f74100ab290 2022-07-12T08:42:04.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740800e600 con 0x7f74100ab290 2022-07-12T08:42:04.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74100adc20 con 0x7f74100ab290 2022-07-12T08:42:04.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f740800e960 con 0x7f74100ab290 2022-07-12T08:42:04.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741014d050 con 0x7f74100ab290 2022-07-12T08:42:04.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741014d330 con 0x7f74100ab290 2022-07-12T08:42:04.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7408019030 con 0x7f74100ab290 2022-07-12T08:42:04.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740d7fa700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73f0053a40 0x7f73f0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:04.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f740806b5e0 con 0x7f74100ab290 2022-07-12T08:42:04.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.196+0000 7f740f7fe700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73f0053a40 0x7f73f0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:04.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.200+0000 7f740f7fe700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73f0053a40 0x7f73f0055f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f740400c000 tx=0x7f7404008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:04.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.204+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7408042440 con 0x7f74100ab290 2022-07-12T08:42:04.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.364+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7410156090 con 0x7f74100ab290 2022-07-12T08:42:04.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f740d7fa700 1 -- 172.21.15.61:0/2608545338 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f7408045a60 con 0x7f74100ab290 2022-07-12T08:42:04.385 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:04.385 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:42:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73f0053a40 msgr2=0x7f73f0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73f0053a40 0x7f73f0055f00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f740400c000 tx=0x7f7404008040).stop 2022-07-12T08:42:04.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 msgr2=0x7f7410156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:04.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 0x7f7410156c30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f741005cd40 tx=0x7f74080014a0).stop 2022-07-12T08:42:04.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 shutdown_connections 2022-07-12T08:42:04.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74100afa60 0x7f7410063860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f73f0053a40 0x7f73f0055f00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74100ab670 0x7f74100ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 --2- 172.21.15.61:0/2608545338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74100ab290 0x7f7410156c30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:04.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 >> 172.21.15.61:0/2608545338 conn(0x7f74100b1950 msgr2=0x7f7410069ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:04.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 shutdown_connections 2022-07-12T08:42:04.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:04.368+0000 7f74153c9700 1 -- 172.21.15.61:0/2608545338 wait complete. 2022-07-12T08:42:04.418 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:42:04.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:41:58.634283+0000', 'modified': '2022-07-12T08:42:00.304215+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4905}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4905': {'gid': 4905, 'name': 'a', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3202017529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3202017529}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3202017529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:42:04.624 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:42:04.625 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:42:04.756 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":4905,"want_state":"up:active","state":"up:active","rank_uptime":6.0955099480000001,"mdsmap_epoch":12,"osdmap_epoch":32,"osdmap_epoch_barrier":31,"uptime":13.325537748} 2022-07-12T08:42:04.757 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 4905, "mdsmap_epoch": 12, "osdmap_epoch": 32, "osdmap_epoch_barrier": 31, "rank_uptime": 6.095509948, "state": "up:active", "uptime": 13.325537748, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:42:04.758 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:42:04.759 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:42:04.760 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:42:04.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:04.761 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:42:04.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:42:04.860 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:42:04.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:42:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:42:04.936 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:42:04.937 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:04.940 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:42:04.968 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:42:04.969 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:42:05.017 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:42:05.035 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:42:05.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:42:05.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:42:05.090+0000 7fe48c022080 -1 init, newargv = 0x560a25ed1530 newargc=9 2022-07-12T08:42:05.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21075]: starting ceph client 2022-07-12T08:42:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:42:05.103 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:42:05.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:05.104 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:42:05.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21075]: starting fuse 2022-07-12T08:42:05.112 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:42:05.113 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:42:05.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:05.115 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:42:05.116 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:42:05.116 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:42:05.117 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:42:05.118 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:42:05.118 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.119 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:42:05.120 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:42:05.121 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:42:05.122 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.122 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:42:05.123 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:42:05.124 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:42:05.124 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:42:05.129 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.130 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:42:05.130 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:42:05.131 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:42:05.132 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:42:05.133 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:42:05.133 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.134 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:42:05.135 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:42:05.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21075.asok 2022-07-12T08:42:05.210 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21075.asok 2022-07-12T08:42:05.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:05.212 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21075.asok status 2022-07-12T08:42:05.346 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:42:05.347 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:42:05.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:42:05.348 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:42:05.348 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:42:05.348 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:42:05.349 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:42:05.349 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21075", 2022-07-12T08:42:05.349 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:42:05.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:42:05.350 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:42:05.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:42:05.351 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4850, 2022-07-12T08:42:05.352 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:42:05.352 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:42:05.355 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:42:05.355 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4850 2022-07-12T08:42:05.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:42:05.356 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:42:05.357 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:42:05.357 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:42:05.358 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3754469628 2022-07-12T08:42:05.358 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:42:05.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:42:05.359 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:42:05.360 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:42:05.360 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:42:05.360 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3754469628 2022-07-12T08:42:05.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:42:05.361 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4850 172.21.15.121:0/3754469628", 2022-07-12T08:42:05.364 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/3754469628", 2022-07-12T08:42:05.364 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:42:05.365 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 12, 2022-07-12T08:42:05.365 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 32, 2022-07-12T08:42:05.365 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:42:05.366 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:42:05.367 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:42:05.367 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:42:05.369 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:05.370 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:42:05.371 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:42:05.391 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:42:05.391 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:42:05.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:05.393 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:42:05.465 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_min_caps_per_client 2022-07-12T08:42:05.578 INFO:teuthology.orchestra.run.smithi061.stdout:{"mds_min_caps_per_client":"100"} 2022-07-12T08:42:05.579 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_min_caps_per_client": "100" } 2022-07-12T08:42:05.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:42:05.580 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:42:05.580 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:42:05.593 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:42:05.594 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:42:05.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:42:05.596 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-07-12T08:42:05.597 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-07-12T08:42:05.598 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:42:05.598 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.599 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-07-12T08:42:05.600 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-12T08:42:05.600 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.601 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-12T08:42:05.601 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-07-12T08:42:05.601 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:42:05.602 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-07-12T08:42:05.603 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-07-12T08:42:05.603 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-07-12T08:42:05.603 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-07-12T08:42:05.604 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-07-12T08:42:05.604 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-07-12T08:42:05.605 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-07-12T08:42:05.605 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:42:15.104 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:42:15.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 Processor -- start 2022-07-12T08:42:15.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 -- start start 2022-07-12T08:42:15.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe5580b1180 0x7fe5580b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580b0120 0x7fe5580ae420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5580afbd0 con 0x7fe5580ae910 2022-07-12T08:42:15.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5580acb60 con 0x7fe5580b1180 2022-07-12T08:42:15.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580ace00 con 0x7fe5580b0120 2022-07-12T08:42:15.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe556d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe5580ae910 0x7fe55813ec70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42674/0 (socket says 172.21.15.61:42674) 2022-07-12T08:42:15.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe556d9d700 1 -- 172.21.15.61:0/174806488 learned_addr learned my addr 172.21.15.61:0/174806488 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:42:15.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe555d9b700 1 --2- 172.21.15.61:0/174806488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580b0120 0x7fe5580ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.204+0000 7fe55659c700 1 --2- 172.21.15.61:0/174806488 >> v2:172.21.15.111:3300/0 conn(0x7fe5580b1180 0x7fe5580b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55559a700 1 -- 172.21.15.61:0/174806488 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486318557 0 0) 0x7fe5580afbd0 con 0x7fe5580ae910 2022-07-12T08:42:15.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55559a700 1 -- 172.21.15.61:0/174806488 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c003260 con 0x7fe5580ae910 2022-07-12T08:42:15.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe555d9b700 1 -- 172.21.15.61:0/174806488 >> v2:172.21.15.111:3300/0 conn(0x7fe5580b1180 msgr2=0x7fe5580b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:15.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe555d9b700 1 --2- 172.21.15.61:0/174806488 >> v2:172.21.15.111:3300/0 conn(0x7fe5580b1180 0x7fe5580b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe555d9b700 1 -- 172.21.15.61:0/174806488 >> v1:172.21.15.61:6789/0 conn(0x7fe5580ae910 legacy=0x7fe55813ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe555d9b700 1 -- 172.21.15.61:0/174806488 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580ad340 con 0x7fe5580b0120 2022-07-12T08:42:15.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe555d9b700 1 --2- 172.21.15.61:0/174806488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580b0120 0x7fe5580ae420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe54c002f70 tx=0x7fe54c006df0).ready entity=mon.2 client_cookie=cfe46802f942ff9a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:15.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55559a700 1 -- 172.21.15.61:0/174806488 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c00d4a0 con 0x7fe5580b0120 2022-07-12T08:42:15.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55559a700 1 -- 172.21.15.61:0/174806488 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c000e90 con 0x7fe5580b0120 2022-07-12T08:42:15.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55559a700 1 -- 172.21.15.61:0/174806488 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c00d7f0 con 0x7fe5580b0120 2022-07-12T08:42:15.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 -- 172.21.15.61:0/174806488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580b0120 msgr2=0x7fe5580ae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 --2- 172.21.15.61:0/174806488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580b0120 0x7fe5580ae420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fe54c002f70 tx=0x7fe54c006df0).stop 2022-07-12T08:42:15.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 -- 172.21.15.61:0/174806488 shutdown_connections 2022-07-12T08:42:15.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 --2- 172.21.15.61:0/174806488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580b0120 0x7fe5580ae420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 --2- 172.21.15.61:0/174806488 >> v2:172.21.15.111:3300/0 conn(0x7fe5580b1180 0x7fe5580b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 -- 172.21.15.61:0/174806488 >> 172.21.15.61:0/174806488 conn(0x7fe5580b5780 msgr2=0x7fe5580b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:15.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 -- 172.21.15.61:0/174806488 shutdown_connections 2022-07-12T08:42:15.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 -- 172.21.15.61:0/174806488 wait complete. 2022-07-12T08:42:15.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 Processor -- start 2022-07-12T08:42:15.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.208+0000 7fe55759e700 1 -- start start 2022-07-12T08:42:15.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580ae910 0x7fe558145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 0x7fe55813efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5580b1180 0x7fe55813f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe558112750 con 0x7fe5580b0120 2022-07-12T08:42:15.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581104a0 con 0x7fe5580b1180 2022-07-12T08:42:15.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581124b0 con 0x7fe5580ae910 2022-07-12T08:42:15.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe555d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 0x7fe55813efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe555d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 0x7fe55813efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54460/0 (socket says 172.21.15.61:54460) 2022-07-12T08:42:15.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe555d9b700 1 -- 172.21.15.61:0/1855754483 learned_addr learned my addr 172.21.15.61:0/1855754483 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:15.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580ae910 0x7fe558145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe556d9d700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5580b1180 0x7fe55813f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 -- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5580b1180 msgr2=0x7fe55813f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5580b1180 0x7fe55813f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 -- 172.21.15.61:0/1855754483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 msgr2=0x7fe55813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 0x7fe55813efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 -- 172.21.15.61:0/1855754483 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c00b180 con 0x7fe5580ae910 2022-07-12T08:42:15.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe556d9d700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5580b1180 0x7fe55813f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:42:15.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe555d9b700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 0x7fe55813efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:42:15.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55659c700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580ae910 0x7fe558145340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe54800e000 tx=0x7fe548006e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:15.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54800ec50 con 0x7fe5580ae910 2022-07-12T08:42:15.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54800edb0 con 0x7fe5580ae910 2022-07-12T08:42:15.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe558145b80 con 0x7fe5580ae910 2022-07-12T08:42:15.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54801a660 con 0x7fe5580ae910 2022-07-12T08:42:15.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe558066a80 con 0x7fe5580ae910 2022-07-12T08:42:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.212+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe558112300 con 0x7fe5580ae910 2022-07-12T08:42:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.216+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe54801a7c0 con 0x7fe5580ae910 2022-07-12T08:42:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.216+0000 7fe5477fe700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe534053ad0 0x7fe534055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.216+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe54806fda0 con 0x7fe5580ae910 2022-07-12T08:42:15.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.216+0000 7fe555d9b700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe534053ad0 0x7fe534055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.216+0000 7fe555d9b700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe534053ad0 0x7fe534055f90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe54c01a000 tx=0x7fe54c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:15.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.224+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe548046bf0 con 0x7fe5580ae910 2022-07-12T08:42:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.392+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5580671c0 con 0x7fe5580ae910 2022-07-12T08:42:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe5477fe700 1 -- 172.21.15.61:0/1855754483 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fe54804a210 con 0x7fe5580ae910 2022-07-12T08:42:15.411 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:15.412 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:42:15.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe534053ad0 msgr2=0x7fe534055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe534053ad0 0x7fe534055f90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe54c01a000 tx=0x7fe54c014000).stop 2022-07-12T08:42:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580ae910 msgr2=0x7fe558145340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580ae910 0x7fe558145340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fe54800e000 tx=0x7fe548006e80).stop 2022-07-12T08:42:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 shutdown_connections 2022-07-12T08:42:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe534053ad0 0x7fe534055f90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5580b1180 0x7fe55813f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5580b0120 0x7fe55813efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 --2- 172.21.15.61:0/1855754483 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5580ae910 0x7fe558145340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 >> 172.21.15.61:0/1855754483 conn(0x7fe5580b5780 msgr2=0x7fe5580614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 shutdown_connections 2022-07-12T08:42:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.400+0000 7fe55759e700 1 -- 172.21.15.61:0/1855754483 wait complete. 2022-07-12T08:42:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:42:15.422 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-12T08:42:15.547 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-12T08:42:15.547 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-12T08:42:15.548 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:42:15.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 Processor -- start 2022-07-12T08:42:15.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 -- start start 2022-07-12T08:42:15.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b6770 0x7ff83c146080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff83c0b4720 0x7ff83c146570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83c0b6b50 con 0x7ff83c146a60 2022-07-12T08:42:15.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83c0b4f10 con 0x7ff83c0b4720 2022-07-12T08:42:15.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff8409c9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0b51b0 con 0x7ff83c0b6770 2022-07-12T08:42:15.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff83a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff83c0b4720 0x7ff83c146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff83b59e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff83c146a60 0x7ff83c05bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42688/0 (socket says 172.21.15.61:42688) 2022-07-12T08:42:15.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff83a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff83c0b4720 0x7ff83c146570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54468/0 (socket says 172.21.15.61:54468) 2022-07-12T08:42:15.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff83b59e700 1 -- 172.21.15.61:0/3397084960 learned_addr learned my addr 172.21.15.61:0/3397084960 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:42:15.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.644+0000 7ff83ad9d700 1 --2- 172.21.15.61:0/3397084960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b6770 0x7ff83c146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff83a59c700 1 -- 172.21.15.61:0/3397084960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b6770 msgr2=0x7ff83c146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:15.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff83a59c700 1 --2- 172.21.15.61:0/3397084960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b6770 0x7ff83c146080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff83a59c700 1 -- 172.21.15.61:0/3397084960 >> v1:172.21.15.61:6789/0 conn(0x7ff83c146a60 legacy=0x7ff83c05bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff83a59c700 1 -- 172.21.15.61:0/3397084960 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0b5450 con 0x7ff83c0b4720 2022-07-12T08:42:15.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff839d9b700 1 -- 172.21.15.61:0/3397084960 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810519106 0 0) 0x7ff83c0b6b50 con 0x7ff83c146a60 2022-07-12T08:42:15.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff83a59c700 1 --2- 172.21.15.61:0/3397084960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c0b4720 0x7ff83c146570 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7ff824002f70 tx=0x7ff824006d90).ready entity=mon.0 client_cookie=f90c1284aafb3b0b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:15.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff839d9b700 1 -- 172.21.15.61:0/3397084960 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82400d4a0 con 0x7ff83c0b4720 2022-07-12T08:42:15.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff839d9b700 1 -- 172.21.15.61:0/3397084960 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff824000e90 con 0x7ff83c0b4720 2022-07-12T08:42:15.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff839d9b700 1 -- 172.21.15.61:0/3397084960 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82400d7f0 con 0x7ff83c0b4720 2022-07-12T08:42:15.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 -- 172.21.15.61:0/3397084960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c0b4720 msgr2=0x7ff83c146570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3397084960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c0b4720 0x7ff83c146570 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7ff824002f70 tx=0x7ff824006d90).stop 2022-07-12T08:42:15.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 -- 172.21.15.61:0/3397084960 shutdown_connections 2022-07-12T08:42:15.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3397084960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c0b4720 0x7ff83c146570 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3397084960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b6770 0x7ff83c146080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 -- 172.21.15.61:0/3397084960 >> 172.21.15.61:0/3397084960 conn(0x7ff83c0b9d50 msgr2=0x7ff83c0b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:15.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 -- 172.21.15.61:0/3397084960 shutdown_connections 2022-07-12T08:42:15.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 -- 172.21.15.61:0/3397084960 wait complete. 2022-07-12T08:42:15.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 Processor -- start 2022-07-12T08:42:15.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.648+0000 7ff8409c9700 1 -- start start 2022-07-12T08:42:15.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b4720 0x7ff83c09d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff83c0b6770 0x7ff83c09d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 0x7ff83c09def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c149d90 con 0x7ff83c146a60 2022-07-12T08:42:15.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c0b2600 con 0x7ff83c0b6770 2022-07-12T08:42:15.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83c09ea10 con 0x7ff83c0b4720 2022-07-12T08:42:15.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 0x7ff83c09def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 0x7ff83c09def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54476/0 (socket says 172.21.15.61:54476) 2022-07-12T08:42:15.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 -- 172.21.15.61:0/3034486778 learned_addr learned my addr 172.21.15.61:0/3034486778 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:15.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83a59c700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff83c0b6770 0x7ff83c09d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83ad9d700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b4720 0x7ff83c09d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 -- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b4720 msgr2=0x7ff83c09d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b4720 0x7ff83c09d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 -- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff83c0b6770 msgr2=0x7ff83c09d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff83c0b6770 0x7ff83c09d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 -- 172.21.15.61:0/3034486778 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82400b120 con 0x7ff83c146a60 2022-07-12T08:42:15.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83a59c700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff83c0b6770 0x7ff83c09d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:15.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83ad9d700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b4720 0x7ff83c09d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:15.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff83b59e700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 0x7ff83c09def0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7ff82c000fe0 tx=0x7ff82c007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:15.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c00ce00 con 0x7ff83c146a60 2022-07-12T08:42:15.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c0072e0 con 0x7ff83c146a60 2022-07-12T08:42:15.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83c09ecb0 con 0x7ff83c146a60 2022-07-12T08:42:15.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c005800 con 0x7ff83c146a60 2022-07-12T08:42:15.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83c147470 con 0x7ff83c146a60 2022-07-12T08:42:15.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83c09ee40 con 0x7ff83c146a60 2022-07-12T08:42:15.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.652+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff82c01a060 con 0x7ff83c146a60 2022-07-12T08:42:15.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.656+0000 7ff833fff700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff820053a40 0x7ff820055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:15.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.656+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff82c06b6c0 con 0x7ff83c146a60 2022-07-12T08:42:15.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.656+0000 7ff83ad9d700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff820053a40 0x7ff820055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:15.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.656+0000 7ff83ad9d700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff820053a40 0x7ff820055f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff83400b000 tx=0x7ff834008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:15.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.660+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff82c042620 con 0x7ff83c146a60 2022-07-12T08:42:15.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.824+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff83c057460 con 0x7ff83c146a60 2022-07-12T08:42:15.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.824+0000 7ff833fff700 1 -- 172.21.15.61:0/3034486778 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7ff82c018020 con 0x7ff83c146a60 2022-07-12T08:42:15.836 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:15.836 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:42:15.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff820053a40 msgr2=0x7ff820055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff820053a40 0x7ff820055f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff83400b000 tx=0x7ff834008040).stop 2022-07-12T08:42:15.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 msgr2=0x7ff83c09def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:15.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 0x7ff83c09def0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7ff82c000fe0 tx=0x7ff82c007bc0).stop 2022-07-12T08:42:15.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 shutdown_connections 2022-07-12T08:42:15.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff83c146a60 0x7ff83c09def0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff820053a40 0x7ff820055f00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff83c0b6770 0x7ff83c09d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 --2- 172.21.15.61:0/3034486778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff83c0b4720 0x7ff83c09d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:15.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 >> 172.21.15.61:0/3034486778 conn(0x7ff83c0b9d50 msgr2=0x7ff83c143290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:15.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 shutdown_connections 2022-07-12T08:42:15.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:15.828+0000 7ff8409c9700 1 -- 172.21.15.61:0/3034486778 wait complete. 2022-07-12T08:42:15.850 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:42:15.852 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-12T08:42:15.967 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-12T08:42:15.969 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-12T08:42:15.970 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:42:16.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.076+0000 7f25a64b5700 1 Processor -- start 2022-07-12T08:42:16.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.076+0000 7f25a64b5700 1 -- start start 2022-07-12T08:42:16.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 0x7f25a005f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 0x7f25a00afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a00abc70 con 0x7f25a00ab290 2022-07-12T08:42:16.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a00aba90 con 0x7f25a00afa60 2022-07-12T08:42:16.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a005c720 con 0x7f25a00ab670 2022-07-12T08:42:16.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a5cb4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 0x7f25a00afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 0x7f25a005f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a5cb4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 0x7f25a00afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33058/0 (socket says 172.21.15.61:33058) 2022-07-12T08:42:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a5cb4700 1 -- 172.21.15.61:0/834402617 learned_addr learned my addr 172.21.15.61:0/834402617 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 0x7f25a005f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36678/0 (socket says 172.21.15.61:36678) 2022-07-12T08:42:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 -- 172.21.15.61:0/834402617 >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 msgr2=0x7f25a00afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 --2- 172.21.15.61:0/834402617 >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 0x7f25a00afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 -- 172.21.15.61:0/834402617 >> v1:172.21.15.111:6789/0 conn(0x7f25a00ab290 legacy=0x7f25a005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 -- 172.21.15.61:0/834402617 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a005c9c0 con 0x7f25a00ab670 2022-07-12T08:42:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a5cb4700 1 --2- 172.21.15.61:0/834402617 >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 0x7f25a00afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:42:16.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a4cb2700 1 --2- 172.21.15.61:0/834402617 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 0x7f25a005f140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f259c0032f0 tx=0x7f259c0091d0).ready entity=mon.2 client_cookie=33edac3da1d7fad7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f2597fff700 1 -- 172.21.15.61:0/834402617 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259c00b6c0 con 0x7f25a00ab670 2022-07-12T08:42:16.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f2597fff700 1 -- 172.21.15.61:0/834402617 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c00b820 con 0x7f25a00ab670 2022-07-12T08:42:16.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f2597fff700 1 -- 172.21.15.61:0/834402617 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259c00b990 con 0x7f25a00ab670 2022-07-12T08:42:16.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- 172.21.15.61:0/834402617 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 msgr2=0x7f25a005f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 --2- 172.21.15.61:0/834402617 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 0x7f25a005f140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f259c0032f0 tx=0x7f259c0091d0).stop 2022-07-12T08:42:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- 172.21.15.61:0/834402617 shutdown_connections 2022-07-12T08:42:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 --2- 172.21.15.61:0/834402617 >> v2:172.21.15.111:3300/0 conn(0x7f25a00afa60 0x7f25a00afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 --2- 172.21.15.61:0/834402617 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00ab670 0x7f25a005f140 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- 172.21.15.61:0/834402617 >> 172.21.15.61:0/834402617 conn(0x7f25a00b1950 msgr2=0x7f25a00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:16.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- 172.21.15.61:0/834402617 shutdown_connections 2022-07-12T08:42:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- 172.21.15.61:0/834402617 wait complete. 2022-07-12T08:42:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 Processor -- start 2022-07-12T08:42:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.080+0000 7f25a64b5700 1 -- start start 2022-07-12T08:42:16.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a00ab290 0x7f25a00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 0x7f25a00673b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00afa60 0x7f25a0159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00aed40 con 0x7f25a00ab670 2022-07-12T08:42:16.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00ae420 con 0x7f25a00ab290 2022-07-12T08:42:16.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a00aef70 con 0x7f25a00afa60 2022-07-12T08:42:16.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 0x7f25a00673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 0x7f25a00673b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54488/0 (socket says 172.21.15.61:54488) 2022-07-12T08:42:16.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 -- 172.21.15.61:0/1460695616 learned_addr learned my addr 172.21.15.61:0/1460695616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a54b3700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a00ab290 0x7f25a00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a5cb4700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00afa60 0x7f25a0159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 -- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00afa60 msgr2=0x7f25a0159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00afa60 0x7f25a0159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 -- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a00ab290 msgr2=0x7f25a00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a00ab290 0x7f25a00b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 -- 172.21.15.61:0/1460695616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259800b1e0 con 0x7f25a00ab670 2022-07-12T08:42:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a5cb4700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00afa60 0x7f25a0159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a4cb2700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 0x7f25a00673b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f259c0032f0 tx=0x7f259c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259c00b6c0 con 0x7f25a00ab670 2022-07-12T08:42:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c009470 con 0x7f25a00ab670 2022-07-12T08:42:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a0066b10 con 0x7f25a00ab670 2022-07-12T08:42:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c021630 con 0x7f25a00ab670 2022-07-12T08:42:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f259c021990 con 0x7f25a00ab670 2022-07-12T08:42:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.084+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a0066e20 con 0x7f25a00ab670 2022-07-12T08:42:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.088+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f259c029070 con 0x7f25a00ab670 2022-07-12T08:42:16.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.088+0000 7f25967fc700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2588053a60 0x7f2588055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.088+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f259c06d440 con 0x7f25a00ab670 2022-07-12T08:42:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.088+0000 7f25a54b3700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2588053a60 0x7f2588055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.088+0000 7f25a54b3700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2588053a60 0x7f2588055f20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f259000d000 tx=0x7f2590000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.096+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f259c0443a0 con 0x7f25a00ab670 2022-07-12T08:42:16.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.264+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a0057460 con 0x7f25a00ab670 2022-07-12T08:42:16.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.264+0000 7f25967fc700 1 -- 172.21.15.61:0/1460695616 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f259c027090 con 0x7f25a00ab670 2022-07-12T08:42:16.277 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:42:16.282 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:42:16.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2588053a60 msgr2=0x7f2588055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2588053a60 0x7f2588055f20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f259000d000 tx=0x7f2590000bc0).stop 2022-07-12T08:42:16.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 msgr2=0x7f25a00673b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 0x7f25a00673b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f259c0032f0 tx=0x7f259c016000).stop 2022-07-12T08:42:16.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 shutdown_connections 2022-07-12T08:42:16.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f25a00afa60 0x7f25a0159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25a00ab670 0x7f25a00673b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2588053a60 0x7f2588055f20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 --2- 172.21.15.61:0/1460695616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a00ab290 0x7f25a00b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 >> 172.21.15.61:0/1460695616 conn(0x7f25a00b1950 msgr2=0x7f25a014d870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:16.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 shutdown_connections 2022-07-12T08:42:16.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.268+0000 7f25a64b5700 1 -- 172.21.15.61:0/1460695616 wait complete. 2022-07-12T08:42:16.306 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:42:16.310 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 2022-07-12T08:42:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 Processor -- start 2022-07-12T08:42:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 -- start start 2022-07-12T08:42:16.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43681471d0 0x7f4368140750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 0x7f43680b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43681465c0 con 0x7f43680b9f70 2022-07-12T08:42:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43680ba350 con 0x7f43681471d0 2022-07-12T08:42:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436e2a2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43681467f0 con 0x7f43680b5a20 2022-07-12T08:42:16.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436c83f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 0x7f43680b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43681471d0 0x7f4368140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43681471d0 0x7f4368140750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33070/0 (socket says 172.21.15.61:33070) 2022-07-12T08:42:16.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 -- 172.21.15.61:0/2500596322 learned_addr learned my addr 172.21.15.61:0/2500596322 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436c83f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 0x7f43680b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36694/0 (socket says 172.21.15.61:36694) 2022-07-12T08:42:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 -- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 msgr2=0x7f43680b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 --2- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 0x7f43680b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 -- 172.21.15.61:0/2500596322 >> v1:172.21.15.111:6789/0 conn(0x7f43680b9f70 legacy=0x7f4368140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 -- 172.21.15.61:0/2500596322 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4368146a90 con 0x7f43681471d0 2022-07-12T08:42:16.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f436c83f700 1 --2- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 0x7f43680b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:42:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.428+0000 7f43677fe700 1 --2- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43681471d0 0x7f4368140750 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4358002f70 tx=0x7f4358006eb0).ready entity=mon.1 client_cookie=e506682a7b2b98d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f4366ffd700 1 -- 172.21.15.61:0/2500596322 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f435800d570 con 0x7f43681471d0 2022-07-12T08:42:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f4366ffd700 1 -- 172.21.15.61:0/2500596322 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358000e90 con 0x7f43681471d0 2022-07-12T08:42:16.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f4366ffd700 1 -- 172.21.15.61:0/2500596322 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f435800d900 con 0x7f43681471d0 2022-07-12T08:42:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43681471d0 msgr2=0x7f4368140750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43681471d0 0x7f4368140750 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f4358002f70 tx=0x7f4358006eb0).stop 2022-07-12T08:42:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- 172.21.15.61:0/2500596322 shutdown_connections 2022-07-12T08:42:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43680b5a20 0x7f43680b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2500596322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43681471d0 0x7f4368140750 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- 172.21.15.61:0/2500596322 >> 172.21.15.61:0/2500596322 conn(0x7f4368067910 msgr2=0x7f43680b9520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:16.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- 172.21.15.61:0/2500596322 shutdown_connections 2022-07-12T08:42:16.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- 172.21.15.61:0/2500596322 wait complete. 2022-07-12T08:42:16.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 Processor -- start 2022-07-12T08:42:16.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- start start 2022-07-12T08:42:16.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43680b5a20 0x7f43680afe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 0x7f43680ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43681471d0 0x7f4368125ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4368152070 con 0x7f43680b9f70 2022-07-12T08:42:16.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43680b7060 con 0x7f43680b5a20 2022-07-12T08:42:16.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43680b4370 con 0x7f43681471d0 2022-07-12T08:42:16.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 0x7f43680ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 0x7f43680ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54502/0 (socket says 172.21.15.61:54502) 2022-07-12T08:42:16.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 -- 172.21.15.61:0/2117527425 learned_addr learned my addr 172.21.15.61:0/2117527425 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f4367fff700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43680b5a20 0x7f43680afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436c83f700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43681471d0 0x7f4368125ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 -- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43681471d0 msgr2=0x7f4368125ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43681471d0 0x7f4368125ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 -- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43680b5a20 msgr2=0x7f43680afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43680b5a20 0x7f43680afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 -- 172.21.15.61:0/2117527425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c00b180 con 0x7f43680b9f70 2022-07-12T08:42:16.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43677fe700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 0x7f43680ae010 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4358002f70 tx=0x7f4358015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f43657fa700 1 -- 172.21.15.61:0/2117527425 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4358006b40 con 0x7f43680b9f70 2022-07-12T08:42:16.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.432+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435800b1e0 con 0x7f43680b9f70 2022-07-12T08:42:16.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43657fa700 1 -- 172.21.15.61:0/2117527425 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435800d8f0 con 0x7f43680b9f70 2022-07-12T08:42:16.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43657fa700 1 -- 172.21.15.61:0/2117527425 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4358011d70 con 0x7f43680b9f70 2022-07-12T08:42:16.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43680b3df0 con 0x7f43680b9f70 2022-07-12T08:42:16.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43657fa700 1 -- 172.21.15.61:0/2117527425 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f435801a040 con 0x7f43680b9f70 2022-07-12T08:42:16.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43657fa700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43480539b0 0x7f4348055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43657fa700 1 -- 172.21.15.61:0/2117527425 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f435806cc00 con 0x7f43680b9f70 2022-07-12T08:42:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f4367fff700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43480539b0 0x7f4348055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 Processor -- start 2022-07-12T08:42:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 -- start start 2022-07-12T08:42:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f4367fff700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43480539b0 0x7f4348055e70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f436000d000 tx=0x7f4360000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b5a00 0x207afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x207c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x1febdf0 con 0x20b10e0 2022-07-12T08:42:16.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1fcc3e0 con 0x207c3c0 2022-07-12T08:42:16.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x2078720 con 0x20b5a00 2022-07-12T08:42:16.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54510/0 (socket says 172.21.15.61:54510) 2022-07-12T08:42:16.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f43677fe700 1 -- 172.21.15.61:0/3363640290 learned_addr learned my addr 172.21.15.61:0/3363640290 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f436c83f700 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x207c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.436+0000 7f4367fff700 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b5a00 0x207afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 -- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b5a00 msgr2=0x207afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b5a00 0x207afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 -- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 msgr2=0x207c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x207c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 -- 172.21.15.61:0/3363640290 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2023a60 con 0x20b10e0 2022-07-12T08:42:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20b5de0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x1fcf6e0 tx=0x7f4358011000).ready entity=mon.0 client_cookie=bef18145c6f90149 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436e2a2700 1 -- 172.21.15.61:0/3363640290 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4358047720 con 0x20b10e0 2022-07-12T08:42:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436e2a2700 1 -- 172.21.15.61:0/3363640290 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358045b20 con 0x20b10e0 2022-07-12T08:42:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436e2a2700 1 -- 172.21.15.61:0/3363640290 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43580468d0 con 0x20b10e0 2022-07-12T08:42:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- 172.21.15.61:0/3363640290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 msgr2=0x20b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20b5de0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x1fcf6e0 tx=0x7f4358011000).stop 2022-07-12T08:42:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- 172.21.15.61:0/3363640290 shutdown_connections 2022-07-12T08:42:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x207c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20b5de0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- 172.21.15.61:0/3363640290 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b5a00 0x207afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- 172.21.15.61:0/3363640290 >> 172.21.15.61:0/3363640290 conn(0x2003bf0 msgr2=0x20b52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- 172.21.15.61:0/3363640290 shutdown_connections 2022-07-12T08:42:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- 172.21.15.61:0/3363640290 wait complete. 2022-07-12T08:42:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- start start 2022-07-12T08:42:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 Processor -- start 2022-07-12T08:42:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x20b2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b10e0 0x20b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 0x20c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2083d60 con 0x20b5a00 2022-07-12T08:42:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x20c01c0 con 0x207c3c0 2022-07-12T08:42:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x20c0460 con 0x20b10e0 2022-07-12T08:42:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43784f9740 10 client.0 ll_register_callbacks cb 0x2033040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:42:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 0x20c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 0x20c0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54518/0 (socket says 172.21.15.61:54518) 2022-07-12T08:42:16.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 -- 172.21.15.61:0/3783837105 learned_addr learned my addr 172.21.15.61:0/3783837105 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f4367fff700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x20b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f43677fe700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b10e0 0x20b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 -- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b10e0 msgr2=0x20b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b10e0 0x20b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 -- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 msgr2=0x20b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x20b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 -- 172.21.15.61:0/3783837105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43680b3df0 con 0x20b5a00 2022-07-12T08:42:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f4367fff700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x20b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f436c83f700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 0x20c0900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f435c00b000 tx=0x7f435c008040).ready entity=mon.0 client_cookie=70d01f281c1c3586 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.440+0000 7f4346ffd700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:42:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4346ffd700 1 -- 172.21.15.61:0/3783837105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f435c013e50 con 0x20b5a00 2022-07-12T08:42:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4346ffd700 1 -- 172.21.15.61:0/3783837105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f435c0119c0 con 0x20b5a00 2022-07-12T08:42:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1fe1e60 con 0x20b5a00 2022-07-12T08:42:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4346ffd700 1 -- 172.21.15.61:0/3783837105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f435c011b80 con 0x20b5a00 2022-07-12T08:42:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 10 client.5118 fetch_fsmap learned FSMap version 12 2022-07-12T08:42:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x1fdb740 con 0x20b5a00 2022-07-12T08:42:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4346ffd700 1 -- 172.21.15.61:0/3783837105 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1fe1e60 con 0x20b5a00 2022-07-12T08:42:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4346ffd700 1 -- 172.21.15.61:0/3783837105 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 12) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f435c011ce0 con 0x20b5a00 2022-07-12T08:42:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 10 client.5118 fetch_fsmap finished waiting for FSMap version 12 2022-07-12T08:42:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 10 client.5118 resolve_mds: resolved ID 'a' to GID 4905 2022-07-12T08:42:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 20 client.5118 populate_metadata read hostname 'smithi061' 2022-07-12T08:42:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20b4aa0 0x20b4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 4 client.5118 mds_command: new command op to 4905 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-12T08:42:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 --> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1fda1d0 con 0x20b4aa0 2022-07-12T08:42:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4367fff700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20b4aa0 0x20b4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4346ffd700 10 client.5118 ms_handle_connect on v2:172.21.15.61:6832/3202017529 2022-07-12T08:42:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.444+0000 7f4367fff700 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20b4aa0 0x20b4e80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=7884b985414eb4f5 server_cookie=5973c23a97db76c3 in_seq=0 out_seq=0 2022-07-12T08:42:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f4346ffd700 1 -- 172.21.15.61:0/3783837105 <== mds.0 v2:172.21.15.61:6832/3202017529 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x1fda1d0 con 0x20b4aa0 2022-07-12T08:42:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f4346ffd700 10 client.5118 handle_command_reply: tid=0 2022-07-12T08:42:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 client.5118 shutdown 2022-07-12T08:42:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 msgr2=0x20c0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 0x20c0900 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f435c00b000 tx=0x7f435c008040).stop 2022-07-12T08:42:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 shutdown_connections 2022-07-12T08:42:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20b4aa0 0x20b4e80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b5a00 0x20c0900 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x20b10e0 0x20b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 --2- 172.21.15.61:0/3783837105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x207c3c0 0x20b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f4346ffd700 0 client.5118 ms_handle_reset on v2:172.21.15.61:6832/3202017529 2022-07-12T08:42:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 >> 172.21.15.61:0/3783837105 conn(0x2003bf0 msgr2=0x20b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 shutdown_connections 2022-07-12T08:42:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 1 -- 172.21.15.61:0/3783837105 wait complete. 2022-07-12T08:42:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.452+0000 7f43784f9740 20 client.5118 trim_cache size 0 max 16384 2022-07-12T08:42:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 Processor -- start 2022-07-12T08:42:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 -- start start 2022-07-12T08:42:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x2078d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20813e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x2082dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x1fd67d0 con 0x20b10e0 2022-07-12T08:42:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x20bff40 con 0x20b5a00 2022-07-12T08:42:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x1fda1d0 con 0x207c3c0 2022-07-12T08:42:16.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20813e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20813e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54522/0 (socket says 172.21.15.61:54522) 2022-07-12T08:42:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 -- 172.21.15.61:0/1267511012 learned_addr learned my addr 172.21.15.61:0/1267511012 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f436c83f700 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x2082dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f4367fff700 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x2078d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 -- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 msgr2=0x2078d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x2078d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 -- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 msgr2=0x2082dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x2082dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 -- 172.21.15.61:0/1267511012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1fe4040 con 0x20b10e0 2022-07-12T08:42:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43677fe700 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20813e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x20bdc90 tx=0x7f4358023000).ready entity=mon.0 client_cookie=de5113c1408f2ef7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f434d7fa700 1 -- 172.21.15.61:0/1267511012 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43580468d0 con 0x20b10e0 2022-07-12T08:42:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f434d7fa700 1 -- 172.21.15.61:0/1267511012 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358046a30 con 0x20b10e0 2022-07-12T08:42:16.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f434d7fa700 1 -- 172.21.15.61:0/1267511012 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f435806c630 con 0x20b10e0 2022-07-12T08:42:16.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 -- 172.21.15.61:0/1267511012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 msgr2=0x20813e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:42:16.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.464+0000 7f43784f9740 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20813e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x20bdc90 tx=0x7f4358023000).stop 2022-07-12T08:42:16.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- 172.21.15.61:0/1267511012 shutdown_connections 2022-07-12T08:42:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x2082dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x20813e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 --2- 172.21.15.61:0/1267511012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x2078d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- 172.21.15.61:0/1267511012 >> 172.21.15.61:0/1267511012 conn(0x2003bf0 msgr2=0x2077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:42:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- 172.21.15.61:0/1267511012 shutdown_connections 2022-07-12T08:42:16.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- 172.21.15.61:0/1267511012 wait complete. 2022-07-12T08:42:16.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- start start 2022-07-12T08:42:16.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 Processor -- start 2022-07-12T08:42:16.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x20770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x2077590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x21136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2077a80 con 0x20b10e0 2022-07-12T08:42:16.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x20d0740 con 0x20b5a00 2022-07-12T08:42:16.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x20b8480 con 0x207c3c0 2022-07-12T08:42:16.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43784f9740 10 client.0 ll_register_callbacks cb 0x2082d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:42:16.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x2077590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x2077590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54530/0 (socket says 172.21.15.61:54530) 2022-07-12T08:42:16.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 -- 172.21.15.61:0/2259065623 learned_addr learned my addr 172.21.15.61:0/2259065623 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:42:16.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f4367fff700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x20770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f436c83f700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x21136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 -- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 msgr2=0x20770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x20770a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 -- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 msgr2=0x21136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:42:16.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x21136c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:42:16.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 -- 172.21.15.61:0/2259065623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4358034040 con 0x20b10e0 2022-07-12T08:42:16.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f436c83f700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x21136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:42:16.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f43677fe700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x2077590 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x20bdc90 tx=0x7f4358037000).ready entity=mon.0 client_cookie=a4a06120a8126af3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:42:16.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f4347fff700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:42:16.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.468+0000 7f4347fff700 1 -- 172.21.15.61:0/2259065623 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4358006440 con 0x20b10e0 2022-07-12T08:42:16.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4347fff700 1 -- 172.21.15.61:0/2259065623 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4358047720 con 0x20b10e0 2022-07-12T08:42:16.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4358045b20 con 0x20b10e0 2022-07-12T08:42:16.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4347fff700 1 -- 172.21.15.61:0/2259065623 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43580065a0 con 0x20b10e0 2022-07-12T08:42:16.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 10 client.5124 fetch_fsmap learned FSMap version 12 2022-07-12T08:42:16.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x1fe0a30 con 0x20b10e0 2022-07-12T08:42:16.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4347fff700 1 -- 172.21.15.61:0/2259065623 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358045b20 con 0x20b10e0 2022-07-12T08:42:16.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4347fff700 1 -- 172.21.15.61:0/2259065623 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 12) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f435801b030 con 0x20b10e0 2022-07-12T08:42:16.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 10 client.5124 fetch_fsmap finished waiting for FSMap version 12 2022-07-12T08:42:16.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 10 client.5124 resolve_mds: resolved ID 'a' to GID 4905 2022-07-12T08:42:16.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 20 client.5124 populate_metadata read hostname 'smithi061' 2022-07-12T08:42:16.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20803d0 0x2113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:42:16.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 4 client.5124 mds_command: new command op to 4905 tid=0[{"prefix": "cache drop"}] 2022-07-12T08:42:16.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 --> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x1ecf590 con 0x20803d0 2022-07-12T08:42:16.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4367fff700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20803d0 0x2113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:42:16.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4347fff700 10 client.5124 ms_handle_connect on v2:172.21.15.61:6832/3202017529 2022-07-12T08:42:16.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:42:16.472+0000 7f4367fff700 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20803d0 0x2113bb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=b6aeb06eb2ca95aa server_cookie=a676f93bfff07749 in_seq=0 out_seq=0 2022-07-12T08:42:31.440 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:31.446 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:31.449 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:31.458 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:42:31.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:00.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f4347fff700 1 -- 172.21.15.61:0/2259065623 <== mds.0 v2:172.21.15.61:6832/3202017529 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+386 (crc 0 0 0) 0x1ecf590 con 0x20803d0 2022-07-12T08:43:00.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f4347fff700 10 client.5124 handle_command_reply: tid=0 2022-07-12T08:43:00.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 client.5124 shutdown 2022-07-12T08:43:00.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 msgr2=0x2077590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:00.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x2077590 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x20bdc90 tx=0x7f4358037000).stop 2022-07-12T08:43:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 shutdown_connections 2022-07-12T08:43:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:6832/3202017529,v1:172.21.15.61:6833/3202017529] conn(0x20803d0 0x2113bb0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x20b5a00 0x21136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x20b10e0 0x2077590 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 --2- 172.21.15.61:0/2259065623 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x207c3c0 0x20770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f4347fff700 0 client.5124 ms_handle_reset on v2:172.21.15.61:6832/3202017529 2022-07-12T08:43:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 >> 172.21.15.61:0/2259065623 conn(0x2003bf0 msgr2=0x2077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 shutdown_connections 2022-07-12T08:43:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 1 -- 172.21.15.61:0/2259065623 wait complete. 2022-07-12T08:43:00.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f43784f9740 20 client.5124 trim_cache size 0 max 16384 2022-07-12T08:43:00.821 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:43:00.822 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-12T08:43:00.822 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-12T08:43:00.822 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(0) Success", 2022-07-12T08:43:00.822 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 920 2022-07-12T08:43:00.822 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:00.823 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-12T08:43:00.823 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-12T08:43:00.823 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-12T08:43:00.823 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:00.824 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-12T08:43:00.824 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 900 2022-07-12T08:43:00.824 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:00.824 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-12T08:43:00.825 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-12T08:43:00.825 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 2507, 2022-07-12T08:43:00.825 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 431776 2022-07-12T08:43:00.825 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:00.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:00.826 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 44.336576923000003 2022-07-12T08:43:00.826 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:43:00.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43480539b0 msgr2=0x7f4348055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:00.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43480539b0 0x7f4348055e70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f436000d000 tx=0x7f4360000f40).stop 2022-07-12T08:43:00.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 msgr2=0x7f43680ae010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:00.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 0x7f43680ae010 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4358002f70 tx=0x7f4358015000).stop 2022-07-12T08:43:00.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 shutdown_connections 2022-07-12T08:43:00.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43681471d0 0x7f4368125ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43680b9f70 0x7f43680ae010 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43480539b0 0x7f4348055e70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 --2- 172.21.15.61:0/2117527425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43680b5a20 0x7f43680afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:00.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.808+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 >> 172.21.15.61:0/2117527425 conn(0x7f4368067910 msgr2=0x7f4368145680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:00.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.812+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 shutdown_connections 2022-07-12T08:43:00.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:00.812+0000 7f436e2a2700 1 -- 172.21.15.61:0/2117527425 wait complete. 2022-07-12T08:43:00.833 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:43:00.834 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:00.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:00.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:00.920 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:00.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:00.989 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:43:00.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:00.991 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:00.991 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:01.000 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:43:01.001 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:43:01.002 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:43:01.002 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:43:01.003 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:01.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21075]: fuse finished with error 0 and tester_r 0 2022-07-12T08:43:01.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:01.029 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:01.029 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:01.085 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:43:01.086 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:43:01.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:43:01.169 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:01.191 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:01.192 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:01.193 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:43:01.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:01.216 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:43:01.217 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:43:01.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:43:01.219 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:43:01.228 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:43:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:01.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:01.231 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:01.232 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:43:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:01.250 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:43:01.251 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:43:01.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:43:01.252 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:43:01.261 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:43:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:01.264 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:43:01.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 Processor -- start 2022-07-12T08:43:01.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- start start 2022-07-12T08:43:01.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b2300 0x7fa5e80b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:01.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5e813fb30 0x7fa5e81434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:01.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80aeec0 con 0x7fa5e80b2c10 2022-07-12T08:43:01.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80af150 con 0x7fa5e813fb30 2022-07-12T08:43:01.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80b2960 con 0x7fa5e80b2300 2022-07-12T08:43:01.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ee1eb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5e813fb30 0x7fa5e81434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:01.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b2300 0x7fa5e80b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:01.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b2300 0x7fa5e80b2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33114/0 (socket says 172.21.15.61:33114) 2022-07-12T08:43:01.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/1056255437 learned_addr learned my addr 172.21.15.61:0/1056255437 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:01.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/1056255437 >> v2:172.21.15.61:3300/0 conn(0x7fa5e813fb30 msgr2=0x7fa5e81434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:01.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 --2- 172.21.15.61:0/1056255437 >> v2:172.21.15.61:3300/0 conn(0x7fa5e813fb30 0x7fa5e81434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:01.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/1056255437 >> v1:172.21.15.111:6789/0 conn(0x7fa5e80b2c10 legacy=0x7fa5e8139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:01.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/1056255437 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8139f20 con 0x7fa5e80b2300 2022-07-12T08:43:01.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ec9e8700 1 -- 172.21.15.61:0/1056255437 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864428755 0 0) 0x7fa5e80aeec0 con 0x7fa5e80b2c10 2022-07-12T08:43:01.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ed1e9700 1 --2- 172.21.15.61:0/1056255437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b2300 0x7fa5e80b2ff0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa5d80032f0 tx=0x7fa5d8009290).ready entity=mon.1 client_cookie=fbbb5b6323f0299d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:01.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ec9e8700 1 -- 172.21.15.61:0/1056255437 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d800b870 con 0x7fa5e80b2300 2022-07-12T08:43:01.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ec9e8700 1 -- 172.21.15.61:0/1056255437 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d800b9d0 con 0x7fa5e80b2300 2022-07-12T08:43:01.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5ec9e8700 1 -- 172.21.15.61:0/1056255437 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d800bb40 con 0x7fa5e80b2300 2022-07-12T08:43:01.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- 172.21.15.61:0/1056255437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b2300 msgr2=0x7fa5e80b2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:01.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/1056255437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b2300 0x7fa5e80b2ff0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa5d80032f0 tx=0x7fa5d8009290).stop 2022-07-12T08:43:01.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- 172.21.15.61:0/1056255437 shutdown_connections 2022-07-12T08:43:01.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/1056255437 >> v2:172.21.15.61:3300/0 conn(0x7fa5e813fb30 0x7fa5e81434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:01.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/1056255437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b2300 0x7fa5e80b2ff0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:01.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- 172.21.15.61:0/1056255437 >> 172.21.15.61:0/1056255437 conn(0x7fa5e8063dc0 msgr2=0x7fa5e80641a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:01.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- 172.21.15.61:0/1056255437 shutdown_connections 2022-07-12T08:43:01.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.376+0000 7fa5efc4e700 1 -- 172.21.15.61:0/1056255437 wait complete. 2022-07-12T08:43:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 Processor -- start 2022-07-12T08:43:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 -- start start 2022-07-12T08:43:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 0x7fa5e8141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 0x7fa5e80b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:01.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e813fb30 0x7fa5e8065930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:01.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e812b430 con 0x7fa5e80b2300 2022-07-12T08:43:01.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e812b660 con 0x7fa5e813fb30 2022-07-12T08:43:01.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5efc4e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e812b900 con 0x7fa5e80b2c10 2022-07-12T08:43:01.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed9ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 0x7fa5e8141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:01.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 0x7fa5e80b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:01.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 0x7fa5e80b4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36740/0 (socket says 172.21.15.61:36740) 2022-07-12T08:43:01.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed9ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 0x7fa5e8141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54546/0 (socket says 172.21.15.61:54546) 2022-07-12T08:43:01.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/228790925 learned_addr learned my addr 172.21.15.61:0/228790925 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:01.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ee1eb700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e813fb30 0x7fa5e8065930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:01.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e813fb30 msgr2=0x7fa5e8065930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:01.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e813fb30 0x7fa5e8065930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:01.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/228790925 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 msgr2=0x7fa5e8141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:01.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 0x7fa5e8141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:01.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed1e9700 1 -- 172.21.15.61:0/228790925 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d8009530 con 0x7fa5e80b2c10 2022-07-12T08:43:01.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.380+0000 7fa5ed9ea700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 0x7fa5e8141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:01.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5ed1e9700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 0x7fa5e80b4c00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa5e8142850 tx=0x7fa5d8016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:01.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d800bb40 con 0x7fa5e80b2c10 2022-07-12T08:43:01.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8021630 con 0x7fa5e80b2c10 2022-07-12T08:43:01.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e80b50f0 con 0x7fa5e80b2c10 2022-07-12T08:43:01.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d800b870 con 0x7fa5e80b2c10 2022-07-12T08:43:01.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e8065e20 con 0x7fa5e80b2c10 2022-07-12T08:43:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e812bac0 con 0x7fa5e80b2c10 2022-07-12T08:43:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa5d8027020 con 0x7fa5e80b2c10 2022-07-12T08:43:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5e6ffd700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a60 0x7fa5cc055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:01.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.384+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5d806d220 con 0x7fa5e80b2c10 2022-07-12T08:43:01.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.388+0000 7fa5ed9ea700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a60 0x7fa5cc055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:01.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.388+0000 7fa5ed9ea700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a60 0x7fa5cc055f20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa5d000d000 tx=0x7fa5d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:01.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.396+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5d80440e0 con 0x7fa5e80b2c10 2022-07-12T08:43:01.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.528+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa5e8057460 con 0x7fa5e80b2c10 2022-07-12T08:43:01.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.528+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/228790925 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v32) v1 ==== 81+0+33693 (secure 0 0 0) 0x7fa5d8015020 con 0x7fa5e80b2c10 2022-07-12T08:43:01.541 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:01.542 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:43:01.551 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 32, 2022-07-12T08:43:01.552 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:43:01.552 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:43:01.553 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:42:01.061999+0000", 2022-07-12T08:43:01.553 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:43:01.554 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:43:01.554 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:43:01.554 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:43:01.555 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:43:01.555 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:43:01.556 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:43:01.556 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:43:01.557 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:43:01.557 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:01.557 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:43:01.558 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:43:01.558 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:43:01.559 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:43:01.559 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:43:01.559 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 6, 2022-07-12T08:43:01.560 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:43:01.560 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:43:01.561 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:43:01.561 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:43:01.562 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:43:01.562 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.562 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:43:01.563 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:43:01.563 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:43:01.564 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:43:01.564 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:43:01.565 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:43:01.565 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:43:01.565 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:43:01.566 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:43:01.566 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:43:01.567 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:43:01.568 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:43:01.569 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:43:01.569 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:43:01.570 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:43:01.570 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:43:01.570 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:43:01.571 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:43:01.571 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:43:01.572 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:43:01.572 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:43:01.573 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:43:01.574 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:43:01.574 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.575 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:43:01.575 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:43:01.576 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:43:01.576 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:43:01.577 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:43:01.577 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:43:01.578 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:43:01.578 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:43:01.579 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:43:01.579 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:43:01.580 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:43:01.580 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:43:01.580 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:43:01.581 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:43:01.581 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:43:01.582 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:43:01.582 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:43:01.583 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:43:01.583 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:43:01.584 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:43:01.584 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:43:01.584 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:43:01.586 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:43:01.586 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:43:01.587 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:43:01.587 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:43:01.587 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:43:01.588 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.588 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:43:01.589 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:43:01.590 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:43:01.590 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:43:01.591 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:43:01.597 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:43:01.599 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:43:01.599 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:43:01.600 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:43:01.600 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:43:01.601 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:43:01.601 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:43:01.602 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:43:01.602 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.603 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:43:01.603 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:43:01.603 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.604 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.604 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.605 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:43:01.605 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:43:01.606 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:43:01.607 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:43:01.607 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:43:01.607 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:43:01.608 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:43:01.608 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:43:01.609 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:43:01.610 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:43:01.610 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:43:01.610 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:43:01.611 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:43:01.611 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:43:01.612 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:43:01.612 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:43:01.613 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:43:01.613 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:43:01.614 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:43:01.614 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:43:01.615 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:43:01.615 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:43:01.616 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:43:01.617 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.617 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:43:01.618 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:43:01.618 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:43:01.618 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:43:01.619 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:43:01.620 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:43:01.620 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:43:01.621 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:43:01.621 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:43:01.622 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:43:01.622 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:43:01.623 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:43:01.624 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:43:01.624 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:43:01.625 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:43:01.625 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:43:01.626 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:43:01.626 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:43:01.627 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:43:01.627 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:43:01.627 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:43:01.628 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:43:01.628 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:43:01.628 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:43:01.629 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:43:01.629 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:43:01.629 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:43:01.630 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.630 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:43:01.631 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:43:01.631 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:43:01.632 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:43:01.632 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:43:01.633 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:43:01.633 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:43:01.634 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:43:01.634 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:43:01.634 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:43:01.635 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:43:01.635 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:43:01.636 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:43:01.636 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:43:01.636 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.637 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.637 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.637 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 5, 2022-07-12T08:43:01.638 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:43:01.638 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:56.415996+0000", 2022-07-12T08:43:01.639 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:43:01.639 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:43:01.640 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:43:01.641 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:43:01.641 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:43:01.642 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:43:01.643 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:43:01.643 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:43:01.643 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:43:01.650 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:43:01.651 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:43:01.651 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:43:01.652 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:43:01.652 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:43:01.652 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:43:01.655 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:43:01.655 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:43:01.655 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:43:01.656 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:43:01.656 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:43:01.656 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.656 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "31", 2022-07-12T08:43:01.657 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:43:01.657 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:43:01.657 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:43:01.657 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:43:01.658 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:43:01.658 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:43:01.659 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:43:01.659 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:43:01.659 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:43:01.660 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:43:01.660 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:43:01.661 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:43:01.662 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:43:01.662 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:43:01.662 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:43:01.663 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:43:01.663 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:43:01.663 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:43:01.664 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:43:01.664 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:43:01.665 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:43:01.665 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:43:01.665 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:43:01.666 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:43:01.666 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:43:01.666 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:43:01.667 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.669 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:43:01.670 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:43:01.670 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:43:01.670 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:43:01.671 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:43:01.671 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:43:01.672 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:43:01.672 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:43:01.673 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:43:01.673 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:43:01.674 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:43:01.674 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:43:01.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:43:01.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:43:01.683 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:43:01.684 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.685 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:43:01.685 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:43:01.686 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:43:01.686 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.687 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.687 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.688 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.689 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 6, 2022-07-12T08:43:01.689 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:43:01.689 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:57.640916+0000", 2022-07-12T08:43:01.693 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:43:01.694 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:43:01.695 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:43:01.695 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:43:01.696 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:43:01.696 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:43:01.697 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:43:01.698 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:43:01.698 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:43:01.699 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:43:01.699 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:43:01.699 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:43:01.700 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:43:01.700 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:43:01.702 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:43:01.702 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:43:01.703 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:43:01.703 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:43:01.704 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:43:01.704 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:43:01.704 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.705 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "32", 2022-07-12T08:43:01.705 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:43:01.705 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:43:01.706 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:43:01.706 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:43:01.707 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:43:01.707 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:43:01.707 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:43:01.708 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:43:01.708 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:43:01.709 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:43:01.709 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:43:01.710 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:43:01.710 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:43:01.711 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:43:01.711 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:43:01.712 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:43:01.712 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:43:01.712 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:43:01.713 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:43:01.714 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:43:01.715 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:43:01.715 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:43:01.716 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:43:01.716 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:43:01.717 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:43:01.717 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:43:01.718 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.718 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:43:01.719 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:43:01.719 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:43:01.720 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:43:01.720 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:43:01.721 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:43:01.721 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:43:01.722 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:43:01.722 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:43:01.723 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:43:01.724 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:43:01.724 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:43:01.725 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:43:01.725 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:43:01.726 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:43:01.727 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.727 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.727 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.728 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:01.728 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:43:01.729 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.729 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:43:01.730 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:43:01.731 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.732 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.732 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.733 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.733 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.734 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.734 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.735 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:01.735 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.736 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.736 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.737 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.737 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.738 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.738 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:43:01.739 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.739 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.740 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.740 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.741 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:43:01.741 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.742 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.742 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.743 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.743 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.744 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.745 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.745 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.746 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:43:01.747 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.747 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.748 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.749 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.749 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:43:01.749 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.750 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.751 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.751 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.752 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.753 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.753 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.754 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.754 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:43:01.759 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.760 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.761 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.761 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:43:01.761 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.762 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.762 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.763 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.763 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.764 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.764 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.765 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.766 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:43:01.766 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.767 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.771 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.771 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.772 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:43:01.772 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:01.773 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.773 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.774 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.774 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:43:01.775 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:43:01.776 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:43:01.776 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:43:01.777 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.777 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.778 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.778 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.779 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.779 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.780 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:43:01.780 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:43:01.780 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.781 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.782 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.784 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.784 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.785 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.786 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.786 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-12T08:43:01.787 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.787 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.788 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.788 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.788 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.789 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.789 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:43:01.789 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.790 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.790 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.791 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.791 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:43:01.792 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.792 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.793 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.793 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.793 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.794 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.794 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.794 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.795 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:43:01.795 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.796 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.796 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.797 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.797 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:43:01.798 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.798 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.799 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.799 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.799 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.800 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.801 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.803 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:43:01.803 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.804 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.804 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.805 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.805 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:43:01.805 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.806 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.806 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.807 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.807 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.808 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.808 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.808 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.809 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:43:01.809 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.810 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.811 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.811 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.812 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:43:01.812 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:01.813 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.813 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.814 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.814 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:43:01.815 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:43:01.815 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:43:01.816 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:43:01.816 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.816 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.817 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.817 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.819 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.819 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.820 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:43:01.820 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:43:01.820 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.820 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.821 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.821 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.821 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.822 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.822 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.822 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 28, 2022-07-12T08:43:01.823 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.823 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.824 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.824 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.825 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.825 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:43:01.826 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.826 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.827 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.828 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:43:01.828 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.828 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.829 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.829 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.829 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.830 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.830 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.830 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.831 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:43:01.831 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.832 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.832 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.833 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.833 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:43:01.834 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.835 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.835 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.835 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.836 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.836 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.836 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.838 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.839 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:43:01.839 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.840 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.847 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.847 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.848 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:43:01.848 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.848 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.849 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.849 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.850 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.850 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.850 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:43:01.850 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.851 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.851 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.851 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:43:01.852 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:01.853 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.853 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.853 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.854 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:43:01.862 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:43:01.862 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:43:01.862 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:43:01.863 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.863 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.863 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.863 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.864 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.864 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.864 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:43:01.865 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:43:01.865 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.866 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.866 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.866 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.867 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.867 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.867 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.867 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:01.868 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.868 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.868 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.868 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.869 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.869 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.869 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:43:01.869 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.870 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.870 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.870 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.870 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:43:01.871 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.871 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.871 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.871 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.872 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.872 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.872 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.872 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.872 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:43:01.873 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.873 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.873 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.874 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:43:01.874 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.874 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.874 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.875 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.875 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.875 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.875 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.875 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.876 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:43:01.876 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.876 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.876 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.877 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.877 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:43:01.877 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.877 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.878 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.878 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.878 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.878 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.879 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.879 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.879 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:43:01.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.880 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.880 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.880 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:43:01.880 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:01.881 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.881 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.881 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.882 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:43:01.882 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:43:01.882 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:43:01.883 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:43:01.883 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.883 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.883 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.884 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.884 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.884 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.884 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:43:01.885 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:43:01.885 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.885 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.885 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.886 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.886 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.886 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.886 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.887 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:01.887 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.887 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.887 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.888 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.888 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.888 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.888 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:43:01.889 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.889 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.889 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.889 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.890 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:43:01.890 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.890 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.891 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.891 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.891 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.891 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.892 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.892 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:43:01.892 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.892 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.893 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.893 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.893 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:43:01.894 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.894 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.894 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.894 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.895 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.895 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.895 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.895 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.896 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:43:01.896 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.897 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.897 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:43:01.897 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.897 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.898 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.898 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.898 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.898 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.899 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.899 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.899 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:43:01.899 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.900 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.900 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.900 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:43:01.901 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:01.901 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.901 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.901 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.902 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:43:01.902 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:43:01.902 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:43:01.903 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:43:01.903 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.903 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.904 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.904 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.904 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.904 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.905 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:43:01.905 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:43:01.905 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.905 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.906 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.906 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.906 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.906 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.907 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.907 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:01.907 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.907 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.907 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.908 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.908 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.908 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.908 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:43:01.909 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.909 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.910 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:43:01.910 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.910 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.910 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.911 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.911 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.911 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.912 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.912 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.912 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:43:01.912 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.912 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.913 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.913 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.913 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:43:01.913 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.914 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.914 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.914 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.914 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.915 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.915 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.915 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.915 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:43:01.915 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.916 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.916 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.916 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.916 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:43:01.916 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.917 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.917 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.917 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.917 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.918 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:43:01.918 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.919 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.919 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.919 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:43:01.920 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:01.920 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.920 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.920 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.920 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:43:01.921 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:43:01.921 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:43:01.921 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:43:01.922 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.922 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.922 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.923 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.923 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.923 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:43:01.923 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:43:01.924 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.924 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.924 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.924 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.925 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.925 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.925 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.925 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 28, 2022-07-12T08:43:01.925 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.926 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.926 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.927 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.927 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.927 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.927 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:43:01.927 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.928 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.928 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.928 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.928 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:43:01.929 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.929 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.929 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.929 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.929 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.930 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.930 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.930 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.930 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:43:01.931 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.931 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.931 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.931 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.932 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:43:01.932 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.932 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.932 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.932 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.933 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.933 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.933 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.933 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.933 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:43:01.934 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.934 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.934 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.934 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.935 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:43:01.935 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.935 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.935 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.936 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.936 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.936 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.936 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.937 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.937 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:43:01.937 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.937 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.937 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.938 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.938 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:43:01.938 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:01.938 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.939 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.939 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.939 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:43:01.940 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:43:01.940 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:43:01.940 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:43:01.940 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.940 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.941 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.941 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.941 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.941 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.942 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:43:01.942 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:43:01.942 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:01.942 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:01.943 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:01.943 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:01.943 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:01.943 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:01.944 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:01.944 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:01.944 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:01.944 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:01.944 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:01.945 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.945 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.945 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.945 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:43:01.945 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.957 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.957 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.957 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:43:01.958 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.959 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.959 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.960 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.961 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:01.962 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.963 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.964 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.964 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:43:01.965 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.965 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.966 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.966 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.967 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:43:01.967 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.968 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.968 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.969 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.969 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:01.969 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.970 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.971 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:43:01.972 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.973 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.974 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.974 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.975 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:43:01.975 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.975 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.977 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.978 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.978 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:01.979 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:01.979 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.979 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:01.980 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:43:01.980 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.980 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.981 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.981 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:01.982 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:43:01.982 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:01.982 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.983 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.984 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:43:01.984 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:43:01.985 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:43:01.985 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:43:01.986 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:01.986 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:01.987 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:01.987 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:01.987 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:01.988 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:01.988 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:43:01.989 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.989 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:43:01.989 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:01.990 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:01.990 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:01.990 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:01.991 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:01.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:43:01.992 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:01.992 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.993 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:43:01.994 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:01.995 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:01.995 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:01.996 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:01.997 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:01.997 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:43:01.997 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:01.998 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:01.998 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:01.999 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:43:01.999 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:02.000 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:02.000 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:02.001 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:02.001 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:02.002 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:43:02.002 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:02.003 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:02.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:02.005 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:43:02.005 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:02.006 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:02.006 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:02.007 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:02.007 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:02.008 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:43:02.008 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:02.010 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:43:02.011 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:02.012 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:02.012 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:02.013 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:02.013 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:02.014 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:43:02.014 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:02.014 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:02.015 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:02.015 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:43:02.016 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:02.016 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:02.017 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:02.017 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:02.017 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:02.019 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:43:02.019 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:02.019 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:02.020 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:02.020 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:43:02.021 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:02.021 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:02.021 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:02.022 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:02.022 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:02.023 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:43:02.023 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:02.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:02.024 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:02.024 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:43:02.024 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:02.025 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:02.025 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:02.036 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:02.036 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:02.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:43:02.038 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:02.039 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:02.039 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:02.040 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:43:02.040 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:43:02.040 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:43:02.041 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:43:02.041 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:43:02.042 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:43:02.042 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:43:02.043 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:43:02.043 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:43:02.044 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:43:02.044 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:43:02.045 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:43:02.045 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:43:02.046 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:02.047 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:02.047 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:43:02.047 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:43:02.048 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:43:02.048 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:43:02.049 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:43:02.049 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:43:02.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a60 msgr2=0x7fa5cc055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a60 0x7fa5cc055f20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa5d000d000 tx=0x7fa5d0009040).stop 2022-07-12T08:43:02.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 msgr2=0x7fa5e80b4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 0x7fa5e80b4c00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa5e8142850 tx=0x7fa5d8016000).stop 2022-07-12T08:43:02.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 shutdown_connections 2022-07-12T08:43:02.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a60 0x7fa5cc055f20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e813fb30 0x7fa5e8065930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b2c10 0x7fa5e80b4c00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 --2- 172.21.15.61:0/228790925 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e80b2300 0x7fa5e8141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 >> 172.21.15.61:0/228790925 conn(0x7fa5e8063dc0 msgr2=0x7fa5e80affd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:02.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 shutdown_connections 2022-07-12T08:43:02.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:01.532+0000 7fa5efc4e700 1 -- 172.21.15.61:0/228790925 wait complete. 2022-07-12T08:43:02.060 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:02.062 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:02.067 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:02.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:02.088 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:02.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:02.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.168+0000 7f19e4cc8700 1 Processor -- start 2022-07-12T08:43:02.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.168+0000 7f19e4cc8700 1 -- start start 2022-07-12T08:43:02.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e013ce60 0x7f19e00b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f19e0142d50 0x7f19e00b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f19e00b60a0 0x7f19e00b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e00b24c0 con 0x7f19e0142d50 2022-07-12T08:43:02.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e013d240 con 0x7f19e00b60a0 2022-07-12T08:43:02.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e01431f0 con 0x7f19e013ce60 2022-07-12T08:43:02.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f19e0142d50 0x7f19e00b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f19e0142d50 0x7f19e00b5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54552/0 (socket says 172.21.15.61:54552) 2022-07-12T08:43:02.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 -- 172.21.15.61:0/2656420402 learned_addr learned my addr 172.21.15.61:0/2656420402 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:02.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ded9d700 1 --2- 172.21.15.61:0/2656420402 >> v2:172.21.15.111:3300/0 conn(0x7f19e00b60a0 0x7f19e00b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19de59c700 1 --2- 172.21.15.61:0/2656420402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e013ce60 0x7f19e00b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 -- 172.21.15.61:0/2656420402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e013ce60 msgr2=0x7f19e00b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:02.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 --2- 172.21.15.61:0/2656420402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e013ce60 0x7f19e00b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 -- 172.21.15.61:0/2656420402 >> v2:172.21.15.111:3300/0 conn(0x7f19e00b60a0 msgr2=0x7f19e00b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:02.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 --2- 172.21.15.61:0/2656420402 >> v2:172.21.15.111:3300/0 conn(0x7f19e00b60a0 0x7f19e00b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 -- 172.21.15.61:0/2656420402 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e0143490 con 0x7f19e0142d50 2022-07-12T08:43:02.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ded9d700 1 --2- 172.21.15.61:0/2656420402 >> v2:172.21.15.111:3300/0 conn(0x7f19e00b60a0 0x7f19e00b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:02.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19ddd9b700 1 --2- 172.21.15.61:0/2656420402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e00b5bb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f19c8002f70 tx=0x7f19c8006eb0).ready entity=mon.0 client_cookie=e97caf3acbb9580b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:02.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19dd59a700 1 -- 172.21.15.61:0/2656420402 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19c800d570 con 0x7f19e0142d50 2022-07-12T08:43:02.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19dd59a700 1 -- 172.21.15.61:0/2656420402 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c8000e90 con 0x7f19e0142d50 2022-07-12T08:43:02.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19dd59a700 1 -- 172.21.15.61:0/2656420402 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19c800d900 con 0x7f19e0142d50 2022-07-12T08:43:02.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2656420402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 msgr2=0x7f19e00b5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2656420402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e00b5bb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f19c8002f70 tx=0x7f19c8006eb0).stop 2022-07-12T08:43:02.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2656420402 shutdown_connections 2022-07-12T08:43:02.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2656420402 >> v2:172.21.15.111:3300/0 conn(0x7f19e00b60a0 0x7f19e00b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2656420402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e00b5bb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2656420402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e013ce60 0x7f19e00b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2656420402 >> 172.21.15.61:0/2656420402 conn(0x7f19e005f9f0 msgr2=0x7f19e0066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:02.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2656420402 shutdown_connections 2022-07-12T08:43:02.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2656420402 wait complete. 2022-07-12T08:43:02.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 Processor -- start 2022-07-12T08:43:02.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.172+0000 7f19e4cc8700 1 -- start start 2022-07-12T08:43:02.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e00b60a0 0x7f19e0147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19e013ce60 0x7f19e0142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e0142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00ae400 con 0x7f19e0142d50 2022-07-12T08:43:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e0142b20 con 0x7f19e013ce60 2022-07-12T08:43:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00ae630 con 0x7f19e00b60a0 2022-07-12T08:43:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e0142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e0142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54562/0 (socket says 172.21.15.61:54562) 2022-07-12T08:43:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 -- 172.21.15.61:0/2635948226 learned_addr learned my addr 172.21.15.61:0/2635948226 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ddd9b700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19e013ce60 0x7f19e0142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19de59c700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e00b60a0 0x7f19e0147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 -- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e00b60a0 msgr2=0x7f19e0147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e00b60a0 0x7f19e0147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 -- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19e013ce60 msgr2=0x7f19e0142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19e013ce60 0x7f19e0142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 -- 172.21.15.61:0/2635948226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c800b1e0 con 0x7f19e0142d50 2022-07-12T08:43:02.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ddd9b700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19e013ce60 0x7f19e0142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:02.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19ded9d700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e0142630 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f19d8006cc0 tx=0x7f19d8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:02.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d8013620 con 0x7f19e0142d50 2022-07-12T08:43:02.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8013780 con 0x7f19e0142d50 2022-07-12T08:43:02.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e0148260 con 0x7f19e0142d50 2022-07-12T08:43:02.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19d8013ae0 con 0x7f19e0142d50 2022-07-12T08:43:02.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e00af5c0 con 0x7f19e0142d50 2022-07-12T08:43:02.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.176+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e0057460 con 0x7f19e0142d50 2022-07-12T08:43:02.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.180+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f19d8013c40 con 0x7f19e0142d50 2022-07-12T08:43:02.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.180+0000 7f19d77fe700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19b80539b0 0x7f19b8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.180+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f19d806bf50 con 0x7f19e0142d50 2022-07-12T08:43:02.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.180+0000 7f19de59c700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19b80539b0 0x7f19b8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.180+0000 7f19de59c700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19b80539b0 0x7f19b8055e70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f19d000c000 tx=0x7f19d0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:02.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.184+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19d8042f00 con 0x7f19e0142d50 2022-07-12T08:43:02.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.344+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19e00b5800 con 0x7f19e0142d50 2022-07-12T08:43:02.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19d77fe700 1 -- 172.21.15.61:0/2635948226 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f19d8046520 con 0x7f19e0142d50 2022-07-12T08:43:02.358 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:02.359 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:42:00.304215+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4905":{"gid":4905,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3202017529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3202017529},{"type":"v1","addr":"172.21.15.61:6833","nonce":3202017529}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:43:02.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19b80539b0 msgr2=0x7f19b8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19b80539b0 0x7f19b8055e70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f19d000c000 tx=0x7f19d0008040).stop 2022-07-12T08:43:02.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 msgr2=0x7f19e0142630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e0142630 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f19d8006cc0 tx=0x7f19d8008040).stop 2022-07-12T08:43:02.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 shutdown_connections 2022-07-12T08:43:02.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19e0142d50 0x7f19e0142630 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19b80539b0 0x7f19b8055e70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19e013ce60 0x7f19e0142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 --2- 172.21.15.61:0/2635948226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19e00b60a0 0x7f19e0147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 >> 172.21.15.61:0/2635948226 conn(0x7f19e005f9f0 msgr2=0x7f19e00b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:02.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 shutdown_connections 2022-07-12T08:43:02.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.348+0000 7f19e4cc8700 1 -- 172.21.15.61:0/2635948226 wait complete. 2022-07-12T08:43:02.368 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 12 2022-07-12T08:43:02.370 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:43:02.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 Processor -- start 2022-07-12T08:43:02.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 -- start start 2022-07-12T08:43:02.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005c050 0x7faf700ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faf700b01f0 0x7faf7014a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf7005cb00 0x7faf7005cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf700b0a40 con 0x7faf7005cb00 2022-07-12T08:43:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7005d430 con 0x7faf700b01f0 2022-07-12T08:43:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf77bfb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700ad2f0 con 0x7faf7005c050 2022-07-12T08:43:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf7005cb00 0x7faf7005cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf7005cb00 0x7faf7005cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54570/0 (socket says 172.21.15.61:54570) 2022-07-12T08:43:02.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 -- 172.21.15.61:0/1639349374 learned_addr learned my addr 172.21.15.61:0/1639349374 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:02.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf75196700 1 --2- 172.21.15.61:0/1639349374 >> v2:172.21.15.111:3300/0 conn(0x7faf700b01f0 0x7faf7014a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf75997700 1 --2- 172.21.15.61:0/1639349374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005c050 0x7faf700ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 -- 172.21.15.61:0/1639349374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005c050 msgr2=0x7faf700ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:02.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 --2- 172.21.15.61:0/1639349374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005c050 0x7faf700ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 -- 172.21.15.61:0/1639349374 >> v2:172.21.15.111:3300/0 conn(0x7faf700b01f0 msgr2=0x7faf7014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:02.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 --2- 172.21.15.61:0/1639349374 >> v2:172.21.15.111:3300/0 conn(0x7faf700b01f0 0x7faf7014a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 -- 172.21.15.61:0/1639349374 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7005c430 con 0x7faf7005cb00 2022-07-12T08:43:02.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf75196700 1 --2- 172.21.15.61:0/1639349374 >> v2:172.21.15.111:3300/0 conn(0x7faf700b01f0 0x7faf7014a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:02.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf76198700 1 --2- 172.21.15.61:0/1639349374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005cb00 0x7faf7005cf40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7faf7005d570 tx=0x7faf68006eb0).ready entity=mon.0 client_cookie=74708d34c0ccb001 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:02.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf74995700 1 -- 172.21.15.61:0/1639349374 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6800e610 con 0x7faf7005cb00 2022-07-12T08:43:02.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf74995700 1 -- 172.21.15.61:0/1639349374 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf68000e90 con 0x7faf7005cb00 2022-07-12T08:43:02.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.476+0000 7faf74995700 1 -- 172.21.15.61:0/1639349374 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6800e9a0 con 0x7faf7005cb00 2022-07-12T08:43:02.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- 172.21.15.61:0/1639349374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005cb00 msgr2=0x7faf7005cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- 172.21.15.61:0/1639349374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005cb00 0x7faf7005cf40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7faf7005d570 tx=0x7faf68006eb0).stop 2022-07-12T08:43:02.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- 172.21.15.61:0/1639349374 shutdown_connections 2022-07-12T08:43:02.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- 172.21.15.61:0/1639349374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005cb00 0x7faf7005cf40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- 172.21.15.61:0/1639349374 >> v2:172.21.15.111:3300/0 conn(0x7faf700b01f0 0x7faf7014a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- 172.21.15.61:0/1639349374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005c050 0x7faf700ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- 172.21.15.61:0/1639349374 >> 172.21.15.61:0/1639349374 conn(0x7faf70062c00 msgr2=0x7faf70062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- 172.21.15.61:0/1639349374 shutdown_connections 2022-07-12T08:43:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- 172.21.15.61:0/1639349374 wait complete. 2022-07-12T08:43:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 Processor -- start 2022-07-12T08:43:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- start start 2022-07-12T08:43:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 0x7faf70143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005cb00 0x7faf7014a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf700b01f0 0x7faf70159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf701499b0 con 0x7faf7005c050 2022-07-12T08:43:02.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf70148e10 con 0x7faf700b01f0 2022-07-12T08:43:02.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf77bfb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf701490b0 con 0x7faf7005cb00 2022-07-12T08:43:02.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf75997700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 0x7faf70143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf75997700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 0x7faf70143180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54572/0 (socket says 172.21.15.61:54572) 2022-07-12T08:43:02.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf75997700 1 -- 172.21.15.61:0/3430425156 learned_addr learned my addr 172.21.15.61:0/3430425156 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:02.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf75196700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005cb00 0x7faf7014a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.480+0000 7faf76198700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf700b01f0 0x7faf70159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75997700 1 -- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005cb00 msgr2=0x7faf7014a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75997700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005cb00 0x7faf7014a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75997700 1 -- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf700b01f0 msgr2=0x7faf70159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:02.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75997700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf700b01f0 0x7faf70159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:02.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75997700 1 -- 172.21.15.61:0/3430425156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60009470 con 0x7faf7005c050 2022-07-12T08:43:02.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75196700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005cb00 0x7faf7014a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:02.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75997700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 0x7faf70143180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7faf6400c000 tx=0x7faf64006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:02.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf64010780 con 0x7faf7005c050 2022-07-12T08:43:02.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf640108e0 con 0x7faf7005c050 2022-07-12T08:43:02.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6800b1e0 con 0x7faf7005c050 2022-07-12T08:43:02.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf64010c40 con 0x7faf7005c050 2022-07-12T08:43:02.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf70143670 con 0x7faf7005c050 2022-07-12T08:43:02.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf70057460 con 0x7faf7005c050 2022-07-12T08:43:02.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faf6401c460 con 0x7faf7005c050 2022-07-12T08:43:02.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf6effd700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf50053a40 0x7faf50055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:02.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faf6406c410 con 0x7faf7005c050 2022-07-12T08:43:02.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.484+0000 7faf75196700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf50053a40 0x7faf50055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:02.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.488+0000 7faf75196700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf50053a40 0x7faf50055f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7faf6000b000 tx=0x7faf60008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:02.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.492+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf64043180 con 0x7faf7005c050 2022-07-12T08:43:02.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:02.656+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faf700447a0 con 0x7faf7005c050 2022-07-12T08:43:03.173 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:43:03.168+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:43:03.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.176+0000 7faf6effd700 1 -- 172.21.15.61:0/3430425156 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v13) v1 ==== 161+0+0 (secure 0 0 0) 0x7faf640467a0 con 0x7faf7005c050 2022-07-12T08:43:03.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf50053a40 msgr2=0x7faf50055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf50053a40 0x7faf50055f00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7faf6000b000 tx=0x7faf60008040).stop 2022-07-12T08:43:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 msgr2=0x7faf70143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 0x7faf70143180 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7faf6400c000 tx=0x7faf64006df0).stop 2022-07-12T08:43:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 shutdown_connections 2022-07-12T08:43:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf50053a40 0x7faf50055f00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf700b01f0 0x7faf70159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf7005cb00 0x7faf7014a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 --2- 172.21.15.61:0/3430425156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf7005c050 0x7faf70143180 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 >> 172.21.15.61:0/3430425156 conn(0x7faf70062c00 msgr2=0x7faf70147b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 shutdown_connections 2022-07-12T08:43:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.180+0000 7faf77bfb700 1 -- 172.21.15.61:0/3430425156 wait complete. 2022-07-12T08:43:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:43:03.208 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:03.234 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:43:03.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.308+0000 7f1f9e44a700 1 Processor -- start 2022-07-12T08:43:03.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.308+0000 7f1f9e44a700 1 -- start start 2022-07-12T08:43:03.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1f9805c8e0 0x7f1f980b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f98146c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f98146110 con 0x7f1f980605c0 2022-07-12T08:43:03.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f9805ccc0 con 0x7f1f9805c8e0 2022-07-12T08:43:03.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9805f350 con 0x7f1f981467e0 2022-07-12T08:43:03.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1f9805c8e0 0x7f1f980b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9cc47700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1f980605c0 0x7f1f981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42812/0 (socket says 172.21.15.61:42812) 2022-07-12T08:43:03.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9cc47700 1 -- 172.21.15.61:0/2915965044 learned_addr learned my addr 172.21.15.61:0/2915965044 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:03.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9dc49700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f98146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513620335 0 0) 0x7f1f98146110 con 0x7f1f980605c0 2022-07-12T08:43:03.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f80003260 con 0x7f1f980605c0 2022-07-12T08:43:03.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955024926 0 0) 0x7f1f80003260 con 0x7f1f980605c0 2022-07-12T08:43:03.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 msgr2=0x7f1f98146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:03.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f98146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 >> v2:172.21.15.61:3300/0 conn(0x7f1f9805c8e0 msgr2=0x7f1f980b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 --2- 172.21.15.61:0/2915965044 >> v2:172.21.15.61:3300/0 conn(0x7f1f9805c8e0 0x7f1f980b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f98146340 con 0x7f1f980605c0 2022-07-12T08:43:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f1f88001660 con 0x7f1f980605c0 2022-07-12T08:43:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 >> v1:172.21.15.61:6789/0 conn(0x7f1f980605c0 legacy=0x7f1f981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f80003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f80003c80 0x7f1f80004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f800049f0 0x7f1f80004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9805ccc0 con 0x7f1f80003c80 2022-07-12T08:43:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9805f350 con 0x7f1f800049f0 2022-07-12T08:43:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8000b750 con 0x7f1f981467e0 2022-07-12T08:43:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9cc47700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f80003c80 0x7f1f80004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f80003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9dc49700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f800049f0 0x7f1f80004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 -- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f800049f0 msgr2=0x7f1f80004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f800049f0 0x7f1f80004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 -- 172.21.15.61:0/2915965044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f80003c80 msgr2=0x7f1f80004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f80003c80 0x7f1f80004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 -- 172.21.15.61:0/2915965044 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f80003260 con 0x7f1f981467e0 2022-07-12T08:43:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9dc49700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f800049f0 0x7f1f80004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9d448700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f80003750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1f9400c000 tx=0x7f1f94006df0).ready entity=mon.2 client_cookie=7aefcf7c408075d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f9400e830 con 0x7f1f981467e0 2022-07-12T08:43:03.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9400e990 con 0x7f1f981467e0 2022-07-12T08:43:03.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f87fff700 1 -- 172.21.15.61:0/2915965044 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f9400ecf0 con 0x7f1f981467e0 2022-07-12T08:43:03.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 msgr2=0x7f1f80003750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2915965044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f80003750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1f9400c000 tx=0x7f1f94006df0).stop 2022-07-12T08:43:03.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9dc49700 1 -- 172.21.15.61:0/2915965044 reap_dead start 2022-07-12T08:43:03.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2915965044 shutdown_connections 2022-07-12T08:43:03.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2915965044 >> 172.21.15.61:0/2915965044 conn(0x7f1f98063a60 msgr2=0x7f1f98063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:03.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2915965044 shutdown_connections 2022-07-12T08:43:03.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.312+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2915965044 wait complete. 2022-07-12T08:43:03.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 Processor -- start 2022-07-12T08:43:03.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 -- start start 2022-07-12T08:43:03.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f980b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f9813f7a0 0x7f1f98065be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f98067020 0x7f1f980674a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f98066bf0 con 0x7f1f9813f7a0 2022-07-12T08:43:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9813dec0 con 0x7f1f98067020 2022-07-12T08:43:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9813e160 con 0x7f1f981467e0 2022-07-12T08:43:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f9813f7a0 0x7f1f98065be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9d448700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f980b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9d448700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f980b63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36800/0 (socket says 172.21.15.61:36800) 2022-07-12T08:43:03.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9d448700 1 -- 172.21.15.61:0/2774198809 learned_addr learned my addr 172.21.15.61:0/2774198809 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:03.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9dc49700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f98067020 0x7f1f980674a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 -- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 msgr2=0x7f1f980b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f980b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 -- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f98067020 msgr2=0x7f1f980674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f98067020 0x7f1f980674a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 -- 172.21.15.61:0/2774198809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9813e400 con 0x7f1f9813f7a0 2022-07-12T08:43:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9cc47700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f9813f7a0 0x7f1f98065be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f1f8800d000 tx=0x7f1f981464d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f880116b0 con 0x7f1f9813f7a0 2022-07-12T08:43:03.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f88011810 con 0x7f1f9813f7a0 2022-07-12T08:43:03.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f980660d0 con 0x7f1f9813f7a0 2022-07-12T08:43:03.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f88011b70 con 0x7f1f9813f7a0 2022-07-12T08:43:03.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.316+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9813f580 con 0x7f1f9813f7a0 2022-07-12T08:43:03.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.320+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f98057460 con 0x7f1f9813f7a0 2022-07-12T08:43:03.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.320+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1f8801b030 con 0x7f1f9813f7a0 2022-07-12T08:43:03.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.320+0000 7f1f867fc700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f780539b0 0x7f1f78055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.320+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1f8806b950 con 0x7f1f9813f7a0 2022-07-12T08:43:03.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.320+0000 7f1f9d448700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f780539b0 0x7f1f78055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.320+0000 7f1f9d448700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f780539b0 0x7f1f78055e70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f1f9401e000 tx=0x7f1f94019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:03.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.328+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f88042750 con 0x7f1f9813f7a0 2022-07-12T08:43:03.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.492+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f98066d80 con 0x7f1f9813f7a0 2022-07-12T08:43:03.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f867fc700 1 -- 172.21.15.61:0/2774198809 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+1779 (secure 0 0 0) 0x7f1f88045d70 con 0x7f1f9813f7a0 2022-07-12T08:43:03.501 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:03.502 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8}],"filesystems":[{"mdsmap":{"epoch":13,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:41:58.634283+0000","modified":"2022-07-12T08:43:03.171141+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-12T08:43:03.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f780539b0 msgr2=0x7f1f78055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f780539b0 0x7f1f78055e70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f1f9401e000 tx=0x7f1f94019000).stop 2022-07-12T08:43:03.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f9813f7a0 msgr2=0x7f1f98065be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f9813f7a0 0x7f1f98065be0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f1f8800d000 tx=0x7f1f981464d0).stop 2022-07-12T08:43:03.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 shutdown_connections 2022-07-12T08:43:03.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f780539b0 0x7f1f78055e70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f98067020 0x7f1f980674a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f9813f7a0 0x7f1f98065be0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 --2- 172.21.15.61:0/2774198809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f981467e0 0x7f1f980b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 >> 172.21.15.61:0/2774198809 conn(0x7f1f98063a60 msgr2=0x7f1f980b29c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:03.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 shutdown_connections 2022-07-12T08:43:03.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.496+0000 7f1f9e44a700 1 -- 172.21.15.61:0/2774198809 wait complete. 2022-07-12T08:43:03.509 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 13 2022-07-12T08:43:03.522 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:43:03.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.620+0000 7f1f1574d700 1 Processor -- start 2022-07-12T08:43:03.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f1574d700 1 -- start start 2022-07-12T08:43:03.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f1574d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1f100992c0 0x7f1f100996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f1574d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f1005f860 0x7f1f1013fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f1574d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1009b160 con 0x7f1f1005cd20 2022-07-12T08:43:03.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f1574d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1009aec0 con 0x7f1f100992c0 2022-07-12T08:43:03.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f1574d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f10140aa0 con 0x7f1f1005f860 2022-07-12T08:43:03.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1f100992c0 0x7f1f100996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f14f4c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1f1005cd20 0x7f1f101402a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42834/0 (socket says 172.21.15.61:42834) 2022-07-12T08:43:03.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f14f4c700 1 -- 172.21.15.61:0/2456725652 learned_addr learned my addr 172.21.15.61:0/2456725652 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:03.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f077fe700 1 --2- 172.21.15.61:0/2456725652 >> v2:172.21.15.111:3300/0 conn(0x7f1f1005f860 0x7f1f1013fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0f7fe700 1 -- 172.21.15.61:0/2456725652 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993456373 0 0) 0x7f1f1009b160 con 0x7f1f1005cd20 2022-07-12T08:43:03.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0f7fe700 1 -- 172.21.15.61:0/2456725652 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef0003260 con 0x7f1f1005cd20 2022-07-12T08:43:03.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0ffff700 1 -- 172.21.15.61:0/2456725652 >> v2:172.21.15.111:3300/0 conn(0x7f1f1005f860 msgr2=0x7f1f1013fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:03.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0ffff700 1 --2- 172.21.15.61:0/2456725652 >> v2:172.21.15.111:3300/0 conn(0x7f1f1005f860 0x7f1f1013fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0ffff700 1 -- 172.21.15.61:0/2456725652 >> v1:172.21.15.61:6789/0 conn(0x7f1f1005cd20 legacy=0x7f1f101402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0f7fe700 1 -- 172.21.15.61:0/2456725652 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1267037616 0 0) 0x7f1ef0003260 con 0x7f1f1005cd20 2022-07-12T08:43:03.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0ffff700 1 -- 172.21.15.61:0/2456725652 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1009ab40 con 0x7f1f100992c0 2022-07-12T08:43:03.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0ffff700 1 --2- 172.21.15.61:0/2456725652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f100996a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1f08001b80 tx=0x7f1f00006df0).ready entity=mon.0 client_cookie=179711111f66fbc3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:03.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0f7fe700 1 -- 172.21.15.61:0/2456725652 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f080019e0 con 0x7f1f100992c0 2022-07-12T08:43:03.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0f7fe700 1 -- 172.21.15.61:0/2456725652 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00000e90 con 0x7f1f100992c0 2022-07-12T08:43:03.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.624+0000 7f1f0f7fe700 1 -- 172.21.15.61:0/2456725652 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f00012460 con 0x7f1f100992c0 2022-07-12T08:43:03.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 -- 172.21.15.61:0/2456725652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 msgr2=0x7f1f100996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 --2- 172.21.15.61:0/2456725652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f100996a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f1f08001b80 tx=0x7f1f00006df0).stop 2022-07-12T08:43:03.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 -- 172.21.15.61:0/2456725652 shutdown_connections 2022-07-12T08:43:03.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 --2- 172.21.15.61:0/2456725652 >> v2:172.21.15.111:3300/0 conn(0x7f1f1005f860 0x7f1f1013fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 --2- 172.21.15.61:0/2456725652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f100996a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 -- 172.21.15.61:0/2456725652 >> 172.21.15.61:0/2456725652 conn(0x7f1f100b1a50 msgr2=0x7f1f100b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:03.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 -- 172.21.15.61:0/2456725652 shutdown_connections 2022-07-12T08:43:03.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 -- 172.21.15.61:0/2456725652 wait complete. 2022-07-12T08:43:03.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 Processor -- start 2022-07-12T08:43:03.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.628+0000 7f1f1574d700 1 -- start start 2022-07-12T08:43:03.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f1005cd20 0x7f1f10067310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f1005f860 0x7f1f1014fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f101501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f10064a40 con 0x7f1f100992c0 2022-07-12T08:43:03.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f10063c30 con 0x7f1f1005f860 2022-07-12T08:43:03.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1014f8c0 con 0x7f1f1005cd20 2022-07-12T08:43:03.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f101501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f101501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54620/0 (socket says 172.21.15.61:54620) 2022-07-12T08:43:03.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 -- 172.21.15.61:0/3219178488 learned_addr learned my addr 172.21.15.61:0/3219178488 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:03.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f077fe700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f1005f860 0x7f1f1014fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f0ffff700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f1005cd20 0x7f1f10067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 -- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f1005cd20 msgr2=0x7f1f10067310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f1005cd20 0x7f1f10067310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 -- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f1005f860 msgr2=0x7f1f1014fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:03.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f1005f860 0x7f1f1014fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:03.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 -- 172.21.15.61:0/3219178488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0000b180 con 0x7f1f100992c0 2022-07-12T08:43:03.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f0ffff700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f1005cd20 0x7f1f10067310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:03.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f14f4c700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f101501e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1f08001bc0 tx=0x7f1f08001ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:03.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f08004ab0 con 0x7f1f100992c0 2022-07-12T08:43:03.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f08004c10 con 0x7f1f100992c0 2022-07-12T08:43:03.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f10063990 con 0x7f1f100992c0 2022-07-12T08:43:03.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f08007170 con 0x7f1f100992c0 2022-07-12T08:43:03.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f100b4900 con 0x7f1f100992c0 2022-07-12T08:43:03.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.632+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f10064bb0 con 0x7f1f100992c0 2022-07-12T08:43:03.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.636+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1f0801b060 con 0x7f1f100992c0 2022-07-12T08:43:03.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.636+0000 7f1f0dffb700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1ef4053a40 0x7f1ef4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:03.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.636+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1f0806bca0 con 0x7f1f100992c0 2022-07-12T08:43:03.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.636+0000 7f1f0ffff700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1ef4053a40 0x7f1ef4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:03.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.636+0000 7f1f0ffff700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1ef4053a40 0x7f1ef4055f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1f00018000 tx=0x7f1f00014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:03.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.640+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f08070070 con 0x7f1f100992c0 2022-07-12T08:43:03.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:03.808+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1f10057460 con 0x7f1f100992c0 2022-07-12T08:43:04.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.184+0000 7f1f0dffb700 1 -- 172.21.15.61:0/3219178488 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1f08019070 con 0x7f1f100992c0 2022-07-12T08:43:04.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1ef4053a40 msgr2=0x7f1ef4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:04.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1ef4053a40 0x7f1ef4055f00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f1f00018000 tx=0x7f1f00014000).stop 2022-07-12T08:43:04.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 msgr2=0x7f1f101501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:04.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f101501e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f1f08001bc0 tx=0x7f1f08001ca0).stop 2022-07-12T08:43:04.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 shutdown_connections 2022-07-12T08:43:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f100992c0 0x7f1f101501e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1ef4053a40 0x7f1ef4055f00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f1005f860 0x7f1f1014fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 --2- 172.21.15.61:0/3219178488 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f1005cd20 0x7f1f10067310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 >> 172.21.15.61:0/3219178488 conn(0x7f1f100b1a50 msgr2=0x7f1f10144650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:04.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 shutdown_connections 2022-07-12T08:43:04.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.188+0000 7f1f1574d700 1 -- 172.21.15.61:0/3219178488 wait complete. 2022-07-12T08:43:04.230 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:43:04.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.344+0000 7ff504fca700 1 Processor -- start 2022-07-12T08:43:04.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.344+0000 7ff504fca700 1 -- start start 2022-07-12T08:43:04.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff5000b1b10 0x7ff500066e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:04.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff5000a9b90 0x7ff500064e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:04.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5000672f0 con 0x7ff5001431f0 2022-07-12T08:43:04.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff500066bc0 con 0x7ff5000b1b10 2022-07-12T08:43:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5000b1ef0 con 0x7ff5000a9b90 2022-07-12T08:43:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fe59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff5000b1b10 0x7ff500066e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff5001431f0 0x7ff500143680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42850/0 (socket says 172.21.15.61:42850) 2022-07-12T08:43:04.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fed9d700 1 -- 172.21.15.61:0/980055474 learned_addr learned my addr 172.21.15.61:0/980055474 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:04.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4f7fff700 1 --2- 172.21.15.61:0/980055474 >> v2:172.21.15.111:3300/0 conn(0x7ff5000a9b90 0x7ff500064e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:04.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fdd9b700 1 -- 172.21.15.61:0/980055474 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534528964 0 0) 0x7ff4f8001490 con 0x7ff5001431f0 2022-07-12T08:43:04.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fdd9b700 1 -- 172.21.15.61:0/980055474 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e0003260 con 0x7ff5001431f0 2022-07-12T08:43:04.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fe59c700 1 -- 172.21.15.61:0/980055474 >> v2:172.21.15.111:3300/0 conn(0x7ff5000a9b90 msgr2=0x7ff500064e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:04.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fe59c700 1 --2- 172.21.15.61:0/980055474 >> v2:172.21.15.111:3300/0 conn(0x7ff5000a9b90 0x7ff500064e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fe59c700 1 -- 172.21.15.61:0/980055474 >> v1:172.21.15.61:6789/0 conn(0x7ff5001431f0 legacy=0x7ff500143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:04.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fe59c700 1 -- 172.21.15.61:0/980055474 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5000b1060 con 0x7ff5000b1b10 2022-07-12T08:43:04.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fe59c700 1 --2- 172.21.15.61:0/980055474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5000b1b10 0x7ff500066e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7ff4f0002f70 tx=0x7ff4f0006df0).ready entity=mon.0 client_cookie=2fc6ce9b8b31ffc1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:04.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fdd9b700 1 -- 172.21.15.61:0/980055474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4f000d4f0 con 0x7ff5000b1b10 2022-07-12T08:43:04.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fdd9b700 1 -- 172.21.15.61:0/980055474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f0000e90 con 0x7ff5000b1b10 2022-07-12T08:43:04.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff4fdd9b700 1 -- 172.21.15.61:0/980055474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4f000d840 con 0x7ff5000b1b10 2022-07-12T08:43:04.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 -- 172.21.15.61:0/980055474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5000b1b10 msgr2=0x7ff500066e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:04.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 --2- 172.21.15.61:0/980055474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5000b1b10 0x7ff500066e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7ff4f0002f70 tx=0x7ff4f0006df0).stop 2022-07-12T08:43:04.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 -- 172.21.15.61:0/980055474 shutdown_connections 2022-07-12T08:43:04.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 --2- 172.21.15.61:0/980055474 >> v2:172.21.15.111:3300/0 conn(0x7ff5000a9b90 0x7ff500064e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 --2- 172.21.15.61:0/980055474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5000b1b10 0x7ff500066e00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.348+0000 7ff504fca700 1 -- 172.21.15.61:0/980055474 >> 172.21.15.61:0/980055474 conn(0x7ff5000b61a0 msgr2=0x7ff5000b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:04.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.352+0000 7ff504fca700 1 -- 172.21.15.61:0/980055474 shutdown_connections 2022-07-12T08:43:04.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.352+0000 7ff504fca700 1 -- 172.21.15.61:0/980055474 wait complete. 2022-07-12T08:43:04.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.352+0000 7ff504fca700 1 Processor -- start 2022-07-12T08:43:04.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.352+0000 7ff504fca700 1 -- start start 2022-07-12T08:43:04.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5000a9b90 0x7ff5000acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:04.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b1b10 0x7ff50014e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:04.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 0x7ff5000b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:04.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000a1400 con 0x7ff5001431f0 2022-07-12T08:43:04.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff500151e00 con 0x7ff5000b1b10 2022-07-12T08:43:04.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5000ac240 con 0x7ff5000a9b90 2022-07-12T08:43:04.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 0x7ff5000b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 0x7ff5000b5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54638/0 (socket says 172.21.15.61:54638) 2022-07-12T08:43:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fe59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5000a9b90 0x7ff5000acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 -- 172.21.15.61:0/1432681010 learned_addr learned my addr 172.21.15.61:0/1432681010 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:04.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4f7fff700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b1b10 0x7ff50014e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:04.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 -- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5000a9b90 msgr2=0x7ff5000acf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5000a9b90 0x7ff5000acf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 -- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b1b10 msgr2=0x7ff50014e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b1b10 0x7ff50014e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:04.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 -- 172.21.15.61:0/1432681010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f000b180 con 0x7ff5001431f0 2022-07-12T08:43:04.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4fed9d700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 0x7ff5000b5a60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff4f8001620 tx=0x7ff4f8007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4f8005ba0 con 0x7ff5001431f0 2022-07-12T08:43:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f8005d00 con 0x7ff5001431f0 2022-07-12T08:43:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5000a1630 con 0x7ff5001431f0 2022-07-12T08:43:04.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4f80072c0 con 0x7ff5001431f0 2022-07-12T08:43:04.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5000ac680 con 0x7ff5001431f0 2022-07-12T08:43:04.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.356+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff500151f70 con 0x7ff5001431f0 2022-07-12T08:43:04.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.360+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff4f8010070 con 0x7ff5001431f0 2022-07-12T08:43:04.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.360+0000 7ff4f77fe700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4e4053ad0 0x7ff4e4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:04.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.360+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff4f806b8b0 con 0x7ff5001431f0 2022-07-12T08:43:04.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.360+0000 7ff4fe59c700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4e4053ad0 0x7ff4e4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:04.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.360+0000 7ff4fe59c700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4e4053ad0 0x7ff4e4055f90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff4f001a000 tx=0x7ff4f0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:04.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.368+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff500151f70 con 0x7ff5001431f0 2022-07-12T08:43:04.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:04.500+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff5000a17c0 con 0x7ff5001431f0 2022-07-12T08:43:05.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.196+0000 7ff4f77fe700 1 -- 172.21.15.61:0/1432681010 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v34) v1 ==== 185+0+0 (secure 0 0 0) 0x7ff5000a17c0 con 0x7ff5001431f0 2022-07-12T08:43:05.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4e4053ad0 msgr2=0x7ff4e4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:05.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4e4053ad0 0x7ff4e4055f90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7ff4f001a000 tx=0x7ff4f0014000).stop 2022-07-12T08:43:05.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 msgr2=0x7ff5000b5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:05.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 0x7ff5000b5a60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff4f8001620 tx=0x7ff4f8007ce0).stop 2022-07-12T08:43:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 shutdown_connections 2022-07-12T08:43:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5001431f0 0x7ff5000b5a60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4e4053ad0 0x7ff4e4055f90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5000b1b10 0x7ff50014e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 --2- 172.21.15.61:0/1432681010 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5000a9b90 0x7ff5000acf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 >> 172.21.15.61:0/1432681010 conn(0x7ff5000b61a0 msgr2=0x7ff5000602e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:05.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 shutdown_connections 2022-07-12T08:43:05.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.200+0000 7ff504fca700 1 -- 172.21.15.61:0/1432681010 wait complete. 2022-07-12T08:43:05.226 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:43:05.231 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:43:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 Processor -- start 2022-07-12T08:43:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 -- start start 2022-07-12T08:43:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b5480 0x7f45f00b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f45f00b05f0 0x7f45f0142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00b3560 con 0x7f45f00b3730 2022-07-12T08:43:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f00b5b80 con 0x7f45f00b05f0 2022-07-12T08:43:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f6714700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00b5e20 con 0x7f45f00b5480 2022-07-12T08:43:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45e7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f45f00b3730 0x7f45f00afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42862/0 (socket says 172.21.15.61:42862) 2022-07-12T08:43:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45e7fff700 1 -- 172.21.15.61:0/4092972907 learned_addr learned my addr 172.21.15.61:0/4092972907 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45f4cb1700 1 --2- 172.21.15.61:0/4092972907 >> v2:172.21.15.111:3300/0 conn(0x7f45f00b05f0 0x7f45f0142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45effff700 1 --2- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b5480 0x7f45f00b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45ef7fe700 1 -- 172.21.15.61:0/4092972907 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354695338 0 0) 0x7f45f00b3560 con 0x7f45f00b3730 2022-07-12T08:43:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.384+0000 7f45ef7fe700 1 -- 172.21.15.61:0/4092972907 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d0003260 con 0x7f45f00b3730 2022-07-12T08:43:05.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f4cb1700 1 -- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b5480 msgr2=0x7f45f00b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:05.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f4cb1700 1 --2- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b5480 0x7f45f00b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f4cb1700 1 -- 172.21.15.61:0/4092972907 >> v1:172.21.15.61:6789/0 conn(0x7f45f00b3730 legacy=0x7f45f00afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:05.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f4cb1700 1 -- 172.21.15.61:0/4092972907 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f00b5860 con 0x7f45f00b05f0 2022-07-12T08:43:05.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f4cb1700 1 --2- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b05f0 0x7f45f0142f90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f45e80032f0 tx=0x7f45e80091d0).ready entity=mon.1 client_cookie=7c795d4e15d5ec1e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:05.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45ef7fe700 1 -- 172.21.15.61:0/4092972907 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e800b600 con 0x7f45f00b05f0 2022-07-12T08:43:05.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45ef7fe700 1 -- 172.21.15.61:0/4092972907 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e800b760 con 0x7f45f00b05f0 2022-07-12T08:43:05.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45ef7fe700 1 -- 172.21.15.61:0/4092972907 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e800b8d0 con 0x7f45f00b05f0 2022-07-12T08:43:05.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 -- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b05f0 msgr2=0x7f45f0142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:05.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 --2- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b05f0 0x7f45f0142f90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f45e80032f0 tx=0x7f45e80091d0).stop 2022-07-12T08:43:05.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 -- 172.21.15.61:0/4092972907 shutdown_connections 2022-07-12T08:43:05.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 --2- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b05f0 0x7f45f0142f90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 --2- 172.21.15.61:0/4092972907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b5480 0x7f45f00b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 -- 172.21.15.61:0/4092972907 >> 172.21.15.61:0/4092972907 conn(0x7f45f00b96b0 msgr2=0x7f45f00b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:05.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 -- 172.21.15.61:0/4092972907 shutdown_connections 2022-07-12T08:43:05.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 -- 172.21.15.61:0/4092972907 wait complete. 2022-07-12T08:43:05.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 Processor -- start 2022-07-12T08:43:05.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.388+0000 7f45f6714700 1 -- start start 2022-07-12T08:43:05.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b05f0 0x7f45f0144e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:05.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 0x7f45f0147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:05.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b5480 0x7f45f01526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:05.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00af600 con 0x7f45f00b3730 2022-07-12T08:43:05.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00af0c0 con 0x7f45f00b5480 2022-07-12T08:43:05.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00af360 con 0x7f45f00b05f0 2022-07-12T08:43:05.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 0x7f45f0147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:05.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f4cb1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b5480 0x7f45f01526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:05.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 0x7f45f0147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54650/0 (socket says 172.21.15.61:54650) 2022-07-12T08:43:05.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 -- 172.21.15.61:0/2376954059 learned_addr learned my addr 172.21.15.61:0/2376954059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:05.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45effff700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b05f0 0x7f45f0144e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:05.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 -- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b05f0 msgr2=0x7f45f0144e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:05.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b05f0 0x7f45f0144e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 -- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b5480 msgr2=0x7f45f01526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:05.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b5480 0x7f45f01526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:05.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 -- 172.21.15.61:0/2376954059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e8009470 con 0x7f45f00b3730 2022-07-12T08:43:05.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f4cb1700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b5480 0x7f45f01526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:05.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45e7fff700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 0x7f45f0147da0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f45d8001110 tx=0x7f45d8007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:05.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45d80070d0 con 0x7f45f00b3730 2022-07-12T08:43:05.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d8007230 con 0x7f45f00b3730 2022-07-12T08:43:05.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45d8005610 con 0x7f45f00b3730 2022-07-12T08:43:05.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f00b0100 con 0x7f45f00b3730 2022-07-12T08:43:05.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.392+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f0151f60 con 0x7f45f00b3730 2022-07-12T08:43:05.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.396+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f0057460 con 0x7f45f00b3730 2022-07-12T08:43:05.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.396+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f45d8010070 con 0x7f45f00b3730 2022-07-12T08:43:05.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.396+0000 7f45edffb700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d4053a40 0x7f45d4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:05.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.396+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f45d806b4f0 con 0x7f45f00b3730 2022-07-12T08:43:05.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.396+0000 7f45effff700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d4053a40 0x7f45d4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:05.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.400+0000 7f45effff700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d4053a40 0x7f45d4055f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f45e000b000 tx=0x7f45e0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:05.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.404+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45d80424a0 con 0x7f45f00b3730 2022-07-12T08:43:05.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:05.528+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f45f00ba5d0 con 0x7f45f00b3730 2022-07-12T08:43:06.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.204+0000 7f45edffb700 1 -- 172.21.15.61:0/2376954059 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v35) v1 ==== 173+0+0 (secure 0 0 0) 0x7f45d8045ac0 con 0x7f45f00b3730 2022-07-12T08:43:06.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d4053a40 msgr2=0x7f45d4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d4053a40 0x7f45d4055f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f45e000b000 tx=0x7f45e0006df0).stop 2022-07-12T08:43:06.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 msgr2=0x7f45f0147da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 0x7f45f0147da0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f45d8001110 tx=0x7f45d8007af0).stop 2022-07-12T08:43:06.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 shutdown_connections 2022-07-12T08:43:06.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d4053a40 0x7f45d4055f00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f00b5480 0x7f45f01526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f00b3730 0x7f45f0147da0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 --2- 172.21.15.61:0/2376954059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f00b05f0 0x7f45f0144e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 >> 172.21.15.61:0/2376954059 conn(0x7f45f00b96b0 msgr2=0x7f45f00b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:06.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 shutdown_connections 2022-07-12T08:43:06.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.208+0000 7f45f6714700 1 -- 172.21.15.61:0/2376954059 wait complete. 2022-07-12T08:43:06.232 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:43:06.236 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:43:06.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.372+0000 7fa1a7fff700 1 Processor -- start 2022-07-12T08:43:06.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.372+0000 7fa1a7fff700 1 -- start start 2022-07-12T08:43:06.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.372+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a80aafa0 0x7fa1a8060eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.372+0000 7fa1a7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a805d440 con 0x7fa1a805d060 2022-07-12T08:43:06.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.372+0000 7fa1a7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a8060c60 con 0x7fa1a8146b00 2022-07-12T08:43:06.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.372+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a80aa230 con 0x7fa1a80aafa0 2022-07-12T08:43:06.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa1a805d060 0x7fa1a805f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42876/0 (socket says 172.21.15.61:42876) 2022-07-12T08:43:06.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a6ffd700 1 -- 172.21.15.61:0/2655258821 learned_addr learned my addr 172.21.15.61:0/2655258821 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:06.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a67fc700 1 --2- 172.21.15.61:0/2655258821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a80aafa0 0x7fa1a8060eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a5ffb700 1 -- 172.21.15.61:0/2655258821 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905990189 0 0) 0x7fa1a805d440 con 0x7fa1a805d060 2022-07-12T08:43:06.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a5ffb700 1 -- 172.21.15.61:0/2655258821 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa18c003260 con 0x7fa1a805d060 2022-07-12T08:43:06.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a67fc700 1 -- 172.21.15.61:0/2655258821 >> v1:172.21.15.111:6789/0 conn(0x7fa1a8146b00 legacy=0x7fa1a8142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a5ffb700 1 -- 172.21.15.61:0/2655258821 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592299257 0 0) 0x7fa1a8060c60 con 0x7fa1a8146b00 2022-07-12T08:43:06.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a67fc700 1 -- 172.21.15.61:0/2655258821 >> v1:172.21.15.61:6789/0 conn(0x7fa1a805d060 legacy=0x7fa1a805f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a67fc700 1 -- 172.21.15.61:0/2655258821 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8146120 con 0x7fa1a80aafa0 2022-07-12T08:43:06.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a67fc700 1 --2- 172.21.15.61:0/2655258821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a80aafa0 0x7fa1a8060eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa1a80aae40 tx=0x7fa198006df0).ready entity=mon.2 client_cookie=1e66fc047cbb87a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a5ffb700 1 -- 172.21.15.61:0/2655258821 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19800d5c0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a5ffb700 1 -- 172.21.15.61:0/2655258821 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa198000e90 con 0x7fa1a80aafa0 2022-07-12T08:43:06.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a5ffb700 1 -- 172.21.15.61:0/2655258821 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19800d910 con 0x7fa1a80aafa0 2022-07-12T08:43:06.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 -- 172.21.15.61:0/2655258821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a80aafa0 msgr2=0x7fa1a8060eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/2655258821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a80aafa0 0x7fa1a8060eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa1a80aae40 tx=0x7fa198006df0).stop 2022-07-12T08:43:06.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 -- 172.21.15.61:0/2655258821 shutdown_connections 2022-07-12T08:43:06.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/2655258821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a80aafa0 0x7fa1a8060eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 -- 172.21.15.61:0/2655258821 >> 172.21.15.61:0/2655258821 conn(0x7fa1a80b1a50 msgr2=0x7fa1a80b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:06.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 -- 172.21.15.61:0/2655258821 shutdown_connections 2022-07-12T08:43:06.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.376+0000 7fa1a7fff700 1 -- 172.21.15.61:0/2655258821 wait complete. 2022-07-12T08:43:06.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 Processor -- start 2022-07-12T08:43:06.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 -- start start 2022-07-12T08:43:06.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a805d060 0x7fa1a81401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 0x7fa1a81444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a8146b00 0x7fa1a81449e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8140e70 con 0x7fa1a80aafa0 2022-07-12T08:43:06.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8144090 con 0x7fa1a8146b00 2022-07-12T08:43:06.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a8145e60 con 0x7fa1a805d060 2022-07-12T08:43:06.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 0x7fa1a81444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 0x7fa1a81444f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54664/0 (socket says 172.21.15.61:54664) 2022-07-12T08:43:06.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 -- 172.21.15.61:0/804092079 learned_addr learned my addr 172.21.15.61:0/804092079 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:06.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a6ffd700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a805d060 0x7fa1a81401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a77fe700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a8146b00 0x7fa1a81449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 -- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a805d060 msgr2=0x7fa1a81401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a805d060 0x7fa1a81401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 -- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a8146b00 msgr2=0x7fa1a81449e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a8146b00 0x7fa1a81449e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 -- 172.21.15.61:0/804092079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19800b180 con 0x7fa1a80aafa0 2022-07-12T08:43:06.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.380+0000 7fa1a67fc700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 0x7fa1a81444f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fa1a80aae40 tx=0x7fa198014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:06.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa198013da0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa198002f70 con 0x7fa1a80aafa0 2022-07-12T08:43:06.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a8145bc0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19801e5f0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a809aed0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a8057460 con 0x7fa1a80aafa0 2022-07-12T08:43:06.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa198025030 con 0x7fa1a80aafa0 2022-07-12T08:43:06.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa197fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1840539b0 0x7fa184055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa19806c7a0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.384+0000 7fa1a6ffd700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1840539b0 0x7fa184055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.388+0000 7fa1a6ffd700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1840539b0 0x7fa184055e70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa19c00d000 tx=0x7fa19c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:06.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.392+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1980439e0 con 0x7fa1a80aafa0 2022-07-12T08:43:06.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.516+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa1a805f330 con 0x7fa1a80aafa0 2022-07-12T08:43:06.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa197fff700 1 -- 172.21.15.61:0/804092079 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa198023020 con 0x7fa1a80aafa0 2022-07-12T08:43:06.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1840539b0 msgr2=0x7fa184055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1840539b0 0x7fa184055e70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fa19c00d000 tx=0x7fa19c009040).stop 2022-07-12T08:43:06.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 msgr2=0x7fa1a81444f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 0x7fa1a81444f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fa1a80aae40 tx=0x7fa198014000).stop 2022-07-12T08:43:06.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 shutdown_connections 2022-07-12T08:43:06.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa1840539b0 0x7fa184055e70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1a8146b00 0x7fa1a81449e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a80aafa0 0x7fa1a81444f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 --2- 172.21.15.61:0/804092079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa1a805d060 0x7fa1a81401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 >> 172.21.15.61:0/804092079 conn(0x7fa1a80b1a50 msgr2=0x7fa1a80b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:06.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 shutdown_connections 2022-07-12T08:43:06.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.524+0000 7fa1a7fff700 1 -- 172.21.15.61:0/804092079 wait complete. 2022-07-12T08:43:06.581 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-12T08:43:06.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.680+0000 7fbbfd98a700 1 Processor -- start 2022-07-12T08:43:06.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.680+0000 7fbbfd98a700 1 -- start start 2022-07-12T08:43:06.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbbf813fe70 0x7fbbf8140300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf80b02e0 con 0x7fbbf80af0b0 2022-07-12T08:43:06.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf8067040 con 0x7fbbf8061030 2022-07-12T08:43:06.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf80af490 con 0x7fbbf813fe70 2022-07-12T08:43:06.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfc988700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbbf80af0b0 0x7fbbf8060920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42888/0 (socket says 172.21.15.61:42888) 2022-07-12T08:43:06.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd189700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbbf813fe70 0x7fbbf8140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfc988700 1 -- 172.21.15.61:0/1062131908 learned_addr learned my addr 172.21.15.61:0/1062131908 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:06.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbf77fe700 1 -- 172.21.15.61:0/1062131908 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592917162 0 0) 0x7fbbf80b02e0 con 0x7fbbf80af0b0 2022-07-12T08:43:06.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd189700 1 -- 172.21.15.61:0/1062131908 >> v1:172.21.15.111:6789/0 conn(0x7fbbf8061030 legacy=0x7fbbf813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd189700 1 -- 172.21.15.61:0/1062131908 >> v1:172.21.15.61:6789/0 conn(0x7fbbf80af0b0 legacy=0x7fbbf8060920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd189700 1 -- 172.21.15.61:0/1062131908 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf8061490 con 0x7fbbf813fe70 2022-07-12T08:43:06.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd189700 1 --2- 172.21.15.61:0/1062131908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf8140300 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fbbe4001110 tx=0x7fbbf00091d0).ready entity=mon.0 client_cookie=c00b956af0d205c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:06.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbf77fe700 1 -- 172.21.15.61:0/1062131908 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbf000b600 con 0x7fbbf813fe70 2022-07-12T08:43:06.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbf77fe700 1 -- 172.21.15.61:0/1062131908 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf000b760 con 0x7fbbf813fe70 2022-07-12T08:43:06.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbf77fe700 1 -- 172.21.15.61:0/1062131908 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbf000b8d0 con 0x7fbbf813fe70 2022-07-12T08:43:06.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- 172.21.15.61:0/1062131908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 msgr2=0x7fbbf8140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/1062131908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf8140300 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fbbe4001110 tx=0x7fbbf00091d0).stop 2022-07-12T08:43:06.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- 172.21.15.61:0/1062131908 shutdown_connections 2022-07-12T08:43:06.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/1062131908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf8140300 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- 172.21.15.61:0/1062131908 >> 172.21.15.61:0/1062131908 conn(0x7fbbf80b5780 msgr2=0x7fbbf80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:06.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- 172.21.15.61:0/1062131908 shutdown_connections 2022-07-12T08:43:06.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.684+0000 7fbbfd98a700 1 -- 172.21.15.61:0/1062131908 wait complete. 2022-07-12T08:43:06.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 Processor -- start 2022-07-12T08:43:06.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 -- start start 2022-07-12T08:43:06.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbbf8061030 0x7fbbf8141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbf80af0b0 0x7fbbf8144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf81453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf813efc0 con 0x7fbbf813fe70 2022-07-12T08:43:06.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8142940 con 0x7fbbf80af0b0 2022-07-12T08:43:06.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd98a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf8142de0 con 0x7fbbf8061030 2022-07-12T08:43:06.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf81453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf81453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54680/0 (socket says 172.21.15.61:54680) 2022-07-12T08:43:06.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 -- 172.21.15.61:0/4250924307 learned_addr learned my addr 172.21.15.61:0/4250924307 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:06.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfc988700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbbf8061030 0x7fbbf8141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbf7fff700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbf80af0b0 0x7fbbf8144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 -- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbbf8061030 msgr2=0x7fbbf8141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbbf8061030 0x7fbbf8141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 -- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbf80af0b0 msgr2=0x7fbbf8144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:06.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbf80af0b0 0x7fbbf8144ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:06.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.688+0000 7fbbfd189700 1 -- 172.21.15.61:0/4250924307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0009470 con 0x7fbbf813fe70 2022-07-12T08:43:06.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbfd189700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf81453c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fbbe4001110 tx=0x7fbbf0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:06.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbf000b600 con 0x7fbbf813fe70 2022-07-12T08:43:06.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf00032d0 con 0x7fbbf813fe70 2022-07-12T08:43:06.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbf001f5e0 con 0x7fbbf813fe70 2022-07-12T08:43:06.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf81391d0 con 0x7fbbf813fe70 2022-07-12T08:43:06.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf8142320 con 0x7fbbf813fe70 2022-07-12T08:43:06.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbbf0026030 con 0x7fbbf813fe70 2022-07-12T08:43:06.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbf5ffb700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbbd8053a60 0x7fbbd8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:06.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fbbf006c610 con 0x7fbbf813fe70 2022-07-12T08:43:06.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf8142ae0 con 0x7fbbf813fe70 2022-07-12T08:43:06.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbfc988700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbbd8053a60 0x7fbbd8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:06.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.692+0000 7fbbfc988700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbbd8053a60 0x7fbbd8055f20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fbbec00d000 tx=0x7fbbec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:06.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.696+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbbf0043890 con 0x7fbbf813fe70 2022-07-12T08:43:06.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:06.864+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7fbbf8057460 con 0x7fbbf813fe70 2022-07-12T08:43:07.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.208+0000 7fbbf5ffb700 1 -- 172.21.15.61:0/4250924307 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v103) v1 ==== 139+0+0 (secure 0 0 0) 0x7fbbf0024030 con 0x7fbbf813fe70 2022-07-12T08:43:07.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbbd8053a60 msgr2=0x7fbbd8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:07.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbbd8053a60 0x7fbbd8055f20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fbbec00d000 tx=0x7fbbec009040).stop 2022-07-12T08:43:07.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 msgr2=0x7fbbf81453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:07.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf81453c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fbbe4001110 tx=0x7fbbf0016000).stop 2022-07-12T08:43:07.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 shutdown_connections 2022-07-12T08:43:07.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbbf813fe70 0x7fbbf81453c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbbd8053a60 0x7fbbd8055f20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbf80af0b0 0x7fbbf8144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 --2- 172.21.15.61:0/4250924307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbbf8061030 0x7fbbf8141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.212+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 >> 172.21.15.61:0/4250924307 conn(0x7fbbf80b5780 msgr2=0x7fbbf813aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:07.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.216+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 shutdown_connections 2022-07-12T08:43:07.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.216+0000 7fbbfd98a700 1 -- 172.21.15.61:0/4250924307 wait complete. 2022-07-12T08:43:07.240 INFO:tasks.cephfs_test_runner:test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-12T08:43:07.241 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-12T08:43:07.244 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-12T08:43:07.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 Processor -- start 2022-07-12T08:43:07.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 -- start start 2022-07-12T08:43:07.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05a80ab290 0x7f05a80ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:07.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a805ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:07.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05a80abc70 con 0x7f05a80afa60 2022-07-12T08:43:07.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05a805f580 con 0x7f05a80ab290 2022-07-12T08:43:07.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a805c720 con 0x7f05a805f140 2022-07-12T08:43:07.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a805ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:07.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a805ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36880/0 (socket says 172.21.15.61:36880) 2022-07-12T08:43:07.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 -- 172.21.15.61:0/2283866894 learned_addr learned my addr 172.21.15.61:0/2283866894 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:07.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a659c700 1 --2- 172.21.15.61:0/2283866894 >> v2:172.21.15.111:3300/0 conn(0x7f05a80ab290 0x7f05a80ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:07.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 -- 172.21.15.61:0/2283866894 >> v2:172.21.15.111:3300/0 conn(0x7f05a80ab290 msgr2=0x7f05a80ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:07.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 --2- 172.21.15.61:0/2283866894 >> v2:172.21.15.111:3300/0 conn(0x7f05a80ab290 0x7f05a80ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 -- 172.21.15.61:0/2283866894 >> v1:172.21.15.111:6789/0 conn(0x7f05a80afa60 legacy=0x7f05a814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:07.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 -- 172.21.15.61:0/2283866894 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a805c9c0 con 0x7f05a805f140 2022-07-12T08:43:07.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a5d9b700 1 --2- 172.21.15.61:0/2283866894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a805ce60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f05980032f0 tx=0x7f0598009290).ready entity=mon.2 client_cookie=e88b1d0ce817cd6d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:07.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a559a700 1 -- 172.21.15.61:0/2283866894 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f059800b8a0 con 0x7f05a805f140 2022-07-12T08:43:07.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a559a700 1 -- 172.21.15.61:0/2283866894 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059800ba00 con 0x7f05a805f140 2022-07-12T08:43:07.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.352+0000 7f05a559a700 1 -- 172.21.15.61:0/2283866894 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f059800bb70 con 0x7f05a805f140 2022-07-12T08:43:07.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/2283866894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 msgr2=0x7f05a805ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:07.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 --2- 172.21.15.61:0/2283866894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a805ce60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f05980032f0 tx=0x7f0598009290).stop 2022-07-12T08:43:07.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/2283866894 shutdown_connections 2022-07-12T08:43:07.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 --2- 172.21.15.61:0/2283866894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a805ce60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 --2- 172.21.15.61:0/2283866894 >> v2:172.21.15.111:3300/0 conn(0x7f05a80ab290 0x7f05a80ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/2283866894 >> 172.21.15.61:0/2283866894 conn(0x7f05a80b1950 msgr2=0x7f05a80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:07.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/2283866894 shutdown_connections 2022-07-12T08:43:07.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/2283866894 wait complete. 2022-07-12T08:43:07.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 Processor -- start 2022-07-12T08:43:07.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- start start 2022-07-12T08:43:07.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a8067400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:07.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05a80ab290 0x7f05a8143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:07.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 0x7f05a8143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:07.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8155e00 con 0x7f05a80afa60 2022-07-12T08:43:07.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8066230 con 0x7f05a80ab290 2022-07-12T08:43:07.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05a8156040 con 0x7f05a805f140 2022-07-12T08:43:07.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 0x7f05a8143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:07.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a5d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05a80ab290 0x7f05a8143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:07.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 0x7f05a8143f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54694/0 (socket says 172.21.15.61:54694) 2022-07-12T08:43:07.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 -- 172.21.15.61:0/52879283 learned_addr learned my addr 172.21.15.61:0/52879283 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:07.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a659c700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a8067400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:07.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 -- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 msgr2=0x7f05a8067400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:07.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a8067400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 -- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05a80ab290 msgr2=0x7f05a8143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:07.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05a80ab290 0x7f05a8143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:07.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 -- 172.21.15.61:0/52879283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0598009530 con 0x7f05a80afa60 2022-07-12T08:43:07.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a5d9b700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05a80ab290 0x7f05a8143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:07.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a6d9d700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 0x7f05a8143f10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f05a805cd40 tx=0x7f05a0007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:07.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05a0007d70 con 0x7f05a80afa60 2022-07-12T08:43:07.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0012400 con 0x7f05a80afa60 2022-07-12T08:43:07.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a80ae370 con 0x7f05a80afa60 2022-07-12T08:43:07.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05a0012760 con 0x7f05a80afa60 2022-07-12T08:43:07.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.356+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a80af020 con 0x7f05a80afa60 2022-07-12T08:43:07.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.360+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a8066080 con 0x7f05a80afa60 2022-07-12T08:43:07.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.360+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f05a0019030 con 0x7f05a80afa60 2022-07-12T08:43:07.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.360+0000 7f059f7fe700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f058c0539b0 0x7f058c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:07.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.360+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f05a006adf0 con 0x7f05a80afa60 2022-07-12T08:43:07.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.360+0000 7f05a659c700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f058c0539b0 0x7f058c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:07.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.360+0000 7f05a659c700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f058c0539b0 0x7f058c055e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f059000b000 tx=0x7f0590006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:07.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.368+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05a0041fd0 con 0x7f05a80afa60 2022-07-12T08:43:07.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:07.536+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7f05a8156200 con 0x7f05a80afa60 2022-07-12T08:43:08.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.212+0000 7f059f7fe700 1 -- 172.21.15.61:0/52879283 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v104) v1 ==== 147+0+0 (secure 0 0 0) 0x7f05a00455f0 con 0x7f05a80afa60 2022-07-12T08:43:08.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f058c0539b0 msgr2=0x7f058c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f058c0539b0 0x7f058c055e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f059000b000 tx=0x7f0590006df0).stop 2022-07-12T08:43:08.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 msgr2=0x7f05a8143f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 0x7f05a8143f10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f05a805cd40 tx=0x7f05a0007bd0).stop 2022-07-12T08:43:08.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 shutdown_connections 2022-07-12T08:43:08.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05a80afa60 0x7f05a8143f10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f058c0539b0 0x7f058c055e70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05a80ab290 0x7f05a8143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 --2- 172.21.15.61:0/52879283 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05a805f140 0x7f05a8067400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 >> 172.21.15.61:0/52879283 conn(0x7f05a80b1950 msgr2=0x7f05a80b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:08.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 shutdown_connections 2022-07-12T08:43:08.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.216+0000 7f05a759e700 1 -- 172.21.15.61:0/52879283 wait complete. 2022-07-12T08:43:08.254 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:43:08.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9267fff700 1 Processor -- start 2022-07-12T08:43:08.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9267fff700 1 -- start start 2022-07-12T08:43:08.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9267fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f926805c120 0x7f92680b1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:08.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9267fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92680b07e0 con 0x7f92680b1e60 2022-07-12T08:43:08.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9267fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9268140bd0 con 0x7f9268060c20 2022-07-12T08:43:08.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9267fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9268061530 con 0x7f926805c120 2022-07-12T08:43:08.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92677fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f92680b1e60 0x7f9268066c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42920/0 (socket says 172.21.15.61:42920) 2022-07-12T08:43:08.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92677fe700 1 -- 172.21.15.61:0/2742847254 learned_addr learned my addr 172.21.15.61:0/2742847254 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:08.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92667fc700 1 --2- 172.21.15.61:0/2742847254 >> v2:172.21.15.111:3300/0 conn(0x7f926805c120 0x7f92680b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:08.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9265ffb700 1 -- 172.21.15.61:0/2742847254 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521928486 0 0) 0x7f92680b07e0 con 0x7f92680b1e60 2022-07-12T08:43:08.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9265ffb700 1 -- 172.21.15.61:0/2742847254 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f924c003260 con 0x7f92680b1e60 2022-07-12T08:43:08.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92667fc700 1 -- 172.21.15.61:0/2742847254 >> v1:172.21.15.111:6789/0 conn(0x7f9268060c20 legacy=0x7f92680641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92667fc700 1 -- 172.21.15.61:0/2742847254 >> v1:172.21.15.61:6789/0 conn(0x7f92680b1e60 legacy=0x7f9268066c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f9265ffb700 1 -- 172.21.15.61:0/2742847254 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438218542 0 0) 0x7f9258001110 con 0x7f9268060c20 2022-07-12T08:43:08.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92667fc700 1 -- 172.21.15.61:0/2742847254 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680b0540 con 0x7f926805c120 2022-07-12T08:43:08.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.372+0000 7f92667fc700 1 --2- 172.21.15.61:0/2742847254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f926805c120 0x7f92680b1970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f925c002f70 tx=0x7f925c006df0).ready entity=mon.1 client_cookie=13e49322375d8588 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:08.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9265ffb700 1 -- 172.21.15.61:0/2742847254 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f925c00d610 con 0x7f926805c120 2022-07-12T08:43:08.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9265ffb700 1 -- 172.21.15.61:0/2742847254 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c000e90 con 0x7f926805c120 2022-07-12T08:43:08.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9265ffb700 1 -- 172.21.15.61:0/2742847254 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f925c00d960 con 0x7f926805c120 2022-07-12T08:43:08.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- 172.21.15.61:0/2742847254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f926805c120 msgr2=0x7f92680b1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 --2- 172.21.15.61:0/2742847254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f926805c120 0x7f92680b1970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f925c002f70 tx=0x7f925c006df0).stop 2022-07-12T08:43:08.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- 172.21.15.61:0/2742847254 shutdown_connections 2022-07-12T08:43:08.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 --2- 172.21.15.61:0/2742847254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f926805c120 0x7f92680b1970 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- 172.21.15.61:0/2742847254 >> 172.21.15.61:0/2742847254 conn(0x7f926805f140 msgr2=0x7f926805f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:08.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- 172.21.15.61:0/2742847254 shutdown_connections 2022-07-12T08:43:08.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- 172.21.15.61:0/2742847254 wait complete. 2022-07-12T08:43:08.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 Processor -- start 2022-07-12T08:43:08.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- start start 2022-07-12T08:43:08.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 0x7f9268066ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:08.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9268060c20 0x7f9268143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:08.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92680b1e60 0x7f9268155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:08.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9268065a70 con 0x7f926805c120 2022-07-12T08:43:08.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680a3680 con 0x7f92680b1e60 2022-07-12T08:43:08.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9267fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9268155870 con 0x7f9268060c20 2022-07-12T08:43:08.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 0x7f9268066ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:08.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 0x7f9268066ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54704/0 (socket says 172.21.15.61:54704) 2022-07-12T08:43:08.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 -- 172.21.15.61:0/2197237138 learned_addr learned my addr 172.21.15.61:0/2197237138 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:08.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f92677fe700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92680b1e60 0x7f9268155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:08.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f92667fc700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9268060c20 0x7f9268143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:08.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 -- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9268060c20 msgr2=0x7f9268143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9268060c20 0x7f9268143b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 -- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92680b1e60 msgr2=0x7f9268155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92680b1e60 0x7f9268155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.376+0000 7f9266ffd700 1 -- 172.21.15.61:0/2197237138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c00b180 con 0x7f926805c120 2022-07-12T08:43:08.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9266ffd700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 0x7f9268066ab0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9258000fe0 tx=0x7f92580018e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:08.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f925800ce10 con 0x7f926805c120 2022-07-12T08:43:08.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925800f470 con 0x7f926805c120 2022-07-12T08:43:08.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9268064eb0 con 0x7f926805c120 2022-07-12T08:43:08.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f925800f7d0 con 0x7f926805c120 2022-07-12T08:43:08.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92680aeb30 con 0x7f926805c120 2022-07-12T08:43:08.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9258019030 con 0x7f926805c120 2022-07-12T08:43:08.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9268065c30 con 0x7f926805c120 2022-07-12T08:43:08.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9257fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9244053af0 0x7f9244055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:08.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f925806aeb0 con 0x7f926805c120 2022-07-12T08:43:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.380+0000 7f92667fc700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9244053af0 0x7f9244055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.384+0000 7f92667fc700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9244053af0 0x7f9244055fb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f925c018000 tx=0x7f925c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:08.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.388+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9258042090 con 0x7f926805c120 2022-07-12T08:43:08.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.520+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9268140bd0 con 0x7f926805c120 2022-07-12T08:43:08.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.528+0000 7f9257fff700 1 -- 172.21.15.61:0/2197237138 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9268140bd0 con 0x7f926805c120 2022-07-12T08:43:08.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9244053af0 msgr2=0x7f9244055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9244053af0 0x7f9244055fb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f925c018000 tx=0x7f925c014000).stop 2022-07-12T08:43:08.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 msgr2=0x7f9268066ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:08.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 0x7f9268066ab0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f9258000fe0 tx=0x7f92580018e0).stop 2022-07-12T08:43:08.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 shutdown_connections 2022-07-12T08:43:08.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9244053af0 0x7f9244055fb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f92680b1e60 0x7f9268155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9268060c20 0x7f9268143b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 --2- 172.21.15.61:0/2197237138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f926805c120 0x7f9268066ab0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:08.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 >> 172.21.15.61:0/2197237138 conn(0x7f926805f140 msgr2=0x7f92680af6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:08.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 shutdown_connections 2022-07-12T08:43:08.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:08.532+0000 7f9267fff700 1 -- 172.21.15.61:0/2197237138 wait complete. 2022-07-12T08:43:08.577 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:43:08.578 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:08.634 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:08.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:08.690 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:08.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:43:08.776 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:43:08.777 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:08.778 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:08.779 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:08.807 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:43:08.808 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:43:08.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:08.809 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:08.810 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:43:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:08.840 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:43:08.841 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:43:08.844 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:08.846 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:08.850 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:43:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:08.895 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:43:08.895 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:43:08.897 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:43:09.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 Processor -- start 2022-07-12T08:43:09.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- start start 2022-07-12T08:43:09.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9a240b5aa0 0x7f9a240b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2405d3d0 con 0x7f9a24147000 2022-07-12T08:43:09.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a240b5e80 con 0x7f9a2405bfc0 2022-07-12T08:43:09.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2414b590 con 0x7f9a240b5aa0 2022-07-12T08:43:09.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9a240b5aa0 0x7f9a240b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9a240b5aa0 0x7f9a240b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54712/0 (socket says 172.21.15.61:54712) 2022-07-12T08:43:09.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2a1c6700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9a24147000 0x7f9a24144140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42934/0 (socket says 172.21.15.61:42934) 2022-07-12T08:43:09.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 -- 172.21.15.61:0/3419990946 learned_addr learned my addr 172.21.15.61:0/3419990946 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:09.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a289c3700 1 -- 172.21.15.61:0/3419990946 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585960365 0 0) 0x7f9a2405d3d0 con 0x7f9a24147000 2022-07-12T08:43:09.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a289c3700 1 -- 172.21.15.61:0/3419990946 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a10003260 con 0x7f9a24147000 2022-07-12T08:43:09.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 -- 172.21.15.61:0/3419990946 >> v1:172.21.15.111:6789/0 conn(0x7f9a2405bfc0 legacy=0x7f9a2414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 -- 172.21.15.61:0/3419990946 >> v1:172.21.15.61:6789/0 conn(0x7f9a24147000 legacy=0x7f9a24144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 -- 172.21.15.61:0/3419990946 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2414a740 con 0x7f9a240b5aa0 2022-07-12T08:43:09.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a299c5700 1 --2- 172.21.15.61:0/3419990946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a240b0380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9a140032f0 tx=0x7f9a14009290).ready entity=mon.0 client_cookie=65716e91de601adb server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:09.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a289c3700 1 -- 172.21.15.61:0/3419990946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a1400b8a0 con 0x7f9a240b5aa0 2022-07-12T08:43:09.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a289c3700 1 -- 172.21.15.61:0/3419990946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1400ba00 con 0x7f9a240b5aa0 2022-07-12T08:43:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a289c3700 1 -- 172.21.15.61:0/3419990946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a1400bb70 con 0x7f9a240b5aa0 2022-07-12T08:43:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- 172.21.15.61:0/3419990946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 msgr2=0x7f9a240b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/3419990946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a240b0380 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f9a140032f0 tx=0x7f9a14009290).stop 2022-07-12T08:43:09.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- 172.21.15.61:0/3419990946 shutdown_connections 2022-07-12T08:43:09.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/3419990946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a240b0380 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- 172.21.15.61:0/3419990946 >> 172.21.15.61:0/3419990946 conn(0x7f9a24063520 msgr2=0x7f9a240b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:09.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- 172.21.15.61:0/3419990946 shutdown_connections 2022-07-12T08:43:09.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.028+0000 7f9a2bc29700 1 -- 172.21.15.61:0/3419990946 wait complete. 2022-07-12T08:43:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 Processor -- start 2022-07-12T08:43:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 -- start start 2022-07-12T08:43:09.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a2405bfc0 0x7f9a240650f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a2409ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a24147000 0x7f9a241597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a24156c60 con 0x7f9a240b5aa0 2022-07-12T08:43:09.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a24156e90 con 0x7f9a2405bfc0 2022-07-12T08:43:09.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a24157130 con 0x7f9a24147000 2022-07-12T08:43:09.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a2409ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a2409ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54720/0 (socket says 172.21.15.61:54720) 2022-07-12T08:43:09.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 -- 172.21.15.61:0/836682224 learned_addr learned my addr 172.21.15.61:0/836682224 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:09.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2a1c6700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a24147000 0x7f9a241597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a299c5700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a2405bfc0 0x7f9a240650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 -- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a24147000 msgr2=0x7f9a241597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a24147000 0x7f9a241597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 -- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a2405bfc0 msgr2=0x7f9a240650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a2405bfc0 0x7f9a240650f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 -- 172.21.15.61:0/836682224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a14009530 con 0x7f9a240b5aa0 2022-07-12T08:43:09.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a291c4700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a2409ee50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9a24062360 tx=0x7f9a18007bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:09.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a18007030 con 0x7f9a240b5aa0 2022-07-12T08:43:09.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18007190 con 0x7f9a240b5aa0 2022-07-12T08:43:09.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a18006510 con 0x7f9a240b5aa0 2022-07-12T08:43:09.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a24064a70 con 0x7f9a240b5aa0 2022-07-12T08:43:09.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.032+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a240a22e0 con 0x7f9a240b5aa0 2022-07-12T08:43:09.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.036+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a24057460 con 0x7f9a240b5aa0 2022-07-12T08:43:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.036+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9a18010030 con 0x7f9a240b5aa0 2022-07-12T08:43:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.036+0000 7f9a22ffd700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a040539b0 0x7f9a04055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.036+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9a1806b6b0 con 0x7f9a240b5aa0 2022-07-12T08:43:09.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.036+0000 7f9a299c5700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a040539b0 0x7f9a04055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.040+0000 7f9a299c5700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a040539b0 0x7f9a04055e70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9a1401b000 tx=0x7f9a14016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:09.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.044+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9a180428f0 con 0x7f9a240b5aa0 2022-07-12T08:43:09.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.168+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f9a2405bda0 con 0x7f9a240b5aa0 2022-07-12T08:43:09.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a22ffd700 1 -- 172.21.15.61:0/836682224 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v35) v1 ==== 81+0+28875 (secure 0 0 0) 0x7f9a18045f10 con 0x7f9a240b5aa0 2022-07-12T08:43:09.179 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:09.179 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:43:09.180 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 35, 2022-07-12T08:43:09.180 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:43:09.180 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:43:09.181 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:43:06.197536+0000", 2022-07-12T08:43:09.181 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:43:09.182 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:43:09.182 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:43:09.182 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:43:09.183 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:43:09.183 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:43:09.183 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:43:09.184 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:43:09.184 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:43:09.185 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:09.185 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:43:09.185 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:43:09.187 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:43:09.187 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:43:09.188 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:43:09.188 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 6, 2022-07-12T08:43:09.189 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:43:09.190 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:43:09.190 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:43:09.191 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:43:09.191 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:43:09.192 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.192 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:43:09.192 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:43:09.193 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:43:09.193 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:43:09.194 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:43:09.194 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:43:09.195 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:43:09.195 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:43:09.195 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:43:09.196 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:43:09.196 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:43:09.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:43:09.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:43:09.199 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:43:09.199 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:43:09.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:43:09.200 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:43:09.201 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:43:09.202 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:43:09.203 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:43:09.203 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:43:09.203 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:43:09.204 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:43:09.204 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.205 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:43:09.205 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:43:09.206 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:43:09.206 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:43:09.217 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:43:09.217 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:43:09.223 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:43:09.223 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:43:09.224 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:43:09.224 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:43:09.224 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:43:09.225 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:43:09.225 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:43:09.226 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:43:09.226 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:43:09.227 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:43:09.227 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:43:09.227 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:43:09.228 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:43:09.228 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:43:09.229 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:43:09.229 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:43:09.230 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:43:09.230 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:43:09.230 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:43:09.231 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:43:09.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:43:09.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.232 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:43:09.233 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:43:09.233 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:43:09.233 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:43:09.233 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:43:09.234 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:43:09.234 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:43:09.234 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:43:09.235 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:43:09.235 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:43:09.235 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:43:09.235 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:43:09.236 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:43:09.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.236 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:43:09.236 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:43:09.237 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.238 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:43:09.238 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:43:09.238 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:43:09.239 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:43:09.239 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:43:09.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:43:09.247 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:43:09.247 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:43:09.247 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:43:09.248 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:43:09.248 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:43:09.249 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:43:09.249 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:43:09.249 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:43:09.250 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:43:09.250 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:43:09.251 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:43:09.251 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:43:09.251 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:43:09.252 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:43:09.252 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:43:09.253 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:43:09.253 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:43:09.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:43:09.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:43:09.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:43:09.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:43:09.255 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:43:09.256 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:43:09.256 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:43:09.257 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:43:09.257 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:43:09.257 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:43:09.258 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:43:09.258 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:43:09.259 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:43:09.259 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:43:09.259 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:43:09.260 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:43:09.260 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:43:09.261 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:43:09.261 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:43:09.261 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:43:09.262 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:43:09.262 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:43:09.263 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:43:09.263 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:43:09.263 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:43:09.264 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:43:09.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:43:09.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.265 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:43:09.265 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:43:09.266 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:43:09.266 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:43:09.266 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:43:09.267 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:43:09.267 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:43:09.267 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:43:09.268 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:43:09.268 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:43:09.269 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:43:09.269 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:43:09.269 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:43:09.270 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:43:09.270 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.271 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.271 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:09.272 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:43:09.272 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.272 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:43:09.273 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:43:09.274 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.274 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.274 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.275 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.276 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.276 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:09.277 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.277 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.277 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.278 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.279 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:43:09.280 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.281 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:43:09.282 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.282 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.283 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:43:09.286 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.287 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:43:09.288 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.288 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.289 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.289 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.290 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.290 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.291 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.291 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:43:09.292 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.293 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.293 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:43:09.294 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.294 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.295 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.295 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.295 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.296 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.297 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:43:09.297 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.298 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.299 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:43:09.300 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:43:09.300 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.301 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.301 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.302 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:43:09.302 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:43:09.303 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:43:09.303 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:43:09.303 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.304 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.304 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.305 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.306 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.306 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:43:09.307 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:43:09.307 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.308 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.308 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.309 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.309 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.310 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.311 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-12T08:43:09.311 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.311 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.312 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.312 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.313 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.313 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:43:09.314 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.314 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.315 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:43:09.316 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.317 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.317 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.318 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.318 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.319 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.319 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.320 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:43:09.320 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.321 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.322 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.322 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:43:09.323 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.323 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.324 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.324 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.324 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.326 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.327 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:43:09.327 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.328 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:43:09.330 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.331 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.331 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.333 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.335 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.335 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:43:09.336 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.336 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.337 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:43:09.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:43:09.339 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.340 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.340 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:43:09.341 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:43:09.341 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:43:09.342 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:43:09.342 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.343 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.343 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.344 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.345 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.345 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:43:09.346 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:43:09.346 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.347 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.347 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.348 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.348 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.349 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.349 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.349 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 28, 2022-07-12T08:43:09.350 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.350 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.351 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.351 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.351 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.352 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.352 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:43:09.352 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.353 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.354 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:43:09.354 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.355 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.355 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.355 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.356 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.356 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:43:09.358 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.359 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:43:09.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.360 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.360 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.361 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.362 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.362 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:43:09.363 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.364 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.364 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.365 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:43:09.365 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.365 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.366 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.366 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.367 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.367 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.368 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.368 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.368 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:43:09.369 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.369 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.370 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.370 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:43:09.371 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:43:09.371 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.372 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.372 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:43:09.373 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:43:09.373 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:43:09.373 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:43:09.374 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.374 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.375 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.375 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.375 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.376 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.376 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:43:09.376 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:43:09.377 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.377 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.377 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.378 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.378 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.379 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.379 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.379 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:09.380 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.380 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.380 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.381 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.382 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:43:09.383 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.383 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.383 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.384 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.384 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:43:09.384 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.385 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.385 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.386 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.386 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.386 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.387 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.387 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:43:09.388 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.388 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.389 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.390 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:43:09.390 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.391 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.391 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.391 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.392 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.392 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.393 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:43:09.394 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.395 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.395 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:43:09.395 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.396 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.396 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.397 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.397 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.398 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.398 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:43:09.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.399 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:43:09.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:43:09.401 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.402 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:43:09.403 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:43:09.403 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:43:09.403 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:43:09.404 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.404 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.404 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.405 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.405 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.406 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.406 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:43:09.406 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:43:09.407 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.407 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.407 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.408 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.409 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.409 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.409 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.410 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:09.410 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.410 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.411 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.411 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.412 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:43:09.413 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:43:09.415 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.415 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.416 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.417 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.418 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.418 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.418 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:43:09.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.419 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.419 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.419 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:43:09.420 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.420 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.420 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.420 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.421 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.421 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.421 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.421 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.421 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:43:09.422 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.422 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.422 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.422 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.423 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:43:09.423 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.423 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.423 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.423 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.424 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.424 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.424 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.425 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:43:09.425 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.426 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.426 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.426 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:43:09.426 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:43:09.427 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.427 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.427 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.427 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:43:09.428 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:43:09.428 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:43:09.428 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:43:09.428 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.428 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.429 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.429 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.429 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.429 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.429 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:43:09.430 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:43:09.430 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.430 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.430 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.431 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.431 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.431 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.431 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.431 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:09.432 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.432 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.432 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.433 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.433 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:43:09.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.434 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.434 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:43:09.434 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.435 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.435 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.435 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.435 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.436 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:43:09.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.436 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.437 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:43:09.437 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.438 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.438 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.438 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.438 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.439 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.439 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.439 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:43:09.439 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.440 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.440 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.440 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.440 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:43:09.440 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.441 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.441 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.441 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.441 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.441 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.442 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.442 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.442 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:43:09.443 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.443 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.443 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.443 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.444 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:43:09.444 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:43:09.444 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.444 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.445 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.445 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:43:09.445 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:43:09.445 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:43:09.445 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:43:09.446 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.446 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.446 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.446 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.447 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.447 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:43:09.447 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:43:09.447 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.448 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.448 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.448 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.448 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.449 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.449 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.449 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 28, 2022-07-12T08:43:09.450 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.450 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.450 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.451 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.451 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:43:09.451 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.452 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.452 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:43:09.453 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.453 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.453 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.453 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.453 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.454 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.454 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.454 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:43:09.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.455 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.455 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:43:09.456 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.456 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.456 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.457 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.457 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.457 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.458 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:43:09.458 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.458 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.458 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.459 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.459 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:43:09.459 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.459 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.460 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.460 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.460 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.460 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.461 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.461 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.461 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:43:09.461 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.463 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.463 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.463 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:43:09.463 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:43:09.464 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.464 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.464 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.464 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:43:09.465 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:43:09.465 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:43:09.465 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:43:09.465 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.465 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.466 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.466 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.467 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.467 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.467 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:43:09.467 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:43:09.468 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:43:09.468 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:43:09.469 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:43:09.469 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:43:09.470 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:43:09.470 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:43:09.471 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:43:09.471 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 29, 2022-07-12T08:43:09.471 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:43:09.472 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:43:09.472 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:43:09.472 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.473 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.473 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.473 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:43:09.474 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.474 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.475 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.475 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:43:09.476 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.476 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.477 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.478 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.478 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:43:09.479 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.479 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.480 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.480 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:43:09.481 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.481 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.482 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.482 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.483 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:43:09.483 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.484 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.484 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.485 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.485 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:43:09.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.486 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.486 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:43:09.488 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.489 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.489 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.490 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.490 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:43:09.491 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.491 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.493 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.494 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.494 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:43:09.495 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:43:09.495 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.495 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:43:09.496 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:43:09.496 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.497 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.497 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.498 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:43:09.498 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:43:09.499 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:43:09.499 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.500 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.500 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.501 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:43:09.501 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:43:09.502 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:43:09.502 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:43:09.503 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:43:09.503 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:43:09.504 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:43:09.504 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:09.505 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.505 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:09.506 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:43:09.506 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.507 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:43:09.507 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.508 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.508 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.509 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.510 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.510 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:43:09.511 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.511 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.511 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.511 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:43:09.512 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.512 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.512 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.512 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.513 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.513 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:43:09.513 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.513 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.513 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.514 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:43:09.514 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.514 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.515 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.515 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.515 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.516 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:43:09.516 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.516 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.516 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.517 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:43:09.517 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.517 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.518 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.518 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.518 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:43:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:43:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.522 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:43:09.522 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:43:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:43:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:43:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.530 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:43:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:09.534 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:43:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:43:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:43:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:43:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:43:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:43:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:43:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:43:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:43:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:43:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:43:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:43:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:43:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:43:09.542 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/3202017529": "2022-07-13T08:43:02.662050+0000", 2022-07-12T08:43:09.543 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/3202017529": "2022-07-13T08:43:02.662050+0000" 2022-07-12T08:43:09.543 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:43:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:43:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:43:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:43:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:43:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:43:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:43:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:43:09.547 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:09.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:43:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:43:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:43:09.549 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:43:09.549 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:43:09.550 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:43:09.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a040539b0 msgr2=0x7f9a04055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a040539b0 0x7f9a04055e70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9a1401b000 tx=0x7f9a14016000).stop 2022-07-12T08:43:09.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 msgr2=0x7f9a2409ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a2409ee50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9a24062360 tx=0x7f9a18007bb0).stop 2022-07-12T08:43:09.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 shutdown_connections 2022-07-12T08:43:09.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9a24147000 0x7f9a241597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9a240b5aa0 0x7f9a2409ee50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a040539b0 0x7f9a04055e70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 --2- 172.21.15.61:0/836682224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a2405bfc0 0x7f9a240650f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 >> 172.21.15.61:0/836682224 conn(0x7f9a24063520 msgr2=0x7f9a241443b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 shutdown_connections 2022-07-12T08:43:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.172+0000 7f9a2bc29700 1 -- 172.21.15.61:0/836682224 wait complete. 2022-07-12T08:43:09.560 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:09.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.668+0000 7f2e7f59e700 1 Processor -- start 2022-07-12T08:43:09.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.668+0000 7f2e7f59e700 1 -- start start 2022-07-12T08:43:09.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.668+0000 7f2e7f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e800ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.668+0000 7f2e7f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e800abc70 con 0x7f2e800afa60 2022-07-12T08:43:09.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.668+0000 7f2e7f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8005c720 con 0x7f2e8005f140 2022-07-12T08:43:09.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8005f580 con 0x7f2e800ab290 2022-07-12T08:43:09.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2e800afa60 0x7f2e80147530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42948/0 (socket says 172.21.15.61:42948) 2022-07-12T08:43:09.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7ed9d700 1 -- 172.21.15.61:0/2340690145 learned_addr learned my addr 172.21.15.61:0/2340690145 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:09.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7e59c700 1 --2- 172.21.15.61:0/2340690145 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e800ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642978214 0 0) 0x7f2e8005c720 con 0x7f2e8005f140 2022-07-12T08:43:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e64003260 con 0x7f2e8005f140 2022-07-12T08:43:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262793985 0 0) 0x7f2e800abc70 con 0x7f2e800afa60 2022-07-12T08:43:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8005c720 con 0x7f2e800afa60 2022-07-12T08:43:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7e59c700 1 -- 172.21.15.61:0/2340690145 >> v1:172.21.15.111:6789/0 conn(0x7f2e8005f140 legacy=0x7f2e8005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7e59c700 1 -- 172.21.15.61:0/2340690145 >> v1:172.21.15.61:6789/0 conn(0x7f2e800afa60 legacy=0x7f2e80147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7e59c700 1 -- 172.21.15.61:0/2340690145 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8005c9c0 con 0x7f2e800ab290 2022-07-12T08:43:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7e59c700 1 --2- 172.21.15.61:0/2340690145 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e800ab670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2e74002f70 tx=0x7f2e74006df0).ready entity=mon.2 client_cookie=771f3b086a3e2c64 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e7400d5c0 con 0x7f2e800ab290 2022-07-12T08:43:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e74000e90 con 0x7f2e800ab290 2022-07-12T08:43:09.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7d59a700 1 -- 172.21.15.61:0/2340690145 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e7400d910 con 0x7f2e800ab290 2022-07-12T08:43:09.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 -- 172.21.15.61:0/2340690145 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 msgr2=0x7f2e800ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/2340690145 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e800ab670 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f2e74002f70 tx=0x7f2e74006df0).stop 2022-07-12T08:43:09.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 -- 172.21.15.61:0/2340690145 shutdown_connections 2022-07-12T08:43:09.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/2340690145 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e800ab670 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 -- 172.21.15.61:0/2340690145 >> 172.21.15.61:0/2340690145 conn(0x7f2e800b1950 msgr2=0x7f2e800b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:09.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 -- 172.21.15.61:0/2340690145 shutdown_connections 2022-07-12T08:43:09.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.672+0000 7f2e7f59e700 1 -- 172.21.15.61:0/2340690145 wait complete. 2022-07-12T08:43:09.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 Processor -- start 2022-07-12T08:43:09.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 -- start start 2022-07-12T08:43:09.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e8005f140 0x7f2e800ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e80145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e800afa60 0x7f2e80145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e800aed80 con 0x7f2e800afa60 2022-07-12T08:43:09.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e80064ca0 con 0x7f2e8005f140 2022-07-12T08:43:09.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e80064f40 con 0x7f2e800ab290 2022-07-12T08:43:09.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e800afa60 0x7f2e80145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e80145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e80145740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36930/0 (socket says 172.21.15.61:36930) 2022-07-12T08:43:09.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 -- 172.21.15.61:0/3469068336 learned_addr learned my addr 172.21.15.61:0/3469068336 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:09.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7e59c700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e8005f140 0x7f2e800ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 -- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e8005f140 msgr2=0x7f2e800ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e8005f140 0x7f2e800ae1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 -- 172.21.15.61:0/3469068336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e800afa60 msgr2=0x7f2e80145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e800afa60 0x7f2e80145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 -- 172.21.15.61:0/3469068336 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7400b180 con 0x7f2e800ab290 2022-07-12T08:43:09.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7dd9b700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e80145740 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f2e70001110 tx=0x7f2e70007ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:09.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e70012400 con 0x7f2e800ab290 2022-07-12T08:43:09.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70012560 con 0x7f2e800ab290 2022-07-12T08:43:09.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e800651e0 con 0x7f2e800ab290 2022-07-12T08:43:09.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e700128c0 con 0x7f2e800ab290 2022-07-12T08:43:09.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.676+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e801560f0 con 0x7f2e800ab290 2022-07-12T08:43:09.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.680+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2e70019030 con 0x7f2e800ab290 2022-07-12T08:43:09.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.680+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e80057460 con 0x7f2e800ab290 2022-07-12T08:43:09.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.680+0000 7f2e6f7fe700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e5c0539b0 0x7f2e5c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:09.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.680+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f2e7006afe0 con 0x7f2e800ab290 2022-07-12T08:43:09.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.680+0000 7f2e7e59c700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e5c0539b0 0x7f2e5c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:09.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.680+0000 7f2e7e59c700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e5c0539b0 0x7f2e5c055e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2e74018000 tx=0x7f2e74014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:09.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.688+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e70042170 con 0x7f2e800ab290 2022-07-12T08:43:09.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.856+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e8006b2b0 con 0x7f2e800ab290 2022-07-12T08:43:09.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e6f7fe700 1 -- 172.21.15.61:0/3469068336 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+1211 (secure 0 0 0) 0x7f2e70017070 con 0x7f2e800ab290 2022-07-12T08:43:09.866 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:09.867 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8},{"gid":5151,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3706199670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3706199670},{"type":"v1","addr":"172.21.15.61:6833","nonce":3706199670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":14}],"filesystems":[]} 2022-07-12T08:43:09.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e5c0539b0 msgr2=0x7f2e5c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e5c0539b0 0x7f2e5c055e70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f2e74018000 tx=0x7f2e74014000).stop 2022-07-12T08:43:09.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 msgr2=0x7f2e80145740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:09.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e80145740 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f2e70001110 tx=0x7f2e70007ac0).stop 2022-07-12T08:43:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 shutdown_connections 2022-07-12T08:43:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e800afa60 0x7f2e80145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e800ab290 0x7f2e80145740 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e5c0539b0 0x7f2e5c055e70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 --2- 172.21.15.61:0/3469068336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e8005f140 0x7f2e800ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:09.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 >> 172.21.15.61:0/3469068336 conn(0x7f2e800b1950 msgr2=0x7f2e800b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:09.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 shutdown_connections 2022-07-12T08:43:09.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:09.860+0000 7f2e7f59e700 1 -- 172.21.15.61:0/3469068336 wait complete. 2022-07-12T08:43:09.898 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-12T08:43:09.901 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:10.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d6def700 1 Processor -- start 2022-07-12T08:43:10.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d6def700 1 -- start start 2022-07-12T08:43:10.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d6def700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7d0143740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d6def700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d00b0cb0 con 0x7fb7d0142f80 2022-07-12T08:43:10.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d4b8b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb7d0142f80 0x7fb7d00af130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42958/0 (socket says 172.21.15.61:42958) 2022-07-12T08:43:10.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d4b8b700 1 -- 172.21.15.61:0/452687882 learned_addr learned my addr 172.21.15.61:0/452687882 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:10.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d00b0f50 con 0x7fb7d00af840 2022-07-12T08:43:10.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0139cf0 con 0x7fb7d00615a0 2022-07-12T08:43:10.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7cffff700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7d0143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457673575 0 0) 0x7fb7d00b0cb0 con 0x7fb7d0142f80 2022-07-12T08:43:10.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.000+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc003260 con 0x7fb7d0142f80 2022-07-12T08:43:10.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793779759 0 0) 0x7fb7d00b0f50 con 0x7fb7d00af840 2022-07-12T08:43:10.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d00b0cb0 con 0x7fb7d00af840 2022-07-12T08:43:10.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237457378 0 0) 0x7fb7bc003260 con 0x7fb7d0142f80 2022-07-12T08:43:10.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 msgr2=0x7fb7d0143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:10.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7d0143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 >> v1:172.21.15.111:6789/0 conn(0x7fb7d00af840 legacy=0x7fb7d00aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d00b11f0 con 0x7fb7d0142f80 2022-07-12T08:43:10.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cffff700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7d0143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:10.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fb7c0001660 con 0x7fb7d0142f80 2022-07-12T08:43:10.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 >> v1:172.21.15.61:6789/0 conn(0x7fb7d0142f80 legacy=0x7fb7d00af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc003910 0x7fb7bc003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7bc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7bc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0139cf0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc00b800 con 0x7fb7bc003910 2022-07-12T08:43:10.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc00b970 con 0x7fb7d00615a0 2022-07-12T08:43:10.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7bc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d4b8b700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc003910 0x7fb7bc003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cffff700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7bc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 -- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 msgr2=0x7fb7bc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d00615a0 0x7fb7bc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 -- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc003910 msgr2=0x7fb7bc003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7bc003910 0x7fb7bc003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 -- 172.21.15.61:0/452687882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c400b180 con 0x7fb7bc0049f0 2022-07-12T08:43:10.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7bc004e70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fb7c8001490 tx=0x7fb7c8007d80).ready entity=mon.0 client_cookie=c79039b87d3ebe5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:10.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7c8001780 con 0x7fb7bc0049f0 2022-07-12T08:43:10.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c8006890 con 0x7fb7bc0049f0 2022-07-12T08:43:10.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7cf7fe700 1 -- 172.21.15.61:0/452687882 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7c8006bf0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 msgr2=0x7fb7bc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d6def700 1 --2- 172.21.15.61:0/452687882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7bc004e70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fb7c8001490 tx=0x7fb7c8007d80).stop 2022-07-12T08:43:10.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d538c700 1 -- 172.21.15.61:0/452687882 reap_dead start 2022-07-12T08:43:10.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 shutdown_connections 2022-07-12T08:43:10.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 >> 172.21.15.61:0/452687882 conn(0x7fb7d005f140 msgr2=0x7fb7d005f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 shutdown_connections 2022-07-12T08:43:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.004+0000 7fb7d6def700 1 -- 172.21.15.61:0/452687882 wait complete. 2022-07-12T08:43:10.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 Processor -- start 2022-07-12T08:43:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 -- start start 2022-07-12T08:43:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7d013a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7d014f9d0 0x7fb7d013f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d013b7c0 0x7fb7d013fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d01401e0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d014fe70 con 0x7fb7d014f9d0 2022-07-12T08:43:10.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0150110 con 0x7fb7d013b7c0 2022-07-12T08:43:10.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7d013a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7d013a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54752/0 (socket says 172.21.15.61:54752) 2022-07-12T08:43:10.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 -- 172.21.15.61:0/2382693225 learned_addr learned my addr 172.21.15.61:0/2382693225 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:10.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7cffff700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7d014f9d0 0x7fb7d013f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d538c700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d013b7c0 0x7fb7d013fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 -- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d013b7c0 msgr2=0x7fb7d013fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d013b7c0 0x7fb7d013fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 -- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7d014f9d0 msgr2=0x7fb7d013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7d014f9d0 0x7fb7d013f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 -- 172.21.15.61:0/2382693225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d01503b0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d538c700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d013b7c0 0x7fb7d013fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d4b8b700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7d013a1a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb7c000d000 tx=0x7fb7d00ad060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7c00116b0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c0011810 con 0x7fb7bc0049f0 2022-07-12T08:43:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d0064ae0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7c0011b70 con 0x7fb7bc0049f0 2022-07-12T08:43:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.008+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d0066540 con 0x7fb7bc0049f0 2022-07-12T08:43:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.012+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d00b1690 con 0x7fb7bc0049f0 2022-07-12T08:43:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.012+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb7c001b030 con 0x7fb7bc0049f0 2022-07-12T08:43:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.012+0000 7fb7cdffb700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb7b0053a60 0x7fb7b0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.012+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb7c006b460 con 0x7fb7bc0049f0 2022-07-12T08:43:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.012+0000 7fb7cffff700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb7b0053a60 0x7fb7b0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.012+0000 7fb7cffff700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb7b0053a60 0x7fb7b0055f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb7c400b000 tx=0x7fb7c4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.020+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7c00426a0 con 0x7fb7bc0049f0 2022-07-12T08:43:10.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.184+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d0150540 con 0x7fb7bc0049f0 2022-07-12T08:43:10.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.184+0000 7fb7cdffb700 1 -- 172.21.15.61:0/2382693225 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+1211 (secure 0 0 0) 0x7fb7c0019030 con 0x7fb7bc0049f0 2022-07-12T08:43:10.205 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:10.206 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4902,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/4144851178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4144851178},{"type":"v1","addr":"172.21.15.111:6835","nonce":4144851178}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":8},{"gid":5151,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3706199670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3706199670},{"type":"v1","addr":"172.21.15.61:6833","nonce":3706199670}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":14}],"filesystems":[]} 2022-07-12T08:43:10.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb7b0053a60 msgr2=0x7fb7b0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb7b0053a60 0x7fb7b0055f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fb7c400b000 tx=0x7fb7c4006df0).stop 2022-07-12T08:43:10.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 msgr2=0x7fb7d013a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7d013a1a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb7c000d000 tx=0x7fb7d00ad060).stop 2022-07-12T08:43:10.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 shutdown_connections 2022-07-12T08:43:10.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb7d013b7c0 0x7fb7d013fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb7b0053a60 0x7fb7b0055f20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb7d014f9d0 0x7fb7d013f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 --2- 172.21.15.61:0/2382693225 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7bc0049f0 0x7fb7d013a1a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 >> 172.21.15.61:0/2382693225 conn(0x7fb7d005f140 msgr2=0x7fb7d00ab1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 shutdown_connections 2022-07-12T08:43:10.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.188+0000 7fb7d6def700 1 -- 172.21.15.61:0/2382693225 wait complete. 2022-07-12T08:43:10.215 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-12T08:43:10.218 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:43:10.218 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:43:10.219 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:43:10.220 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:43:10.220 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:43:10.221 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:43:10.221 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:43:10.222 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:43:10.240 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:43:10.243 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:43:10.245 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:43:10.246 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:43:10.283 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:43:10.285 DEBUG:teuthology.parallel:result is None 2022-07-12T08:43:10.288 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:43:10.290 DEBUG:teuthology.parallel:result is None 2022-07-12T08:43:10.291 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:43:10.350 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:43:10.371 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:43:10.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 Processor -- start 2022-07-12T08:43:10.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 -- start start 2022-07-12T08:43:10.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa39c0b3c60 0x7fa39c145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa39c067860 0x7fa39c146760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c0afca0 con 0x7fa39c067860 2022-07-12T08:43:10.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c05d440 con 0x7fa39c0b3c60 2022-07-12T08:43:10.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.392+0000 7fa39b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0aff00 con 0x7fa39c05d060 2022-07-12T08:43:10.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa39c067860 0x7fa39c146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39a59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa399d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa39c0b3c60 0x7fa39c145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa39c067860 0x7fa39c146760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54776/0 (socket says 172.21.15.61:54776) 2022-07-12T08:43:10.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39a59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c145680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:36968/0 (socket says 172.21.15.61:36968) 2022-07-12T08:43:10.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 -- 172.21.15.61:0/227818181 learned_addr learned my addr 172.21.15.61:0/227818181 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:10.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 -- 172.21.15.61:0/227818181 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 msgr2=0x7fa39c145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:10.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 --2- 172.21.15.61:0/227818181 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 -- 172.21.15.61:0/227818181 >> v2:172.21.15.111:3300/0 conn(0x7fa39c0b3c60 msgr2=0x7fa39c145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:10.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 --2- 172.21.15.61:0/227818181 >> v2:172.21.15.111:3300/0 conn(0x7fa39c0b3c60 0x7fa39c145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 -- 172.21.15.61:0/227818181 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c146060 con 0x7fa39c067860 2022-07-12T08:43:10.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39ad9d700 1 --2- 172.21.15.61:0/227818181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c067860 0x7fa39c146760 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fa390002f70 tx=0x7fa390006eb0).ready entity=mon.0 client_cookie=a2bed95cb467de5d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:10.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39959a700 1 -- 172.21.15.61:0/227818181 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39000d570 con 0x7fa39c067860 2022-07-12T08:43:10.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39959a700 1 -- 172.21.15.61:0/227818181 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa390000e90 con 0x7fa39c067860 2022-07-12T08:43:10.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39959a700 1 -- 172.21.15.61:0/227818181 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa39000d900 con 0x7fa39c067860 2022-07-12T08:43:10.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 -- 172.21.15.61:0/227818181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c067860 msgr2=0x7fa39c146760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 --2- 172.21.15.61:0/227818181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c067860 0x7fa39c146760 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fa390002f70 tx=0x7fa390006eb0).stop 2022-07-12T08:43:10.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 -- 172.21.15.61:0/227818181 shutdown_connections 2022-07-12T08:43:10.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 --2- 172.21.15.61:0/227818181 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c067860 0x7fa39c146760 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 --2- 172.21.15.61:0/227818181 >> v2:172.21.15.111:3300/0 conn(0x7fa39c0b3c60 0x7fa39c145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 --2- 172.21.15.61:0/227818181 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 -- 172.21.15.61:0/227818181 >> 172.21.15.61:0/227818181 conn(0x7fa39c0b67c0 msgr2=0x7fa39c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:10.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 -- 172.21.15.61:0/227818181 shutdown_connections 2022-07-12T08:43:10.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.396+0000 7fa39b59e700 1 -- 172.21.15.61:0/227818181 wait complete. 2022-07-12T08:43:10.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 Processor -- start 2022-07-12T08:43:10.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 -- start start 2022-07-12T08:43:10.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c09d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c067860 0x7fa39c090dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 0x7fa39c0912b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c08fa70 con 0x7fa39c0b3c60 2022-07-12T08:43:10.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c152c10 con 0x7fa39c067860 2022-07-12T08:43:10.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c152970 con 0x7fa39c05d060 2022-07-12T08:43:10.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 0x7fa39c0912b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 0x7fa39c0912b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54782/0 (socket says 172.21.15.61:54782) 2022-07-12T08:43:10.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 -- 172.21.15.61:0/645933184 learned_addr learned my addr 172.21.15.61:0/645933184 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:10.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39a59c700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c09d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa399d9b700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c067860 0x7fa39c090dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 -- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 msgr2=0x7fa39c09d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c09d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 -- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c067860 msgr2=0x7fa39c090dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c067860 0x7fa39c090dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 -- 172.21.15.61:0/645933184 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39000b1e0 con 0x7fa39c0b3c60 2022-07-12T08:43:10.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39ad9d700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 0x7fa39c0912b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fa39c09d0a0 tx=0x7fa390015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa390011610 con 0x7fa39c0b3c60 2022-07-12T08:43:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa390011770 con 0x7fa39c0b3c60 2022-07-12T08:43:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c152650 con 0x7fa39c0b3c60 2022-07-12T08:43:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa390011ad0 con 0x7fa39c0b3c60 2022-07-12T08:43:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.400+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c09d980 con 0x7fa39c0b3c60 2022-07-12T08:43:10.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.404+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa390025030 con 0x7fa39c0b3c60 2022-07-12T08:43:10.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.404+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c1527e0 con 0x7fa39c0b3c60 2022-07-12T08:43:10.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.404+0000 7fa38b7fe700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa378053af0 0x7fa378055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:10.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.404+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa39006ca30 con 0x7fa39c0b3c60 2022-07-12T08:43:10.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.404+0000 7fa39a59c700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa378053af0 0x7fa378055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:10.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.404+0000 7fa39a59c700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa378053af0 0x7fa378055fb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa39400b000 tx=0x7fa3940091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:10.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.408+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa390043c70 con 0x7fa39c0b3c60 2022-07-12T08:43:10.446 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2022-07-12T08:43:10.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:10.552+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fa39c09d0e0 con 0x7fa39c0b3c60 2022-07-12T08:43:11.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.412+0000 7fa38b7fe700 1 -- 172.21.15.61:0/645933184 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa39c09d0e0 con 0x7fa39c0b3c60 2022-07-12T08:43:11.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.416+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa378053af0 msgr2=0x7fa378055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.416+0000 7fa39b59e700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa378053af0 0x7fa378055fb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa39400b000 tx=0x7fa3940091d0).stop 2022-07-12T08:43:11.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.416+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 msgr2=0x7fa39c0912b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.416+0000 7fa39b59e700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 0x7fa39c0912b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fa39c09d0a0 tx=0x7fa390015000).stop 2022-07-12T08:43:11.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.416+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 shutdown_connections 2022-07-12T08:43:11.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa39c0b3c60 0x7fa39c0912b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa378053af0 0x7fa378055fb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa39c067860 0x7fa39c090dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 --2- 172.21.15.61:0/645933184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa39c05d060 0x7fa39c09d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 >> 172.21.15.61:0/645933184 conn(0x7fa39c0b67c0 msgr2=0x7fa39c09ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:11.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 shutdown_connections 2022-07-12T08:43:11.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.420+0000 7fa39b59e700 1 -- 172.21.15.61:0/645933184 wait complete. 2022-07-12T08:43:11.439 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:43:11.453 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:43:11.454 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:43:11.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.556+0000 7fd70ffff700 1 Processor -- start 2022-07-12T08:43:11.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.556+0000 7fd70ffff700 1 -- start start 2022-07-12T08:43:11.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7100b6850 0x7fd71009ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:11.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd710146470 con 0x7fd7100b6470 2022-07-12T08:43:11.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd710146710 con 0x7fd71009d370 2022-07-12T08:43:11.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7101469b0 con 0x7fd7100b6850 2022-07-12T08:43:11.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd7100b6470 0x7fd710142120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43006/0 (socket says 172.21.15.61:43006) 2022-07-12T08:43:11.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70effd700 1 -- 172.21.15.61:0/2440334322 learned_addr learned my addr 172.21.15.61:0/2440334322 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70f7fe700 1 --2- 172.21.15.61:0/2440334322 >> v2:172.21.15.61:3300/0 conn(0x7fd7100b6850 0x7fd71009ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70e7fc700 1 -- 172.21.15.61:0/2440334322 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544482850 0 0) 0x7fd710146470 con 0x7fd7100b6470 2022-07-12T08:43:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70f7fe700 1 -- 172.21.15.61:0/2440334322 >> v1:172.21.15.111:6789/0 conn(0x7fd71009d370 legacy=0x7fd71009e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70f7fe700 1 -- 172.21.15.61:0/2440334322 >> v1:172.21.15.61:6789/0 conn(0x7fd7100b6470 legacy=0x7fd710142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70f7fe700 1 -- 172.21.15.61:0/2440334322 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd710146e50 con 0x7fd7100b6850 2022-07-12T08:43:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70f7fe700 1 --2- 172.21.15.61:0/2440334322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100b6850 0x7fd71009ea30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd7080032f0 tx=0x7fd7080091d0).ready entity=mon.0 client_cookie=cd54ca4a3e09ba7f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:11.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70e7fc700 1 -- 172.21.15.61:0/2440334322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd708009ea0 con 0x7fd7100b6850 2022-07-12T08:43:11.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70e7fc700 1 -- 172.21.15.61:0/2440334322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70800b8a0 con 0x7fd7100b6850 2022-07-12T08:43:11.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70e7fc700 1 -- 172.21.15.61:0/2440334322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd70800ba10 con 0x7fd7100b6850 2022-07-12T08:43:11.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- 172.21.15.61:0/2440334322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100b6850 msgr2=0x7fd71009ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 --2- 172.21.15.61:0/2440334322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100b6850 0x7fd71009ea30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd7080032f0 tx=0x7fd7080091d0).stop 2022-07-12T08:43:11.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- 172.21.15.61:0/2440334322 shutdown_connections 2022-07-12T08:43:11.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 --2- 172.21.15.61:0/2440334322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7100b6850 0x7fd71009ea30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- 172.21.15.61:0/2440334322 >> 172.21.15.61:0/2440334322 conn(0x7fd7100b9d50 msgr2=0x7fd7100b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- 172.21.15.61:0/2440334322 shutdown_connections 2022-07-12T08:43:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.560+0000 7fd70ffff700 1 -- 172.21.15.61:0/2440334322 wait complete. 2022-07-12T08:43:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 Processor -- start 2022-07-12T08:43:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 -- start start 2022-07-12T08:43:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 0x7fd710151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7100b6470 0x7fd7101522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7100b6850 0x7fd7100aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71009f450 con 0x7fd71009d370 2022-07-12T08:43:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd710152aa0 con 0x7fd7100b6850 2022-07-12T08:43:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd710152d40 con 0x7fd7100b6470 2022-07-12T08:43:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 0x7fd710151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 0x7fd710151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54794/0 (socket says 172.21.15.61:54794) 2022-07-12T08:43:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 -- 172.21.15.61:0/225004818 learned_addr learned my addr 172.21.15.61:0/225004818 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70f7fe700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7100b6850 0x7fd7100aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd707fff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7100b6470 0x7fd7101522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 -- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7100b6470 msgr2=0x7fd7101522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7100b6470 0x7fd7101522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 -- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7100b6850 msgr2=0x7fd7100aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7100b6850 0x7fd7100aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 -- 172.21.15.61:0/225004818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd708009470 con 0x7fd71009d370 2022-07-12T08:43:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.564+0000 7fd70effd700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 0x7fd710151df0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fd700000f90 tx=0x7fd7000018a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7000070d0 con 0x7fd71009d370 2022-07-12T08:43:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd700007230 con 0x7fd71009d370 2022-07-12T08:43:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd710152830 con 0x7fd71009d370 2022-07-12T08:43:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7000073a0 con 0x7fd71009d370 2022-07-12T08:43:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd710140ac0 con 0x7fd71009d370 2022-07-12T08:43:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd710143f40 con 0x7fd71009d370 2022-07-12T08:43:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd70001a060 con 0x7fd71009d370 2022-07-12T08:43:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70cff9700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6f40539b0 0x7fd6f4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd70006b250 con 0x7fd71009d370 2022-07-12T08:43:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.568+0000 7fd707fff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6f40539b0 0x7fd6f4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.572+0000 7fd707fff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6f40539b0 0x7fd6f4055e70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd6f800d000 tx=0x7fd6f8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.576+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd710143f40 con 0x7fd71009d370 2022-07-12T08:43:11.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.744+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd710143f40 con 0x7fd71009d370 2022-07-12T08:43:11.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.744+0000 7fd70cff9700 1 -- 172.21.15.61:0/225004818 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fd710143f40 con 0x7fd71009d370 2022-07-12T08:43:11.775 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:11.775 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:43:11.776 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:43:11.776 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.777 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:43:11.777 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:43:11.778 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.778 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:43:11.779 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.779 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.780 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.780 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.781 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.782 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:43:11.782 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:43:11.783 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.783 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:43:11.784 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.786 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.787 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.787 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.788 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.788 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.788 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:43:11.789 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:43:11.789 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.790 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.790 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.791 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.791 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.792 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.794 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.798 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:43:11.798 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:43:11.798 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.799 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.799 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.800 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.800 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.801 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.801 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:43:11.802 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:43:11.802 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.803 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.803 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.803 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.804 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.804 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.806 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.806 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:43:11.807 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:43:11.807 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.807 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.814 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.815 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.819 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.820 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.820 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.821 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:43:11.821 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:43:11.821 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.822 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.822 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.823 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.823 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.824 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.825 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:43:11.825 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:43:11.825 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.826 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.827 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.827 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.827 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.828 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.828 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:43:11.829 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:43:11.829 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.831 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.831 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.832 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.832 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.833 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.833 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.833 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:43:11.834 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:43:11.834 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.834 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.834 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.835 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.835 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.836 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.837 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.837 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:43:11.837 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:43:11.838 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.838 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:43:11.839 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:43:11.839 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:43:11.839 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.839 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.840 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.840 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:43:11.841 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:43:11.841 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.841 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:43:11.842 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:43:11.843 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:43:11.843 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.843 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.844 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.844 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:43:11.844 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:43:11.844 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.845 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:43:11.845 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:43:11.845 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:43:11.846 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.846 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.847 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.847 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:43:11.847 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:43:11.847 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.848 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:43:11.848 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:43:11.849 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:43:11.849 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.849 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.850 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.850 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.851 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:43:11.851 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:43:11.852 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.852 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:43:11.853 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:43:11.853 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.853 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:11.857 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:11.858 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:43:11.858 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:43:11.859 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:43:11.859 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:43:11.859 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:43:11.860 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:43:11.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:11.862 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:43:11.862 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:43:11.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6f40539b0 msgr2=0x7fd6f4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6f40539b0 0x7fd6f4055e70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd6f800d000 tx=0x7fd6f8007b90).stop 2022-07-12T08:43:11.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 msgr2=0x7fd710151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:11.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 0x7fd710151df0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fd700000f90 tx=0x7fd7000018a0).stop 2022-07-12T08:43:11.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 shutdown_connections 2022-07-12T08:43:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd6f40539b0 0x7fd6f4055e70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7100b6850 0x7fd7100aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7100b6470 0x7fd7101522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 --2- 172.21.15.61:0/225004818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd71009d370 0x7fd710151df0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:11.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 >> 172.21.15.61:0/225004818 conn(0x7fd7100b9d50 msgr2=0x7fd7100b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:11.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 shutdown_connections 2022-07-12T08:43:11.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:11.748+0000 7fd70ffff700 1 -- 172.21.15.61:0/225004818 wait complete. 2022-07-12T08:43:11.888 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:43:12.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 Processor -- start 2022-07-12T08:43:12.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 -- start start 2022-07-12T08:43:12.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0e900b4250 0x7f0e90066440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900ad1c0 0x7f0e90140050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e900b4630 con 0x7f0e900b2480 2022-07-12T08:43:12.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e90066930 con 0x7f0e900b4250 2022-07-12T08:43:12.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900acce0 con 0x7f0e900ad1c0 2022-07-12T08:43:12.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0e900b4250 0x7f0e90066440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0e900b4250 0x7f0e90066440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54802/0 (socket says 172.21.15.61:54802) 2022-07-12T08:43:12.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 -- 172.21.15.61:0/2063535942 learned_addr learned my addr 172.21.15.61:0/2063535942 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 -- 172.21.15.61:0/2063535942 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900ad1c0 msgr2=0x7f0e90140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:12.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96a68700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900ad1c0 0x7f0e90140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900ad1c0 0x7f0e90140050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 -- 172.21.15.61:0/2063535942 >> v1:172.21.15.111:6789/0 conn(0x7f0e900b2480 legacy=0x7f0e90144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:12.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 -- 172.21.15.61:0/2063535942 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e90065fa0 con 0x7f0e900b4250 2022-07-12T08:43:12.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96a68700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900ad1c0 0x7f0e90140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:43:12.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e96267700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900b4250 0x7f0e90066440 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0e900b8510 tx=0x7f0e88009290).ready entity=mon.0 client_cookie=bf6b0f421dc44243 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e95265700 1 -- 172.21.15.61:0/2063535942 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e8800b860 con 0x7f0e900b4250 2022-07-12T08:43:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e95265700 1 -- 172.21.15.61:0/2063535942 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8800b9c0 con 0x7f0e900b4250 2022-07-12T08:43:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e95265700 1 -- 172.21.15.61:0/2063535942 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e8800bb70 con 0x7f0e900b4250 2022-07-12T08:43:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 -- 172.21.15.61:0/2063535942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900b4250 msgr2=0x7f0e90066440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.004+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900b4250 0x7f0e90066440 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0e900b8510 tx=0x7f0e88009290).stop 2022-07-12T08:43:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 -- 172.21.15.61:0/2063535942 shutdown_connections 2022-07-12T08:43:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900ad1c0 0x7f0e90140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2063535942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900b4250 0x7f0e90066440 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 -- 172.21.15.61:0/2063535942 >> 172.21.15.61:0/2063535942 conn(0x7f0e90064e10 msgr2=0x7f0e90065220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 -- 172.21.15.61:0/2063535942 shutdown_connections 2022-07-12T08:43:12.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 -- 172.21.15.61:0/2063535942 wait complete. 2022-07-12T08:43:12.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 Processor -- start 2022-07-12T08:43:12.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.008+0000 7f0e984cb700 1 -- start start 2022-07-12T08:43:12.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 0x7f0e90142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e900b2480 0x7f0e9013d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900b4250 0x7f0e900b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900b53b0 con 0x7f0e900ad1c0 2022-07-12T08:43:12.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e90142330 con 0x7f0e900b2480 2022-07-12T08:43:12.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900b68e0 con 0x7f0e900b4250 2022-07-12T08:43:12.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 0x7f0e90142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 0x7f0e90142cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54808/0 (socket says 172.21.15.61:54808) 2022-07-12T08:43:12.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 -- 172.21.15.61:0/2204830360 learned_addr learned my addr 172.21.15.61:0/2204830360 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e95a66700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e900b2480 0x7f0e9013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96a68700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900b4250 0x7f0e900b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 -- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900b4250 msgr2=0x7f0e900b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900b4250 0x7f0e900b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 -- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e900b2480 msgr2=0x7f0e9013d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e900b2480 0x7f0e9013d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 -- 172.21.15.61:0/2204830360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e88009530 con 0x7f0e900ad1c0 2022-07-12T08:43:12.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e96267700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 0x7f0e90142cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f0e900b8510 tx=0x7f0e88016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e88021630 con 0x7f0e900ad1c0 2022-07-12T08:43:12.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e88021790 con 0x7f0e900ad1c0 2022-07-12T08:43:12.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9013de60 con 0x7f0e900ad1c0 2022-07-12T08:43:12.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e88021af0 con 0x7f0e900ad1c0 2022-07-12T08:43:12.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e90067840 con 0x7f0e900ad1c0 2022-07-12T08:43:12.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e900b5520 con 0x7f0e900ad1c0 2022-07-12T08:43:12.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.012+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0e8800f070 con 0x7f0e900ad1c0 2022-07-12T08:43:12.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.016+0000 7f0e8f7fe700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0e74053ad0 0x7f0e74055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.016+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0e8806d7b0 con 0x7f0e900ad1c0 2022-07-12T08:43:12.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.016+0000 7f0e95a66700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0e74053ad0 0x7f0e74055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.016+0000 7f0e95a66700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0e74053ad0 0x7f0e74055f90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0e7800a000 tx=0x7f0e78008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.020+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0e88044a40 con 0x7f0e900ad1c0 2022-07-12T08:43:12.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.144+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e900b6700 con 0x7f0e900ad1c0 2022-07-12T08:43:12.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.144+0000 7f0e8f7fe700 1 -- 172.21.15.61:0/2204830360 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13229 (secure 0 0 0) 0x7f0e88015030 con 0x7f0e900ad1c0 2022-07-12T08:43:12.159 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:12.159 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:43:11.410335+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:43:12.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0e74053ad0 msgr2=0x7f0e74055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0e74053ad0 0x7f0e74055f90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f0e7800a000 tx=0x7f0e78008040).stop 2022-07-12T08:43:12.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 msgr2=0x7f0e90142cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 0x7f0e90142cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f0e900b8510 tx=0x7f0e88016000).stop 2022-07-12T08:43:12.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 shutdown_connections 2022-07-12T08:43:12.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e900b4250 0x7f0e900b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0e74053ad0 0x7f0e74055f90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e900b2480 0x7f0e9013d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 --2- 172.21.15.61:0/2204830360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e900ad1c0 0x7f0e90142cc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 >> 172.21.15.61:0/2204830360 conn(0x7f0e90064e10 msgr2=0x7f0e9013ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 shutdown_connections 2022-07-12T08:43:12.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.148+0000 7f0e984cb700 1 -- 172.21.15.61:0/2204830360 wait complete. 2022-07-12T08:43:12.218 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:43:12.219 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:43:12.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7efc86700 1 Processor -- start 2022-07-12T08:43:12.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7efc86700 1 -- start start 2022-07-12T08:43:12.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7efc86700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe7e80b3ea0 0x7fe7e80b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7efc86700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80b3120 con 0x7fe7e805c810 2022-07-12T08:43:12.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7efc86700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80b33c0 con 0x7fe7e80b9db0 2022-07-12T08:43:12.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7efc86700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e805bd10 con 0x7fe7e80b3ea0 2022-07-12T08:43:12.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ee223700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe7e805c810 0x7fe7e80b4c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43038/0 (socket says 172.21.15.61:43038) 2022-07-12T08:43:12.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ed221700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe7e80b3ea0 0x7fe7e80b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ee223700 1 -- 172.21.15.61:0/760088018 learned_addr learned my addr 172.21.15.61:0/760088018 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:12.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7eca20700 1 -- 172.21.15.61:0/760088018 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707919084 0 0) 0x7fe7e80b3120 con 0x7fe7e805c810 2022-07-12T08:43:12.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7eca20700 1 -- 172.21.15.61:0/760088018 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d0003260 con 0x7fe7e805c810 2022-07-12T08:43:12.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7eca20700 1 -- 172.21.15.61:0/760088018 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649517626 0 0) 0x7fe7e80b33c0 con 0x7fe7e80b9db0 2022-07-12T08:43:12.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ed221700 1 -- 172.21.15.61:0/760088018 >> v1:172.21.15.111:6789/0 conn(0x7fe7e80b9db0 legacy=0x7fe7e805c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ed221700 1 -- 172.21.15.61:0/760088018 >> v1:172.21.15.61:6789/0 conn(0x7fe7e805c810 legacy=0x7fe7e80b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ed221700 1 -- 172.21.15.61:0/760088018 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e805cc60 con 0x7fe7e80b3ea0 2022-07-12T08:43:12.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.332+0000 7fe7ed221700 1 --2- 172.21.15.61:0/760088018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e80b3ea0 0x7fe7e80b2a40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fe7e80b1230 tx=0x7fe7d8006df0).ready entity=mon.0 client_cookie=f2a4dece01af9c93 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7eca20700 1 -- 172.21.15.61:0/760088018 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7d800d610 con 0x7fe7e80b3ea0 2022-07-12T08:43:12.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7eca20700 1 -- 172.21.15.61:0/760088018 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d8000e90 con 0x7fe7e80b3ea0 2022-07-12T08:43:12.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7eca20700 1 -- 172.21.15.61:0/760088018 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7d800d960 con 0x7fe7e80b3ea0 2022-07-12T08:43:12.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 -- 172.21.15.61:0/760088018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e80b3ea0 msgr2=0x7fe7e80b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 --2- 172.21.15.61:0/760088018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e80b3ea0 0x7fe7e80b2a40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fe7e80b1230 tx=0x7fe7d8006df0).stop 2022-07-12T08:43:12.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 -- 172.21.15.61:0/760088018 shutdown_connections 2022-07-12T08:43:12.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 --2- 172.21.15.61:0/760088018 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e80b3ea0 0x7fe7e80b2a40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 -- 172.21.15.61:0/760088018 >> 172.21.15.61:0/760088018 conn(0x7fe7e8067960 msgr2=0x7fe7e80b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 -- 172.21.15.61:0/760088018 shutdown_connections 2022-07-12T08:43:12.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 -- 172.21.15.61:0/760088018 wait complete. 2022-07-12T08:43:12.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 Processor -- start 2022-07-12T08:43:12.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.336+0000 7fe7efc86700 1 -- start start 2022-07-12T08:43:12.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 0x7fe7e80af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e80b3ea0 0x7fe7e8147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e80b9db0 0x7fe7e8151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8152240 con 0x7fe7e805c810 2022-07-12T08:43:12.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8155380 con 0x7fe7e80b9db0 2022-07-12T08:43:12.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e80ae3a0 con 0x7fe7e80b3ea0 2022-07-12T08:43:12.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 0x7fe7e80af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 0x7fe7e80af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54824/0 (socket says 172.21.15.61:54824) 2022-07-12T08:43:12.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 -- 172.21.15.61:0/2537082266 learned_addr learned my addr 172.21.15.61:0/2537082266 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7ed221700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e80b3ea0 0x7fe7e8147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7ee223700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e80b9db0 0x7fe7e8151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 -- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e80b3ea0 msgr2=0x7fe7e8147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e80b3ea0 0x7fe7e8147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 -- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e80b9db0 msgr2=0x7fe7e8151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e80b9db0 0x7fe7e8151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 -- 172.21.15.61:0/2537082266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d800b180 con 0x7fe7e805c810 2022-07-12T08:43:12.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7eda22700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 0x7fe7e80af260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe7e8114390 tx=0x7fe7e4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7e4012400 con 0x7fe7e805c810 2022-07-12T08:43:12.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e4012560 con 0x7fe7e805c810 2022-07-12T08:43:12.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e80ae640 con 0x7fe7e805c810 2022-07-12T08:43:12.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7e40128c0 con 0x7fe7e805c810 2022-07-12T08:43:12.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.340+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e80ae020 con 0x7fe7e805c810 2022-07-12T08:43:12.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.344+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e8057460 con 0x7fe7e805c810 2022-07-12T08:43:12.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.344+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe7e4019030 con 0x7fe7e805c810 2022-07-12T08:43:12.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.344+0000 7fe7d6ffd700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c8053a40 0x7fe7c8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.344+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe7e406afe0 con 0x7fe7e805c810 2022-07-12T08:43:12.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.344+0000 7fe7ed221700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c8053a40 0x7fe7c8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.344+0000 7fe7ed221700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c8053a40 0x7fe7c8055f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe7d801a000 tx=0x7fe7d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.352+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe7e4042180 con 0x7fe7e805c810 2022-07-12T08:43:12.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.476+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe7e806a940 con 0x7fe7e805c810 2022-07-12T08:43:12.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.476+0000 7fe7d6ffd700 1 -- 172.21.15.61:0/2537082266 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v38) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe7e40457a0 con 0x7fe7e805c810 2022-07-12T08:43:12.498 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:43:12.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c8053a40 msgr2=0x7fe7c8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c8053a40 0x7fe7c8055f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fe7d801a000 tx=0x7fe7d8014000).stop 2022-07-12T08:43:12.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 msgr2=0x7fe7e80af260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 0x7fe7e80af260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fe7e8114390 tx=0x7fe7e4007c50).stop 2022-07-12T08:43:12.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 shutdown_connections 2022-07-12T08:43:12.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe7c8053a40 0x7fe7c8055f00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7e80b9db0 0x7fe7e8151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7e80b3ea0 0x7fe7e8147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 --2- 172.21.15.61:0/2537082266 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7e805c810 0x7fe7e80af260 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 >> 172.21.15.61:0/2537082266 conn(0x7fe7e8067960 msgr2=0x7fe7e8146660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 shutdown_connections 2022-07-12T08:43:12.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.480+0000 7fe7efc86700 1 -- 172.21.15.61:0/2537082266 wait complete. 2022-07-12T08:43:12.509 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:43:12.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.620+0000 7fbb1b59e700 1 Processor -- start 2022-07-12T08:43:12.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.620+0000 7fbb1b59e700 1 -- start start 2022-07-12T08:43:12.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b2c40 0x7fbb1c0aec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb1c0b1180 0x7fbb1c13fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1c0ae2c0 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1c0ae520 con 0x7fbb1c0b1180 2022-07-12T08:43:12.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0ae7c0 con 0x7fbb1c0b2c40 2022-07-12T08:43:12.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb1c0b1180 0x7fbb1c13fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb1c0b1180 0x7fbb1c13fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54834/0 (socket says 172.21.15.61:54834) 2022-07-12T08:43:12.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 -- 172.21.15.61:0/2602597173 learned_addr learned my addr 172.21.15.61:0/2602597173 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1a59c700 1 --2- 172.21.15.61:0/2602597173 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b2c40 0x7fbb1c0aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 -- 172.21.15.61:0/2602597173 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b2c40 msgr2=0x7fbb1c0aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:12.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 --2- 172.21.15.61:0/2602597173 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b2c40 0x7fbb1c0aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 -- 172.21.15.61:0/2602597173 >> v1:172.21.15.111:6789/0 conn(0x7fbb1c0b20f0 legacy=0x7fbb1c142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 -- 172.21.15.61:0/2602597173 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c0b3020 con 0x7fbb1c0b1180 2022-07-12T08:43:12.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1ad9d700 1 --2- 172.21.15.61:0/2602597173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b1180 0x7fbb1c13fb30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fbb14002f70 tx=0x7fbb14006eb0).ready entity=mon.0 client_cookie=b92d27b82263aff8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1959a700 1 -- 172.21.15.61:0/2602597173 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb1400d570 con 0x7fbb1c0b1180 2022-07-12T08:43:12.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1959a700 1 -- 172.21.15.61:0/2602597173 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb14000e90 con 0x7fbb1c0b1180 2022-07-12T08:43:12.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1959a700 1 -- 172.21.15.61:0/2602597173 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb1400d900 con 0x7fbb1c0b1180 2022-07-12T08:43:12.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2602597173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b1180 msgr2=0x7fbb1c13fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2602597173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b1180 0x7fbb1c13fb30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fbb14002f70 tx=0x7fbb14006eb0).stop 2022-07-12T08:43:12.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2602597173 shutdown_connections 2022-07-12T08:43:12.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2602597173 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b1180 0x7fbb1c13fb30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2602597173 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b2c40 0x7fbb1c0aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2602597173 >> 172.21.15.61:0/2602597173 conn(0x7fbb1c061f50 msgr2=0x7fbb1c063a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2602597173 shutdown_connections 2022-07-12T08:43:12.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.624+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2602597173 wait complete. 2022-07-12T08:43:12.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 Processor -- start 2022-07-12T08:43:12.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 -- start start 2022-07-12T08:43:12.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b1180 0x7fbb1c13b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 0x7fbb1c13e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb1c0b2c40 0x7fbb1c13ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c13db10 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c13ddb0 con 0x7fbb1c0b2c40 2022-07-12T08:43:12.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c13e250 con 0x7fbb1c0b1180 2022-07-12T08:43:12.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 0x7fbb1c13e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 0x7fbb1c13e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54838/0 (socket says 172.21.15.61:54838) 2022-07-12T08:43:12.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 -- 172.21.15.61:0/2489600086 learned_addr learned my addr 172.21.15.61:0/2489600086 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1a59c700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b1180 0x7fbb1c13b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1ad9d700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb1c0b2c40 0x7fbb1c13ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 -- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b1180 msgr2=0x7fbb1c13b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b1180 0x7fbb1c13b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 -- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb1c0b2c40 msgr2=0x7fbb1c13ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb1c0b2c40 0x7fbb1c13ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 -- 172.21.15.61:0/2489600086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1400b1e0 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb19d9b700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 0x7fbb1c13e990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fbb1c0aeb40 tx=0x7fbb04007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb04012400 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb04012560 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c13ba20 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb040128c0 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.628+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c136b70 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.632+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c13df70 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.632+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbb04019030 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.632+0000 7fbb137fe700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb00053a60 0x7fbb00055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.632+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fbb0406ae20 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.632+0000 7fbb1a59c700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb00053a60 0x7fbb00055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.632+0000 7fbb1a59c700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb00053a60 0x7fbb00055f20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbb0c00b000 tx=0x7fbb0c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.640+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb040420b0 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbb1c057460 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb137fe700 1 -- 172.21.15.61:0/2489600086 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v38) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbb040456d0 con 0x7fbb1c0b20f0 2022-07-12T08:43:12.774 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:12.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb00053a60 msgr2=0x7fbb00055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb00053a60 0x7fbb00055f20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fbb0c00b000 tx=0x7fbb0c0091d0).stop 2022-07-12T08:43:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 msgr2=0x7fbb1c13e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 0x7fbb1c13e990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fbb1c0aeb40 tx=0x7fbb04007c20).stop 2022-07-12T08:43:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 shutdown_connections 2022-07-12T08:43:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb00053a60 0x7fbb00055f20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb1c0b2c40 0x7fbb1c13ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb1c0b20f0 0x7fbb1c13e990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 --2- 172.21.15.61:0/2489600086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb1c0b1180 0x7fbb1c13b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 >> 172.21.15.61:0/2489600086 conn(0x7fbb1c061f50 msgr2=0x7fbb1c0b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 shutdown_connections 2022-07-12T08:43:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.768+0000 7fbb1b59e700 1 -- 172.21.15.61:0/2489600086 wait complete. 2022-07-12T08:43:12.798 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:43:12.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.896+0000 7f15aa0ea700 1 Processor -- start 2022-07-12T08:43:12.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.896+0000 7f15aa0ea700 1 -- start start 2022-07-12T08:43:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a405c050 0x7f15a40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f15a414a8f0 0x7f15a41443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40ad2f0 con 0x7f15a40b01f0 2022-07-12T08:43:12.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15a40b0a40 con 0x7f15a414a8f0 2022-07-12T08:43:12.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a405c430 con 0x7f15a405c050 2022-07-12T08:43:12.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f15a414a8f0 0x7f15a41443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f15a414a8f0 0x7f15a41443c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54848/0 (socket says 172.21.15.61:54848) 2022-07-12T08:43:12.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 -- 172.21.15.61:0/304406905 learned_addr learned my addr 172.21.15.61:0/304406905 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a37fe700 1 --2- 172.21.15.61:0/304406905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a405c050 0x7f15a40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 -- 172.21.15.61:0/304406905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a405c050 msgr2=0x7f15a40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 --2- 172.21.15.61:0/304406905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a405c050 0x7f15a40ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 -- 172.21.15.61:0/304406905 >> v1:172.21.15.111:6789/0 conn(0x7f15a40b01f0 legacy=0x7f15a405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 -- 172.21.15.61:0/304406905 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40b2420 con 0x7f15a414a8f0 2022-07-12T08:43:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a3fff700 1 --2- 172.21.15.61:0/304406905 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a414a8f0 0x7f15a41443c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1594002f70 tx=0x7f1594006eb0).ready entity=mon.0 client_cookie=73935e2cd1cbfece server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a27fc700 1 -- 172.21.15.61:0/304406905 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f159400d570 con 0x7f15a414a8f0 2022-07-12T08:43:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a27fc700 1 -- 172.21.15.61:0/304406905 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1594000e90 con 0x7f15a414a8f0 2022-07-12T08:43:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15a27fc700 1 -- 172.21.15.61:0/304406905 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f159400d900 con 0x7f15a414a8f0 2022-07-12T08:43:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- 172.21.15.61:0/304406905 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a414a8f0 msgr2=0x7f15a41443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/304406905 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a414a8f0 0x7f15a41443c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f1594002f70 tx=0x7f1594006eb0).stop 2022-07-12T08:43:12.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- 172.21.15.61:0/304406905 shutdown_connections 2022-07-12T08:43:12.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/304406905 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a414a8f0 0x7f15a41443c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/304406905 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a405c050 0x7f15a40ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- 172.21.15.61:0/304406905 >> 172.21.15.61:0/304406905 conn(0x7f15a4062c00 msgr2=0x7f15a4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:12.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- 172.21.15.61:0/304406905 shutdown_connections 2022-07-12T08:43:12.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.900+0000 7f15aa0ea700 1 -- 172.21.15.61:0/304406905 wait complete. 2022-07-12T08:43:12.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 Processor -- start 2022-07-12T08:43:12.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 -- start start 2022-07-12T08:43:12.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 0x7f15a40af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a40b01f0 0x7f15a4142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a414a8f0 0x7f15a40ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a414a610 con 0x7f15a405c050 2022-07-12T08:43:12.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a4148f00 con 0x7f15a414a8f0 2022-07-12T08:43:12.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15aa0ea700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a40add00 con 0x7f15a40b01f0 2022-07-12T08:43:12.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15a37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 0x7f15a40af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15a37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 0x7f15a40af050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54850/0 (socket says 172.21.15.61:54850) 2022-07-12T08:43:12.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15a37fe700 1 -- 172.21.15.61:0/2045356543 learned_addr learned my addr 172.21.15.61:0/2045356543 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:12.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15a3fff700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a414a8f0 0x7f15a40ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.904+0000 7f15a2ffd700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a40b01f0 0x7f15a4142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a37fe700 1 -- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a40b01f0 msgr2=0x7f15a4142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a37fe700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a40b01f0 0x7f15a4142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a37fe700 1 -- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a414a8f0 msgr2=0x7f15a40ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:12.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a37fe700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a414a8f0 0x7f15a40ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:12.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a37fe700 1 -- 172.21.15.61:0/2045356543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159400b1e0 con 0x7f15a405c050 2022-07-12T08:43:12.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a37fe700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 0x7f15a40af050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f159c00c000 tx=0x7f159c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f159c00c6a0 con 0x7f15a405c050 2022-07-12T08:43:12.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f159c00c800 con 0x7f15a405c050 2022-07-12T08:43:12.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a4149130 con 0x7f15a405c050 2022-07-12T08:43:12.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f159c00cb60 con 0x7f15a405c050 2022-07-12T08:43:12.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a4144300 con 0x7f15a405c050 2022-07-12T08:43:12.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a40adec0 con 0x7f15a405c050 2022-07-12T08:43:12.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f159c022070 con 0x7f15a405c050 2022-07-12T08:43:12.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a0ff9700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1588053920 0x7f1588055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:12.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f159c06c110 con 0x7f15a405c050 2022-07-12T08:43:12.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.908+0000 7f15a2ffd700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1588053920 0x7f1588055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:12.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.912+0000 7f15a2ffd700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1588053920 0x7f1588055de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f158c00d000 tx=0x7f158c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:12.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:12.916+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f15a40adec0 con 0x7f15a405c050 2022-07-12T08:43:13.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.040+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f15a40adec0 con 0x7f15a405c050 2022-07-12T08:43:13.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15a0ff9700 1 -- 172.21.15.61:0/2045356543 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v38) v1 ==== 84+0+143 (secure 0 0 0) 0x7f15a40adec0 con 0x7f15a405c050 2022-07-12T08:43:13.060 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:13.060 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:43:13.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:13.061 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:43:13.061 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:43:13.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:13.062 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:43:13.062 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:43:13.062 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:43:13.063 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:13.063 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:43:13.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1588053920 msgr2=0x7f1588055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:13.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1588053920 0x7f1588055de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f158c00d000 tx=0x7f158c007b90).stop 2022-07-12T08:43:13.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 msgr2=0x7f15a40af050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:13.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 0x7f15a40af050 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f159c00c000 tx=0x7f159c0091d0).stop 2022-07-12T08:43:13.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 shutdown_connections 2022-07-12T08:43:13.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1588053920 0x7f1588055de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15a414a8f0 0x7f15a40ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f15a40b01f0 0x7f15a4142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 --2- 172.21.15.61:0/2045356543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15a405c050 0x7f15a40af050 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 >> 172.21.15.61:0/2045356543 conn(0x7f15a4062c00 msgr2=0x7f15a4145fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:13.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 shutdown_connections 2022-07-12T08:43:13.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.044+0000 7f15aa0ea700 1 -- 172.21.15.61:0/2045356543 wait complete. 2022-07-12T08:43:13.071 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:43:13.071 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:43:13.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 Processor -- start 2022-07-12T08:43:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- start start 2022-07-12T08:43:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f23e00b0df0 0x7f23e0146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23e0065490 0x7f23e00b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:13.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e005f1e0 con 0x7f23e00b6550 2022-07-12T08:43:13.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e00b11d0 con 0x7f23e0065490 2022-07-12T08:43:13.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e0060170 con 0x7f23e00b0df0 2022-07-12T08:43:13.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ded9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f23e00b6550 0x7f23e013fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43076/0 (socket says 172.21.15.61:43076) 2022-07-12T08:43:13.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ded9d700 1 -- 172.21.15.61:0/4190557480 learned_addr learned my addr 172.21.15.61:0/4190557480 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:13.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23d5d9b700 1 --2- 172.21.15.61:0/4190557480 >> v2:172.21.15.61:3300/0 conn(0x7f23e0065490 0x7f23e00b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:13.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23de59c700 1 --2- 172.21.15.61:0/4190557480 >> v2:172.21.15.111:3300/0 conn(0x7f23e00b0df0 0x7f23e0146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:13.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ddd9b700 1 -- 172.21.15.61:0/4190557480 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130940467 0 0) 0x7f23e005f1e0 con 0x7f23e00b6550 2022-07-12T08:43:13.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ddd9b700 1 -- 172.21.15.61:0/4190557480 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4003260 con 0x7f23e00b6550 2022-07-12T08:43:13.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ddd9b700 1 -- 172.21.15.61:0/4190557480 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417229847 0 0) 0x7f23c4003260 con 0x7f23e00b6550 2022-07-12T08:43:13.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23d5d9b700 1 -- 172.21.15.61:0/4190557480 >> v2:172.21.15.111:3300/0 conn(0x7f23e00b0df0 msgr2=0x7f23e0146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:13.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23d5d9b700 1 --2- 172.21.15.61:0/4190557480 >> v2:172.21.15.111:3300/0 conn(0x7f23e00b0df0 0x7f23e0146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23d5d9b700 1 -- 172.21.15.61:0/4190557480 >> v1:172.21.15.61:6789/0 conn(0x7f23e00b6550 legacy=0x7f23e013fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:13.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23d5d9b700 1 -- 172.21.15.61:0/4190557480 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e005f410 con 0x7f23e0065490 2022-07-12T08:43:13.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23d5d9b700 1 --2- 172.21.15.61:0/4190557480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e0065490 0x7f23e00b6060 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f23d8001b80 tx=0x7f23c8006df0).ready entity=mon.0 client_cookie=8e4b4d4caa54db1d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:13.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ddd9b700 1 -- 172.21.15.61:0/4190557480 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23d80019e0 con 0x7f23e0065490 2022-07-12T08:43:13.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ddd9b700 1 -- 172.21.15.61:0/4190557480 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c8000e90 con 0x7f23e0065490 2022-07-12T08:43:13.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23ddd9b700 1 -- 172.21.15.61:0/4190557480 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c8012460 con 0x7f23e0065490 2022-07-12T08:43:13.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- 172.21.15.61:0/4190557480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e0065490 msgr2=0x7f23e00b6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:13.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 --2- 172.21.15.61:0/4190557480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e0065490 0x7f23e00b6060 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f23d8001b80 tx=0x7f23c8006df0).stop 2022-07-12T08:43:13.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- 172.21.15.61:0/4190557480 shutdown_connections 2022-07-12T08:43:13.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 --2- 172.21.15.61:0/4190557480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e0065490 0x7f23e00b6060 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 --2- 172.21.15.61:0/4190557480 >> v2:172.21.15.111:3300/0 conn(0x7f23e00b0df0 0x7f23e0146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- 172.21.15.61:0/4190557480 >> 172.21.15.61:0/4190557480 conn(0x7f23e0063ad0 msgr2=0x7f23e0063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:13.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- 172.21.15.61:0/4190557480 shutdown_connections 2022-07-12T08:43:13.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.180+0000 7f23df59e700 1 -- 172.21.15.61:0/4190557480 wait complete. 2022-07-12T08:43:13.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 Processor -- start 2022-07-12T08:43:13.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 -- start start 2022-07-12T08:43:13.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e00b0df0 0x7f23e00b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:13.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 0x7f23e0065dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:13.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f23e0133e90 0x7f23e00662c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:13.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e0133a60 con 0x7f23e00b6550 2022-07-12T08:43:13.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e013cf00 con 0x7f23e00b0df0 2022-07-12T08:43:13.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23df59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e00658c0 con 0x7f23e0133e90 2022-07-12T08:43:13.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 0x7f23e0065dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:13.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 0x7f23e0065dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54866/0 (socket says 172.21.15.61:54866) 2022-07-12T08:43:13.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 -- 172.21.15.61:0/3929433854 learned_addr learned my addr 172.21.15.61:0/3929433854 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:13.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23de59c700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e00b0df0 0x7f23e00b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:13.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23ded9d700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f23e0133e90 0x7f23e00662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:13.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 -- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f23e0133e90 msgr2=0x7f23e00662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:13.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f23e0133e90 0x7f23e00662c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 -- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e00b0df0 msgr2=0x7f23e00b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:13.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e00b0df0 0x7f23e00b2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:13.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 -- 172.21.15.61:0/3929433854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c800b180 con 0x7f23e00b6550 2022-07-12T08:43:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.184+0000 7f23d5d9b700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 0x7f23e0065dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f23d8001b80 tx=0x7f23c8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c8016070 con 0x7f23e00b6550 2022-07-12T08:43:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c8002f50 con 0x7f23e00b6550 2022-07-12T08:43:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e0067660 con 0x7f23e00b6550 2022-07-12T08:43:13.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f23c8012460 con 0x7f23e00b6550 2022-07-12T08:43:13.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e0066d60 con 0x7f23e00b6550 2022-07-12T08:43:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e013d070 con 0x7f23e00b6550 2022-07-12T08:43:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f23c8025030 con 0x7f23e00b6550 2022-07-12T08:43:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23d7fff700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f23b80539b0 0x7f23b8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f23c806ca60 con 0x7f23e00b6550 2022-07-12T08:43:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23de59c700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f23b80539b0 0x7f23b8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.188+0000 7f23de59c700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f23b80539b0 0x7f23b8055e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f23d000b000 tx=0x7f23d0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.192+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f23e013d070 con 0x7f23e00b6550 2022-07-12T08:43:13.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:13.328+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f23e013d070 con 0x7f23e00b6550 2022-07-12T08:43:14.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.252+0000 7f23d7fff700 1 -- 172.21.15.61:0/3929433854 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v39) v1 ==== 133+0+0 (secure 0 0 0) 0x7f23e013d070 con 0x7f23e00b6550 2022-07-12T08:43:14.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f23b80539b0 msgr2=0x7f23b8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:14.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f23b80539b0 0x7f23b8055e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f23d000b000 tx=0x7f23d0006df0).stop 2022-07-12T08:43:14.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 msgr2=0x7f23e0065dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:14.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 0x7f23e0065dd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f23d8001b80 tx=0x7f23c8014000).stop 2022-07-12T08:43:14.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 shutdown_connections 2022-07-12T08:43:14.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f23e0133e90 0x7f23e00662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23e00b6550 0x7f23e0065dd0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f23b80539b0 0x7f23b8055e70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 --2- 172.21.15.61:0/3929433854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23e00b0df0 0x7f23e00b2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 >> 172.21.15.61:0/3929433854 conn(0x7f23e0063ad0 msgr2=0x7f23e01494d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:14.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 shutdown_connections 2022-07-12T08:43:14.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.256+0000 7f23df59e700 1 -- 172.21.15.61:0/3929433854 wait complete. 2022-07-12T08:43:14.314 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:43:14.315 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:43:14.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.436+0000 7fe61dab3700 1 Processor -- start 2022-07-12T08:43:14.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.436+0000 7fe61dab3700 1 -- start start 2022-07-12T08:43:14.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61dab3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b1eb0 0x7fe618142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61dab3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe61805bd10 0x7fe618066290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61dab3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618065b90 con 0x7fe6180b7b50 2022-07-12T08:43:14.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61dab3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6180b7f30 con 0x7fe61805bd10 2022-07-12T08:43:14.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61dab3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe618066780 con 0x7fe6180b1eb0 2022-07-12T08:43:14.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe61805bd10 0x7fe618066290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:14.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe617fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b1eb0 0x7fe618142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:14.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe61805bd10 0x7fe618066290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33450/0 (socket says 172.21.15.61:33450) 2022-07-12T08:43:14.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe617fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b1eb0 0x7fe618142ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:37070/0 (socket says 172.21.15.61:37070) 2022-07-12T08:43:14.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2230287014 learned_addr learned my addr 172.21.15.61:0/2230287014 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:14.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61cab1700 1 --1- 172.21.15.61:0/2230287014 >> v1:172.21.15.111:6789/0 conn(0x7fe6180b7b50 0x7fe618145f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56692/0 (socket says 172.21.15.61:56692) 2022-07-12T08:43:14.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b1eb0 msgr2=0x7fe618142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:14.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 --2- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b1eb0 0x7fe618142ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2230287014 >> v1:172.21.15.111:6789/0 conn(0x7fe6180b7b50 legacy=0x7fe618145f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:14.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2230287014 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61805d1c0 con 0x7fe61805bd10 2022-07-12T08:43:14.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe61d2b2700 1 --2- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe61805bd10 0x7fe618066290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe6100032f0 tx=0x7fe610009290).ready entity=mon.1 client_cookie=979d47cd4f3750f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:14.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe6177fe700 1 -- 172.21.15.61:0/2230287014 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe61000b8a0 con 0x7fe61805bd10 2022-07-12T08:43:14.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe6177fe700 1 -- 172.21.15.61:0/2230287014 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61000ba00 con 0x7fe61805bd10 2022-07-12T08:43:14.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.440+0000 7fe6177fe700 1 -- 172.21.15.61:0/2230287014 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe61000bb70 con 0x7fe61805bd10 2022-07-12T08:43:14.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe61805bd10 msgr2=0x7fe618066290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:14.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe61805bd10 0x7fe618066290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe6100032f0 tx=0x7fe610009290).stop 2022-07-12T08:43:14.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- 172.21.15.61:0/2230287014 shutdown_connections 2022-07-12T08:43:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe61805bd10 0x7fe618066290 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2230287014 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b1eb0 0x7fe618142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- 172.21.15.61:0/2230287014 >> 172.21.15.61:0/2230287014 conn(0x7fe618065650 msgr2=0x7fe6180b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:14.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- 172.21.15.61:0/2230287014 shutdown_connections 2022-07-12T08:43:14.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- 172.21.15.61:0/2230287014 wait complete. 2022-07-12T08:43:14.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 Processor -- start 2022-07-12T08:43:14.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- start start 2022-07-12T08:43:14.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe61805bd10 0x7fe618143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:14.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6180b1eb0 0x7fe6181556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:14.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b7b50 0x7fe6180af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:14.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe618153040 con 0x7fe61805bd10 2022-07-12T08:43:14.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe618155bc0 con 0x7fe6180b1eb0 2022-07-12T08:43:14.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61dab3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6180b0f90 con 0x7fe6180b7b50 2022-07-12T08:43:14.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61cab1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe61805bd10 0x7fe618143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe617fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6180b1eb0 0x7fe6181556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61cab1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe61805bd10 0x7fe618143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54880/0 (socket says 172.21.15.61:54880) 2022-07-12T08:43:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61d2b2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b7b50 0x7fe6180af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.444+0000 7fe61cab1700 1 -- 172.21.15.61:0/2629757737 learned_addr learned my addr 172.21.15.61:0/2629757737 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:14.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6180b1eb0 msgr2=0x7fe6181556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:14.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61d2b2700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6180b1eb0 0x7fe6181556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2629757737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe61805bd10 msgr2=0x7fe618143e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:14.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61d2b2700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe61805bd10 0x7fe618143e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:14.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61d2b2700 1 -- 172.21.15.61:0/2629757737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe610009530 con 0x7fe6180b7b50 2022-07-12T08:43:14.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe617fff700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6180b1eb0 0x7fe6181556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:14.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61d2b2700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b7b50 0x7fe6180af330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe6100032f0 tx=0x7fe610016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:14.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe61000b8a0 con 0x7fe6180b7b50 2022-07-12T08:43:14.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6180b1230 con 0x7fe6180b7b50 2022-07-12T08:43:14.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe610021440 con 0x7fe6180b7b50 2022-07-12T08:43:14.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618146250 con 0x7fe6180b7b50 2022-07-12T08:43:14.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6100217a0 con 0x7fe6180b7b50 2022-07-12T08:43:14.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe618057460 con 0x7fe6180b7b50 2022-07-12T08:43:14.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.448+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe610027020 con 0x7fe6180b7b50 2022-07-12T08:43:14.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.452+0000 7fe615ffb700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe5f8053a40 0x7fe5f8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:14.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.452+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fe61006dc40 con 0x7fe6180b7b50 2022-07-12T08:43:14.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.452+0000 7fe61cab1700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe5f8053a40 0x7fe5f8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:14.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.452+0000 7fe61cab1700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe5f8053a40 0x7fe5f8055f00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fe60800d000 tx=0x7fe608007c40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:14.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.460+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe610044d70 con 0x7fe6180b7b50 2022-07-12T08:43:14.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:14.588+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fe61805d3f0 con 0x7fe6180b7b50 2022-07-12T08:43:15.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.268+0000 7fe615ffb700 1 -- 172.21.15.61:0/2629757737 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v40) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe610015070 con 0x7fe6180b7b50 2022-07-12T08:43:15.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe5f8053a40 msgr2=0x7fe5f8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe5f8053a40 0x7fe5f8055f00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fe60800d000 tx=0x7fe608007c40).stop 2022-07-12T08:43:15.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b7b50 msgr2=0x7fe6180af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b7b50 0x7fe6180af330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fe6100032f0 tx=0x7fe610016000).stop 2022-07-12T08:43:15.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 shutdown_connections 2022-07-12T08:43:15.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6180b7b50 0x7fe6180af330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe5f8053a40 0x7fe5f8055f00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6180b1eb0 0x7fe6181556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 --2- 172.21.15.61:0/2629757737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe61805bd10 0x7fe618143e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 >> 172.21.15.61:0/2629757737 conn(0x7fe618065650 msgr2=0x7fe61813f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:15.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 shutdown_connections 2022-07-12T08:43:15.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.272+0000 7fe61dab3700 1 -- 172.21.15.61:0/2629757737 wait complete. 2022-07-12T08:43:15.285 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:43:15.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:43:15.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 Processor -- start 2022-07-12T08:43:15.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 -- start start 2022-07-12T08:43:15.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe900067160 0x7fe9000b2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe90005c420 0x7fe9001473f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe900067540 con 0x7fe9000afef0 2022-07-12T08:43:15.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9000b0a60 con 0x7fe900067160 2022-07-12T08:43:15.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe9081eb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9000b5af0 con 0x7fe90005c420 2022-07-12T08:43:15.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905786700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe9000afef0 0x7fe90005bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43106/0 (socket says 172.21.15.61:43106) 2022-07-12T08:43:15.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905786700 1 -- 172.21.15.61:0/126580426 learned_addr learned my addr 172.21.15.61:0/126580426 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:15.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905f87700 1 --2- 172.21.15.61:0/126580426 >> v2:172.21.15.111:3300/0 conn(0x7fe900067160 0x7fe9000b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe906788700 1 --2- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe90005c420 0x7fe9001473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe904f85700 1 -- 172.21.15.61:0/126580426 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210444133 0 0) 0x7fe900067540 con 0x7fe9000afef0 2022-07-12T08:43:15.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe904f85700 1 -- 172.21.15.61:0/126580426 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e8003260 con 0x7fe9000afef0 2022-07-12T08:43:15.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905f87700 1 -- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe90005c420 msgr2=0x7fe9001473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:15.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905f87700 1 --2- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe90005c420 0x7fe9001473f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905f87700 1 -- 172.21.15.61:0/126580426 >> v1:172.21.15.61:6789/0 conn(0x7fe9000afef0 legacy=0x7fe90005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905f87700 1 -- 172.21.15.61:0/126580426 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000b0400 con 0x7fe900067160 2022-07-12T08:43:15.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe905f87700 1 --2- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe900067160 0x7fe9000b2940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe8fc002f70 tx=0x7fe8fc006df0).ready entity=mon.1 client_cookie=15402617780960be server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:15.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe904f85700 1 -- 172.21.15.61:0/126580426 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8fc00d4a0 con 0x7fe900067160 2022-07-12T08:43:15.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe904f85700 1 -- 172.21.15.61:0/126580426 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc000e90 con 0x7fe900067160 2022-07-12T08:43:15.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.420+0000 7fe904f85700 1 -- 172.21.15.61:0/126580426 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8fc00d7f0 con 0x7fe900067160 2022-07-12T08:43:15.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe900067160 msgr2=0x7fe9000b2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 --2- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe900067160 0x7fe9000b2940 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe8fc002f70 tx=0x7fe8fc006df0).stop 2022-07-12T08:43:15.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- 172.21.15.61:0/126580426 shutdown_connections 2022-07-12T08:43:15.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 --2- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe90005c420 0x7fe9001473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 --2- 172.21.15.61:0/126580426 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe900067160 0x7fe9000b2940 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- 172.21.15.61:0/126580426 >> 172.21.15.61:0/126580426 conn(0x7fe9000b94d0 msgr2=0x7fe9000b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:15.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- 172.21.15.61:0/126580426 shutdown_connections 2022-07-12T08:43:15.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- 172.21.15.61:0/126580426 wait complete. 2022-07-12T08:43:15.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 Processor -- start 2022-07-12T08:43:15.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- start start 2022-07-12T08:43:15.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe90005c420 0x7fe9001440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe900067160 0x7fe900146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 0x7fe9000ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9000acb60 con 0x7fe9000afef0 2022-07-12T08:43:15.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe900145080 con 0x7fe90005c420 2022-07-12T08:43:15.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.424+0000 7fe9081eb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9000acd50 con 0x7fe900067160 2022-07-12T08:43:15.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe906788700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 0x7fe9000ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe906788700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 0x7fe9000ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54896/0 (socket says 172.21.15.61:54896) 2022-07-12T08:43:15.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe906788700 1 -- 172.21.15.61:0/80922829 learned_addr learned my addr 172.21.15.61:0/80922829 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:15.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe900067160 0x7fe900146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905f87700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe90005c420 0x7fe9001440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 -- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe90005c420 msgr2=0x7fe9001440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe90005c420 0x7fe9001440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 -- 172.21.15.61:0/80922829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 msgr2=0x7fe9000ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 0x7fe9000ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 -- 172.21.15.61:0/80922829 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc00b180 con 0x7fe900067160 2022-07-12T08:43:15.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe906788700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 0x7fe9000ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:15.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe905786700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe900067160 0x7fe900146410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe8f0001110 tx=0x7fe8f0007a80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:15.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8f000cda0 con 0x7fe900067160 2022-07-12T08:43:15.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f00070d0 con 0x7fe900067160 2022-07-12T08:43:15.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8f0007430 con 0x7fe900067160 2022-07-12T08:43:15.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9000afb20 con 0x7fe900067160 2022-07-12T08:43:15.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe900144e40 con 0x7fe900067160 2022-07-12T08:43:15.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.428+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe900057460 con 0x7fe900067160 2022-07-12T08:43:15.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.432+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe8f0010070 con 0x7fe900067160 2022-07-12T08:43:15.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.432+0000 7fe8eeffd700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8e0053af0 0x7fe8e0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.432+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fe8f006bc90 con 0x7fe900067160 2022-07-12T08:43:15.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.432+0000 7fe905f87700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8e0053af0 0x7fe8e0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.432+0000 7fe905f87700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8e0053af0 0x7fe8e0055fb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe8fc01a000 tx=0x7fe8fc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:15.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.440+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8f0042c60 con 0x7fe900067160 2022-07-12T08:43:15.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.608+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fe900144e40 con 0x7fe900067160 2022-07-12T08:43:15.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.620+0000 7fe8eeffd700 1 -- 172.21.15.61:0/80922829 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 7 and data pool 8 v16) v1 ==== 187+0+0 (secure 0 0 0) 0x7fe900144e40 con 0x7fe900067160 2022-07-12T08:43:15.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.624+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8e0053af0 msgr2=0x7fe8e0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.624+0000 7fe9081eb700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8e0053af0 0x7fe8e0055fb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe8fc01a000 tx=0x7fe8fc014000).stop 2022-07-12T08:43:15.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.624+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe900067160 msgr2=0x7fe900146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.624+0000 7fe9081eb700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe900067160 0x7fe900146410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fe8f0001110 tx=0x7fe8f0007a80).stop 2022-07-12T08:43:15.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 shutdown_connections 2022-07-12T08:43:15.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9000afef0 0x7fe9000ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe900067160 0x7fe900146410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8e0053af0 0x7fe8e0055fb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 --2- 172.21.15.61:0/80922829 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe90005c420 0x7fe9001440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 >> 172.21.15.61:0/80922829 conn(0x7fe9000b94d0 msgr2=0x7fe90014cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:15.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 shutdown_connections 2022-07-12T08:43:15.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.628+0000 7fe9081eb700 1 -- 172.21.15.61:0/80922829 wait complete. 2022-07-12T08:43:15.639 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 7 and data pool 8 2022-07-12T08:43:15.657 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:43:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cc670700 1 Processor -- start 2022-07-12T08:43:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cc670700 1 -- start start 2022-07-12T08:43:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cc670700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 0x7f32c40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cc670700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40ad2f0 con 0x7f32c40b01f0 2022-07-12T08:43:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cc670700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c405c430 con 0x7f32c414a8f0 2022-07-12T08:43:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cc670700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40b0a40 con 0x7f32c405c050 2022-07-12T08:43:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c9c0b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f32c40b01f0 0x7f32c405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43128/0 (socket says 172.21.15.61:43128) 2022-07-12T08:43:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c9c0b700 1 -- 172.21.15.61:0/158765750 learned_addr learned my addr 172.21.15.61:0/158765750 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32ca40c700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 0x7f32c40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797802491 0 0) 0x7f32c40ad2f0 con 0x7f32c40b01f0 2022-07-12T08:43:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac003260 con 0x7f32c40b01f0 2022-07-12T08:43:15.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274342667 0 0) 0x7f32ac003260 con 0x7f32c40b01f0 2022-07-12T08:43:15.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 msgr2=0x7f32c40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 0x7f32c40ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 >> v1:172.21.15.111:6789/0 conn(0x7f32c414a8f0 legacy=0x7f32c41443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c40b2420 con 0x7f32c40b01f0 2022-07-12T08:43:15.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f32b4001660 con 0x7f32c40b01f0 2022-07-12T08:43:15.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 >> v1:172.21.15.61:6789/0 conn(0x7f32c40b01f0 legacy=0x7f32c405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ac003780 0x7f32ac003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 0x7f32ac004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32ac0049f0 0x7f32ac004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40b0a40 con 0x7f32ac0049f0 2022-07-12T08:43:15.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac00b780 con 0x7f32ac003780 2022-07-12T08:43:15.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac00b940 con 0x7f32c405c050 2022-07-12T08:43:15.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32ac0049f0 0x7f32ac004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32ca40c700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ac003780 0x7f32ac003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c9c0b700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 0x7f32ac004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 -- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 msgr2=0x7f32ac004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:15.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c405c050 0x7f32ac004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 -- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ac003780 msgr2=0x7f32ac003ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:15.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32ac003780 0x7f32ac003ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 -- 172.21.15.61:0/158765750 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac003260 con 0x7f32ac0049f0 2022-07-12T08:43:15.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32cac0d700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32ac0049f0 0x7f32ac004e70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f32c4063b30 tx=0x7f32bc007d80).ready entity=mon.0 client_cookie=9a6a4ecb890f5342 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:15.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32bc001620 con 0x7f32ac0049f0 2022-07-12T08:43:15.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc006890 con 0x7f32ac0049f0 2022-07-12T08:43:15.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.768+0000 7f32c940a700 1 -- 172.21.15.61:0/158765750 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32bc006bf0 con 0x7f32ac0049f0 2022-07-12T08:43:15.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 -- 172.21.15.61:0/158765750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32ac0049f0 msgr2=0x7f32ac004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 --2- 172.21.15.61:0/158765750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32ac0049f0 0x7f32ac004e70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f32c4063b30 tx=0x7f32bc007d80).stop 2022-07-12T08:43:15.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cac0d700 1 -- 172.21.15.61:0/158765750 reap_dead start 2022-07-12T08:43:15.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 -- 172.21.15.61:0/158765750 shutdown_connections 2022-07-12T08:43:15.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 -- 172.21.15.61:0/158765750 >> 172.21.15.61:0/158765750 conn(0x7f32c4062c00 msgr2=0x7f32c4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:15.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 -- 172.21.15.61:0/158765750 shutdown_connections 2022-07-12T08:43:15.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 -- 172.21.15.61:0/158765750 wait complete. 2022-07-12T08:43:15.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 Processor -- start 2022-07-12T08:43:15.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.772+0000 7f32cc670700 1 -- start start 2022-07-12T08:43:15.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32c40ae570 0x7f32c4066d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 0x7f32c4149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c4144720 0x7f32c4144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40ae950 con 0x7f32c4149970 2022-07-12T08:43:15.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40663b0 con 0x7f32c40ae570 2022-07-12T08:43:15.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c4066650 con 0x7f32c4144720 2022-07-12T08:43:15.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 0x7f32c4149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 0x7f32c4149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54924/0 (socket says 172.21.15.61:54924) 2022-07-12T08:43:15.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 -- 172.21.15.61:0/4089366117 learned_addr learned my addr 172.21.15.61:0/4089366117 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:15.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32ca40c700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32c40ae570 0x7f32c4066d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cac0d700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c4144720 0x7f32c4144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 -- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c4144720 msgr2=0x7f32c4144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c4144720 0x7f32c4144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 -- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32c40ae570 msgr2=0x7f32c4066d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32c40ae570 0x7f32c4066d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 -- 172.21.15.61:0/4089366117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c40668f0 con 0x7f32c4149970 2022-07-12T08:43:15.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32ca40c700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32c40ae570 0x7f32c4066d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:15.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32c9c0b700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 0x7f32c4149df0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f32b4001ec0 tx=0x7f32b4001e10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:15.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b4011850 con 0x7f32c4149970 2022-07-12T08:43:15.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b40119b0 con 0x7f32c4149970 2022-07-12T08:43:15.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c4145090 con 0x7f32c4149970 2022-07-12T08:43:15.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b4011d10 con 0x7f32c4149970 2022-07-12T08:43:15.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.776+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c4063f40 con 0x7f32c4149970 2022-07-12T08:43:15.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.780+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f32b401b030 con 0x7f32c4149970 2022-07-12T08:43:15.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.780+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c4067c20 con 0x7f32c4149970 2022-07-12T08:43:15.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.780+0000 7f32b37fe700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32a40539d0 0x7f32a4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:15.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.780+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f32b406bb70 con 0x7f32c4149970 2022-07-12T08:43:15.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.780+0000 7f32ca40c700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32a40539d0 0x7f32a4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:15.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.780+0000 7f32ca40c700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32a40539d0 0x7f32a4055e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f32c000b000 tx=0x7f32c0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:15.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.784+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f32b40429c0 con 0x7f32c4149970 2022-07-12T08:43:15.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.916+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32c4066a80 con 0x7f32c4149970 2022-07-12T08:43:15.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.916+0000 7f32b37fe700 1 -- 172.21.15.61:0/4089366117 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f32b4019030 con 0x7f32c4149970 2022-07-12T08:43:15.925 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:15.926 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":41,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:43:15.614946+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:43:13.384889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-12T08:43:14.650247+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:43:15.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32a40539d0 msgr2=0x7f32a4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32a40539d0 0x7f32a4055e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f32c000b000 tx=0x7f32c0008040).stop 2022-07-12T08:43:15.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 msgr2=0x7f32c4149df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:15.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 0x7f32c4149df0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f32b4001ec0 tx=0x7f32b4001e10).stop 2022-07-12T08:43:15.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 shutdown_connections 2022-07-12T08:43:15.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32c4144720 0x7f32c4144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32c4149970 0x7f32c4149df0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32a40539d0 0x7f32a4055e90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 --2- 172.21.15.61:0/4089366117 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32c40ae570 0x7f32c4066d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:15.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 >> 172.21.15.61:0/4089366117 conn(0x7f32c4062c00 msgr2=0x7f32c40b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:15.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 shutdown_connections 2022-07-12T08:43:15.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:15.920+0000 7f32cc670700 1 -- 172.21.15.61:0/4089366117 wait complete. 2022-07-12T08:43:15.949 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:43:16.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d8d3d700 1 Processor -- start 2022-07-12T08:43:16.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d8d3d700 1 -- start start 2022-07-12T08:43:16.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d8d3d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d40605c0 0x7fe1d40b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d8d3d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d4146110 con 0x7fe1d405c8e0 2022-07-12T08:43:16.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d8d3d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d405ccc0 con 0x7fe1d41467e0 2022-07-12T08:43:16.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d8d3d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d405f350 con 0x7fe1d40605c0 2022-07-12T08:43:16.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe1d405c8e0 0x7fe1d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43150/0 (socket says 172.21.15.61:43150) 2022-07-12T08:43:16.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d37fe700 1 -- 172.21.15.61:0/4150902248 learned_addr learned my addr 172.21.15.61:0/4150902248 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:16.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/4150902248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d40605c0 0x7fe1d40b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d27fc700 1 -- 172.21.15.61:0/4150902248 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230056111 0 0) 0x7fe1d4146110 con 0x7fe1d405c8e0 2022-07-12T08:43:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d27fc700 1 -- 172.21.15.61:0/4150902248 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b4003260 con 0x7fe1d405c8e0 2022-07-12T08:43:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/4150902248 >> v1:172.21.15.111:6789/0 conn(0x7fe1d41467e0 legacy=0x7fe1d4141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d27fc700 1 -- 172.21.15.61:0/4150902248 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908969541 0 0) 0x7fe1d405ccc0 con 0x7fe1d41467e0 2022-07-12T08:43:16.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/4150902248 >> v1:172.21.15.61:6789/0 conn(0x7fe1d405c8e0 legacy=0x7fe1d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/4150902248 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4146340 con 0x7fe1d40605c0 2022-07-12T08:43:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.052+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/4150902248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d40605c0 0x7fe1d40b1410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe1cc0032f0 tx=0x7fe1cc0091d0).ready entity=mon.2 client_cookie=3a1ea5ee7bc5c15b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d27fc700 1 -- 172.21.15.61:0/4150902248 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1cc00b6c0 con 0x7fe1d40605c0 2022-07-12T08:43:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d27fc700 1 -- 172.21.15.61:0/4150902248 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc00b820 con 0x7fe1d40605c0 2022-07-12T08:43:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d27fc700 1 -- 172.21.15.61:0/4150902248 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1cc00b990 con 0x7fe1d40605c0 2022-07-12T08:43:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/4150902248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d40605c0 msgr2=0x7fe1d40b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/4150902248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d40605c0 0x7fe1d40b1410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe1cc0032f0 tx=0x7fe1cc0091d0).stop 2022-07-12T08:43:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/4150902248 shutdown_connections 2022-07-12T08:43:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/4150902248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d40605c0 0x7fe1d40b1410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/4150902248 >> 172.21.15.61:0/4150902248 conn(0x7fe1d4063a60 msgr2=0x7fe1d4063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/4150902248 shutdown_connections 2022-07-12T08:43:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/4150902248 wait complete. 2022-07-12T08:43:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.056+0000 7fe1d8d3d700 1 Processor -- start 2022-07-12T08:43:16.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 -- start start 2022-07-12T08:43:16.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d405c8e0 0x7fe1d41445c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 0x7fe1d40ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d41467e0 0x7fe1d40aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40ae590 con 0x7fe1d40605c0 2022-07-12T08:43:16.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d40aedb0 con 0x7fe1d405c8e0 2022-07-12T08:43:16.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d41143b0 con 0x7fe1d41467e0 2022-07-12T08:43:16.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 0x7fe1d40ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 0x7fe1d40ae8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54940/0 (socket says 172.21.15.61:54940) 2022-07-12T08:43:16.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/356585126 learned_addr learned my addr 172.21.15.61:0/356585126 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:16.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d37fe700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d405c8e0 0x7fe1d41445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d3fff700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d41467e0 0x7fe1d40aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d41467e0 msgr2=0x7fe1d40aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d41467e0 0x7fe1d40aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d405c8e0 msgr2=0x7fe1d41445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d405c8e0 0x7fe1d41445c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/356585126 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc009470 con 0x7fe1d40605c0 2022-07-12T08:43:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 0x7fe1d40ae8c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe1d41136c0 tx=0x7fe1cc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1cc00b990 con 0x7fe1d40605c0 2022-07-12T08:43:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc011e20 con 0x7fe1d40605c0 2022-07-12T08:43:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d40af4c0 con 0x7fe1d40605c0 2022-07-12T08:43:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1cc00b6c0 con 0x7fe1d40605c0 2022-07-12T08:43:16.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.060+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d4144f20 con 0x7fe1d40605c0 2022-07-12T08:43:16.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.064+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe1cc026030 con 0x7fe1d40605c0 2022-07-12T08:43:16.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.064+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d4057460 con 0x7fe1d40605c0 2022-07-12T08:43:16.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.064+0000 7fe1d0ff9700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1b8053a40 0x7fe1b8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.064+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe1cc06caa0 con 0x7fe1d40605c0 2022-07-12T08:43:16.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.064+0000 7fe1d37fe700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1b8053a40 0x7fe1b8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.064+0000 7fe1d37fe700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1b8053a40 0x7fe1b8055f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe1bc00d000 tx=0x7fe1bc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.072+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe1cc043a00 con 0x7fe1d40605c0 2022-07-12T08:43:16.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.192+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1d4114570 con 0x7fe1d40605c0 2022-07-12T08:43:16.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.192+0000 7fe1d0ff9700 1 -- 172.21.15.61:0/356585126 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16099 (secure 0 0 0) 0x7fe1cc024090 con 0x7fe1d40605c0 2022-07-12T08:43:16.201 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:16.203 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":41,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:43:15.614946+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:43:13.384889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-12T08:43:14.650247+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:43:16.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1b8053a40 msgr2=0x7fe1b8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1b8053a40 0x7fe1b8055f00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe1bc00d000 tx=0x7fe1bc009040).stop 2022-07-12T08:43:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 msgr2=0x7fe1d40ae8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 0x7fe1d40ae8c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe1d41136c0 tx=0x7fe1cc016000).stop 2022-07-12T08:43:16.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 shutdown_connections 2022-07-12T08:43:16.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1d41467e0 0x7fe1d40aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1d40605c0 0x7fe1d40ae8c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1b8053a40 0x7fe1b8055f00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 --2- 172.21.15.61:0/356585126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1d405c8e0 0x7fe1d41445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 >> 172.21.15.61:0/356585126 conn(0x7fe1d4063a60 msgr2=0x7fe1d4145220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:16.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 shutdown_connections 2022-07-12T08:43:16.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.196+0000 7fe1d8d3d700 1 -- 172.21.15.61:0/356585126 wait complete. 2022-07-12T08:43:16.220 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:43:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f2dd94700 1 Processor -- start 2022-07-12T08:43:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f2dd94700 1 -- start start 2022-07-12T08:43:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f2dd94700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6f280b2a40 0x7f6f2805c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f2dd94700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2805cd00 con 0x7f6f280b9db0 2022-07-12T08:43:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f2dd94700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f280b3120 con 0x7f6f280b3ea0 2022-07-12T08:43:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f2dd94700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f2805bd10 con 0x7f6f280b2a40 2022-07-12T08:43:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f277fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6f280b9db0 0x7f6f2805c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43162/0 (socket says 172.21.15.61:43162) 2022-07-12T08:43:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f277fe700 1 -- 172.21.15.61:0/3793224803 learned_addr learned my addr 172.21.15.61:0/3793224803 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f27fff700 1 --2- 172.21.15.61:0/3793224803 >> v2:172.21.15.111:3300/0 conn(0x7f6f280b2a40 0x7f6f2805c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f267fc700 1 -- 172.21.15.61:0/3793224803 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997629234 0 0) 0x7f6f2805cd00 con 0x7f6f280b9db0 2022-07-12T08:43:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f267fc700 1 -- 172.21.15.61:0/3793224803 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f14003260 con 0x7f6f280b9db0 2022-07-12T08:43:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f27fff700 1 -- 172.21.15.61:0/3793224803 >> v1:172.21.15.111:6789/0 conn(0x7f6f280b3ea0 legacy=0x7f6f280b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f27fff700 1 -- 172.21.15.61:0/3793224803 >> v1:172.21.15.61:6789/0 conn(0x7f6f280b9db0 legacy=0x7f6f2805c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f27fff700 1 -- 172.21.15.61:0/3793224803 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f280b33c0 con 0x7f6f280b2a40 2022-07-12T08:43:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f27fff700 1 --2- 172.21.15.61:0/3793224803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f2805c810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6f280b1230 tx=0x7f6f200091d0).ready entity=mon.1 client_cookie=a2b8c41e95c7006b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f267fc700 1 -- 172.21.15.61:0/3793224803 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f2000b680 con 0x7f6f280b2a40 2022-07-12T08:43:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.328+0000 7f6f267fc700 1 -- 172.21.15.61:0/3793224803 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2000b7e0 con 0x7f6f280b2a40 2022-07-12T08:43:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f267fc700 1 -- 172.21.15.61:0/3793224803 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f2000b950 con 0x7f6f280b2a40 2022-07-12T08:43:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- 172.21.15.61:0/3793224803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 msgr2=0x7f6f2805c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/3793224803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f2805c810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6f280b1230 tx=0x7f6f200091d0).stop 2022-07-12T08:43:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- 172.21.15.61:0/3793224803 shutdown_connections 2022-07-12T08:43:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/3793224803 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f2805c810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- 172.21.15.61:0/3793224803 >> 172.21.15.61:0/3793224803 conn(0x7f6f28067960 msgr2=0x7f6f280b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- 172.21.15.61:0/3793224803 shutdown_connections 2022-07-12T08:43:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- 172.21.15.61:0/3793224803 wait complete. 2022-07-12T08:43:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 Processor -- start 2022-07-12T08:43:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- start start 2022-07-12T08:43:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f280af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f280b3ea0 0x7f6f28146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 0x7f6f280acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280ad050 con 0x7f6f280b9db0 2022-07-12T08:43:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28114930 con 0x7f6f280b2a40 2022-07-12T08:43:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f2dd94700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280aecb0 con 0x7f6f280b3ea0 2022-07-12T08:43:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.332+0000 7f6f27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 0x7f6f280acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 0x7f6f280acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54954/0 (socket says 172.21.15.61:54954) 2022-07-12T08:43:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 -- 172.21.15.61:0/2351336996 learned_addr learned my addr 172.21.15.61:0/2351336996 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f277fe700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f280af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f26ffd700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f280b3ea0 0x7f6f28146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 -- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f280b3ea0 msgr2=0x7f6f28146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f280b3ea0 0x7f6f28146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 -- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 msgr2=0x7f6f280af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f280af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 -- 172.21.15.61:0/2351336996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f20009470 con 0x7f6f280b9db0 2022-07-12T08:43:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f277fe700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f280af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f27fff700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 0x7f6f280acb60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f6f280b1230 tx=0x7f6f20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f2000b680 con 0x7f6f280b9db0 2022-07-12T08:43:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f20021630 con 0x7f6f280b9db0 2022-07-12T08:43:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f20021990 con 0x7f6f280b9db0 2022-07-12T08:43:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f280aef50 con 0x7f6f280b9db0 2022-07-12T08:43:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f28113290 con 0x7f6f280b9db0 2022-07-12T08:43:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6f20027020 con 0x7f6f280b9db0 2022-07-12T08:43:16.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.336+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f28057460 con 0x7f6f280b9db0 2022-07-12T08:43:16.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.340+0000 7f6f24ff9700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f08053ad0 0x7f6f08055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.340+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6f2006d250 con 0x7f6f280b9db0 2022-07-12T08:43:16.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.340+0000 7f6f277fe700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f08053ad0 0x7f6f08055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.340+0000 7f6f277fe700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f08053ad0 0x7f6f08055f90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6f1800d000 tx=0x7f6f18009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.344+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f200441b0 con 0x7f6f280b9db0 2022-07-12T08:43:16.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.508+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6f2805f630 con 0x7f6f280b9db0 2022-07-12T08:43:16.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f24ff9700 1 -- 172.21.15.61:0/2351336996 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6f200150b0 con 0x7f6f280b9db0 2022-07-12T08:43:16.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f08053ad0 msgr2=0x7f6f08055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f08053ad0 0x7f6f08055f90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6f1800d000 tx=0x7f6f18009040).stop 2022-07-12T08:43:16.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 msgr2=0x7f6f280acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 0x7f6f280acb60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f6f280b1230 tx=0x7f6f20016000).stop 2022-07-12T08:43:16.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 shutdown_connections 2022-07-12T08:43:16.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f280b9db0 0x7f6f280acb60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f280b3ea0 0x7f6f28146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f08053ad0 0x7f6f08055f90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 --2- 172.21.15.61:0/2351336996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f280b2a40 0x7f6f280af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 >> 172.21.15.61:0/2351336996 conn(0x7f6f28067960 msgr2=0x7f6f280b7760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:16.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 shutdown_connections 2022-07-12T08:43:16.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.628+0000 7f6f2dd94700 1 -- 172.21.15.61:0/2351336996 wait complete. 2022-07-12T08:43:16.667 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:16.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 Processor -- start 2022-07-12T08:43:16.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 -- start start 2022-07-12T08:43:16.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f95c00af280 0x7f95c0060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95c00b3f60 0x7f95c00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00aecb0 con 0x7f95c009bce0 2022-07-12T08:43:16.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c00afd70 con 0x7f95c00af280 2022-07-12T08:43:16.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6a17700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c0143b00 con 0x7f95c00b3f60 2022-07-12T08:43:16.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5a15700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f95c00af280 0x7f95c0060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5214700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f95c009bce0 0x7f95c00af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43176/0 (socket says 172.21.15.61:43176) 2022-07-12T08:43:16.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5214700 1 -- 172.21.15.61:0/801652792 learned_addr learned my addr 172.21.15.61:0/801652792 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:16.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c6216700 1 --2- 172.21.15.61:0/801652792 >> v2:172.21.15.111:3300/0 conn(0x7f95c00b3f60 0x7f95c00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5a15700 1 -- 172.21.15.61:0/801652792 >> v2:172.21.15.111:3300/0 conn(0x7f95c00b3f60 msgr2=0x7f95c00b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:16.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5a15700 1 --2- 172.21.15.61:0/801652792 >> v2:172.21.15.111:3300/0 conn(0x7f95c00b3f60 0x7f95c00b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5a15700 1 -- 172.21.15.61:0/801652792 >> v1:172.21.15.61:6789/0 conn(0x7f95c009bce0 legacy=0x7f95c00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5a15700 1 -- 172.21.15.61:0/801652792 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c01435c0 con 0x7f95c00af280 2022-07-12T08:43:16.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c5a15700 1 --2- 172.21.15.61:0/801652792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0060680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f95bc002f70 tx=0x7f95bc006d90).ready entity=mon.0 client_cookie=33ef55c17c118701 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c4a13700 1 -- 172.21.15.61:0/801652792 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95bc00d4a0 con 0x7f95c00af280 2022-07-12T08:43:16.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c4a13700 1 -- 172.21.15.61:0/801652792 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95bc000e90 con 0x7f95c00af280 2022-07-12T08:43:16.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.784+0000 7f95c4a13700 1 -- 172.21.15.61:0/801652792 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95bc00d7f0 con 0x7f95c00af280 2022-07-12T08:43:16.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 -- 172.21.15.61:0/801652792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 msgr2=0x7f95c0060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 --2- 172.21.15.61:0/801652792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0060680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f95bc002f70 tx=0x7f95bc006d90).stop 2022-07-12T08:43:16.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 -- 172.21.15.61:0/801652792 shutdown_connections 2022-07-12T08:43:16.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 --2- 172.21.15.61:0/801652792 >> v2:172.21.15.111:3300/0 conn(0x7f95c00b3f60 0x7f95c00b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 --2- 172.21.15.61:0/801652792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0060680 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 -- 172.21.15.61:0/801652792 >> 172.21.15.61:0/801652792 conn(0x7f95c00b92c0 msgr2=0x7f95c00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:16.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 -- 172.21.15.61:0/801652792 shutdown_connections 2022-07-12T08:43:16.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 -- 172.21.15.61:0/801652792 wait complete. 2022-07-12T08:43:16.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 Processor -- start 2022-07-12T08:43:16.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.788+0000 7f95c6a17700 1 -- start start 2022-07-12T08:43:16.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95c009bce0 0x7f95c009e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95c00b3f60 0x7f95c0152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c013e2a0 con 0x7f95c00af280 2022-07-12T08:43:16.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00b31e0 con 0x7f95c00b3f60 2022-07-12T08:43:16.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c0152030 con 0x7f95c009bce0 2022-07-12T08:43:16.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54966/0 (socket says 172.21.15.61:54966) 2022-07-12T08:43:16.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 -- 172.21.15.61:0/1489829935 learned_addr learned my addr 172.21.15.61:0/1489829935 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:16.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5a15700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95c009bce0 0x7f95c009e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6216700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95c00b3f60 0x7f95c0152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 -- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95c009bce0 msgr2=0x7f95c009e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95c009bce0 0x7f95c009e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 -- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95c00b3f60 msgr2=0x7f95c0152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95c00b3f60 0x7f95c0152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 -- 172.21.15.61:0/1489829935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc00b120 con 0x7f95c00af280 2022-07-12T08:43:16.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c5214700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0152750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f95c013e3e0 tx=0x7f95b0007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95b00070d0 con 0x7f95c00af280 2022-07-12T08:43:16.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b0007230 con 0x7f95c00af280 2022-07-12T08:43:16.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c00b1db0 con 0x7f95c00af280 2022-07-12T08:43:16.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95b0005610 con 0x7f95c00af280 2022-07-12T08:43:16.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c009eec0 con 0x7f95c00af280 2022-07-12T08:43:16.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c00b3000 con 0x7f95c00af280 2022-07-12T08:43:16.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.792+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f95b0010070 con 0x7f95c00af280 2022-07-12T08:43:16.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.796+0000 7f95aeffd700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95a0053a40 0x7f95a0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:16.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.796+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f95b006b3f0 con 0x7f95c00af280 2022-07-12T08:43:16.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.796+0000 7f95c5a15700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95a0053a40 0x7f95a0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:16.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.796+0000 7f95c5a15700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95a0053a40 0x7f95a0055f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f95bc01a000 tx=0x7f95bc013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:16.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.800+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f95b00422b0 con 0x7f95c00af280 2022-07-12T08:43:16.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.960+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95c0057460 con 0x7f95c00af280 2022-07-12T08:43:16.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.960+0000 7f95aeffd700 1 -- 172.21.15.61:0/1489829935 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7f95b0018070 con 0x7f95c00af280 2022-07-12T08:43:16.970 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:16.971 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:16.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95a0053a40 msgr2=0x7f95a0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95a0053a40 0x7f95a0055f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f95bc01a000 tx=0x7f95bc013000).stop 2022-07-12T08:43:16.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 msgr2=0x7f95c0152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:16.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0152750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f95c013e3e0 tx=0x7f95b0007c10).stop 2022-07-12T08:43:16.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 shutdown_connections 2022-07-12T08:43:16.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95a0053a40 0x7f95a0055f00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95c00b3f60 0x7f95c0152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95c00af280 0x7f95c0152750 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 --2- 172.21.15.61:0/1489829935 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95c009bce0 0x7f95c009e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:16.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 >> 172.21.15.61:0/1489829935 conn(0x7f95c00b92c0 msgr2=0x7f95c00b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:16.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 shutdown_connections 2022-07-12T08:43:16.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:16.964+0000 7f95c6a17700 1 -- 172.21.15.61:0/1489829935 wait complete. 2022-07-12T08:43:16.983 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:16.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2022-07-12T08:43:16.988 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:43:17.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.084+0000 7f326a8db700 1 Processor -- start 2022-07-12T08:43:17.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.084+0000 7f326a8db700 1 -- start start 2022-07-12T08:43:17.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f3264060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f32640b3f60 0x7f32640b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32640aecb0 con 0x7f326409bce0 2022-07-12T08:43:17.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32640afd70 con 0x7f32640b3f60 2022-07-12T08:43:17.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3264143b00 con 0x7f32640af280 2022-07-12T08:43:17.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f32640b3f60 0x7f32640b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f32640b3f60 0x7f32640b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33550/0 (socket says 172.21.15.61:33550) 2022-07-12T08:43:17.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 -- 172.21.15.61:0/1567692536 learned_addr learned my addr 172.21.15.61:0/1567692536 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:17.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f32690d8700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f326409bce0 0x7f32640af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56794/0 (socket says 172.21.15.61:56794) 2022-07-12T08:43:17.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f32698d9700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f3264060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 -- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 msgr2=0x7f3264060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:17.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f3264060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 -- 172.21.15.61:0/1567692536 >> v1:172.21.15.111:6789/0 conn(0x7f326409bce0 legacy=0x7f32640af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 -- 172.21.15.61:0/1567692536 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32641435c0 con 0x7f32640b3f60 2022-07-12T08:43:17.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f32698d9700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f3264060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:17.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a0da700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32640b4810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f325c0032f0 tx=0x7f325c009290).ready entity=mon.1 client_cookie=82d151db6d670c08 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f32688d7700 1 -- 172.21.15.61:0/1567692536 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325c00b8a0 con 0x7f32640b3f60 2022-07-12T08:43:17.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f32688d7700 1 -- 172.21.15.61:0/1567692536 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c00ba00 con 0x7f32640b3f60 2022-07-12T08:43:17.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f32688d7700 1 -- 172.21.15.61:0/1567692536 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325c00bb70 con 0x7f32640b3f60 2022-07-12T08:43:17.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 msgr2=0x7f32640b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32640b4810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f325c0032f0 tx=0x7f325c009290).stop 2022-07-12T08:43:17.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- 172.21.15.61:0/1567692536 shutdown_connections 2022-07-12T08:43:17.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32640b4810 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 --2- 172.21.15.61:0/1567692536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f3264060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- 172.21.15.61:0/1567692536 >> 172.21.15.61:0/1567692536 conn(0x7f32640b92c0 msgr2=0x7f32640b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:17.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- 172.21.15.61:0/1567692536 shutdown_connections 2022-07-12T08:43:17.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.088+0000 7f326a8db700 1 -- 172.21.15.61:0/1567692536 wait complete. 2022-07-12T08:43:17.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 Processor -- start 2022-07-12T08:43:17.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 -- start start 2022-07-12T08:43:17.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 0x7f3264152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f32641525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32641406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32640b18c0 con 0x7f326409bce0 2022-07-12T08:43:17.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32640b1b60 con 0x7f32640b3f60 2022-07-12T08:43:17.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32640b15e0 con 0x7f32640af280 2022-07-12T08:43:17.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 0x7f3264152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 0x7f3264152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54978/0 (socket says 172.21.15.61:54978) 2022-07-12T08:43:17.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 -- 172.21.15.61:0/686752867 learned_addr learned my addr 172.21.15.61:0/686752867 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:17.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a0da700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32641406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32690d8700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f32641525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 -- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 msgr2=0x7f32641525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f32641525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 -- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 msgr2=0x7f32641406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32641406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 -- 172.21.15.61:0/686752867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325400b120 con 0x7f326409bce0 2022-07-12T08:43:17.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32690d8700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f32641525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:17.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f32698d9700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 0x7f3264152100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f325400c000 tx=0x7f3254008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3254012600 con 0x7f326409bce0 2022-07-12T08:43:17.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3254012760 con 0x7f326409bce0 2022-07-12T08:43:17.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3254012ac0 con 0x7f326409bce0 2022-07-12T08:43:17.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c009530 con 0x7f326409bce0 2022-07-12T08:43:17.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.092+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32640b0700 con 0x7f326409bce0 2022-07-12T08:43:17.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.096+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32640b1d20 con 0x7f326409bce0 2022-07-12T08:43:17.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.096+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3254022070 con 0x7f326409bce0 2022-07-12T08:43:17.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.096+0000 7f3262ffd700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32440539d0 0x7f3244055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.096+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f325406c330 con 0x7f326409bce0 2022-07-12T08:43:17.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.096+0000 7f32690d8700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32440539d0 0x7f3244055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.096+0000 7f32690d8700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32440539d0 0x7f3244055e90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f325800b000 tx=0x7f32580070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.100+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3254043340 con 0x7f326409bce0 2022-07-12T08:43:17.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.232+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3264057460 con 0x7f326409bce0 2022-07-12T08:43:17.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.232+0000 7f3262ffd700 1 -- 172.21.15.61:0/686752867 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f3254046960 con 0x7f326409bce0 2022-07-12T08:43:17.241 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:17.241 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:43:16.617857+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:43:13.384889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-12T08:43:14.650247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:43:17.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32440539d0 msgr2=0x7f3244055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32440539d0 0x7f3244055e90 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f325800b000 tx=0x7f32580070e0).stop 2022-07-12T08:43:17.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 msgr2=0x7f3264152100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 0x7f3264152100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f325400c000 tx=0x7f3254008040).stop 2022-07-12T08:43:17.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 shutdown_connections 2022-07-12T08:43:17.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f32440539d0 0x7f3244055e90 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b3f60 0x7f32641406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640af280 0x7f32641525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 --2- 172.21.15.61:0/686752867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f326409bce0 0x7f3264152100 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 >> 172.21.15.61:0/686752867 conn(0x7f32640b92c0 msgr2=0x7f326405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:17.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 shutdown_connections 2022-07-12T08:43:17.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.236+0000 7f326a8db700 1 -- 172.21.15.61:0/686752867 wait complete. 2022-07-12T08:43:17.258 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:43:17.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 Processor -- start 2022-07-12T08:43:17.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 -- start start 2022-07-12T08:43:17.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f115c1427b0 0x7f115c0618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f115c138f40 0x7f115c140370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115c142b90 con 0x7f115c0b02f0 2022-07-12T08:43:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115c060ef0 con 0x7f115c1427b0 2022-07-12T08:43:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115bfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f115c061160 con 0x7f115c138f40 2022-07-12T08:43:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115a7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f115c0b02f0 0x7f115c138830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43206/0 (socket says 172.21.15.61:43206) 2022-07-12T08:43:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f115c1427b0 0x7f115c0618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115a7fc700 1 -- 172.21.15.61:0/3275780369 learned_addr learned my addr 172.21.15.61:0/3275780369 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115b7fe700 1 --2- 172.21.15.61:0/3275780369 >> v2:172.21.15.111:3300/0 conn(0x7f115c138f40 0x7f115c140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f1159ffb700 1 -- 172.21.15.61:0/3275780369 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157004293 0 0) 0x7f115c142b90 con 0x7f115c0b02f0 2022-07-12T08:43:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f1159ffb700 1 -- 172.21.15.61:0/3275780369 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1140003260 con 0x7f115c0b02f0 2022-07-12T08:43:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115affd700 1 -- 172.21.15.61:0/3275780369 >> v2:172.21.15.111:3300/0 conn(0x7f115c138f40 msgr2=0x7f115c140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115affd700 1 --2- 172.21.15.61:0/3275780369 >> v2:172.21.15.111:3300/0 conn(0x7f115c138f40 0x7f115c140370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115affd700 1 -- 172.21.15.61:0/3275780369 >> v1:172.21.15.61:6789/0 conn(0x7f115c0b02f0 legacy=0x7f115c138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.372+0000 7f115affd700 1 -- 172.21.15.61:0/3275780369 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c061400 con 0x7f115c1427b0 2022-07-12T08:43:17.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115affd700 1 --2- 172.21.15.61:0/3275780369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c1427b0 0x7f115c0618d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f114c002f70 tx=0x7f114c006df0).ready entity=mon.0 client_cookie=f8504da15ef3e607 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f1159ffb700 1 -- 172.21.15.61:0/3275780369 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114c00d4f0 con 0x7f115c1427b0 2022-07-12T08:43:17.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f1159ffb700 1 -- 172.21.15.61:0/3275780369 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c000e90 con 0x7f115c1427b0 2022-07-12T08:43:17.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f1159ffb700 1 -- 172.21.15.61:0/3275780369 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114c00d840 con 0x7f115c1427b0 2022-07-12T08:43:17.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 -- 172.21.15.61:0/3275780369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c1427b0 msgr2=0x7f115c0618d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 --2- 172.21.15.61:0/3275780369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c1427b0 0x7f115c0618d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f114c002f70 tx=0x7f114c006df0).stop 2022-07-12T08:43:17.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 -- 172.21.15.61:0/3275780369 shutdown_connections 2022-07-12T08:43:17.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 --2- 172.21.15.61:0/3275780369 >> v2:172.21.15.111:3300/0 conn(0x7f115c138f40 0x7f115c140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 --2- 172.21.15.61:0/3275780369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c1427b0 0x7f115c0618d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 -- 172.21.15.61:0/3275780369 >> 172.21.15.61:0/3275780369 conn(0x7f115c0b5860 msgr2=0x7f115c0b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:17.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 -- 172.21.15.61:0/3275780369 shutdown_connections 2022-07-12T08:43:17.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 -- 172.21.15.61:0/3275780369 wait complete. 2022-07-12T08:43:17.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 Processor -- start 2022-07-12T08:43:17.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.376+0000 7f115bfff700 1 -- start start 2022-07-12T08:43:17.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 0x7f115c14f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f115c138f40 0x7f115c145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f115c1427b0 0x7f115c150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c14fd50 con 0x7f115c0b02f0 2022-07-12T08:43:17.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115c152e50 con 0x7f115c138f40 2022-07-12T08:43:17.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115c0aea30 con 0x7f115c1427b0 2022-07-12T08:43:17.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 0x7f115c14f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 0x7f115c14f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:54992/0 (socket says 172.21.15.61:54992) 2022-07-12T08:43:17.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 -- 172.21.15.61:0/2439516841 learned_addr learned my addr 172.21.15.61:0/2439516841 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:17.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115b7fe700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f115c1427b0 0x7f115c150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115a7fc700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f115c138f40 0x7f115c145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 -- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f115c1427b0 msgr2=0x7f115c150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f115c1427b0 0x7f115c150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 -- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f115c138f40 msgr2=0x7f115c145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f115c138f40 0x7f115c145820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 -- 172.21.15.61:0/2439516841 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c00b180 con 0x7f115c0b02f0 2022-07-12T08:43:17.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115b7fe700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f115c1427b0 0x7f115c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:17.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115affd700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 0x7f115c14f860 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f114c002f70 tx=0x7f114c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114c00de60 con 0x7f115c0b02f0 2022-07-12T08:43:17.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114c01e400 con 0x7f115c0b02f0 2022-07-12T08:43:17.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115c0ae790 con 0x7f115c0b02f0 2022-07-12T08:43:17.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114c01e760 con 0x7f115c0b02f0 2022-07-12T08:43:17.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f115c145d10 con 0x7f115c0b02f0 2022-07-12T08:43:17.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f115c142e00 con 0x7f115c0b02f0 2022-07-12T08:43:17.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.380+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f114c025030 con 0x7f115c0b02f0 2022-07-12T08:43:17.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.384+0000 7f114bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f11380539b0 0x7f1138055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.384+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f114c06caa0 con 0x7f115c0b02f0 2022-07-12T08:43:17.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.384+0000 7f115a7fc700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f11380539b0 0x7f1138055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.384+0000 7f115a7fc700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f11380539b0 0x7f1138055e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f115000b000 tx=0x7f1150009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.388+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f114c043a00 con 0x7f115c0b02f0 2022-07-12T08:43:17.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.556+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f115c152fc0 con 0x7f115c0b02f0 2022-07-12T08:43:17.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.564+0000 7f114bfff700 1 -- 172.21.15.61:0/2439516841 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f114c023020 con 0x7f115c0b02f0 2022-07-12T08:43:17.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f11380539b0 msgr2=0x7f1138055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f11380539b0 0x7f1138055e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f115000b000 tx=0x7f1150009040).stop 2022-07-12T08:43:17.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 msgr2=0x7f115c14f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 0x7f115c14f860 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f114c002f70 tx=0x7f114c014000).stop 2022-07-12T08:43:17.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 shutdown_connections 2022-07-12T08:43:17.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f115c1427b0 0x7f115c150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f11380539b0 0x7f1138055e70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f115c138f40 0x7f115c145820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 --2- 172.21.15.61:0/2439516841 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f115c0b02f0 0x7f115c14f860 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 >> 172.21.15.61:0/2439516841 conn(0x7f115c0b5860 msgr2=0x7f115c0b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:17.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 shutdown_connections 2022-07-12T08:43:17.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.568+0000 7f115bfff700 1 -- 172.21.15.61:0/2439516841 wait complete. 2022-07-12T08:43:17.589 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:43:17.591 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:43:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4ec39700 1 Processor -- start 2022-07-12T08:43:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4ec39700 1 -- start start 2022-07-12T08:43:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4ec39700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce4805cd20 0x7fce48146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4ec39700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4805ff60 con 0x7fce480af490 2022-07-12T08:43:17.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4ec39700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce48146200 con 0x7fce4805d100 2022-07-12T08:43:17.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4ec39700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce481464a0 con 0x7fce4805cd20 2022-07-12T08:43:17.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4d436700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fce480af490 0x7fce4805f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43218/0 (socket says 172.21.15.61:43218) 2022-07-12T08:43:17.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4d436700 1 -- 172.21.15.61:0/3188688165 learned_addr learned my addr 172.21.15.61:0/3188688165 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:17.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4dc37700 1 --2- 172.21.15.61:0/3188688165 >> v2:172.21.15.111:3300/0 conn(0x7fce4805cd20 0x7fce48146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.728+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084848248 0 0) 0x7fce4805ff60 con 0x7fce480af490 2022-07-12T08:43:17.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce30003260 con 0x7fce480af490 2022-07-12T08:43:17.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985805230 0 0) 0x7fce48146200 con 0x7fce4805d100 2022-07-12T08:43:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce4805ff60 con 0x7fce4805d100 2022-07-12T08:43:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4dc37700 1 -- 172.21.15.61:0/3188688165 >> v1:172.21.15.111:6789/0 conn(0x7fce4805d100 legacy=0x7fce4805f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4dc37700 1 -- 172.21.15.61:0/3188688165 >> v1:172.21.15.61:6789/0 conn(0x7fce480af490 legacy=0x7fce4805f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4dc37700 1 -- 172.21.15.61:0/3188688165 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce48146740 con 0x7fce4805cd20 2022-07-12T08:43:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4dc37700 1 --2- 172.21.15.61:0/3188688165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4805cd20 0x7fce48146be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fce38002f70 tx=0x7fce38006df0).ready entity=mon.1 client_cookie=be9a7492224f7a8d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce3800d540 con 0x7fce4805cd20 2022-07-12T08:43:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce38000e90 con 0x7fce4805cd20 2022-07-12T08:43:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4cc35700 1 -- 172.21.15.61:0/3188688165 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce3800d890 con 0x7fce4805cd20 2022-07-12T08:43:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 -- 172.21.15.61:0/3188688165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4805cd20 msgr2=0x7fce48146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 --2- 172.21.15.61:0/3188688165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4805cd20 0x7fce48146be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fce38002f70 tx=0x7fce38006df0).stop 2022-07-12T08:43:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 -- 172.21.15.61:0/3188688165 shutdown_connections 2022-07-12T08:43:17.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 --2- 172.21.15.61:0/3188688165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce4805cd20 0x7fce48146be0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 -- 172.21.15.61:0/3188688165 >> 172.21.15.61:0/3188688165 conn(0x7fce480b9e20 msgr2=0x7fce480b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:17.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 -- 172.21.15.61:0/3188688165 shutdown_connections 2022-07-12T08:43:17.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 -- 172.21.15.61:0/3188688165 wait complete. 2022-07-12T08:43:17.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 Processor -- start 2022-07-12T08:43:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.732+0000 7fce4ec39700 1 -- start start 2022-07-12T08:43:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce4805cd20 0x7fce480b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 0x7fce48151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce480af490 0x7fce48148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce48152cb0 con 0x7fce4805d100 2022-07-12T08:43:17.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4809b8f0 con 0x7fce480af490 2022-07-12T08:43:17.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce48155df0 con 0x7fce4805cd20 2022-07-12T08:43:17.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 0x7fce48151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 0x7fce48151c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55006/0 (socket says 172.21.15.61:55006) 2022-07-12T08:43:17.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 -- 172.21.15.61:0/4209571409 learned_addr learned my addr 172.21.15.61:0/4209571409 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:17.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4e438700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce480af490 0x7fce48148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4dc37700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce4805cd20 0x7fce480b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 -- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce4805cd20 msgr2=0x7fce480b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce4805cd20 0x7fce480b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 -- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce480af490 msgr2=0x7fce48148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce480af490 0x7fce48148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 -- 172.21.15.61:0/4209571409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3800b180 con 0x7fce4805d100 2022-07-12T08:43:17.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4e438700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce480af490 0x7fce48148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:17.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4d436700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 0x7fce48151c40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fce44001110 tx=0x7fce44007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce440070d0 con 0x7fce4805d100 2022-07-12T08:43:17.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44007230 con 0x7fce4805d100 2022-07-12T08:43:17.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce480b0e20 con 0x7fce4805d100 2022-07-12T08:43:17.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce44005610 con 0x7fce4805d100 2022-07-12T08:43:17.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.736+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce48098d00 con 0x7fce4805d100 2022-07-12T08:43:17.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.740+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce48057460 con 0x7fce4805d100 2022-07-12T08:43:17.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.740+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fce44010070 con 0x7fce4805d100 2022-07-12T08:43:17.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.740+0000 7fce36ffd700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fce28053a60 0x7fce28055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:17.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.740+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fce4406b5d0 con 0x7fce4805d100 2022-07-12T08:43:17.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.740+0000 7fce4dc37700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fce28053a60 0x7fce28055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:17.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.740+0000 7fce4dc37700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fce28053a60 0x7fce28055f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fce3801a000 tx=0x7fce38014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:17.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.744+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fce440423a0 con 0x7fce4805d100 2022-07-12T08:43:17.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.908+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fce48155fc0 con 0x7fce4805d100 2022-07-12T08:43:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.916+0000 7fce36ffd700 1 -- 172.21.15.61:0/4209571409 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7fce440459c0 con 0x7fce4805d100 2022-07-12T08:43:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fce28053a60 msgr2=0x7fce28055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fce28053a60 0x7fce28055f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fce3801a000 tx=0x7fce38014000).stop 2022-07-12T08:43:17.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 msgr2=0x7fce48151c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:17.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 0x7fce48151c40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fce44001110 tx=0x7fce44007ad0).stop 2022-07-12T08:43:17.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 shutdown_connections 2022-07-12T08:43:17.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fce28053a60 0x7fce28055f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce480af490 0x7fce48148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce4805d100 0x7fce48151c40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 --2- 172.21.15.61:0/4209571409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce4805cd20 0x7fce480b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 >> 172.21.15.61:0/4209571409 conn(0x7fce480b9e20 msgr2=0x7fce480b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:17.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 shutdown_connections 2022-07-12T08:43:17.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:17.920+0000 7fce4ec39700 1 -- 172.21.15.61:0/4209571409 wait complete. 2022-07-12T08:43:17.933 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:43:17.941 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:43:18.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 Processor -- start 2022-07-12T08:43:18.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 -- start start 2022-07-12T08:43:18.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1260141e50 0x7f12601422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12600b56c0 con 0x7f12600b52e0 2022-07-12T08:43:18.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1260142f00 con 0x7f1260141e50 2022-07-12T08:43:18.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f1268358700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12600b1320 con 0x7f12600b4050 2022-07-12T08:43:18.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12660f4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f12600b52e0 0x7f12600b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43232/0 (socket says 172.21.15.61:43232) 2022-07-12T08:43:18.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12660f4700 1 -- 172.21.15.61:0/2662827335 learned_addr learned my addr 172.21.15.61:0/2662827335 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:18.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12668f5700 1 --2- 172.21.15.61:0/2662827335 >> v2:172.21.15.111:3300/0 conn(0x7f1260141e50 0x7f12601422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12658f3700 1 --2- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12650f2700 1 -- 172.21.15.61:0/2662827335 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222681493 0 0) 0x7f12600b56c0 con 0x7f12600b52e0 2022-07-12T08:43:18.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12650f2700 1 -- 172.21.15.61:0/2662827335 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1248003260 con 0x7f12600b52e0 2022-07-12T08:43:18.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12668f5700 1 -- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 msgr2=0x7f12600b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:18.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12668f5700 1 --2- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12668f5700 1 -- 172.21.15.61:0/2662827335 >> v1:172.21.15.61:6789/0 conn(0x7f12600b52e0 legacy=0x7f12600b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12668f5700 1 -- 172.21.15.61:0/2662827335 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12600b4bd0 con 0x7f1260141e50 2022-07-12T08:43:18.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12668f5700 1 --2- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12601422c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f12600b1460 tx=0x7f12580091d0).ready entity=mon.1 client_cookie=b02fff367d7b6b2b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:18.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12650f2700 1 -- 172.21.15.61:0/2662827335 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f125800b600 con 0x7f1260141e50 2022-07-12T08:43:18.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12650f2700 1 -- 172.21.15.61:0/2662827335 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125800b760 con 0x7f1260141e50 2022-07-12T08:43:18.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.052+0000 7f12650f2700 1 -- 172.21.15.61:0/2662827335 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f125800b8d0 con 0x7f1260141e50 2022-07-12T08:43:18.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 -- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 msgr2=0x7f12601422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 --2- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12601422c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f12600b1460 tx=0x7f12580091d0).stop 2022-07-12T08:43:18.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 -- 172.21.15.61:0/2662827335 shutdown_connections 2022-07-12T08:43:18.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 --2- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12601422c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 --2- 172.21.15.61:0/2662827335 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 -- 172.21.15.61:0/2662827335 >> 172.21.15.61:0/2662827335 conn(0x7f1260067020 msgr2=0x7f1260067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:18.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 -- 172.21.15.61:0/2662827335 shutdown_connections 2022-07-12T08:43:18.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 -- 172.21.15.61:0/2662827335 wait complete. 2022-07-12T08:43:18.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 Processor -- start 2022-07-12T08:43:18.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.056+0000 7f1268358700 1 -- start start 2022-07-12T08:43:18.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 0x7f1260147cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12600af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260146230 con 0x7f12600b52e0 2022-07-12T08:43:18.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260147b10 con 0x7f1260141e50 2022-07-12T08:43:18.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1260147870 con 0x7f12600b4050 2022-07-12T08:43:18.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 0x7f1260147cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 0x7f1260147cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55022/0 (socket says 172.21.15.61:55022) 2022-07-12T08:43:18.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 -- 172.21.15.61:0/3227726131 learned_addr learned my addr 172.21.15.61:0/3227726131 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:18.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12668f5700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12600af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12660f4700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 -- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 msgr2=0x7f12600b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 -- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 msgr2=0x7f12600af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12600af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 -- 172.21.15.61:0/3227726131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1258009470 con 0x7f12600b52e0 2022-07-12T08:43:18.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12668f5700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12600af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:18.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12658f3700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 0x7f1260147cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f125400c000 tx=0x7f1254006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:18.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1254010780 con 0x7f12600b52e0 2022-07-12T08:43:18.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12540108e0 con 0x7f12600b52e0 2022-07-12T08:43:18.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1260144510 con 0x7f12600b52e0 2022-07-12T08:43:18.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1254010c40 con 0x7f12600b52e0 2022-07-12T08:43:18.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.060+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12600afa10 con 0x7f12600b52e0 2022-07-12T08:43:18.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.064+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12601446a0 con 0x7f12600b52e0 2022-07-12T08:43:18.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.064+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f125401c460 con 0x7f12600b52e0 2022-07-12T08:43:18.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.064+0000 7f12537fe700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1240053ad0 0x7f1240055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.064+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f125406c280 con 0x7f12600b52e0 2022-07-12T08:43:18.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.064+0000 7f12660f4700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1240053ad0 0x7f1240055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.064+0000 7f12660f4700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1240053ad0 0x7f1240055f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f125c00d000 tx=0x7f125c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:18.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.072+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1254043180 con 0x7f12600b52e0 2022-07-12T08:43:18.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.236+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1260057460 con 0x7f12600b52e0 2022-07-12T08:43:18.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.244+0000 7f12537fe700 1 -- 172.21.15.61:0/3227726131 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f12540467a0 con 0x7f12600b52e0 2022-07-12T08:43:18.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1240053ad0 msgr2=0x7f1240055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1240053ad0 0x7f1240055f90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f125c00d000 tx=0x7f125c009040).stop 2022-07-12T08:43:18.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 msgr2=0x7f1260147cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 0x7f1260147cd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f125400c000 tx=0x7f1254006df0).stop 2022-07-12T08:43:18.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 shutdown_connections 2022-07-12T08:43:18.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1240053ad0 0x7f1240055f90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1260141e50 0x7f12600af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12600b52e0 0x7f1260147cd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 --2- 172.21.15.61:0/3227726131 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12600b4050 0x7f12600b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 >> 172.21.15.61:0/3227726131 conn(0x7f1260067020 msgr2=0x7f1260142260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:18.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 shutdown_connections 2022-07-12T08:43:18.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.248+0000 7f1268358700 1 -- 172.21.15.61:0/3227726131 wait complete. 2022-07-12T08:43:18.266 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:43:18.270 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:18.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.364+0000 7fba0d5d8700 1 Processor -- start 2022-07-12T08:43:18.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.364+0000 7fba0d5d8700 1 -- start start 2022-07-12T08:43:18.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.364+0000 7fba0d5d8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 0x7fba0805f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.364+0000 7fba0d5d8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fba080b3760 0x7fba08146230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba08143810 con 0x7fba08146720 2022-07-12T08:43:18.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba080606d0 con 0x7fba080b3760 2022-07-12T08:43:18.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba080ad0f0 con 0x7fba0805f600 2022-07-12T08:43:18.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba077fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fba08146720 0x7fba081420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43250/0 (socket says 172.21.15.61:43250) 2022-07-12T08:43:18.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba077fe700 1 -- 172.21.15.61:0/4082371539 learned_addr learned my addr 172.21.15.61:0/4082371539 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:18.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 --2- 172.21.15.61:0/4082371539 >> v2:172.21.15.111:3300/0 conn(0x7fba080b3760 0x7fba08146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba06ffd700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 0x7fba0805f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069337076 0 0) 0x7fba08143810 con 0x7fba08146720 2022-07-12T08:43:18.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec003260 con 0x7fba08146720 2022-07-12T08:43:18.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701570878 0 0) 0x7fb9ec003260 con 0x7fba08146720 2022-07-12T08:43:18.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 msgr2=0x7fba0805f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:18.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 0x7fba0805f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 >> v2:172.21.15.111:3300/0 conn(0x7fba080b3760 msgr2=0x7fba08146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:18.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 --2- 172.21.15.61:0/4082371539 >> v2:172.21.15.111:3300/0 conn(0x7fba080b3760 0x7fba08146230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba080acd70 con 0x7fba08146720 2022-07-12T08:43:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fb9f8001660 con 0x7fba08146720 2022-07-12T08:43:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 >> v1:172.21.15.61:6789/0 conn(0x7fba08146720 legacy=0x7fba081420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 0x7fb9ec003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9ec003c80 0x7fb9ec004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9ec0049f0 0x7fb9ec004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080606d0 con 0x7fb9ec003c80 2022-07-12T08:43:18.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba080ad0f0 con 0x7fb9ec0049f0 2022-07-12T08:43:18.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec00b750 con 0x7fba0805f600 2022-07-12T08:43:18.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9ec003c80 0x7fb9ec004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba06ffd700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 0x7fb9ec003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba077fe700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9ec0049f0 0x7fb9ec004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 -- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 msgr2=0x7fb9ec003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:18.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fba0805f600 0x7fb9ec003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 -- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9ec0049f0 msgr2=0x7fb9ec004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:18.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9ec0049f0 0x7fb9ec004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 -- 172.21.15.61:0/4082371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec003260 con 0x7fb9ec003c80 2022-07-12T08:43:18.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba077fe700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9ec0049f0 0x7fb9ec004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba067fc700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9ec003c80 0x7fb9ec004500 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fb9ec0054d0 tx=0x7fb9f0006df0).ready entity=mon.0 client_cookie=12644cd4a44f4771 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f000e830 con 0x7fb9ec003c80 2022-07-12T08:43:18.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f000be90 con 0x7fb9ec003c80 2022-07-12T08:43:18.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba05ffb700 1 -- 172.21.15.61:0/4082371539 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f000eb80 con 0x7fb9ec003c80 2022-07-12T08:43:18.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- 172.21.15.61:0/4082371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9ec003c80 msgr2=0x7fb9ec004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/4082371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9ec003c80 0x7fb9ec004500 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fb9ec0054d0 tx=0x7fb9f0006df0).stop 2022-07-12T08:43:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba077fe700 1 -- 172.21.15.61:0/4082371539 reap_dead start 2022-07-12T08:43:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- 172.21.15.61:0/4082371539 shutdown_connections 2022-07-12T08:43:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- 172.21.15.61:0/4082371539 >> 172.21.15.61:0/4082371539 conn(0x7fba080b3d50 msgr2=0x7fba080b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- 172.21.15.61:0/4082371539 shutdown_connections 2022-07-12T08:43:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.368+0000 7fba0d5d8700 1 -- 172.21.15.61:0/4082371539 wait complete. 2022-07-12T08:43:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 Processor -- start 2022-07-12T08:43:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 -- start start 2022-07-12T08:43:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9ec003c80 0x7fba0809cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 0x7fba08155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08155580 0x7fba0809b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0809c9b0 con 0x7fba08096730 2022-07-12T08:43:18.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba08067760 con 0x7fba08155580 2022-07-12T08:43:18.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba08067a00 con 0x7fb9ec003c80 2022-07-12T08:43:18.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 0x7fba08155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 0x7fba08155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55042/0 (socket says 172.21.15.61:55042) 2022-07-12T08:43:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 -- 172.21.15.61:0/2197965339 learned_addr learned my addr 172.21.15.61:0/2197965339 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba06ffd700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9ec003c80 0x7fba0809cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba077fe700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08155580 0x7fba0809b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 -- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9ec003c80 msgr2=0x7fba0809cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9ec003c80 0x7fba0809cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 -- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08155580 msgr2=0x7fba0809b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08155580 0x7fba0809b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 -- 172.21.15.61:0/2197965339 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba08155a00 con 0x7fba08096730 2022-07-12T08:43:18.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba067fc700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 0x7fba08155db0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fb9ec0054d0 tx=0x7fb9f0019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:18.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f0012d60 con 0x7fba08096730 2022-07-12T08:43:18.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f0000e90 con 0x7fba08096730 2022-07-12T08:43:18.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0809b980 con 0x7fba08096730 2022-07-12T08:43:18.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f001c620 con 0x7fba08096730 2022-07-12T08:43:18.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.372+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0809be80 con 0x7fba08096730 2022-07-12T08:43:18.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.376+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba08096c10 con 0x7fba08096730 2022-07-12T08:43:18.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.376+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb9f001c780 con 0x7fba08096730 2022-07-12T08:43:18.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.376+0000 7fb9fffff700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9e0053a60 0x7fb9e0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:18.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.376+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb9f006da90 con 0x7fba08096730 2022-07-12T08:43:18.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.376+0000 7fba06ffd700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9e0053a60 0x7fb9e0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:18.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.376+0000 7fba06ffd700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9e0053a60 0x7fb9e0055f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fba0000b000 tx=0x7fba000091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:18.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.384+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb9f0044880 con 0x7fba08096730 2022-07-12T08:43:18.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.544+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba08067bc0 con 0x7fba08096730 2022-07-12T08:43:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fb9fffff700 1 -- 172.21.15.61:0/2197965339 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7fb9f0047ea0 con 0x7fba08096730 2022-07-12T08:43:18.560 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:18.560 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9e0053a60 msgr2=0x7fb9e0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9e0053a60 0x7fb9e0055f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fba0000b000 tx=0x7fba000091d0).stop 2022-07-12T08:43:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 msgr2=0x7fba08155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 0x7fba08155db0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fb9ec0054d0 tx=0x7fb9f0019000).stop 2022-07-12T08:43:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 shutdown_connections 2022-07-12T08:43:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9e0053a60 0x7fb9e0055f20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba08155580 0x7fba0809b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba08096730 0x7fba08155db0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 --2- 172.21.15.61:0/2197965339 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9ec003c80 0x7fba0809cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 >> 172.21.15.61:0/2197965339 conn(0x7fba080b3d50 msgr2=0x7fba080ad470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 shutdown_connections 2022-07-12T08:43:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:18.548+0000 7fba0d5d8700 1 -- 172.21.15.61:0/2197965339 wait complete. 2022-07-12T08:43:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:18.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:43:15.615120+0000', 'modified': '2022-07-12T08:43:16.618859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'a', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/4199688387', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 4199688387}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 4199688387}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:43:18.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:43:18.776 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:43:18.895 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":3.2668022720000001,"mdsmap_epoch":17,"osdmap_epoch":42,"osdmap_epoch_barrier":42,"uptime":8.5120984699999998} 2022-07-12T08:43:18.902 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5241, "mdsmap_epoch": 17, "osdmap_epoch": 42, "osdmap_epoch_barrier": 42, "rank_uptime": 3.266802272, "state": "up:active", "uptime": 8.51209847, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:43:18.909 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:43:19.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bb59e700 1 Processor -- start 2022-07-12T08:43:19.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bb59e700 1 -- start start 2022-07-12T08:43:19.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bb59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f83bc05c740 0x7f83bc098070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc05d220 con 0x7f83bc05f340 2022-07-12T08:43:19.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bb59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83bc143460 con 0x7f83bc0b1210 2022-07-12T08:43:19.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bb59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83bc0aaa00 con 0x7f83bc05c740 2022-07-12T08:43:19.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f83bc05f340 0x7f83bc142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43270/0 (socket says 172.21.15.61:43270) 2022-07-12T08:43:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83bad9d700 1 -- 172.21.15.61:0/2036580733 learned_addr learned my addr 172.21.15.61:0/2036580733 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.024+0000 7f83ba59c700 1 --2- 172.21.15.61:0/2036580733 >> v2:172.21.15.111:3300/0 conn(0x7f83bc05c740 0x7f83bc098070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359199428 0 0) 0x7f83bc143460 con 0x7f83bc0b1210 2022-07-12T08:43:19.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f839c003260 con 0x7f83bc0b1210 2022-07-12T08:43:19.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361202378 0 0) 0x7f83bc05d220 con 0x7f83bc05f340 2022-07-12T08:43:19.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83bc143460 con 0x7f83bc05f340 2022-07-12T08:43:19.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83ba59c700 1 -- 172.21.15.61:0/2036580733 >> v1:172.21.15.111:6789/0 conn(0x7f83bc0b1210 legacy=0x7f83bc142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83ba59c700 1 -- 172.21.15.61:0/2036580733 >> v1:172.21.15.61:6789/0 conn(0x7f83bc05f340 legacy=0x7f83bc142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83ba59c700 1 -- 172.21.15.61:0/2036580733 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc09aca0 con 0x7f83bc05c740 2022-07-12T08:43:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83ba59c700 1 --2- 172.21.15.61:0/2036580733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05c740 0x7f83bc098070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f83ac002f70 tx=0x7f83ac006df0).ready entity=mon.1 client_cookie=f8b4516a29ccd988 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83ac00d540 con 0x7f83bc05c740 2022-07-12T08:43:19.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac000e90 con 0x7f83bc05c740 2022-07-12T08:43:19.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83b9d9b700 1 -- 172.21.15.61:0/2036580733 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83ac00d890 con 0x7f83bc05c740 2022-07-12T08:43:19.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 -- 172.21.15.61:0/2036580733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05c740 msgr2=0x7f83bc098070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2036580733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05c740 0x7f83bc098070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f83ac002f70 tx=0x7f83ac006df0).stop 2022-07-12T08:43:19.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 -- 172.21.15.61:0/2036580733 shutdown_connections 2022-07-12T08:43:19.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2036580733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05c740 0x7f83bc098070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 -- 172.21.15.61:0/2036580733 >> 172.21.15.61:0/2036580733 conn(0x7f83bc0b1950 msgr2=0x7f83bc0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 -- 172.21.15.61:0/2036580733 shutdown_connections 2022-07-12T08:43:19.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 -- 172.21.15.61:0/2036580733 wait complete. 2022-07-12T08:43:19.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 Processor -- start 2022-07-12T08:43:19.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.028+0000 7f83bb59e700 1 -- start start 2022-07-12T08:43:19.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83bc05c740 0x7f83bc14fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05f340 0x7f83bc140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 0x7f83bc0ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc130ce0 con 0x7f83bc0b1210 2022-07-12T08:43:19.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0ac970 con 0x7f83bc05f340 2022-07-12T08:43:19.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc0acc00 con 0x7f83bc05c740 2022-07-12T08:43:19.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 0x7f83bc0ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 0x7f83bc0ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55058/0 (socket says 172.21.15.61:55058) 2022-07-12T08:43:19.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 -- 172.21.15.61:0/2912390326 learned_addr learned my addr 172.21.15.61:0/2912390326 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:19.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83ba59c700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83bc05c740 0x7f83bc14fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83b3fff700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05f340 0x7f83bc140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 -- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83bc05c740 msgr2=0x7f83bc14fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83bc05c740 0x7f83bc14fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 -- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05f340 msgr2=0x7f83bc140c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05f340 0x7f83bc140c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 -- 172.21.15.61:0/2912390326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac00b180 con 0x7f83bc0b1210 2022-07-12T08:43:19.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bad9d700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 0x7f83bc0ac480 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f83b4001490 tx=0x7f83b4007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83b4005ba0 con 0x7f83bc0b1210 2022-07-12T08:43:19.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b4005d00 con 0x7f83bc0b1210 2022-07-12T08:43:19.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc0acea0 con 0x7f83bc0b1210 2022-07-12T08:43:19.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83b40072c0 con 0x7f83bc0b1210 2022-07-12T08:43:19.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.032+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83bc13e8d0 con 0x7f83bc0b1210 2022-07-12T08:43:19.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.036+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83bc1532e0 con 0x7f83bc0b1210 2022-07-12T08:43:19.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.036+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f83b401a070 con 0x7f83bc0b1210 2022-07-12T08:43:19.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.036+0000 7f83b37fe700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83a00539b0 0x7f83a0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.036+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f83b406b3d0 con 0x7f83bc0b1210 2022-07-12T08:43:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.036+0000 7f83ba59c700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83a00539b0 0x7f83a0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.036+0000 7f83ba59c700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83a00539b0 0x7f83a0055e70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f83ac01a000 tx=0x7f83ac014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.040+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f83bc1532e0 con 0x7f83bc0b1210 2022-07-12T08:43:19.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.168+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83bc1532e0 con 0x7f83bc0b1210 2022-07-12T08:43:19.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83b37fe700 1 -- 172.21.15.61:0/2912390326 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f83bc1532e0 con 0x7f83bc0b1210 2022-07-12T08:43:19.196 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:19.197 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:43:16.617857+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:43:13.384889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-12T08:43:14.650247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:43:19.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83a00539b0 msgr2=0x7f83a0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83a00539b0 0x7f83a0055e70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f83ac01a000 tx=0x7f83ac014000).stop 2022-07-12T08:43:19.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 msgr2=0x7f83bc0ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 0x7f83bc0ac480 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f83b4001490 tx=0x7f83b4007e70).stop 2022-07-12T08:43:19.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 shutdown_connections 2022-07-12T08:43:19.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83bc0b1210 0x7f83bc0ac480 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83a00539b0 0x7f83a0055e70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83bc05f340 0x7f83bc140c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 --2- 172.21.15.61:0/2912390326 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83bc05c740 0x7f83bc14fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 >> 172.21.15.61:0/2912390326 conn(0x7f83bc0b1950 msgr2=0x7f83bc0b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 shutdown_connections 2022-07-12T08:43:19.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.172+0000 7f83bb59e700 1 -- 172.21.15.61:0/2912390326 wait complete. 2022-07-12T08:43:19.211 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:43:13.384889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:43:14.650247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:43:19.213 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:43:19.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 Processor -- start 2022-07-12T08:43:19.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 -- start start 2022-07-12T08:43:19.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f674c05d060 0x7f674c0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142ab0 0x7f674c0afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c05d440 con 0x7f674c142f10 2022-07-12T08:43:19.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f674c143d80 con 0x7f674c05d060 2022-07-12T08:43:19.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.320+0000 7f675270f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c143ab0 con 0x7f674c142ab0 2022-07-12T08:43:19.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675170d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f674c05d060 0x7f674c0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6751f0e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f674c142f10 0x7f674c1433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56886/0 (socket says 172.21.15.61:56886) 2022-07-12T08:43:19.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6751f0e700 1 -- 172.21.15.61:0/604062961 learned_addr learned my addr 172.21.15.61:0/604062961 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:19.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6750f0c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142ab0 0x7f674c0afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6737fff700 1 -- 172.21.15.61:0/604062961 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773739427 0 0) 0x7f674c05d440 con 0x7f674c142f10 2022-07-12T08:43:19.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6737fff700 1 -- 172.21.15.61:0/604062961 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6730003260 con 0x7f674c142f10 2022-07-12T08:43:19.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675170d700 1 -- 172.21.15.61:0/604062961 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142ab0 msgr2=0x7f674c0afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:19.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675170d700 1 --2- 172.21.15.61:0/604062961 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142ab0 0x7f674c0afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675170d700 1 -- 172.21.15.61:0/604062961 >> v1:172.21.15.111:6789/0 conn(0x7f674c142f10 legacy=0x7f674c1433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675170d700 1 -- 172.21.15.61:0/604062961 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c0674b0 con 0x7f674c05d060 2022-07-12T08:43:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6750f0c700 1 --2- 172.21.15.61:0/604062961 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142ab0 0x7f674c0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675170d700 1 --2- 172.21.15.61:0/604062961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c0b0130 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f674c05d5d0 tx=0x7f6740006df0).ready entity=mon.0 client_cookie=87e0d6fbd5032e1e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6737fff700 1 -- 172.21.15.61:0/604062961 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f674000d570 con 0x7f674c05d060 2022-07-12T08:43:19.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6737fff700 1 -- 172.21.15.61:0/604062961 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6740000e90 con 0x7f674c05d060 2022-07-12T08:43:19.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f6737fff700 1 -- 172.21.15.61:0/604062961 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f674000d8c0 con 0x7f674c05d060 2022-07-12T08:43:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 -- 172.21.15.61:0/604062961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 msgr2=0x7f674c0b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 --2- 172.21.15.61:0/604062961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c0b0130 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f674c05d5d0 tx=0x7f6740006df0).stop 2022-07-12T08:43:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 -- 172.21.15.61:0/604062961 shutdown_connections 2022-07-12T08:43:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 --2- 172.21.15.61:0/604062961 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142ab0 0x7f674c0afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 --2- 172.21.15.61:0/604062961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c0b0130 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 -- 172.21.15.61:0/604062961 >> 172.21.15.61:0/604062961 conn(0x7f674c0b6900 msgr2=0x7f674c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 -- 172.21.15.61:0/604062961 shutdown_connections 2022-07-12T08:43:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 -- 172.21.15.61:0/604062961 wait complete. 2022-07-12T08:43:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 Processor -- start 2022-07-12T08:43:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.324+0000 7f675270f700 1 -- start start 2022-07-12T08:43:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c09ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142f10 0x7f674c09e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f674c09d790 0x7f674c152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c09cf40 con 0x7f674c05d060 2022-07-12T08:43:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c09e780 con 0x7f674c09d790 2022-07-12T08:43:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c09d160 con 0x7f674c142f10 2022-07-12T08:43:19.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c09ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c09ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55068/0 (socket says 172.21.15.61:55068) 2022-07-12T08:43:19.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 -- 172.21.15.61:0/2063371539 learned_addr learned my addr 172.21.15.61:0/2063371539 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:19.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f6751f0e700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f674c09d790 0x7f674c152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f6750f0c700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142f10 0x7f674c09e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 -- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142f10 msgr2=0x7f674c09e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142f10 0x7f674c09e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 -- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f674c09d790 msgr2=0x7f674c152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f674c09d790 0x7f674c152980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 -- 172.21.15.61:0/2063371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674800b180 con 0x7f674c05d060 2022-07-12T08:43:19.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f6751f0e700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f674c09d790 0x7f674c152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:19.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f6750f0c700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142f10 0x7f674c09e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:19.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675170d700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c09ee60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f674c09d080 tx=0x7f6740014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6740002f70 con 0x7f674c05d060 2022-07-12T08:43:19.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674001e400 con 0x7f674c05d060 2022-07-12T08:43:19.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674000b180 con 0x7f674c05d060 2022-07-12T08:43:19.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f674001e760 con 0x7f674c05d060 2022-07-12T08:43:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c09ca60 con 0x7f674c05d060 2022-07-12T08:43:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c057460 con 0x7f674c05d060 2022-07-12T08:43:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.328+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6740025030 con 0x7f674c05d060 2022-07-12T08:43:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.332+0000 7f673effd700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6728053a40 0x7f6728055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.332+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f674006cd40 con 0x7f674c05d060 2022-07-12T08:43:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.332+0000 7f6750f0c700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6728053a40 0x7f6728055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.332+0000 7f6750f0c700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6728053a40 0x7f6728055f00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f674800b000 tx=0x7f6748008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.340+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6740043ca0 con 0x7f674c05d060 2022-07-12T08:43:19.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.464+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f674c09bec0 con 0x7f674c05d060 2022-07-12T08:43:19.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.464+0000 7f673effd700 1 -- 172.21.15.61:0/2063371539 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6740023020 con 0x7f674c05d060 2022-07-12T08:43:19.474 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:43:19.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6728053a40 msgr2=0x7f6728055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6728053a40 0x7f6728055f00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f674800b000 tx=0x7f6748008040).stop 2022-07-12T08:43:19.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 msgr2=0x7f674c09ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c09ee60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f674c09d080 tx=0x7f6740014000).stop 2022-07-12T08:43:19.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 shutdown_connections 2022-07-12T08:43:19.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6728053a40 0x7f6728055f00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f674c09d790 0x7f674c152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f674c142f10 0x7f674c09e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 --2- 172.21.15.61:0/2063371539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f674c05d060 0x7f674c09ee60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 >> 172.21.15.61:0/2063371539 conn(0x7f674c0b6900 msgr2=0x7f674c060000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 shutdown_connections 2022-07-12T08:43:19.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.468+0000 7f675270f700 1 -- 172.21.15.61:0/2063371539 wait complete. 2022-07-12T08:43:19.508 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:43:19.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.624+0000 7f75fad77700 1 Processor -- start 2022-07-12T08:43:19.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.624+0000 7f75fad77700 1 -- start start 2022-07-12T08:43:19.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.624+0000 7f75fad77700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f4142eb0 0x7f75f405c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75f40b6130 0x7f75f40b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f4143500 con 0x7f75f405d130 2022-07-12T08:43:19.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40b65c0 con 0x7f75f40b6130 2022-07-12T08:43:19.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4143290 con 0x7f75f4142eb0 2022-07-12T08:43:19.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f9574700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f75f405d130 0x7f75f40b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43296/0 (socket says 172.21.15.61:43296) 2022-07-12T08:43:19.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f9574700 1 -- 172.21.15.61:0/3732889622 learned_addr learned my addr 172.21.15.61:0/3732889622 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:19.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fa576700 1 --2- 172.21.15.61:0/3732889622 >> v2:172.21.15.111:3300/0 conn(0x7f75f40b6130 0x7f75f40b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f9d75700 1 --2- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f4142eb0 0x7f75f405c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f8d73700 1 -- 172.21.15.61:0/3732889622 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716365346 0 0) 0x7f75f4143500 con 0x7f75f405d130 2022-07-12T08:43:19.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f8d73700 1 -- 172.21.15.61:0/3732889622 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc003260 con 0x7f75f405d130 2022-07-12T08:43:19.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fa576700 1 -- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f4142eb0 msgr2=0x7f75f405c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:19.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fa576700 1 --2- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f4142eb0 0x7f75f405c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fa576700 1 -- 172.21.15.61:0/3732889622 >> v1:172.21.15.61:6789/0 conn(0x7f75f405d130 legacy=0x7f75f40b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fa576700 1 -- 172.21.15.61:0/3732889622 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4143760 con 0x7f75f40b6130 2022-07-12T08:43:19.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fa576700 1 --2- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f40b6130 0x7f75f40b1ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f75ec0032f0 tx=0x7f75ec0091d0).ready entity=mon.1 client_cookie=8aebfd18eabd4e45 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f8d73700 1 -- 172.21.15.61:0/3732889622 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ec00b600 con 0x7f75f40b6130 2022-07-12T08:43:19.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f8d73700 1 -- 172.21.15.61:0/3732889622 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec00b760 con 0x7f75f40b6130 2022-07-12T08:43:19.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75f8d73700 1 -- 172.21.15.61:0/3732889622 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ec00b8d0 con 0x7f75f40b6130 2022-07-12T08:43:19.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f40b6130 msgr2=0x7f75f40b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 --2- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f40b6130 0x7f75f40b1ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f75ec0032f0 tx=0x7f75ec0091d0).stop 2022-07-12T08:43:19.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- 172.21.15.61:0/3732889622 shutdown_connections 2022-07-12T08:43:19.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 --2- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f40b6130 0x7f75f40b1ad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 --2- 172.21.15.61:0/3732889622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f4142eb0 0x7f75f405c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- 172.21.15.61:0/3732889622 >> 172.21.15.61:0/3732889622 conn(0x7f75f4066d70 msgr2=0x7f75f4067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- 172.21.15.61:0/3732889622 shutdown_connections 2022-07-12T08:43:19.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.628+0000 7f75fad77700 1 -- 172.21.15.61:0/3732889622 wait complete. 2022-07-12T08:43:19.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 Processor -- start 2022-07-12T08:43:19.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 -- start start 2022-07-12T08:43:19.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f405d130 0x7f75f41442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 0x7f75f4155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f4142eb0 0x7f75f40ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40affb0 con 0x7f75f40b6130 2022-07-12T08:43:19.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f4133e70 con 0x7f75f4142eb0 2022-07-12T08:43:19.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fad77700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40ad0b0 con 0x7f75f405d130 2022-07-12T08:43:19.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 0x7f75f4155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 0x7f75f4155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55084/0 (socket says 172.21.15.61:55084) 2022-07-12T08:43:19.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 -- 172.21.15.61:0/3183734556 learned_addr learned my addr 172.21.15.61:0/3183734556 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:19.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9d75700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f405d130 0x7f75f41442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75fa576700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f4142eb0 0x7f75f40ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 -- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f405d130 msgr2=0x7f75f41442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f405d130 0x7f75f41442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 -- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f4142eb0 msgr2=0x7f75f40ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f4142eb0 0x7f75f40ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 -- 172.21.15.61:0/3183734556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec009470 con 0x7f75f40b6130 2022-07-12T08:43:19.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.632+0000 7f75f9574700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 0x7f75f4155070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f75e4001110 tx=0x7f75e4007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e40070d0 con 0x7f75f40b6130 2022-07-12T08:43:19.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4007230 con 0x7f75f40b6130 2022-07-12T08:43:19.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f40ad350 con 0x7f75f40b6130 2022-07-12T08:43:19.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e4005610 con 0x7f75f40b6130 2022-07-12T08:43:19.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f41461d0 con 0x7f75f40b6130 2022-07-12T08:43:19.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f75e401a060 con 0x7f75f40b6130 2022-07-12T08:43:19.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f4133c90 con 0x7f75f40b6130 2022-07-12T08:43:19.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75e2ffd700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f75d4053a40 0x7f75d4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f75e406b340 con 0x7f75f40b6130 2022-07-12T08:43:19.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75f9d75700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f75d4053a40 0x7f75d4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.636+0000 7f75f9d75700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f75d4053a40 0x7f75d4055f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f75f000b000 tx=0x7f75f0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.640+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75e4042190 con 0x7f75f40b6130 2022-07-12T08:43:19.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.764+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f75f40b3810 con 0x7f75f40b6130 2022-07-12T08:43:19.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.764+0000 7f75e2ffd700 1 -- 172.21.15.61:0/3183734556 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7f75e4018020 con 0x7f75f40b6130 2022-07-12T08:43:19.774 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:19.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f75d4053a40 msgr2=0x7f75d4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f75d4053a40 0x7f75d4055f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f75f000b000 tx=0x7f75f0006df0).stop 2022-07-12T08:43:19.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 msgr2=0x7f75f4155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 0x7f75f4155070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f75e4001110 tx=0x7f75e4007b80).stop 2022-07-12T08:43:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 shutdown_connections 2022-07-12T08:43:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f75d4053a40 0x7f75d4055f00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75f4142eb0 0x7f75f40ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75f40b6130 0x7f75f4155070 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 --2- 172.21.15.61:0/3183734556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75f405d130 0x7f75f41442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 >> 172.21.15.61:0/3183734556 conn(0x7f75f4066d70 msgr2=0x7f75f413ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 shutdown_connections 2022-07-12T08:43:19.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.768+0000 7f75fad77700 1 -- 172.21.15.61:0/3183734556 wait complete. 2022-07-12T08:43:19.809 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:43:19.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.948+0000 7f2ff2bfe700 1 Processor -- start 2022-07-12T08:43:19.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.948+0000 7f2ff2bfe700 1 -- start start 2022-07-12T08:43:19.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2fec0b3730 0x7f2fec1436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec0b3560 con 0x7f2fec0b5480 2022-07-12T08:43:19.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fec0b5b80 con 0x7f2fec0b05f0 2022-07-12T08:43:19.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fec0b5e20 con 0x7f2fec0b3730 2022-07-12T08:43:19.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff119b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2fec0b3730 0x7f2fec1436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff099a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2fec0b5480 0x7f2fec0afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43308/0 (socket says 172.21.15.61:43308) 2022-07-12T08:43:19.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff099a700 1 -- 172.21.15.61:0/2491332322 learned_addr learned my addr 172.21.15.61:0/2491332322 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:19.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff119b700 1 -- 172.21.15.61:0/2491332322 >> v1:172.21.15.111:6789/0 conn(0x7f2fec0b05f0 legacy=0x7f2fec142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2feb7fe700 1 -- 172.21.15.61:0/2491332322 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378317357 0 0) 0x7f2fec0b3560 con 0x7f2fec0b5480 2022-07-12T08:43:19.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff119b700 1 -- 172.21.15.61:0/2491332322 >> v1:172.21.15.61:6789/0 conn(0x7f2fec0b5480 legacy=0x7f2fec0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff119b700 1 -- 172.21.15.61:0/2491332322 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec0b5860 con 0x7f2fec0b3730 2022-07-12T08:43:19.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff119b700 1 --2- 172.21.15.61:0/2491332322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b3730 0x7f2fec1436a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2fe0001110 tx=0x7f2fe40091d0).ready entity=mon.0 client_cookie=ef3defedd9b1ec56 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2feb7fe700 1 -- 172.21.15.61:0/2491332322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe400b600 con 0x7f2fec0b3730 2022-07-12T08:43:19.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2feb7fe700 1 -- 172.21.15.61:0/2491332322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe400b760 con 0x7f2fec0b3730 2022-07-12T08:43:19.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2feb7fe700 1 -- 172.21.15.61:0/2491332322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe400b8d0 con 0x7f2fec0b3730 2022-07-12T08:43:19.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/2491332322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b3730 msgr2=0x7f2fec1436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/2491332322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b3730 0x7f2fec1436a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2fe0001110 tx=0x7f2fe40091d0).stop 2022-07-12T08:43:19.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/2491332322 shutdown_connections 2022-07-12T08:43:19.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/2491332322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b3730 0x7f2fec1436a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/2491332322 >> 172.21.15.61:0/2491332322 conn(0x7f2fec0b96b0 msgr2=0x7f2fec0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:19.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/2491332322 shutdown_connections 2022-07-12T08:43:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.952+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/2491332322 wait complete. 2022-07-12T08:43:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 Processor -- start 2022-07-12T08:43:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 -- start start 2022-07-12T08:43:19.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fec0b05f0 0x7f2fec155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fec0b3730 0x7f2fec1458f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 0x7f2fec13deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:19.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0ad140 con 0x7f2fec0b5480 2022-07-12T08:43:19.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0ae980 con 0x7f2fec0b3730 2022-07-12T08:43:19.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff2bfe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec0acc00 con 0x7f2fec0b05f0 2022-07-12T08:43:19.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 0x7f2fec13deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 0x7f2fec13deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55100/0 (socket says 172.21.15.61:55100) 2022-07-12T08:43:19.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 -- 172.21.15.61:0/1701765692 learned_addr learned my addr 172.21.15.61:0/1701765692 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:19.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff099a700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fec0b05f0 0x7f2fec155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2febfff700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fec0b3730 0x7f2fec1458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:19.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 -- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fec0b05f0 msgr2=0x7f2fec155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fec0b05f0 0x7f2fec155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 -- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fec0b3730 msgr2=0x7f2fec1458f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:19.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fec0b3730 0x7f2fec1458f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:19.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 -- 172.21.15.61:0/1701765692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe4009470 con 0x7f2fec0b5480 2022-07-12T08:43:19.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2ff119b700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 0x7f2fec13deb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2fe0001110 tx=0x7f2fe4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:19.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.956+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe400b600 con 0x7f2fec0b5480 2022-07-12T08:43:19.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe40032d0 con 0x7f2fec0b5480 2022-07-12T08:43:19.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe401f5e0 con 0x7f2fec0b5480 2022-07-12T08:43:19.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec0acea0 con 0x7f2fec0b5480 2022-07-12T08:43:20.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec0b0140 con 0x7f2fec0b5480 2022-07-12T08:43:20.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fec057460 con 0x7f2fec0b5480 2022-07-12T08:43:20.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2fe4026030 con 0x7f2fec0b5480 2022-07-12T08:43:20.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2fe9ffb700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fcc053920 0x7f2fcc055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2fe406ca10 con 0x7f2fec0b5480 2022-07-12T08:43:20.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.960+0000 7f2ff099a700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fcc053920 0x7f2fcc055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.964+0000 7f2ff099a700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fcc053920 0x7f2fcc055de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f2fdc00d000 tx=0x7f2fdc007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:19.964+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2fe40438d0 con 0x7f2fec0b5480 2022-07-12T08:43:20.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.092+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2fec0ba5d0 con 0x7f2fec0b5480 2022-07-12T08:43:20.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2fe9ffb700 1 -- 172.21.15.61:0/1701765692 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2fe4024020 con 0x7f2fec0b5480 2022-07-12T08:43:20.113 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:20.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fcc053920 msgr2=0x7f2fcc055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fcc053920 0x7f2fcc055de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f2fdc00d000 tx=0x7f2fdc007c00).stop 2022-07-12T08:43:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 msgr2=0x7f2fec13deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 0x7f2fec13deb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2fe0001110 tx=0x7f2fe4016000).stop 2022-07-12T08:43:20.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 shutdown_connections 2022-07-12T08:43:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fec0b5480 0x7f2fec13deb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fcc053920 0x7f2fcc055de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fec0b3730 0x7f2fec1458f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 --2- 172.21.15.61:0/1701765692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fec0b05f0 0x7f2fec155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 >> 172.21.15.61:0/1701765692 conn(0x7f2fec0b96b0 msgr2=0x7f2fec0b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:20.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 shutdown_connections 2022-07-12T08:43:20.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.096+0000 7f2ff2bfe700 1 -- 172.21.15.61:0/1701765692 wait complete. 2022-07-12T08:43:20.123 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:43:20.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72fcb0e700 1 Processor -- start 2022-07-12T08:43:20.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72fcb0e700 1 -- start start 2022-07-12T08:43:20.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72fcb0e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72f80ab730 0x7f72f8144860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72fcb0e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80b0500 con 0x7f72f814aad0 2022-07-12T08:43:20.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72fcb0e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f805d0b0 con 0x7f72f80b0120 2022-07-12T08:43:20.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72fcb0e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80b1040 con 0x7f72f80ab730 2022-07-12T08:43:20.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f6d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f72f814aad0 0x7f72f814af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43326/0 (socket says 172.21.15.61:43326) 2022-07-12T08:43:20.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f6d9d700 1 -- 172.21.15.61:0/47298315 learned_addr learned my addr 172.21.15.61:0/47298315 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:20.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f5d9b700 1 --2- 172.21.15.61:0/47298315 >> v2:172.21.15.111:3300/0 conn(0x7f72f80ab730 0x7f72f8144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f559a700 1 -- 172.21.15.61:0/47298315 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517574377 0 0) 0x7f72f80b0500 con 0x7f72f814aad0 2022-07-12T08:43:20.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f559a700 1 -- 172.21.15.61:0/47298315 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72dc003260 con 0x7f72f814aad0 2022-07-12T08:43:20.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f5d9b700 1 -- 172.21.15.61:0/47298315 >> v1:172.21.15.111:6789/0 conn(0x7f72f80b0120 legacy=0x7f72f8144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f5d9b700 1 -- 172.21.15.61:0/47298315 >> v1:172.21.15.61:6789/0 conn(0x7f72f814aad0 legacy=0x7f72f814af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f5d9b700 1 -- 172.21.15.61:0/47298315 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f805d350 con 0x7f72f80ab730 2022-07-12T08:43:20.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f5d9b700 1 --2- 172.21.15.61:0/47298315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f8144860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f72ec002f70 tx=0x7f72ec006df0).ready entity=mon.1 client_cookie=6d8d91729c5b5b37 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f559a700 1 -- 172.21.15.61:0/47298315 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ec00d610 con 0x7f72f80ab730 2022-07-12T08:43:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f559a700 1 -- 172.21.15.61:0/47298315 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec000e90 con 0x7f72f80ab730 2022-07-12T08:43:20.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.240+0000 7f72f559a700 1 -- 172.21.15.61:0/47298315 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ec00d960 con 0x7f72f80ab730 2022-07-12T08:43:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 -- 172.21.15.61:0/47298315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 msgr2=0x7f72f8144860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/47298315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f8144860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f72ec002f70 tx=0x7f72ec006df0).stop 2022-07-12T08:43:20.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 -- 172.21.15.61:0/47298315 shutdown_connections 2022-07-12T08:43:20.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/47298315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f8144860 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 -- 172.21.15.61:0/47298315 >> 172.21.15.61:0/47298315 conn(0x7f72f80b5290 msgr2=0x7f72f80b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:20.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 -- 172.21.15.61:0/47298315 shutdown_connections 2022-07-12T08:43:20.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 -- 172.21.15.61:0/47298315 wait complete. 2022-07-12T08:43:20.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 Processor -- start 2022-07-12T08:43:20.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.244+0000 7f72fcb0e700 1 -- start start 2022-07-12T08:43:20.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f80a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 0x7f72f80a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72f814aad0 0x7f72f8159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f809f0f0 con 0x7f72f80b0120 2022-07-12T08:43:20.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80645e0 con 0x7f72f80ab730 2022-07-12T08:43:20.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8159d00 con 0x7f72f814aad0 2022-07-12T08:43:20.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 0x7f72f80a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 0x7f72f80a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55112/0 (socket says 172.21.15.61:55112) 2022-07-12T08:43:20.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 -- 172.21.15.61:0/3842389106 learned_addr learned my addr 172.21.15.61:0/3842389106 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:20.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f6d9d700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72f814aad0 0x7f72f8159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f659c700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f80a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 -- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72f814aad0 msgr2=0x7f72f8159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72f814aad0 0x7f72f8159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 -- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 msgr2=0x7f72f80a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f80a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 -- 172.21.15.61:0/3842389106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec00b180 con 0x7f72f80b0120 2022-07-12T08:43:20.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72f5d9b700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 0x7f72f80a01c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f72ec002f70 tx=0x7f72ec014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ec016070 con 0x7f72f80b0120 2022-07-12T08:43:20.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec00de90 con 0x7f72f80b0120 2022-07-12T08:43:20.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f809f4f0 con 0x7f72f80b0120 2022-07-12T08:43:20.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ec013da0 con 0x7f72f80b0120 2022-07-12T08:43:20.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.248+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f809fa20 con 0x7f72f80b0120 2022-07-12T08:43:20.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.252+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f72ec025030 con 0x7f72f80b0120 2022-07-12T08:43:20.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.252+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8057460 con 0x7f72f80b0120 2022-07-12T08:43:20.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.252+0000 7f72e77fe700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72d4053a60 0x7f72d4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.252+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f72ec06ce10 con 0x7f72f80b0120 2022-07-12T08:43:20.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.252+0000 7f72f659c700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72d4053a60 0x7f72d4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.252+0000 7f72f659c700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72d4053a60 0x7f72d4055f20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f72e800d000 tx=0x7f72e8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.260+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72ec070030 con 0x7f72f80b0120 2022-07-12T08:43:20.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.380+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72f80447a0 con 0x7f72f80b0120 2022-07-12T08:43:20.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72e77fe700 1 -- 172.21.15.61:0/3842389106 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7f72ec023070 con 0x7f72f80b0120 2022-07-12T08:43:20.395 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:20.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72d4053a60 msgr2=0x7f72d4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72d4053a60 0x7f72d4055f20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f72e800d000 tx=0x7f72e8007b90).stop 2022-07-12T08:43:20.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 msgr2=0x7f72f80a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 0x7f72f80a01c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f72ec002f70 tx=0x7f72ec014000).stop 2022-07-12T08:43:20.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 shutdown_connections 2022-07-12T08:43:20.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72f814aad0 0x7f72f8159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72f80b0120 0x7f72f80a01c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72d4053a60 0x7f72d4055f20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 --2- 172.21.15.61:0/3842389106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72f80ab730 0x7f72f80a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 >> 172.21.15.61:0/3842389106 conn(0x7f72f80b5290 msgr2=0x7f72f80b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:20.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 shutdown_connections 2022-07-12T08:43:20.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.384+0000 7f72fcb0e700 1 -- 172.21.15.61:0/3842389106 wait complete. 2022-07-12T08:43:20.409 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:43:20.410 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:20.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.512+0000 7fb98b59e700 1 Processor -- start 2022-07-12T08:43:20.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.512+0000 7fb98b59e700 1 -- start start 2022-07-12T08:43:20.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.512+0000 7fb98b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb98c0ab670 0x7fb98c05f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.512+0000 7fb98b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98c0abc70 con 0x7fb98c0afa60 2022-07-12T08:43:20.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.512+0000 7fb98b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb98c05c720 con 0x7fb98c0ab290 2022-07-12T08:43:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.512+0000 7fb98b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98c0aba90 con 0x7fb98c0ab670 2022-07-12T08:43:20.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb98c0ab670 0x7fb98c05f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb98c0ab670 0x7fb98c05f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55118/0 (socket says 172.21.15.61:55118) 2022-07-12T08:43:20.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 -- 172.21.15.61:0/2607813409 learned_addr learned my addr 172.21.15.61:0/2607813409 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 -- 172.21.15.61:0/2607813409 >> v1:172.21.15.111:6789/0 conn(0x7fb98c0ab290 legacy=0x7fb98c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:20.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 -- 172.21.15.61:0/2607813409 >> v1:172.21.15.61:6789/0 conn(0x7fb98c0afa60 legacy=0x7fb98c147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 -- 172.21.15.61:0/2607813409 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c05c9c0 con 0x7fb98c0ab670 2022-07-12T08:43:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 --2- 172.21.15.61:0/2607813409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab670 0x7fb98c05f140 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fb98c0af510 tx=0x7fb97c009290).ready entity=mon.0 client_cookie=2cd340aa8f28a210 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98959a700 1 -- 172.21.15.61:0/2607813409 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb97c00b860 con 0x7fb98c0ab670 2022-07-12T08:43:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98959a700 1 -- 172.21.15.61:0/2607813409 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c00b9c0 con 0x7fb98c0ab670 2022-07-12T08:43:20.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98959a700 1 -- 172.21.15.61:0/2607813409 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb97c00bb30 con 0x7fb98c0ab670 2022-07-12T08:43:20.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- 172.21.15.61:0/2607813409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab670 msgr2=0x7fb98c05f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 --2- 172.21.15.61:0/2607813409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab670 0x7fb98c05f140 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fb98c0af510 tx=0x7fb97c009290).stop 2022-07-12T08:43:20.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- 172.21.15.61:0/2607813409 shutdown_connections 2022-07-12T08:43:20.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 --2- 172.21.15.61:0/2607813409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab670 0x7fb98c05f140 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- 172.21.15.61:0/2607813409 >> 172.21.15.61:0/2607813409 conn(0x7fb98c0b1950 msgr2=0x7fb98c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:20.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- 172.21.15.61:0/2607813409 shutdown_connections 2022-07-12T08:43:20.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- 172.21.15.61:0/2607813409 wait complete. 2022-07-12T08:43:20.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 Processor -- start 2022-07-12T08:43:20.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- start start 2022-07-12T08:43:20.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 0x7fb98c0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb98c0ab670 0x7fb98c099c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb98c0afa60 0x7fb98c065f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c0ae6f0 con 0x7fb98c0ab290 2022-07-12T08:43:20.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c0ae1b0 con 0x7fb98c0ab670 2022-07-12T08:43:20.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c0ae450 con 0x7fb98c0afa60 2022-07-12T08:43:20.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 0x7fb98c0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 0x7fb98c0aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55124/0 (socket says 172.21.15.61:55124) 2022-07-12T08:43:20.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98a59c700 1 -- 172.21.15.61:0/1761611815 learned_addr learned my addr 172.21.15.61:0/1761611815 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:20.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb98ad9d700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb98c0afa60 0x7fb98c065f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.516+0000 7fb989d9b700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb98c0ab670 0x7fb98c099c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98a59c700 1 -- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb98c0afa60 msgr2=0x7fb98c065f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98a59c700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb98c0afa60 0x7fb98c065f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98a59c700 1 -- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb98c0ab670 msgr2=0x7fb98c099c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98a59c700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb98c0ab670 0x7fb98c099c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98a59c700 1 -- 172.21.15.61:0/1761611815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c009530 con 0x7fb98c0ab290 2022-07-12T08:43:20.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb989d9b700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb98c0ab670 0x7fb98c099c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:20.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98a59c700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 0x7fb98c0aee40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fb980001fa0 tx=0x7fb980007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb980007030 con 0x7fb98c0ab290 2022-07-12T08:43:20.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980007190 con 0x7fb98c0ab290 2022-07-12T08:43:20.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb980006510 con 0x7fb98c0ab290 2022-07-12T08:43:20.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb98c066420 con 0x7fb98c0ab290 2022-07-12T08:43:20.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98c063c90 con 0x7fb98c0ab290 2022-07-12T08:43:20.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98c0640c0 con 0x7fb98c0ab290 2022-07-12T08:43:20.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb980010030 con 0x7fb98c0ab290 2022-07-12T08:43:20.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb97b7fe700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb968053ad0 0x7fb968055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:20.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb98006bc40 con 0x7fb98c0ab290 2022-07-12T08:43:20.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.520+0000 7fb989d9b700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb968053ad0 0x7fb968055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:20.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.524+0000 7fb989d9b700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb968053ad0 0x7fb968055f90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fb97c019000 tx=0x7fb97c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:20.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.524+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb980042a90 con 0x7fb98c0ab290 2022-07-12T08:43:20.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.688+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb98c057460 con 0x7fb98c0ab290 2022-07-12T08:43:20.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.688+0000 7fb97b7fe700 1 -- 172.21.15.61:0/1761611815 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7fb9800460b0 con 0x7fb98c0ab290 2022-07-12T08:43:20.700 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:20.701 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:20.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb968053ad0 msgr2=0x7fb968055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb968053ad0 0x7fb968055f90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fb97c019000 tx=0x7fb97c016000).stop 2022-07-12T08:43:20.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 msgr2=0x7fb98c0aee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:20.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 0x7fb98c0aee40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fb980001fa0 tx=0x7fb980007c30).stop 2022-07-12T08:43:20.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 shutdown_connections 2022-07-12T08:43:20.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb98c0afa60 0x7fb98c065f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb968053ad0 0x7fb968055f90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb98c0ab670 0x7fb98c099c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 --2- 172.21.15.61:0/1761611815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb98c0ab290 0x7fb98c0aee40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:20.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 >> 172.21.15.61:0/1761611815 conn(0x7fb98c0b1950 msgr2=0x7fb98c14d870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:20.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 shutdown_connections 2022-07-12T08:43:20.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:20.692+0000 7fb98b59e700 1 -- 172.21.15.61:0/1761611815 wait complete. 2022-07-12T08:43:20.710 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:20.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:43:15.615120+0000', 'modified': '2022-07-12T08:43:16.618859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'a', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/4199688387', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 4199688387}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 4199688387}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:43:20.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:43:20.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:43:21.086 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":5.4540292609999996,"mdsmap_epoch":17,"osdmap_epoch":42,"osdmap_epoch_barrier":42,"uptime":10.699325896} 2022-07-12T08:43:21.086 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5241, "mdsmap_epoch": 17, "osdmap_epoch": 42, "osdmap_epoch_barrier": 42, "rank_uptime": 5.454029261, "state": "up:active", "uptime": 10.699325896, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:43:21.087 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:43:21.087 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:43:21.089 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:43:21.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.091 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:21.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:43:21.140 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:43:21.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:43:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:43:21.209 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:43:21.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.211 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:43:21.221 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:43:21.222 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:21.279 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:43:21.308 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:43:21.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:43:21.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:43:21.354+0000 7f3dddeeb080 -1 init, newargv = 0x55e517fa9700 newargc=9 2022-07-12T08:43:21.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21169]: starting ceph client 2022-07-12T08:43:21.365 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:43:21.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21169]: starting fuse 2022-07-12T08:43:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:43:21.370 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.370 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:43:21.377 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:43:21.387 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:43:21.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.389 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:43:21.392 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:43:21.394 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:43:21.395 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:43:21.395 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:43:21.396 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.397 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:43:21.407 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:43:21.407 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:43:21.408 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.409 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:43:21.409 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:43:21.410 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:43:21.411 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:43:21.412 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.412 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:43:21.419 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:43:21.420 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:43:21.422 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:43:21.422 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:43:21.435 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.436 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:43:21.437 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:43:21.489 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21169.asok 2022-07-12T08:43:21.491 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21169.asok 2022-07-12T08:43:21.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.493 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21169.asok status 2022-07-12T08:43:21.602 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:43:21.602 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:43:21.603 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:43:21.603 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:43:21.604 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:43:21.604 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:43:21.605 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:43:21.605 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21169", 2022-07-12T08:43:21.605 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:43:21.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:21.606 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:43:21.607 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:43:21.607 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5078, 2022-07-12T08:43:21.607 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:43:21.608 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:43:21.608 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:43:21.609 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5078 2022-07-12T08:43:21.609 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:21.612 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:43:21.612 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:43:21.613 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:43:21.613 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 771087494 2022-07-12T08:43:21.614 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:43:21.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:21.615 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:43:21.615 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:43:21.615 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:43:21.616 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 771087494 2022-07-12T08:43:21.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:21.617 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5078 172.21.15.121:0/771087494", 2022-07-12T08:43:21.617 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/771087494", 2022-07-12T08:43:21.618 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:43:21.618 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 17, 2022-07-12T08:43:21.626 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 42, 2022-07-12T08:43:21.627 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:43:21.627 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:43:21.628 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:43:21.628 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:43:21.630 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.631 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:21.632 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:21.672 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:43:21.673 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:43:21.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.698 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:21.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:21.749 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:21.749 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:21.759 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:43:21.760 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:43:21.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:43:21.764 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-07-12T08:43:21.765 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-07-12T08:43:21.765 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:43:21.766 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.770 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-07-12T08:43:21.770 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-12T08:43:21.771 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.771 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-12T08:43:21.772 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-07-12T08:43:21.772 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:21.773 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-07-12T08:43:21.773 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-07-12T08:43:21.774 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-07-12T08:43:21.774 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-07-12T08:43:21.775 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-07-12T08:43:21.775 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-07-12T08:43:21.776 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-07-12T08:43:21.776 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:43:31.086 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:31.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.436+0000 7efc1c218700 1 Processor -- start 2022-07-12T08:43:31.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.436+0000 7efc1c218700 1 -- start start 2022-07-12T08:43:31.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efc140b3b40 0x7efc140ad700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc14143720 0x7efc14143b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc14142fe0 con 0x7efc140b3f20 2022-07-12T08:43:31.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc140adbf0 con 0x7efc14143720 2022-07-12T08:43:31.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc14143280 con 0x7efc140b3b40 2022-07-12T08:43:31.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1a7b5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc14143720 0x7efc14143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc197b3700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efc140b3f20 0x7efc1405bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43354/0 (socket says 172.21.15.61:43354) 2022-07-12T08:43:31.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc197b3700 1 -- 172.21.15.61:0/2069625723 learned_addr learned my addr 172.21.15.61:0/2069625723 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:31.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc19fb4700 1 --2- 172.21.15.61:0/2069625723 >> v2:172.21.15.111:3300/0 conn(0x7efc140b3b40 0x7efc140ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc18fb2700 1 -- 172.21.15.61:0/2069625723 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187097436 0 0) 0x7efc14142fe0 con 0x7efc140b3f20 2022-07-12T08:43:31.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc18fb2700 1 -- 172.21.15.61:0/2069625723 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc00003260 con 0x7efc140b3f20 2022-07-12T08:43:31.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1a7b5700 1 -- 172.21.15.61:0/2069625723 >> v2:172.21.15.111:3300/0 conn(0x7efc140b3b40 msgr2=0x7efc140ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:31.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1a7b5700 1 --2- 172.21.15.61:0/2069625723 >> v2:172.21.15.111:3300/0 conn(0x7efc140b3b40 0x7efc140ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1a7b5700 1 -- 172.21.15.61:0/2069625723 >> v1:172.21.15.61:6789/0 conn(0x7efc140b3f20 legacy=0x7efc1405bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1a7b5700 1 -- 172.21.15.61:0/2069625723 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc14142ce0 con 0x7efc14143720 2022-07-12T08:43:31.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc19fb4700 1 --2- 172.21.15.61:0/2069625723 >> v2:172.21.15.111:3300/0 conn(0x7efc140b3b40 0x7efc140ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:31.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1a7b5700 1 --2- 172.21.15.61:0/2069625723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc14143720 0x7efc14143b40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7efc0c002f70 tx=0x7efc0c006df0).ready entity=mon.0 client_cookie=5411632e32c8d3c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:31.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc18fb2700 1 -- 172.21.15.61:0/2069625723 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc0c00d570 con 0x7efc14143720 2022-07-12T08:43:31.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc18fb2700 1 -- 172.21.15.61:0/2069625723 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c000e90 con 0x7efc14143720 2022-07-12T08:43:31.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc18fb2700 1 -- 172.21.15.61:0/2069625723 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc0c00d8c0 con 0x7efc14143720 2022-07-12T08:43:31.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- 172.21.15.61:0/2069625723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc14143720 msgr2=0x7efc14143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 --2- 172.21.15.61:0/2069625723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc14143720 0x7efc14143b40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7efc0c002f70 tx=0x7efc0c006df0).stop 2022-07-12T08:43:31.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- 172.21.15.61:0/2069625723 shutdown_connections 2022-07-12T08:43:31.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 --2- 172.21.15.61:0/2069625723 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc14143720 0x7efc14143b40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 --2- 172.21.15.61:0/2069625723 >> v2:172.21.15.111:3300/0 conn(0x7efc140b3b40 0x7efc140ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- 172.21.15.61:0/2069625723 >> 172.21.15.61:0/2069625723 conn(0x7efc14064b90 msgr2=0x7efc14064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:31.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- 172.21.15.61:0/2069625723 shutdown_connections 2022-07-12T08:43:31.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 -- 172.21.15.61:0/2069625723 wait complete. 2022-07-12T08:43:31.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.440+0000 7efc1c218700 1 Processor -- start 2022-07-12T08:43:31.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 -- start start 2022-07-12T08:43:31.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc140b3b40 0x7efc14060210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 0x7efc140b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc14143720 0x7efc140b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc140aedc0 con 0x7efc140b3f20 2022-07-12T08:43:31.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc140b6930 con 0x7efc140b3b40 2022-07-12T08:43:31.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc14144c50 con 0x7efc14143720 2022-07-12T08:43:31.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 0x7efc140b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 0x7efc140b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55142/0 (socket says 172.21.15.61:55142) 2022-07-12T08:43:31.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 -- 172.21.15.61:0/1587705421 learned_addr learned my addr 172.21.15.61:0/1587705421 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:31.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc19fb4700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc140b3b40 0x7efc14060210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1a7b5700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc14143720 0x7efc140b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 -- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc14143720 msgr2=0x7efc140b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc14143720 0x7efc140b5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 -- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc140b3b40 msgr2=0x7efc14060210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc140b3b40 0x7efc14060210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 -- 172.21.15.61:0/1587705421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c00b180 con 0x7efc140b3f20 2022-07-12T08:43:31.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc19fb4700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc140b3b40 0x7efc14060210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:31.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc197b3700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 0x7efc140b5510 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7efc08001110 tx=0x7efc08007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:31.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc080072e0 con 0x7efc140b3f20 2022-07-12T08:43:31.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08007440 con 0x7efc140b3f20 2022-07-12T08:43:31.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc140b5120 con 0x7efc140b3f20 2022-07-12T08:43:31.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc08005800 con 0x7efc140b3f20 2022-07-12T08:43:31.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.444+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc140b5ef0 con 0x7efc140b3f20 2022-07-12T08:43:31.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.448+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc14057460 con 0x7efc140b3f20 2022-07-12T08:43:31.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.448+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efc0801a060 con 0x7efc140b3f20 2022-07-12T08:43:31.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.448+0000 7efc12ffd700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efbf4053af0 0x7efbf4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.448+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efc0806b850 con 0x7efc140b3f20 2022-07-12T08:43:31.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.448+0000 7efc19fb4700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efbf4053af0 0x7efbf4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.448+0000 7efc19fb4700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efbf4053af0 0x7efbf4055fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7efc0400d000 tx=0x7efc04008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:31.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.452+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc080426a0 con 0x7efc140b3f20 2022-07-12T08:43:31.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.620+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc140008d0 con 0x7efc140b3f20 2022-07-12T08:43:31.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.620+0000 7efc12ffd700 1 -- 172.21.15.61:0/1587705421 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7efc08018020 con 0x7efc140b3f20 2022-07-12T08:43:31.629 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:31.629 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:31.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.620+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efbf4053af0 msgr2=0x7efbf4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.620+0000 7efc1c218700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efbf4053af0 0x7efbf4055fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7efc0400d000 tx=0x7efc04008040).stop 2022-07-12T08:43:31.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 msgr2=0x7efc140b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 0x7efc140b5510 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7efc08001110 tx=0x7efc08007b80).stop 2022-07-12T08:43:31.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 shutdown_connections 2022-07-12T08:43:31.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc14143720 0x7efc140b5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc140b3f20 0x7efc140b5510 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efbf4053af0 0x7efbf4055fb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 --2- 172.21.15.61:0/1587705421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc140b3b40 0x7efc14060210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 >> 172.21.15.61:0/1587705421 conn(0x7efc14064b90 msgr2=0x7efc1405c500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:31.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 shutdown_connections 2022-07-12T08:43:31.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.624+0000 7efc1c218700 1 -- 172.21.15.61:0/1587705421 wait complete. 2022-07-12T08:43:31.642 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:31.652 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-12T08:43:31.798 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-12T08:43:31.802 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-12T08:43:31.803 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.928+0000 7fe67ea7a700 1 Processor -- start 2022-07-12T08:43:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.928+0000 7fe67ea7a700 1 -- start start 2022-07-12T08:43:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe67805c050 0x7fe6780ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6780b01f0 0x7fe67814a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780ad2f0 con 0x7fe67805cb00 2022-07-12T08:43:31.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6780b0a40 con 0x7fe67805c050 2022-07-12T08:43:31.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67805c430 con 0x7fe6780b01f0 2022-07-12T08:43:31.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe67805c050 0x7fe6780ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe677fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6780b01f0 0x7fe67814a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:31.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe67805c050 0x7fe6780ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33722/0 (socket says 172.21.15.61:33722) 2022-07-12T08:43:31.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 -- 172.21.15.61:0/215169125 learned_addr learned my addr 172.21.15.61:0/215169125 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:31.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67d017700 1 --1- 172.21.15.61:0/215169125 >> v1:172.21.15.111:6789/0 conn(0x7fe67805cb00 0x7fe6781443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:56970/0 (socket says 172.21.15.61:56970) 2022-07-12T08:43:31.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 -- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6780b01f0 msgr2=0x7fe67814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:31.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 --2- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6780b01f0 0x7fe67814a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 -- 172.21.15.61:0/215169125 >> v1:172.21.15.111:6789/0 conn(0x7fe67805cb00 legacy=0x7fe6781443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 -- 172.21.15.61:0/215169125 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780b2420 con 0x7fe67805c050 2022-07-12T08:43:31.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67c816700 1 --2- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe6780ac710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe6700032f0 tx=0x7fe670009290).ready entity=mon.1 client_cookie=e6b2fdbac797862f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:31.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe6777fe700 1 -- 172.21.15.61:0/215169125 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe67000b8a0 con 0x7fe67805c050 2022-07-12T08:43:31.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe6777fe700 1 -- 172.21.15.61:0/215169125 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000ba00 con 0x7fe67805c050 2022-07-12T08:43:31.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe6777fe700 1 -- 172.21.15.61:0/215169125 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe67000bb70 con 0x7fe67805c050 2022-07-12T08:43:31.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 msgr2=0x7fe6780ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:31.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe6780ac710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fe6700032f0 tx=0x7fe670009290).stop 2022-07-12T08:43:31.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- 172.21.15.61:0/215169125 shutdown_connections 2022-07-12T08:43:31.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6780b01f0 0x7fe67814a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/215169125 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe6780ac710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:31.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- 172.21.15.61:0/215169125 >> 172.21.15.61:0/215169125 conn(0x7fe678062c00 msgr2=0x7fe678062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:31.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- 172.21.15.61:0/215169125 shutdown_connections 2022-07-12T08:43:31.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.932+0000 7fe67ea7a700 1 -- 172.21.15.61:0/215169125 wait complete. 2022-07-12T08:43:31.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 Processor -- start 2022-07-12T08:43:31.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 -- start start 2022-07-12T08:43:31.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe678066bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:31.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe67805cb00 0x7fe6780b4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 0x7fe6781564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780661d0 con 0x7fe6780b01f0 2022-07-12T08:43:32.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678066470 con 0x7fe67805c050 2022-07-12T08:43:32.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe678066710 con 0x7fe67805cb00 2022-07-12T08:43:32.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 0x7fe6781564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 0x7fe6781564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55158/0 (socket says 172.21.15.61:55158) 2022-07-12T08:43:32.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 -- 172.21.15.61:0/1609236322 learned_addr learned my addr 172.21.15.61:0/1609236322 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67c816700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe678066bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe677fff700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe67805cb00 0x7fe6780b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 -- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe67805cb00 msgr2=0x7fe6780b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe67805cb00 0x7fe6780b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 -- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 msgr2=0x7fe678066bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe678066bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 -- 172.21.15.61:0/1609236322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670009530 con 0x7fe6780b01f0 2022-07-12T08:43:32.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67d017700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 0x7fe6781564f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fe6780b30c0 tx=0x7fe668007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe66800ce30 con 0x7fe6780b01f0 2022-07-12T08:43:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6680072e0 con 0x7fe6780b01f0 2022-07-12T08:43:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67814d020 con 0x7fe6780b01f0 2022-07-12T08:43:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe668005740 con 0x7fe6780b01f0 2022-07-12T08:43:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.936+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67814df20 con 0x7fe6780b01f0 2022-07-12T08:43:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.940+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6780668d0 con 0x7fe6780b01f0 2022-07-12T08:43:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.940+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe66801a070 con 0x7fe6780b01f0 2022-07-12T08:43:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.940+0000 7fe675ffb700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6580539b0 0x7fe658055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.940+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe66806b250 con 0x7fe6780b01f0 2022-07-12T08:43:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.940+0000 7fe67c816700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6580539b0 0x7fe658055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.940+0000 7fe67c816700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6580539b0 0x7fe658055e70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fe670019000 tx=0x7fe670016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:31.948+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe668042040 con 0x7fe6780b01f0 2022-07-12T08:43:32.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe678057460 con 0x7fe6780b01f0 2022-07-12T08:43:32.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe675ffb700 1 -- 172.21.15.61:0/1609236322 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7fe668045660 con 0x7fe6780b01f0 2022-07-12T08:43:32.119 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:32.120 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:32.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6580539b0 msgr2=0x7fe658055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6580539b0 0x7fe658055e70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fe670019000 tx=0x7fe670016000).stop 2022-07-12T08:43:32.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 msgr2=0x7fe6781564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 0x7fe6781564f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fe6780b30c0 tx=0x7fe668007bc0).stop 2022-07-12T08:43:32.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 shutdown_connections 2022-07-12T08:43:32.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6780b01f0 0x7fe6781564f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe67805cb00 0x7fe6780b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6580539b0 0x7fe658055e70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 --2- 172.21.15.61:0/1609236322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe67805c050 0x7fe678066bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 >> 172.21.15.61:0/1609236322 conn(0x7fe678062c00 msgr2=0x7fe678147370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 shutdown_connections 2022-07-12T08:43:32.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.112+0000 7fe67ea7a700 1 -- 172.21.15.61:0/1609236322 wait complete. 2022-07-12T08:43:32.135 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:32.136 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-12T08:43:32.169 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:32.217 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:32.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:32.235 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:32.268 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-12T08:43:32.269 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-12T08:43:32.270 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:43:32.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:43:32.312 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:32.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 Processor -- start 2022-07-12T08:43:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 -- start start 2022-07-12T08:43:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b9eb0 0x7fa77c0b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa77c0b4840 0x7fa77c1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0ba290 con 0x7fa77c0b3180 2022-07-12T08:43:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c0b1410 con 0x7fa77c0b4840 2022-07-12T08:43:32.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa7809c9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c0b4010 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa77c0b4840 0x7fa77c1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77a59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b9eb0 0x7fa77c0b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa77c0b4840 0x7fa77c1420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33740/0 (socket says 172.21.15.61:33740) 2022-07-12T08:43:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 -- 172.21.15.61:0/4078744632 learned_addr learned my addr 172.21.15.61:0/4078744632 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 -- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b9eb0 msgr2=0x7fa77c0b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 --2- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b9eb0 0x7fa77c0b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 -- 172.21.15.61:0/4078744632 >> v1:172.21.15.111:6789/0 conn(0x7fa77c0b3180 legacy=0x7fa77c0b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 -- 172.21.15.61:0/4078744632 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0b42b0 con 0x7fa77c0b4840 2022-07-12T08:43:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77ad9d700 1 --2- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b4840 0x7fa77c1420b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fa774002f70 tx=0x7fa774006eb0).ready entity=mon.1 client_cookie=eebcde6028672ec5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77959a700 1 -- 172.21.15.61:0/4078744632 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa77400d570 con 0x7fa77c0b4840 2022-07-12T08:43:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77959a700 1 -- 172.21.15.61:0/4078744632 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa774000e90 con 0x7fa77c0b4840 2022-07-12T08:43:32.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.440+0000 7fa77959a700 1 -- 172.21.15.61:0/4078744632 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa77400d900 con 0x7fa77c0b4840 2022-07-12T08:43:32.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 -- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b4840 msgr2=0x7fa77c1420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 --2- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b4840 0x7fa77c1420b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fa774002f70 tx=0x7fa774006eb0).stop 2022-07-12T08:43:32.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 -- 172.21.15.61:0/4078744632 shutdown_connections 2022-07-12T08:43:32.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 --2- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b4840 0x7fa77c1420b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 --2- 172.21.15.61:0/4078744632 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b9eb0 0x7fa77c0b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 -- 172.21.15.61:0/4078744632 >> 172.21.15.61:0/4078744632 conn(0x7fa77c0678c0 msgr2=0x7fa77c0b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 -- 172.21.15.61:0/4078744632 shutdown_connections 2022-07-12T08:43:32.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 -- 172.21.15.61:0/4078744632 wait complete. 2022-07-12T08:43:32.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 Processor -- start 2022-07-12T08:43:32.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.444+0000 7fa7809c9700 1 -- start start 2022-07-12T08:43:32.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b3180 0x7fa77c155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b4840 0x7fa77c0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 0x7fa77c13cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c13e0d0 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c13f7f0 con 0x7fa77c0b3180 2022-07-12T08:43:32.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c146580 con 0x7fa77c0b4840 2022-07-12T08:43:32.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 0x7fa77c13cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa779d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b4840 0x7fa77c0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 0x7fa77c13cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55172/0 (socket says 172.21.15.61:55172) 2022-07-12T08:43:32.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 -- 172.21.15.61:0/1699091089 learned_addr learned my addr 172.21.15.61:0/1699091089 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77a59c700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b3180 0x7fa77c155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 -- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b4840 msgr2=0x7fa77c0b6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b4840 0x7fa77c0b6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 -- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b3180 msgr2=0x7fa77c155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b3180 0x7fa77c155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 -- 172.21.15.61:0/1699091089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77400b1e0 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa779d9b700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b4840 0x7fa77c0b6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:32.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa77ad9d700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 0x7fa77c13cce0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa77c152150 tx=0x7fa774015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7740068b0 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77400d8f0 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c13fa30 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa774011d30 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.448+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c145080 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.452+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c13fbf0 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.452+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa774025030 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.452+0000 7fa7737fe700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa760053a40 0x7fa760055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.452+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa77406cfe0 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.452+0000 7fa77a59c700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa760053a40 0x7fa760055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.452+0000 7fa77a59c700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa760053a40 0x7fa760055f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa76c006000 tx=0x7fa76c006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.460+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa774043e90 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.632+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa77c057460 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.632+0000 7fa7737fe700 1 -- 172.21.15.61:0/1699091089 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7fa774023090 con 0x7fa77c0b9eb0 2022-07-12T08:43:32.659 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:32.660 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:32.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa760053a40 msgr2=0x7fa760055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa760053a40 0x7fa760055f00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fa76c006000 tx=0x7fa76c006d90).stop 2022-07-12T08:43:32.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 msgr2=0x7fa77c13cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 0x7fa77c13cce0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa77c152150 tx=0x7fa774015000).stop 2022-07-12T08:43:32.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 shutdown_connections 2022-07-12T08:43:32.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa77c0b9eb0 0x7fa77c13cce0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa77c0b4840 0x7fa77c0b6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa760053a40 0x7fa760055f00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 --2- 172.21.15.61:0/1699091089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa77c0b3180 0x7fa77c155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 >> 172.21.15.61:0/1699091089 conn(0x7fa77c0678c0 msgr2=0x7fa77c0b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 shutdown_connections 2022-07-12T08:43:32.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.636+0000 7fa7809c9700 1 -- 172.21.15.61:0/1699091089 wait complete. 2022-07-12T08:43:32.682 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:32.683 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 2022-07-12T08:43:32.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeef59e700 1 Processor -- start 2022-07-12T08:43:32.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeef59e700 1 -- start start 2022-07-12T08:43:32.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeef59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0060e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeef59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef014a560 con 0x7fdef005cd20 2022-07-12T08:43:32.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeef59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef014b340 con 0x7fdef01465a0 2022-07-12T08:43:32.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeef59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0097d40 con 0x7fdef005d100 2022-07-12T08:43:32.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeee59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdef005cd20 0x7fdef014ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43394/0 (socket says 172.21.15.61:43394) 2022-07-12T08:43:32.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeee59c700 1 -- 172.21.15.61:0/3970204737 learned_addr learned my addr 172.21.15.61:0/3970204737 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:32.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.796+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/3970204737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0060e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeed59a700 1 -- 172.21.15.61:0/3970204737 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919326133 0 0) 0x7fdef014a560 con 0x7fdef005cd20 2022-07-12T08:43:32.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeed59a700 1 -- 172.21.15.61:0/3970204737 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded4003260 con 0x7fdef005cd20 2022-07-12T08:43:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeedd9b700 1 -- 172.21.15.61:0/3970204737 >> v1:172.21.15.111:6789/0 conn(0x7fdef01465a0 legacy=0x7fdef01469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeedd9b700 1 -- 172.21.15.61:0/3970204737 >> v1:172.21.15.61:6789/0 conn(0x7fdef005cd20 legacy=0x7fdef014ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeedd9b700 1 -- 172.21.15.61:0/3970204737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef00aafa0 con 0x7fdef005d100 2022-07-12T08:43:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/3970204737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0060e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fdee00032f0 tx=0x7fdee00091d0).ready entity=mon.2 client_cookie=79f3fc931ac8427a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeed59a700 1 -- 172.21.15.61:0/3970204737 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee000b6c0 con 0x7fdef005d100 2022-07-12T08:43:32.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeed59a700 1 -- 172.21.15.61:0/3970204737 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000b820 con 0x7fdef005d100 2022-07-12T08:43:32.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeed59a700 1 -- 172.21.15.61:0/3970204737 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee000b990 con 0x7fdef005d100 2022-07-12T08:43:32.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- 172.21.15.61:0/3970204737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 msgr2=0x7fdef0060e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3970204737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0060e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fdee00032f0 tx=0x7fdee00091d0).stop 2022-07-12T08:43:32.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- 172.21.15.61:0/3970204737 shutdown_connections 2022-07-12T08:43:32.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3970204737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0060e80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- 172.21.15.61:0/3970204737 >> 172.21.15.61:0/3970204737 conn(0x7fdef00b1a50 msgr2=0x7fdef00b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- 172.21.15.61:0/3970204737 shutdown_connections 2022-07-12T08:43:32.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- 172.21.15.61:0/3970204737 wait complete. 2022-07-12T08:43:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 Processor -- start 2022-07-12T08:43:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- start start 2022-07-12T08:43:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef005cd20 0x7fdef005fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 0x7fdef01597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef014a2e0 con 0x7fdef01465a0 2022-07-12T08:43:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00ac910 con 0x7fdef005cd20 2022-07-12T08:43:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeef59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef00acbb0 con 0x7fdef005d100 2022-07-12T08:43:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeeed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 0x7fdef01597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeeed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 0x7fdef01597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55186/0 (socket says 172.21.15.61:55186) 2022-07-12T08:43:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.800+0000 7fdeeed9d700 1 -- 172.21.15.61:0/3036259042 learned_addr learned my addr 172.21.15.61:0/3036259042 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeee59c700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef005cd20 0x7fdef005fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeeed9d700 1 -- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 msgr2=0x7fdef0142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeeed9d700 1 -- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef005cd20 msgr2=0x7fdef005fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef005cd20 0x7fdef005fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeeed9d700 1 -- 172.21.15.61:0/3036259042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee0009470 con 0x7fdef01465a0 2022-07-12T08:43:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 0x7fdef01597b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fdef0061480 tx=0x7fdee8007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdedf7fe700 1 -- 172.21.15.61:0/3036259042 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee800ce30 con 0x7fdef01465a0 2022-07-12T08:43:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdedf7fe700 1 -- 172.21.15.61:0/3036259042 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8007240 con 0x7fdef01465a0 2022-07-12T08:43:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef0149d20 con 0x7fdef01465a0 2022-07-12T08:43:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdedf7fe700 1 -- 172.21.15.61:0/3036259042 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee8006510 con 0x7fdef01465a0 2022-07-12T08:43:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef00602d0 con 0x7fdef01465a0 2022-07-12T08:43:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdedf7fe700 1 -- 172.21.15.61:0/3036259042 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdee8010030 con 0x7fdef01465a0 2022-07-12T08:43:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdedf7fe700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdecc053af0 0x7fdecc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.804+0000 7fdedf7fe700 1 -- 172.21.15.61:0/3036259042 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdee806ba90 con 0x7fdef01465a0 2022-07-12T08:43:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeee59c700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdecc053af0 0x7fdecc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 Processor -- start 2022-07-12T08:43:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeee59c700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdecc053af0 0x7fdecc055fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fdee400b000 tx=0x7fdee4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 -- start start 2022-07-12T08:43:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2755da0 0x271ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x2716ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x2717090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x268bdf0 con 0x271b7b0 2022-07-12T08:43:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x266c3e0 con 0x2755da0 2022-07-12T08:43:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x2718bb0 con 0x2751b50 2022-07-12T08:43:32.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x2716ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x2716ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55192/0 (socket says 172.21.15.61:55192) 2022-07-12T08:43:32.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 -- 172.21.15.61:0/4197014929 learned_addr learned my addr 172.21.15.61:0/4197014929 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeee59c700 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2755da0 0x271ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x2717090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 -- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 msgr2=0x2717090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x2717090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 -- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2755da0 msgr2=0x271ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2755da0 0x271ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 -- 172.21.15.61:0/4197014929 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x26c3a60 con 0x271b7b0 2022-07-12T08:43:32.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x2716ba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fdee0019000 tx=0x7fdee0016000).ready entity=mon.0 client_cookie=39ecf85181923ac6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeef59e700 1 -- 172.21.15.61:0/4197014929 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee0006cc0 con 0x271b7b0 2022-07-12T08:43:32.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeef59e700 1 -- 172.21.15.61:0/4197014929 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000bd30 con 0x271b7b0 2022-07-12T08:43:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.808+0000 7fdeef59e700 1 -- 172.21.15.61:0/4197014929 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee00205f0 con 0x271b7b0 2022-07-12T08:43:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- 172.21.15.61:0/4197014929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 msgr2=0x2716ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x2716ba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fdee0019000 tx=0x7fdee0016000).stop 2022-07-12T08:43:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- 172.21.15.61:0/4197014929 shutdown_connections 2022-07-12T08:43:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x2717090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x2716ba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4197014929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2755da0 0x271ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- 172.21.15.61:0/4197014929 >> 172.21.15.61:0/4197014929 conn(0x26a3bf0 msgr2=0x27554e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- 172.21.15.61:0/4197014929 shutdown_connections 2022-07-12T08:43:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- 172.21.15.61:0/4197014929 wait complete. 2022-07-12T08:43:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- start start 2022-07-12T08:43:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 Processor -- start 2022-07-12T08:43:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x274e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x274e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2755da0 0x271f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x27601a0 con 0x271b7b0 2022-07-12T08:43:32.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x274fd20 con 0x2751b50 2022-07-12T08:43:32.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x27212f0 con 0x2755da0 2022-07-12T08:43:32.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdefeddc740 10 client.0 ll_register_callbacks cb 0x26d3040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:43:32.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x274e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x274e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55196/0 (socket says 172.21.15.61:55196) 2022-07-12T08:43:32.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 -- 172.21.15.61:0/1519891643 learned_addr learned my addr 172.21.15.61:0/1519891643 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2755da0 0x271f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x274e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 -- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2755da0 msgr2=0x271f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2755da0 0x271f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 -- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 msgr2=0x274e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x274e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 -- 172.21.15.61:0/1519891643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef00aafa0 con 0x271b7b0 2022-07-12T08:43:32.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdeee59c700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x274e0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fdee4017000 tx=0x7fdee4007000).ready entity=mon.0 client_cookie=d021500c359aad72 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.812+0000 7fdec97fa700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:43:32.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 1 -- 172.21.15.61:0/1519891643 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee401d070 con 0x271b7b0 2022-07-12T08:43:32.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 1 -- 172.21.15.61:0/1519891643 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdee4018c20 con 0x271b7b0 2022-07-12T08:43:32.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2681e60 con 0x271b7b0 2022-07-12T08:43:32.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 1 -- 172.21.15.61:0/1519891643 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee40074c0 con 0x271b7b0 2022-07-12T08:43:32.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 10 client.5436 fetch_fsmap learned FSMap version 17 2022-07-12T08:43:32.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x267b740 con 0x271b7b0 2022-07-12T08:43:32.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 1 -- 172.21.15.61:0/1519891643 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2681e60 con 0x271b7b0 2022-07-12T08:43:32.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 1 -- 172.21.15.61:0/1519891643 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 17) v1 ==== 1247+0+0 (secure 0 0 0) 0x7fdee4022960 con 0x271b7b0 2022-07-12T08:43:32.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 10 client.5436 fetch_fsmap finished waiting for FSMap version 17 2022-07-12T08:43:32.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 10 client.5436 resolve_mds: resolved ID 'a' to GID 5241 2022-07-12T08:43:32.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 20 client.5436 populate_metadata read hostname 'smithi061' 2022-07-12T08:43:32.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x271fdd0 0x27638a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 4 client.5436 mds_command: new command op to 5241 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-12T08:43:32.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 --> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x267a1d0 con 0x271fdd0 2022-07-12T08:43:32.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x271fdd0 0x27638a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 10 client.5436 ms_handle_connect on v2:172.21.15.61:6832/4199688387 2022-07-12T08:43:32.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x271fdd0 0x27638a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=70ee324a08982379 server_cookie=f2b9ffe357b92d60 in_seq=0 out_seq=0 2022-07-12T08:43:32.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 1 -- 172.21.15.61:0/1519891643 <== mds.0 v2:172.21.15.61:6832/4199688387 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x267a1d0 con 0x271fdd0 2022-07-12T08:43:32.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 10 client.5436 handle_command_reply: tid=0 2022-07-12T08:43:32.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 client.5436 shutdown 2022-07-12T08:43:32.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 msgr2=0x274e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x274e0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fdee4017000 tx=0x7fdee4007000).stop 2022-07-12T08:43:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 shutdown_connections 2022-07-12T08:43:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x271fdd0 0x27638a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2755da0 0x271f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x274e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 --2- 172.21.15.61:0/1519891643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x271b7b0 0x274e0d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdec97fa700 0 client.5436 ms_handle_reset on v2:172.21.15.61:6832/4199688387 2022-07-12T08:43:32.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 >> 172.21.15.61:0/1519891643 conn(0x26a3bf0 msgr2=0x274ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 shutdown_connections 2022-07-12T08:43:32.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 1 -- 172.21.15.61:0/1519891643 wait complete. 2022-07-12T08:43:32.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.816+0000 7fdefeddc740 20 client.5436 trim_cache size 0 max 16384 2022-07-12T08:43:32.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 Processor -- start 2022-07-12T08:43:32.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 -- start start 2022-07-12T08:43:32.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x271b7b0 0x2718180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x2720650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x274dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2720510 con 0x2755da0 2022-07-12T08:43:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x26e3b20 con 0x2751b50 2022-07-12T08:43:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x2718dd0 con 0x271b7b0 2022-07-12T08:43:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdeeed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x274dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.832+0000 7fdeeed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x274dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55206/0 (socket says 172.21.15.61:55206) 2022-07-12T08:43:32.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 -- 172.21.15.61:0/690021211 learned_addr learned my addr 172.21.15.61:0/690021211 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x2720650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeee59c700 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x271b7b0 0x2718180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 -- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x271b7b0 msgr2=0x2718180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x271b7b0 0x2718180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 -- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 msgr2=0x2720650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x2720650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 -- 172.21.15.61:0/690021211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x271f770 con 0x2755da0 2022-07-12T08:43:32.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x274dcd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x2722400 tx=0x7fdee8003040).ready entity=mon.0 client_cookie=f076449f0750fbc0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdecb7fe700 1 -- 172.21.15.61:0/690021211 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee8044a50 con 0x2755da0 2022-07-12T08:43:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdecb7fe700 1 -- 172.21.15.61:0/690021211 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8045ba0 con 0x2755da0 2022-07-12T08:43:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdecb7fe700 1 -- 172.21.15.61:0/690021211 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee80424a0 con 0x2755da0 2022-07-12T08:43:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 -- 172.21.15.61:0/690021211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 msgr2=0x274dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x274dcd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x2722400 tx=0x7fdee8003040).stop 2022-07-12T08:43:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 -- 172.21.15.61:0/690021211 shutdown_connections 2022-07-12T08:43:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x274dcd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x2751b50 0x2720650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 --2- 172.21.15.61:0/690021211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x271b7b0 0x2718180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 -- 172.21.15.61:0/690021211 >> 172.21.15.61:0/690021211 conn(0x26a3bf0 msgr2=0x2721830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 -- 172.21.15.61:0/690021211 shutdown_connections 2022-07-12T08:43:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 -- 172.21.15.61:0/690021211 wait complete. 2022-07-12T08:43:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 -- start start 2022-07-12T08:43:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.836+0000 7fdefeddc740 1 Processor -- start 2022-07-12T08:43:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x271b7b0 0x27ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x27ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x276f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2770c80 con 0x2755da0 2022-07-12T08:43:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x27b7fc0 con 0x271b7b0 2022-07-12T08:43:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x2755a90 con 0x2751b50 2022-07-12T08:43:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 10 client.0 ll_register_callbacks cb 0x2722f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:43:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x276f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x276f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55214/0 (socket says 172.21.15.61:55214) 2022-07-12T08:43:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeedd9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x27ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 -- 172.21.15.61:0/4134829609 learned_addr learned my addr 172.21.15.61:0/4134829609 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeee59c700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x271b7b0 0x27ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 -- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 msgr2=0x27ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x27ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 -- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x271b7b0 msgr2=0x27ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x271b7b0 0x27ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:32.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 -- 172.21.15.61:0/4134829609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee8022040 con 0x2755da0 2022-07-12T08:43:32.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeedd9b700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x27ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:32.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeeed9d700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x276f900 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x2722400 tx=0x7fdee8027000).ready entity=mon.0 client_cookie=84d785b6afbb5e0f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:32.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:43:32.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 1 -- 172.21.15.61:0/4134829609 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee8045610 con 0x2755da0 2022-07-12T08:43:32.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 1 -- 172.21.15.61:0/4134829609 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdee8045770 con 0x2755da0 2022-07-12T08:43:32.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdee000bd30 con 0x2755da0 2022-07-12T08:43:32.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 1 -- 172.21.15.61:0/4134829609 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdee8045930 con 0x2755da0 2022-07-12T08:43:32.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 10 client.5442 fetch_fsmap learned FSMap version 17 2022-07-12T08:43:32.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x2684040 con 0x2755da0 2022-07-12T08:43:32.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 1 -- 172.21.15.61:0/4134829609 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000bd30 con 0x2755da0 2022-07-12T08:43:32.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 1 -- 172.21.15.61:0/4134829609 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 17) v1 ==== 1247+0+0 (secure 0 0 0) 0x7fdee8032040 con 0x2755da0 2022-07-12T08:43:32.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 10 client.5442 fetch_fsmap finished waiting for FSMap version 17 2022-07-12T08:43:32.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 10 client.5442 resolve_mds: resolved ID 'a' to GID 5241 2022-07-12T08:43:32.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 20 client.5442 populate_metadata read hostname 'smithi061' 2022-07-12T08:43:32.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x274f8b0 0x276fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:32.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 4 client.5442 mds_command: new command op to 5241 tid=0[{"prefix": "cache drop"}] 2022-07-12T08:43:32.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 --> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x256f590 con 0x274f8b0 2022-07-12T08:43:32.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeee59c700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x274f8b0 0x276fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:32.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeee59c700 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x274f8b0 0x276fdf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=59efe153bfa68122 server_cookie=3e914620ece1ec04 in_seq=0 out_seq=0 2022-07-12T08:43:32.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:32.840+0000 7fdeca7fc700 10 client.5442 ms_handle_connect on v2:172.21.15.61:6832/4199688387 2022-07-12T08:43:44.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeca7fc700 1 -- 172.21.15.61:0/4134829609 <== mds.0 v2:172.21.15.61:6832/4199688387 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+386 (crc 0 0 0) 0x256f590 con 0x274f8b0 2022-07-12T08:43:44.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeca7fc700 10 client.5442 handle_command_reply: tid=0 2022-07-12T08:43:44.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 client.5442 shutdown 2022-07-12T08:43:44.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 msgr2=0x276f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x276f900 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x2722400 tx=0x7fdee8027000).stop 2022-07-12T08:43:44.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 shutdown_connections 2022-07-12T08:43:44.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2755da0 0x276f900 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.61:6832/4199688387,v1:172.21.15.61:6833/4199688387] conn(0x274f8b0 0x276fdf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2751b50 0x27ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 --2- 172.21.15.61:0/4134829609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x271b7b0 0x27ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeca7fc700 0 client.5442 ms_handle_reset on v2:172.21.15.61:6832/4199688387 2022-07-12T08:43:44.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 >> 172.21.15.61:0/4134829609 conn(0x26a3bf0 msgr2=0x2717f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 shutdown_connections 2022-07-12T08:43:44.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 1 -- 172.21.15.61:0/4134829609 wait complete. 2022-07-12T08:43:44.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdefeddc740 20 client.5442 trim_cache size 0 max 16384 2022-07-12T08:43:44.057 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:43:44.057 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-12T08:43:44.058 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-12T08:43:44.058 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(0) Success", 2022-07-12T08:43:44.058 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 220 2022-07-12T08:43:44.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:44.059 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-12T08:43:44.059 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-12T08:43:44.060 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-12T08:43:44.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:44.060 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-12T08:43:44.060 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 0 2022-07-12T08:43:44.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:44.061 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-12T08:43:44.061 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-12T08:43:44.061 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 20076, 2022-07-12T08:43:44.062 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 3105184 2022-07-12T08:43:44.062 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:43:44.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:43:44.063 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 11.190192959999999 2022-07-12T08:43:44.063 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:43:44.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdecc053af0 msgr2=0x7fdecc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdecc053af0 0x7fdecc055fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fdee400b000 tx=0x7fdee4009040).stop 2022-07-12T08:43:44.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 msgr2=0x7fdef01597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 0x7fdef01597b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fdef0061480 tx=0x7fdee8007c20).stop 2022-07-12T08:43:44.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 shutdown_connections 2022-07-12T08:43:44.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef01465a0 0x7fdef01597b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef005d100 0x7fdef0142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdecc053af0 0x7fdecc055fb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 --2- 172.21.15.61:0/3036259042 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef005cd20 0x7fdef005fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 >> 172.21.15.61:0/3036259042 conn(0x7fdef00b1a50 msgr2=0x7fdef00ab5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 shutdown_connections 2022-07-12T08:43:44.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.032+0000 7fdeef59e700 1 -- 172.21.15.61:0/3036259042 wait complete. 2022-07-12T08:43:44.104 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-12T08:43:44.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:44.105 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:44.105 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:44.118 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-12T08:43:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:43:44.121 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-12T08:43:44.123 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:43:44.123 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:43:44.124 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:44.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:44.158 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:44.159 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:44.217 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:43:44.226 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:43:44.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:43:44.227 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:44.255 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:43:44.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.352+0000 7f4bc4c9e700 1 Processor -- start 2022-07-12T08:43:44.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.352+0000 7f4bc4c9e700 1 -- start start 2022-07-12T08:43:44.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00aecb0 con 0x7f4bc009bce0 2022-07-12T08:43:44.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc01435c0 con 0x7f4bc00af280 2022-07-12T08:43:44.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00afd70 con 0x7f4bc00b3f60 2022-07-12T08:43:44.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbeffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4bc009bce0 0x7f4bc00af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43440/0 (socket says 172.21.15.61:43440) 2022-07-12T08:43:44.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbeffd700 1 -- 172.21.15.61:0/3024998694 learned_addr learned my addr 172.21.15.61:0/3024998694 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:44.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbffff700 1 --2- 172.21.15.61:0/3024998694 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012027995 0 0) 0x7f4bc00aecb0 con 0x7f4bc009bce0 2022-07-12T08:43:44.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bac003260 con 0x7f4bc009bce0 2022-07-12T08:43:44.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840572747 0 0) 0x7f4bc01435c0 con 0x7f4bc00af280 2022-07-12T08:43:44.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc00aecb0 con 0x7f4bc00af280 2022-07-12T08:43:44.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbffff700 1 -- 172.21.15.61:0/3024998694 >> v1:172.21.15.111:6789/0 conn(0x7f4bc00af280 legacy=0x7f4bc0060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbffff700 1 -- 172.21.15.61:0/3024998694 >> v1:172.21.15.61:6789/0 conn(0x7f4bc009bce0 legacy=0x7f4bc00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbffff700 1 -- 172.21.15.61:0/3024998694 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc0143860 con 0x7f4bc00b3f60 2022-07-12T08:43:44.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbffff700 1 --2- 172.21.15.61:0/3024998694 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc00b4810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4bb80032f0 tx=0x7f4bb80091d0).ready entity=mon.2 client_cookie=d4d503d8068e0f3d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb800b640 con 0x7f4bc00b3f60 2022-07-12T08:43:44.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb800b7a0 con 0x7f4bc00b3f60 2022-07-12T08:43:44.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/3024998694 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb800b910 con 0x7f4bc00b3f60 2022-07-12T08:43:44.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/3024998694 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 msgr2=0x7f4bc00b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/3024998694 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc00b4810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4bb80032f0 tx=0x7f4bb80091d0).stop 2022-07-12T08:43:44.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/3024998694 shutdown_connections 2022-07-12T08:43:44.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/3024998694 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc00b4810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/3024998694 >> 172.21.15.61:0/3024998694 conn(0x7f4bc00b92c0 msgr2=0x7f4bc00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/3024998694 shutdown_connections 2022-07-12T08:43:44.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.356+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/3024998694 wait complete. 2022-07-12T08:43:44.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 Processor -- start 2022-07-12T08:43:44.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 -- start start 2022-07-12T08:43:44.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc009bce0 0x7f4bc009ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 0x7f4bc0143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc005bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc009e0c0 con 0x7f4bc00af280 2022-07-12T08:43:44.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc009e360 con 0x7f4bc009bce0 2022-07-12T08:43:44.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bc4c9e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc009e600 con 0x7f4bc00b3f60 2022-07-12T08:43:44.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 0x7f4bc0143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 0x7f4bc0143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55228/0 (socket says 172.21.15.61:55228) 2022-07-12T08:43:44.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc005bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1042270222 learned_addr learned my addr 172.21.15.61:0/1042270222 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:44.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbf7fe700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc009bce0 0x7f4bc009ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 msgr2=0x7f4bc005bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc005bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc009bce0 msgr2=0x7f4bc009ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc009bce0 0x7f4bc009ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1042270222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009470 con 0x7f4bc00af280 2022-07-12T08:43:44.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbffff700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc005bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:44.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.360+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 0x7f4bc0143f70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f4bb4001110 tx=0x7f4bb4007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb400ce80 con 0x7f4bc00af280 2022-07-12T08:43:44.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb40070d0 con 0x7f4bc00af280 2022-07-12T08:43:44.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc005c430 con 0x7f4bc00af280 2022-07-12T08:43:44.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb4007430 con 0x7f4bc00af280 2022-07-12T08:43:44.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc00b0e50 con 0x7f4bc00af280 2022-07-12T08:43:44.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4bb4010070 con 0x7f4bc00af280 2022-07-12T08:43:44.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc0155400 con 0x7f4bc00af280 2022-07-12T08:43:44.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbcff9700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ba0053a60 0x7f4ba0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4bb406b3b0 con 0x7f4bc00af280 2022-07-12T08:43:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.364+0000 7f4bbf7fe700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ba0053a60 0x7f4ba0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.368+0000 7f4bbf7fe700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ba0053a60 0x7f4ba0055f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4bb000d000 tx=0x7f4bb0007b20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.372+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4bb4042310 con 0x7f4bc00af280 2022-07-12T08:43:44.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.492+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bc006b2b0 con 0x7f4bc00af280 2022-07-12T08:43:44.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1042270222 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f4bb4018070 con 0x7f4bc00af280 2022-07-12T08:43:44.501 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:44.501 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:43:16.617857+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:43:13.384889+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-12T08:43:14.650247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:43:44.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ba0053a60 msgr2=0x7f4ba0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ba0053a60 0x7f4ba0055f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4bb000d000 tx=0x7f4bb0007b20).stop 2022-07-12T08:43:44.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 msgr2=0x7f4bc0143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 0x7f4bc0143f70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f4bb4001110 tx=0x7f4bb4007aa0).stop 2022-07-12T08:43:44.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 shutdown_connections 2022-07-12T08:43:44.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4bc00b3f60 0x7f4bc005bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00af280 0x7f4bc0143f70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ba0053a60 0x7f4ba0055f20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 --2- 172.21.15.61:0/1042270222 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc009bce0 0x7f4bc009ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 >> 172.21.15.61:0/1042270222 conn(0x7f4bc00b92c0 msgr2=0x7f4bc00b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 shutdown_connections 2022-07-12T08:43:44.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.496+0000 7f4bc4c9e700 1 -- 172.21.15.61:0/1042270222 wait complete. 2022-07-12T08:43:44.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:43:13.384889+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:43:14.650247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:43:44.535 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:43:44.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 Processor -- start 2022-07-12T08:43:44.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 -- start start 2022-07-12T08:43:44.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe80b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8fe8142ab0 0x7f8fe80afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8fe8142f10 0x7f8fe81433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe8143d80 con 0x7f8fe8142f10 2022-07-12T08:43:44.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe805d440 con 0x7f8fe8142ab0 2022-07-12T08:43:44.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fef1cf700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe80674b0 con 0x7f8fe805d060 2022-07-12T08:43:44.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8fe8142f10 0x7f8fe81433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8fe8142f10 0x7f8fe81433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55236/0 (socket says 172.21.15.61:55236) 2022-07-12T08:43:44.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 -- 172.21.15.61:0/476423854 learned_addr learned my addr 172.21.15.61:0/476423854 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:44.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fed9cc700 1 --2- 172.21.15.61:0/476423854 >> v2:172.21.15.111:3300/0 conn(0x7f8fe8142ab0 0x7f8fe80afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee1cd700 1 --2- 172.21.15.61:0/476423854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe80b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 -- 172.21.15.61:0/476423854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 msgr2=0x7f8fe80b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:44.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 --2- 172.21.15.61:0/476423854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe80b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 -- 172.21.15.61:0/476423854 >> v2:172.21.15.111:3300/0 conn(0x7f8fe8142ab0 msgr2=0x7f8fe80afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:44.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 --2- 172.21.15.61:0/476423854 >> v2:172.21.15.111:3300/0 conn(0x7f8fe8142ab0 0x7f8fe80afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 -- 172.21.15.61:0/476423854 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe8067750 con 0x7f8fe8142f10 2022-07-12T08:43:44.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fed9cc700 1 --2- 172.21.15.61:0/476423854 >> v2:172.21.15.111:3300/0 conn(0x7f8fe8142ab0 0x7f8fe80afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:43:44.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.640+0000 7f8fee9ce700 1 --2- 172.21.15.61:0/476423854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe8142f10 0x7f8fe81433a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8fe00032f0 tx=0x7f8fe0009290).ready entity=mon.0 client_cookie=338743621c95587b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fed1cb700 1 -- 172.21.15.61:0/476423854 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe000b680 con 0x7f8fe8142f10 2022-07-12T08:43:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fed1cb700 1 -- 172.21.15.61:0/476423854 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe000b7e0 con 0x7f8fe8142f10 2022-07-12T08:43:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fed1cb700 1 -- 172.21.15.61:0/476423854 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe000b950 con 0x7f8fe8142f10 2022-07-12T08:43:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/476423854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe8142f10 msgr2=0x7f8fe81433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/476423854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe8142f10 0x7f8fe81433a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f8fe00032f0 tx=0x7f8fe0009290).stop 2022-07-12T08:43:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/476423854 shutdown_connections 2022-07-12T08:43:44.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/476423854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe8142f10 0x7f8fe81433a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/476423854 >> v2:172.21.15.111:3300/0 conn(0x7f8fe8142ab0 0x7f8fe80afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/476423854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe80b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/476423854 >> 172.21.15.61:0/476423854 conn(0x7f8fe80b6900 msgr2=0x7f8fe8066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/476423854 shutdown_connections 2022-07-12T08:43:44.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/476423854 wait complete. 2022-07-12T08:43:44.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 Processor -- start 2022-07-12T08:43:44.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- start start 2022-07-12T08:43:44.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe8149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fe8142f10 0x7f8fe809dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 0x7f8fe809e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe8156790 con 0x7f8fe809e250 2022-07-12T08:43:44.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe809bee0 con 0x7f8fe8142f10 2022-07-12T08:43:44.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe809ebd0 con 0x7f8fe805d060 2022-07-12T08:43:44.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 0x7f8fe809e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 0x7f8fe809e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55242/0 (socket says 172.21.15.61:55242) 2022-07-12T08:43:44.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 -- 172.21.15.61:0/2415049299 learned_addr learned my addr 172.21.15.61:0/2415049299 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:44.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fed9cc700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fe8142f10 0x7f8fe809dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee1cd700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe8149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 -- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 msgr2=0x7f8fe8149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe8149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 -- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fe8142f10 msgr2=0x7f8fe809dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fe8142f10 0x7f8fe809dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 -- 172.21.15.61:0/2415049299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe400b180 con 0x7f8fe809e250 2022-07-12T08:43:44.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fed9cc700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fe8142f10 0x7f8fe809dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:43:44.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee1cd700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe8149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:43:44.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fee9ce700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 0x7f8fe809e6e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f8fe81568d0 tx=0x7f8fe0017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe001f070 con 0x7f8fe809e250 2022-07-12T08:43:44.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0011e70 con 0x7f8fe809e250 2022-07-12T08:43:44.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe0009530 con 0x7f8fe809e250 2022-07-12T08:43:44.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe000b680 con 0x7f8fe809e250 2022-07-12T08:43:44.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.644+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe809d250 con 0x7f8fe809e250 2022-07-12T08:43:44.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.648+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe81564d0 con 0x7f8fe809e250 2022-07-12T08:43:44.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.648+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8fe000f070 con 0x7f8fe809e250 2022-07-12T08:43:44.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.648+0000 7f8fd77fe700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8fc80539b0 0x7f8fc8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.648+0000 7f8fee1cd700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8fc80539b0 0x7f8fc8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.648+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8fe006d520 con 0x7f8fe809e250 2022-07-12T08:43:44.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.648+0000 7f8fee1cd700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8fc80539b0 0x7f8fc8055e70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f8fd800b000 tx=0x7f8fd8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.656+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8fe81564d0 con 0x7f8fe809e250 2022-07-12T08:43:44.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.776+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8fe81564d0 con 0x7f8fe809e250 2022-07-12T08:43:44.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.776+0000 7f8fd77fe700 1 -- 172.21.15.61:0/2415049299 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8fe81564d0 con 0x7f8fe809e250 2022-07-12T08:43:44.793 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:43:44.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8fc80539b0 msgr2=0x7f8fc8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8fc80539b0 0x7f8fc8055e70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f8fd800b000 tx=0x7f8fd8008040).stop 2022-07-12T08:43:44.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 msgr2=0x7f8fe809e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 0x7f8fe809e6e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f8fe81568d0 tx=0x7f8fe0017000).stop 2022-07-12T08:43:44.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 shutdown_connections 2022-07-12T08:43:44.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fe809e250 0x7f8fe809e6e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8fc80539b0 0x7f8fc8055e70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fe8142f10 0x7f8fe809dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 --2- 172.21.15.61:0/2415049299 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8fe805d060 0x7f8fe8149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 >> 172.21.15.61:0/2415049299 conn(0x7f8fe80b6900 msgr2=0x7f8fe805f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 shutdown_connections 2022-07-12T08:43:44.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.780+0000 7f8fef1cf700 1 -- 172.21.15.61:0/2415049299 wait complete. 2022-07-12T08:43:44.803 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:43:44.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.900+0000 7f3c6cf5b700 1 Processor -- start 2022-07-12T08:43:44.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.900+0000 7f3c6cf5b700 1 -- start start 2022-07-12T08:43:44.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.900+0000 7f3c6cf5b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c680b5920 0x7f3c6814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.900+0000 7f3c6cf5b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6805c3e0 con 0x7f3c680b0280 2022-07-12T08:43:44.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.900+0000 7f3c6cf5b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c680b5d00 con 0x7f3c680aa830 2022-07-12T08:43:44.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.900+0000 7f3c6cf5b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6805c610 con 0x7f3c680b5920 2022-07-12T08:43:44.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c66ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3c680b0280 0x7f3c6814aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43466/0 (socket says 172.21.15.61:43466) 2022-07-12T08:43:44.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c66ffd700 1 -- 172.21.15.61:0/1078973369 learned_addr learned my addr 172.21.15.61:0/1078973369 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:44.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c677fe700 1 --2- 172.21.15.61:0/1078973369 >> v2:172.21.15.61:3300/0 conn(0x7f3c680b5920 0x7f3c6814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c667fc700 1 -- 172.21.15.61:0/1078973369 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302977045 0 0) 0x7f3c6805c3e0 con 0x7f3c680b0280 2022-07-12T08:43:44.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c667fc700 1 -- 172.21.15.61:0/1078973369 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c48003260 con 0x7f3c680b0280 2022-07-12T08:43:44.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c677fe700 1 -- 172.21.15.61:0/1078973369 >> v1:172.21.15.111:6789/0 conn(0x7f3c680aa830 legacy=0x7f3c680aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c667fc700 1 -- 172.21.15.61:0/1078973369 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 503990582 0 0) 0x7f3c48003260 con 0x7f3c680b0280 2022-07-12T08:43:44.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c677fe700 1 -- 172.21.15.61:0/1078973369 >> v1:172.21.15.61:6789/0 conn(0x7f3c680b0280 legacy=0x7f3c6814aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c677fe700 1 -- 172.21.15.61:0/1078973369 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6805c8b0 con 0x7f3c680b5920 2022-07-12T08:43:44.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c677fe700 1 --2- 172.21.15.61:0/1078973369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680b5920 0x7f3c6814a9e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f3c50001800 tx=0x7f3c58006df0).ready entity=mon.0 client_cookie=24425807ee39f128 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c667fc700 1 -- 172.21.15.61:0/1078973369 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c50001660 con 0x7f3c680b5920 2022-07-12T08:43:44.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c667fc700 1 -- 172.21.15.61:0/1078973369 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c58000e90 con 0x7f3c680b5920 2022-07-12T08:43:44.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c667fc700 1 -- 172.21.15.61:0/1078973369 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c58011400 con 0x7f3c680b5920 2022-07-12T08:43:44.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1078973369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680b5920 msgr2=0x7f3c6814a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1078973369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680b5920 0x7f3c6814a9e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f3c50001800 tx=0x7f3c58006df0).stop 2022-07-12T08:43:44.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1078973369 shutdown_connections 2022-07-12T08:43:44.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1078973369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680b5920 0x7f3c6814a9e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1078973369 >> 172.21.15.61:0/1078973369 conn(0x7f3c68063340 msgr2=0x7f3c680b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:44.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1078973369 shutdown_connections 2022-07-12T08:43:44.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1078973369 wait complete. 2022-07-12T08:43:44.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 Processor -- start 2022-07-12T08:43:44.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.904+0000 7f3c6cf5b700 1 -- start start 2022-07-12T08:43:44.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 0x7f3c680aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c680b0280 0x7f3c68146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c680b5920 0x7f3c68146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68149630 con 0x7f3c680aa830 2022-07-12T08:43:44.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c681498d0 con 0x7f3c680b5920 2022-07-12T08:43:44.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c68149b70 con 0x7f3c680b0280 2022-07-12T08:43:44.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 0x7f3c680aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 0x7f3c680aafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55254/0 (socket says 172.21.15.61:55254) 2022-07-12T08:43:44.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 -- 172.21.15.61:0/1314034159 learned_addr learned my addr 172.21.15.61:0/1314034159 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:44.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c66ffd700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c680b0280 0x7f3c68146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c67fff700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c680b5920 0x7f3c68146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:44.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 -- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c680b0280 msgr2=0x7f3c68146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c680b0280 0x7f3c68146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 -- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c680b5920 msgr2=0x7f3c68146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:44.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c680b5920 0x7f3c68146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:44.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 -- 172.21.15.61:0/1314034159 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5800b180 con 0x7f3c680aa830 2022-07-12T08:43:44.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c677fe700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 0x7f3c680aafa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3c50001800 tx=0x7f3c680af5d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:44.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c58015070 con 0x7f3c680aa830 2022-07-12T08:43:44.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5800db10 con 0x7f3c680aa830 2022-07-12T08:43:44.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c68064290 con 0x7f3c680aa830 2022-07-12T08:43:44.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c58011400 con 0x7f3c680aa830 2022-07-12T08:43:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c680af050 con 0x7f3c680aa830 2022-07-12T08:43:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.908+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c68159460 con 0x7f3c680aa830 2022-07-12T08:43:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.912+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3c58024030 con 0x7f3c680aa830 2022-07-12T08:43:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.912+0000 7f3c64ff9700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c4c053ad0 0x7f3c4c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:44.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.912+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3c5806c970 con 0x7f3c680aa830 2022-07-12T08:43:45.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.912+0000 7f3c66ffd700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c4c053ad0 0x7f3c4c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.912+0000 7f3c66ffd700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c4c053ad0 0x7f3c4c055f90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f3c5000d000 tx=0x7f3c50000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:44.916+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c58043840 con 0x7f3c680aa830 2022-07-12T08:43:45.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.036+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3c68057460 con 0x7f3c680aa830 2022-07-12T08:43:45.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.036+0000 7f3c64ff9700 1 -- 172.21.15.61:0/1314034159 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3c58022070 con 0x7f3c680aa830 2022-07-12T08:43:45.048 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:45.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c4c053ad0 msgr2=0x7f3c4c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c4c053ad0 0x7f3c4c055f90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f3c5000d000 tx=0x7f3c50000bc0).stop 2022-07-12T08:43:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 msgr2=0x7f3c680aafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 0x7f3c680aafa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f3c50001800 tx=0x7f3c680af5d0).stop 2022-07-12T08:43:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 shutdown_connections 2022-07-12T08:43:45.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c4c053ad0 0x7f3c4c055f90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c680b5920 0x7f3c68146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c680b0280 0x7f3c68146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 --2- 172.21.15.61:0/1314034159 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c680aa830 0x7f3c680aafa0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 >> 172.21.15.61:0/1314034159 conn(0x7f3c68063340 msgr2=0x7f3c680b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 shutdown_connections 2022-07-12T08:43:45.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.040+0000 7f3c6cf5b700 1 -- 172.21.15.61:0/1314034159 wait complete. 2022-07-12T08:43:45.064 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:43:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.180+0000 7f1e81aa6700 1 Processor -- start 2022-07-12T08:43:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.180+0000 7f1e81aa6700 1 -- start start 2022-07-12T08:43:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1e7c067050 0x7f1e7c060870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1e7c0a9450 0x7f1e7c0a9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c0b1140 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c0b17f0 con 0x7f1e7c067050 2022-07-12T08:43:45.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c0a9d40 con 0x7f1e7c0a9450 2022-07-12T08:43:45.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1e7c067050 0x7f1e7c060870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1e7c067050 0x7f1e7c060870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55264/0 (socket says 172.21.15.61:55264) 2022-07-12T08:43:45.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 -- 172.21.15.61:0/3163335489 learned_addr learned my addr 172.21.15.61:0/3163335489 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:45.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7bfff700 1 --2- 172.21.15.61:0/3163335489 >> v2:172.21.15.111:3300/0 conn(0x7f1e7c0a9450 0x7f1e7c0a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 -- 172.21.15.61:0/3163335489 >> v2:172.21.15.111:3300/0 conn(0x7f1e7c0a9450 msgr2=0x7f1e7c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:45.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 --2- 172.21.15.61:0/3163335489 >> v2:172.21.15.111:3300/0 conn(0x7f1e7c0a9450 0x7f1e7c0a9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 -- 172.21.15.61:0/3163335489 >> v1:172.21.15.111:6789/0 conn(0x7f1e7c0b0b40 legacy=0x7f1e7c05f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 -- 172.21.15.61:0/3163335489 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c067430 con 0x7f1e7c067050 2022-07-12T08:43:45.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7b7fe700 1 --2- 172.21.15.61:0/3163335489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c067050 0x7f1e7c060870 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f1e6c002f70 tx=0x7f1e6c006eb0).ready entity=mon.0 client_cookie=91cc9e845a4600e6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7a7fc700 1 -- 172.21.15.61:0/3163335489 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e6c00d610 con 0x7f1e7c067050 2022-07-12T08:43:45.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7a7fc700 1 -- 172.21.15.61:0/3163335489 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c000e90 con 0x7f1e7c067050 2022-07-12T08:43:45.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e7a7fc700 1 -- 172.21.15.61:0/3163335489 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e6c00d9a0 con 0x7f1e7c067050 2022-07-12T08:43:45.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- 172.21.15.61:0/3163335489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c067050 msgr2=0x7f1e7c060870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/3163335489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c067050 0x7f1e7c060870 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f1e6c002f70 tx=0x7f1e6c006eb0).stop 2022-07-12T08:43:45.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- 172.21.15.61:0/3163335489 shutdown_connections 2022-07-12T08:43:45.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/3163335489 >> v2:172.21.15.111:3300/0 conn(0x7f1e7c0a9450 0x7f1e7c0a9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/3163335489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c067050 0x7f1e7c060870 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- 172.21.15.61:0/3163335489 >> 172.21.15.61:0/3163335489 conn(0x7f1e7c0b61a0 msgr2=0x7f1e7c0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- 172.21.15.61:0/3163335489 shutdown_connections 2022-07-12T08:43:45.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.184+0000 7f1e81aa6700 1 -- 172.21.15.61:0/3163335489 wait complete. 2022-07-12T08:43:45.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 Processor -- start 2022-07-12T08:43:45.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 -- start start 2022-07-12T08:43:45.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e7c067050 0x7f1e7c13f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1e7c0a9450 0x7f1e7c13f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 0x7f1e7c142c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c143850 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0abba0 con 0x7f1e7c067050 2022-07-12T08:43:45.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e81aa6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0ac8d0 con 0x7f1e7c0a9450 2022-07-12T08:43:45.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 0x7f1e7c142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 0x7f1e7c142c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55272/0 (socket says 172.21.15.61:55272) 2022-07-12T08:43:45.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 -- 172.21.15.61:0/2761234466 learned_addr learned my addr 172.21.15.61:0/2761234466 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:45.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7b7fe700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e7c067050 0x7f1e7c13f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7affd700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1e7c0a9450 0x7f1e7c13f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 -- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1e7c0a9450 msgr2=0x7f1e7c13f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1e7c0a9450 0x7f1e7c13f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 -- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e7c067050 msgr2=0x7f1e7c13f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e7c067050 0x7f1e7c13f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 -- 172.21.15.61:0/2761234466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c00b1e0 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e7bfff700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 0x7f1e7c142c30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1e7400a000 tx=0x7f1e74008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.188+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e74013620 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e74013780 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c143a80 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e74013ae0 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c0acd30 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c0aca70 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1e74013c40 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e78ff9700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e600539b0 0x7f1e60055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1e7406bd70 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.192+0000 7f1e7b7fe700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e600539b0 0x7f1e60055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.196+0000 7f1e7b7fe700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e600539b0 0x7f1e60055e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1e6c01a000 tx=0x7f1e6c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.200+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e7c0aca70 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.320+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e7c0aca40 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.320+0000 7f1e78ff9700 1 -- 172.21.15.61:0/2761234466 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1e74042bd0 con 0x7f1e7c0b0b40 2022-07-12T08:43:45.331 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:45.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e600539b0 msgr2=0x7f1e60055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e600539b0 0x7f1e60055e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1e6c01a000 tx=0x7f1e6c015000).stop 2022-07-12T08:43:45.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 msgr2=0x7f1e7c142c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 0x7f1e7c142c30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f1e7400a000 tx=0x7f1e74008040).stop 2022-07-12T08:43:45.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 shutdown_connections 2022-07-12T08:43:45.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e7c0b0b40 0x7f1e7c142c30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1e7c0a9450 0x7f1e7c13f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e600539b0 0x7f1e60055e70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 --2- 172.21.15.61:0/2761234466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e7c067050 0x7f1e7c13f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 >> 172.21.15.61:0/2761234466 conn(0x7f1e7c0b61a0 msgr2=0x7f1e7c0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 shutdown_connections 2022-07-12T08:43:45.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.324+0000 7f1e81aa6700 1 -- 172.21.15.61:0/2761234466 wait complete. 2022-07-12T08:43:45.345 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:43:45.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 Processor -- start 2022-07-12T08:43:45.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 -- start start 2022-07-12T08:43:45.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2c80adc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2c80acb60 0x7fb2c8140140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c80b21d0 con 0x7fb2c80b3a90 2022-07-12T08:43:45.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c805d4d0 con 0x7fb2c80acb60 2022-07-12T08:43:45.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2cd7ff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80b8290 con 0x7fb2c80b7eb0 2022-07-12T08:43:45.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c67fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb2c80b3a90 0x7fb2c8144310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43496/0 (socket says 172.21.15.61:43496) 2022-07-12T08:43:45.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c67fc700 1 -- 172.21.15.61:0/1380095096 learned_addr learned my addr 172.21.15.61:0/1380095096 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:45.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/1380095096 >> v2:172.21.15.111:3300/0 conn(0x7fb2c80acb60 0x7fb2c8140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2c80adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931835477 0 0) 0x7fb2c80b21d0 con 0x7fb2c80b3a90 2022-07-12T08:43:45.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8003260 con 0x7fb2c80b3a90 2022-07-12T08:43:45.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153639569 0 0) 0x7fb2a8003260 con 0x7fb2c80b3a90 2022-07-12T08:43:45.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 msgr2=0x7fb2c80adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:45.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2c80adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 >> v2:172.21.15.111:3300/0 conn(0x7fb2c80acb60 msgr2=0x7fb2c8140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:45.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 --2- 172.21.15.61:0/1380095096 >> v2:172.21.15.111:3300/0 conn(0x7fb2c80acb60 0x7fb2c8140140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c805d230 con 0x7fb2c80b3a90 2022-07-12T08:43:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fb2b0001660 con 0x7fb2c80b3a90 2022-07-12T08:43:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2c80adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 >> v1:172.21.15.61:6789/0 conn(0x7fb2c80b3a90 legacy=0x7fb2c8144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2a8003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2a8003c60 0x7fb2a8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2a80049f0 0x7fb2a8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c805d4d0 con 0x7fb2a80049f0 2022-07-12T08:43:45.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80b8290 con 0x7fb2a8003c60 2022-07-12T08:43:45.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a800b750 con 0x7fb2c80b7eb0 2022-07-12T08:43:45.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2a80049f0 0x7fb2a8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c67fc700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2a8003c60 0x7fb2a8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2a8003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 -- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 msgr2=0x7fb2a8003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:45.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2c80b7eb0 0x7fb2a8003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 -- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2a8003c60 msgr2=0x7fb2a8004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:43:45.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2a8003c60 0x7fb2a8004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 -- 172.21.15.61:0/1380095096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a8003260 con 0x7fb2a80049f0 2022-07-12T08:43:45.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2a80049f0 0x7fb2a8004e70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fb2a80054d0 tx=0x7fb2c00091d0).ready entity=mon.0 client_cookie=af6db7f65add11e4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c000e7e0 con 0x7fb2a80049f0 2022-07-12T08:43:45.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c000e940 con 0x7fb2a80049f0 2022-07-12T08:43:45.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.460+0000 7fb2c5ffb700 1 -- 172.21.15.61:0/1380095096 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c000eca0 con 0x7fb2a80049f0 2022-07-12T08:43:45.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/1380095096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2a80049f0 msgr2=0x7fb2a8004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/1380095096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2a80049f0 0x7fb2a8004e70 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fb2a80054d0 tx=0x7fb2c00091d0).stop 2022-07-12T08:43:45.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2c77fe700 1 -- 172.21.15.61:0/1380095096 reap_dead start 2022-07-12T08:43:45.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/1380095096 shutdown_connections 2022-07-12T08:43:45.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/1380095096 >> 172.21.15.61:0/1380095096 conn(0x7fb2c8065850 msgr2=0x7fb2c80b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/1380095096 shutdown_connections 2022-07-12T08:43:45.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/1380095096 wait complete. 2022-07-12T08:43:45.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 Processor -- start 2022-07-12T08:43:45.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- start start 2022-07-12T08:43:45.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2a80049f0 0x7fb2c8111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 0x7fb2c80afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8151320 0x7fb2c81517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80b0ce0 con 0x7fb2c80af340 2022-07-12T08:43:45.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80b0420 con 0x7fb2c8151320 2022-07-12T08:43:45.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2cd7ff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c80ae8a0 con 0x7fb2a80049f0 2022-07-12T08:43:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2c67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 0x7fb2c80afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2c67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 0x7fb2c80afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55292/0 (socket says 172.21.15.61:55292) 2022-07-12T08:43:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2c67fc700 1 -- 172.21.15.61:0/2494634405 learned_addr learned my addr 172.21.15.61:0/2494634405 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:45.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2a80049f0 0x7fb2c8111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.464+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8151320 0x7fb2c81517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c67fc700 1 -- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2a80049f0 msgr2=0x7fb2c8111150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c67fc700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2a80049f0 0x7fb2c8111150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c67fc700 1 -- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8151320 msgr2=0x7fb2c81517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c67fc700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8151320 0x7fb2c81517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c67fc700 1 -- 172.21.15.61:0/2494634405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c80aeb40 con 0x7fb2c80af340 2022-07-12T08:43:45.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2a80049f0 0x7fb2c8111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:45.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c77fe700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8151320 0x7fb2c81517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:43:45.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c67fc700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 0x7fb2c80afbd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fb2b000d000 tx=0x7fb2c80b31b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2b0011850 con 0x7fb2c80af340 2022-07-12T08:43:45.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b00119b0 con 0x7fb2c80af340 2022-07-12T08:43:45.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c8143d30 con 0x7fb2c80af340 2022-07-12T08:43:45.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2b0011d10 con 0x7fb2c80af340 2022-07-12T08:43:45.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c8151e30 con 0x7fb2c80af340 2022-07-12T08:43:45.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c8057460 con 0x7fb2c80af340 2022-07-12T08:43:45.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb2b001b030 con 0x7fb2c80af340 2022-07-12T08:43:45.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2bffff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ac053ad0 0x7fb2ac055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb2b006b930 con 0x7fb2c80af340 2022-07-12T08:43:45.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.468+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ac053ad0 0x7fb2ac055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.472+0000 7fb2c6ffd700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ac053ad0 0x7fb2ac055f90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb2b800c000 tx=0x7fb2b8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.472+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb2b0042890 con 0x7fb2c80af340 2022-07-12T08:43:45.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.596+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2c80630a0 con 0x7fb2c80af340 2022-07-12T08:43:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.596+0000 7fb2bffff700 1 -- 172.21.15.61:0/2494634405 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb2b0019030 con 0x7fb2c80af340 2022-07-12T08:43:45.617 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:43:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ac053ad0 msgr2=0x7fb2ac055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ac053ad0 0x7fb2ac055f90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fb2b800c000 tx=0x7fb2b8008040).stop 2022-07-12T08:43:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 msgr2=0x7fb2c80afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 0x7fb2c80afbd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fb2b000d000 tx=0x7fb2c80b31b0).stop 2022-07-12T08:43:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 shutdown_connections 2022-07-12T08:43:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb2ac053ad0 0x7fb2ac055f90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2c8151320 0x7fb2c81517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2c80af340 0x7fb2c80afbd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 --2- 172.21.15.61:0/2494634405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2a80049f0 0x7fb2c8111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 >> 172.21.15.61:0/2494634405 conn(0x7fb2c8065850 msgr2=0x7fb2c8065d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 shutdown_connections 2022-07-12T08:43:45.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.600+0000 7fb2cd7ff700 1 -- 172.21.15.61:0/2494634405 wait complete. 2022-07-12T08:43:45.629 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:43:45.630 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:43:45.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 Processor -- start 2022-07-12T08:43:45.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- start start 2022-07-12T08:43:45.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b0b10 0x7ff844140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440b32c0 con 0x7ff8440b01e0 2022-07-12T08:43:45.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8440a93f0 con 0x7ff8440b2ee0 2022-07-12T08:43:45.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff844060b50 con 0x7ff8440b0b10 2022-07-12T08:43:45.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff848ba4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff8440b01e0 0x7ff8440aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43518/0 (socket says 172.21.15.61:43518) 2022-07-12T08:43:45.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff848ba4700 1 -- 172.21.15.61:0/1483252914 learned_addr learned my addr 172.21.15.61:0/1483252914 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:43:45.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff83b7fe700 1 --2- 172.21.15.61:0/1483252914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b0b10 0x7ff844140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730178684 0 0) 0x7ff8440b32c0 con 0x7ff8440b01e0 2022-07-12T08:43:45.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff824003260 con 0x7ff8440b01e0 2022-07-12T08:43:45.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980486255 0 0) 0x7ff8440a93f0 con 0x7ff8440b2ee0 2022-07-12T08:43:45.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8440b32c0 con 0x7ff8440b2ee0 2022-07-12T08:43:45.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1483252914 >> v1:172.21.15.111:6789/0 conn(0x7ff8440b2ee0 legacy=0x7ff8441403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1483252914 >> v1:172.21.15.61:6789/0 conn(0x7ff8440b01e0 legacy=0x7ff8440aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1483252914 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8440a9690 con 0x7ff8440b0b10 2022-07-12T08:43:45.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff83b7fe700 1 --2- 172.21.15.61:0/1483252914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b0b10 0x7ff844140ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff82c002f70 tx=0x7ff82c006df0).ready entity=mon.2 client_cookie=c86b96d5fd4c79ee server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c00d5c0 con 0x7ff8440b0b10 2022-07-12T08:43:45.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c000e90 con 0x7ff8440b0b10 2022-07-12T08:43:45.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff8437fe700 1 -- 172.21.15.61:0/1483252914 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c00d910 con 0x7ff8440b0b10 2022-07-12T08:43:45.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- 172.21.15.61:0/1483252914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b0b10 msgr2=0x7ff844140ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 --2- 172.21.15.61:0/1483252914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b0b10 0x7ff844140ae0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7ff82c002f70 tx=0x7ff82c006df0).stop 2022-07-12T08:43:45.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- 172.21.15.61:0/1483252914 shutdown_connections 2022-07-12T08:43:45.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 --2- 172.21.15.61:0/1483252914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b0b10 0x7ff844140ae0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- 172.21.15.61:0/1483252914 >> 172.21.15.61:0/1483252914 conn(0x7ff844063ab0 msgr2=0x7ff844063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- 172.21.15.61:0/1483252914 shutdown_connections 2022-07-12T08:43:45.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.736+0000 7ff84a607700 1 -- 172.21.15.61:0/1483252914 wait complete. 2022-07-12T08:43:45.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 Processor -- start 2022-07-12T08:43:45.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 -- start start 2022-07-12T08:43:45.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8440b01e0 0x7ff844141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 0x7ff844129d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b2ee0 0x7ff84412a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440b3fa0 con 0x7ff8440b0b10 2022-07-12T08:43:45.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440b41d0 con 0x7ff8440b01e0 2022-07-12T08:43:45.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8440b4470 con 0x7ff8440b2ee0 2022-07-12T08:43:45.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 0x7ff844129d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 0x7ff844129d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55302/0 (socket says 172.21.15.61:55302) 2022-07-12T08:43:45.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1434590744 learned_addr learned my addr 172.21.15.61:0/1434590744 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:43:45.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff843fff700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8440b01e0 0x7ff844141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff848ba4700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b2ee0 0x7ff84412a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b2ee0 msgr2=0x7ff84412a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b2ee0 0x7ff84412a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8440b01e0 msgr2=0x7ff844141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8440b01e0 0x7ff844141ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 -- 172.21.15.61:0/1434590744 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c00b180 con 0x7ff8440b0b10 2022-07-12T08:43:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff83b7fe700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 0x7ff844129d30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ff82c002f70 tx=0x7ff82c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c013de0 con 0x7ff8440b0b10 2022-07-12T08:43:45.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff82c01e400 con 0x7ff8440b0b10 2022-07-12T08:43:45.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84412b1b0 con 0x7ff8440b0b10 2022-07-12T08:43:45.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff82c01e760 con 0x7ff8440b0b10 2022-07-12T08:43:45.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.740+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff844142d60 con 0x7ff8440b0b10 2022-07-12T08:43:45.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.744+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff844057460 con 0x7ff8440b0b10 2022-07-12T08:43:45.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.744+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff82c025030 con 0x7ff8440b0b10 2022-07-12T08:43:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.744+0000 7ff841ffb700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff828053ad0 0x7ff828055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:43:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.744+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff82c06ca40 con 0x7ff8440b0b10 2022-07-12T08:43:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.744+0000 7ff843fff700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff828053ad0 0x7ff828055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:43:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.744+0000 7ff843fff700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff828053ad0 0x7ff828055f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7ff83400d000 tx=0x7ff834007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:43:45.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.752+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff82c0439a0 con 0x7ff8440b0b10 2022-07-12T08:43:45.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8440679a0 con 0x7ff8440b0b10 2022-07-12T08:43:45.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff841ffb700 1 -- 172.21.15.61:0/1434590744 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7ff82c023070 con 0x7ff8440b0b10 2022-07-12T08:43:45.937 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:43:45.937 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:43:45.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff828053ad0 msgr2=0x7ff828055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff828053ad0 0x7ff828055f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7ff83400d000 tx=0x7ff834007ad0).stop 2022-07-12T08:43:45.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 msgr2=0x7ff844129d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:43:45.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 0x7ff844129d30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ff82c002f70 tx=0x7ff82c014000).stop 2022-07-12T08:43:45.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 shutdown_connections 2022-07-12T08:43:45.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8440b2ee0 0x7ff84412a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8440b0b10 0x7ff844129d30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff828053ad0 0x7ff828055f90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 --2- 172.21.15.61:0/1434590744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8440b01e0 0x7ff844141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:43:45.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 >> 172.21.15.61:0/1434590744 conn(0x7ff844063ab0 msgr2=0x7ff84405f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:43:45.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 shutdown_connections 2022-07-12T08:43:45.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:43:45.924+0000 7ff84a607700 1 -- 172.21.15.61:0/1434590744 wait complete. 2022-07-12T08:43:45.956 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:43:46.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:43:15.615120+0000', 'modified': '2022-07-12T08:43:16.618859+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'a', 'rank': 0, 'incarnation': 16, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/4199688387', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 4199688387}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 4199688387}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:43:46.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:43:46.177 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:43:46.305 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5241,"want_state":"up:active","state":"up:active","rank_uptime":30.66876993,"mdsmap_epoch":17,"osdmap_epoch":42,"osdmap_epoch_barrier":42,"uptime":35.914067952000003} 2022-07-12T08:43:46.306 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5241, "mdsmap_epoch": 17, "osdmap_epoch": 42, "osdmap_epoch_barrier": 42, "rank_uptime": 30.66876993, "state": "up:active", "uptime": 35.914067952, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:43:46.307 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:43:46.308 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:43:46.309 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:43:46.309 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:46.310 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:46.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:43:46.399 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:43:46.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:43:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:43:46.467 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:43:46.468 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:46.473 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:43:46.528 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:43:46.529 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:46.556 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:43:46.591 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:43:46.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:43:46.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:43:46.618+0000 7f514a09e080 -1 init, newargv = 0x559afd322780 newargc=9ceph-fuse[21245]: starting ceph client 2022-07-12T08:43:46.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-07-12T08:43:46.636 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21245]: starting fuse 2022-07-12T08:43:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:43:46.680 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:43:46.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:46.681 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:43:46.795 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:43:46.806 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:43:46.827 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:46.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:43:46.846 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:43:46.846 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:43:46.847 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:43:46.847 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:43:46.847 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:46.848 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:43:46.848 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:43:46.849 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:43:46.849 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:46.850 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:43:46.850 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:43:46.850 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:43:46.851 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:43:46.851 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:46.852 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:43:46.852 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:43:46.852 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:43:46.853 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:43:46.853 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:43:46.914 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:43:46.916 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:43:46.921 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:43:46.991 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21245.asok 2022-07-12T08:43:46.992 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21245.asok 2022-07-12T08:43:46.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:47.007 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21245.asok status 2022-07-12T08:43:47.155 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:43:47.155 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:43:47.156 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:43:47.156 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:43:47.157 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:43:47.157 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:43:47.157 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:43:47.158 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21245", 2022-07-12T08:43:47.158 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:43:47.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:47.159 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:43:47.160 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:43:47.160 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5443, 2022-07-12T08:43:47.160 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:43:47.161 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:43:47.161 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:43:47.161 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5443 2022-07-12T08:43:47.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:47.168 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:43:47.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:43:47.169 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:43:47.170 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 123273476 2022-07-12T08:43:47.170 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:43:47.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:47.171 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:43:47.171 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:43:47.172 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:43:47.172 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 123273476 2022-07-12T08:43:47.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:43:47.173 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5443 172.21.15.121:0/123273476", 2022-07-12T08:43:47.173 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/123273476", 2022-07-12T08:43:47.179 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:43:47.179 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 17, 2022-07-12T08:43:47.181 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 42, 2022-07-12T08:43:47.181 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:43:47.181 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:43:47.185 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:43:47.186 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:43:47.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:47.189 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:43:47.190 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:43:47.261 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:43:47.261 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:43:47.262 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:43:47.263 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:02.344 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:02.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:02.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:02.390 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:02.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:32.458 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:32.462 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:32.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:32.473 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:32.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:44:35.385 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:44:35.387 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:35.407 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:35.431 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:35.453 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:35.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:35.499 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:44:35.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:35.500 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:35.500 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:35.509 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:44:35.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:44:35.512 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:44:35.513 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:44:35.514 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:35.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21245]: fuse finished with error 0 and tester_r 0 2022-07-12T08:44:35.570 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:35.571 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:35.571 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:35.579 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:44:35.580 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:44:35.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:44:35.689 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:35.760 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:35.760 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:35.761 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:44:35.771 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:44:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:35.773 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:44:35.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:44:35.777 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:44:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:35.825 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:44:35.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:35.826 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:35.826 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:44:35.833 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:44:35.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:35.836 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:44:35.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:44:35.837 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:44:35.844 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:44:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:35.846 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:44:35.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.952+0000 7fa855f3a700 1 Processor -- start 2022-07-12T08:44:35.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.952+0000 7fa855f3a700 1 -- start start 2022-07-12T08:44:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa8500b5aa0 0x7fa8500b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa850147000 0x7fa8501474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85005d3d0 con 0x7fa85005bfc0 2022-07-12T08:44:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85014b590 con 0x7fa8500b5aa0 2022-07-12T08:44:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8500b5e80 con 0x7fa850147000 2022-07-12T08:44:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa85005bfc0 0x7fa85014ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43528/0 (socket says 172.21.15.61:43528) 2022-07-12T08:44:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84effd700 1 -- 172.21.15.61:0/207108654 learned_addr learned my addr 172.21.15.61:0/207108654 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84f7fe700 1 --2- 172.21.15.61:0/207108654 >> v2:172.21.15.61:3300/0 conn(0x7fa8500b5aa0 0x7fa8500b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84e7fc700 1 -- 172.21.15.61:0/207108654 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147011357 0 0) 0x7fa85005d3d0 con 0x7fa85005bfc0 2022-07-12T08:44:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84ffff700 1 --2- 172.21.15.61:0/207108654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa850147000 0x7fa8501474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84e7fc700 1 -- 172.21.15.61:0/207108654 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa83c003260 con 0x7fa85005bfc0 2022-07-12T08:44:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84e7fc700 1 -- 172.21.15.61:0/207108654 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3456322882 0 0) 0x7fa83c003260 con 0x7fa85005bfc0 2022-07-12T08:44:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84f7fe700 1 -- 172.21.15.61:0/207108654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa850147000 msgr2=0x7fa8501474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:35.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84f7fe700 1 --2- 172.21.15.61:0/207108654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa850147000 0x7fa8501474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:35.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84f7fe700 1 -- 172.21.15.61:0/207108654 >> v1:172.21.15.61:6789/0 conn(0x7fa85005bfc0 legacy=0x7fa85014ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:35.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84f7fe700 1 -- 172.21.15.61:0/207108654 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85014a740 con 0x7fa8500b5aa0 2022-07-12T08:44:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84f7fe700 1 --2- 172.21.15.61:0/207108654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8500b5aa0 0x7fa8500b0380 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fa840001800 tx=0x7fa844006df0).ready entity=mon.0 client_cookie=b3fa43df17158143 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84e7fc700 1 -- 172.21.15.61:0/207108654 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa840001660 con 0x7fa8500b5aa0 2022-07-12T08:44:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84e7fc700 1 -- 172.21.15.61:0/207108654 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa844000e90 con 0x7fa8500b5aa0 2022-07-12T08:44:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa84e7fc700 1 -- 172.21.15.61:0/207108654 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa844012460 con 0x7fa8500b5aa0 2022-07-12T08:44:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- 172.21.15.61:0/207108654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8500b5aa0 msgr2=0x7fa8500b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 --2- 172.21.15.61:0/207108654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8500b5aa0 0x7fa8500b0380 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fa840001800 tx=0x7fa844006df0).stop 2022-07-12T08:44:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- 172.21.15.61:0/207108654 shutdown_connections 2022-07-12T08:44:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 --2- 172.21.15.61:0/207108654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa850147000 0x7fa8501474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:35.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 --2- 172.21.15.61:0/207108654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8500b5aa0 0x7fa8500b0380 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:35.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- 172.21.15.61:0/207108654 >> 172.21.15.61:0/207108654 conn(0x7fa850063520 msgr2=0x7fa8500b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:35.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- 172.21.15.61:0/207108654 shutdown_connections 2022-07-12T08:44:35.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.956+0000 7fa855f3a700 1 -- 172.21.15.61:0/207108654 wait complete. 2022-07-12T08:44:35.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 Processor -- start 2022-07-12T08:44:35.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 -- start start 2022-07-12T08:44:35.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa85005bfc0 0x7fa850149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:35.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8500b5aa0 0x7fa8500aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:35.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 0x7fa850066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:35.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8500af300 con 0x7fa850147000 2022-07-12T08:44:35.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa850065f10 con 0x7fa8500b5aa0 2022-07-12T08:44:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8500661b0 con 0x7fa85005bfc0 2022-07-12T08:44:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 0x7fa850066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 0x7fa850066650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55318/0 (socket says 172.21.15.61:55318) 2022-07-12T08:44:35.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 -- 172.21.15.61:0/1139344852 learned_addr learned my addr 172.21.15.61:0/1139344852 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:35.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84f7fe700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa85005bfc0 0x7fa850149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:35.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84effd700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8500b5aa0 0x7fa8500aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:35.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 -- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa85005bfc0 msgr2=0x7fa850149d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:35.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa85005bfc0 0x7fa850149d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:35.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 -- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8500b5aa0 msgr2=0x7fa8500aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:35.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8500b5aa0 0x7fa8500aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:35.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 -- 172.21.15.61:0/1139344852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84400b180 con 0x7fa850147000 2022-07-12T08:44:35.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84ffff700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 0x7fa850066650 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa848006e00 tx=0x7fa8480091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:35.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa84800c7f0 con 0x7fa850147000 2022-07-12T08:44:35.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84800c950 con 0x7fa850147000 2022-07-12T08:44:35.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa84800ccb0 con 0x7fa850147000 2022-07-12T08:44:35.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa850066b40 con 0x7fa850147000 2022-07-12T08:44:35.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.960+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8500ae2e0 con 0x7fa850147000 2022-07-12T08:44:35.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.964+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8500af150 con 0x7fa850147000 2022-07-12T08:44:35.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.964+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa848022070 con 0x7fa850147000 2022-07-12T08:44:35.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.964+0000 7fa84cff9700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa830053af0 0x7fa830055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:35.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.964+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa84806c4d0 con 0x7fa850147000 2022-07-12T08:44:35.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.964+0000 7fa84f7fe700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa830053af0 0x7fa830055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:35.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.964+0000 7fa84f7fe700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa830053af0 0x7fa830055fb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa844018000 tx=0x7fa844014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:35.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:35.972+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa848043430 con 0x7fa850147000 2022-07-12T08:44:36.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.096+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa850057460 con 0x7fa850147000 2022-07-12T08:44:36.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.096+0000 7fa84cff9700 1 -- 172.21.15.61:0/1139344852 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v42) v1 ==== 81+0+33693 (secure 0 0 0) 0x7fa848016090 con 0x7fa850147000 2022-07-12T08:44:36.103 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:36.103 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:44:36.104 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 42, 2022-07-12T08:44:36.104 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:44:36.104 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:44:36.104 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:43:16.617857+0000", 2022-07-12T08:44:36.105 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:44:36.105 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:44:36.105 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:44:36.106 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:44:36.106 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:44:36.106 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:44:36.106 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:44:36.107 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:44:36.107 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:44:36.107 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:36.107 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:44:36.108 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:44:36.108 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:44:36.108 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:44:36.108 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:44:36.109 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 8, 2022-07-12T08:44:36.109 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:44:36.109 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:44:36.109 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:44:36.109 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:44:36.110 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:44:36.110 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.110 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:44:36.110 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:44:36.111 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:44:36.111 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:44:36.111 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:44:36.111 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:44:36.112 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:44:36.112 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:44:36.112 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:44:36.112 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:44:36.112 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:44:36.113 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:44:36.113 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:44:36.113 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:44:36.113 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:44:36.114 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:44:36.114 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:44:36.114 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:44:36.114 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:44:36.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:44:36.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:44:36.115 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:44:36.115 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:44:36.116 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.116 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:44:36.116 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:44:36.116 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:44:36.117 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:44:36.117 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:44:36.117 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:44:36.117 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:44:36.117 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:44:36.118 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:44:36.118 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:44:36.118 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:44:36.118 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:44:36.119 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:44:36.119 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:44:36.119 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:44:36.119 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:44:36.120 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:44:36.120 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:44:36.120 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:44:36.120 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:44:36.120 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:44:36.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:44:36.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:44:36.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:44:36.121 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:44:36.122 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:44:36.122 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:44:36.122 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.122 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:44:36.123 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:44:36.123 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:44:36.123 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:44:36.123 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:44:36.123 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:44:36.124 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:44:36.124 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:44:36.124 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:44:36.124 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:44:36.125 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:44:36.125 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:44:36.125 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:44:36.126 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.126 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:44:36.126 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:44:36.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.127 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.127 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.127 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:44:36.128 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:44:36.128 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:44:36.128 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:44:36.128 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:44:36.129 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:44:36.129 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:44:36.129 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:44:36.129 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:44:36.130 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:44:36.130 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:44:36.130 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:44:36.130 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:44:36.130 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:44:36.131 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:44:36.131 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:44:36.131 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:44:36.131 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:44:36.132 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:44:36.132 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:44:36.132 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:44:36.132 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:44:36.133 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:44:36.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:44:36.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:44:36.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:44:36.134 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:44:36.134 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:44:36.134 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:44:36.134 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:44:36.135 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:44:36.135 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:44:36.135 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:44:36.135 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:44:36.135 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:44:36.136 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:44:36.136 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:44:36.136 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:44:36.136 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:44:36.137 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:44:36.137 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:44:36.137 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:44:36.137 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:44:36.137 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:44:36.138 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:44:36.138 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:44:36.138 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:44:36.138 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:44:36.139 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:44:36.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:44:36.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.139 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:44:36.139 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:44:36.140 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:44:36.140 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:44:36.140 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:44:36.140 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:44:36.141 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:44:36.141 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:44:36.141 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:44:36.141 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:44:36.142 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:44:36.142 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:44:36.142 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:44:36.142 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:44:36.142 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.143 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 7, 2022-07-12T08:44:36.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:44:36.144 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:43:13.384889+0000", 2022-07-12T08:44:36.144 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:44:36.144 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:44:36.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:44:36.145 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:44:36.145 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:44:36.145 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:44:36.145 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:44:36.145 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:44:36.146 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:44:36.146 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:44:36.146 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:44:36.146 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:44:36.147 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:44:36.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:44:36.147 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:44:36.147 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:44:36.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:44:36.148 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:44:36.148 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:44:36.148 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:44:36.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "41", 2022-07-12T08:44:36.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:44:36.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:44:36.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:44:36.150 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:44:36.150 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:44:36.150 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:44:36.150 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:44:36.151 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:44:36.151 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:44:36.151 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:44:36.151 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:44:36.151 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:44:36.152 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:44:36.152 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:44:36.152 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:44:36.152 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:44:36.153 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:44:36.153 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:44:36.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:44:36.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:44:36.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:44:36.154 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:44:36.154 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:44:36.154 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:44:36.154 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:44:36.155 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:44:36.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.155 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:44:36.155 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:44:36.155 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:44:36.156 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:44:36.156 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:44:36.156 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:44:36.156 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:44:36.157 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:44:36.157 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:44:36.157 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:44:36.157 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:44:36.157 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:44:36.158 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:44:36.158 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:44:36.158 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:44:36.158 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.159 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:44:36.159 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:44:36.159 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:44:36.159 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.159 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.160 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.160 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 8, 2022-07-12T08:44:36.160 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:44:36.161 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:43:14.650247+0000", 2022-07-12T08:44:36.161 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:44:36.161 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:44:36.162 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:44:36.162 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:44:36.162 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:44:36.162 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:44:36.163 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:44:36.163 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:44:36.163 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:44:36.163 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:44:36.163 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:44:36.164 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:44:36.164 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:44:36.164 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:44:36.164 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:44:36.165 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:44:36.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:44:36.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:44:36.165 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:44:36.166 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:44:36.166 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.166 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "42", 2022-07-12T08:44:36.166 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:44:36.166 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:44:36.167 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:44:36.167 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:44:36.167 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:44:36.167 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:44:36.168 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:44:36.168 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:44:36.168 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:44:36.168 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:44:36.169 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:44:36.169 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:44:36.169 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:44:36.169 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:44:36.169 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:44:36.170 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:44:36.170 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:44:36.170 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:44:36.170 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:44:36.171 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:44:36.171 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:44:36.171 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:44:36.171 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:44:36.172 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:44:36.172 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:44:36.172 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:44:36.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.173 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:44:36.173 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:44:36.173 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:44:36.173 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:44:36.173 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:44:36.174 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:44:36.174 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:44:36.175 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:44:36.175 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:44:36.176 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:44:36.176 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:44:36.176 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:44:36.176 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:44:36.177 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:44:36.177 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:44:36.177 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.178 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.178 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.179 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:36.179 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:44:36.179 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.179 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:44:36.180 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:44:36.180 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.181 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.181 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.182 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.182 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.183 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.183 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.183 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:36.184 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.184 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.184 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.185 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.185 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:44:36.185 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.186 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.186 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.186 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.186 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:44:36.187 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.187 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.187 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.188 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:44:36.190 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.190 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.190 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.191 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.191 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:44:36.192 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.192 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.193 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.193 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.194 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.194 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.195 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:44:36.196 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.196 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:44:36.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.198 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.199 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.200 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.200 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.201 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.201 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:44:36.202 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.202 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.203 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:44:36.204 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:36.204 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.205 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.206 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:44:36.206 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:44:36.207 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:44:36.207 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:44:36.207 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.208 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.208 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.209 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.210 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:44:36.210 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:44:36.211 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.211 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.212 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.212 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.213 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.213 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.213 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.214 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:36.215 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.215 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.215 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:44:36.218 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.218 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:44:36.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.220 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.220 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.221 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.221 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.222 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.222 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.222 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:44:36.222 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.223 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.223 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.223 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:44:36.223 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.224 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.224 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.225 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.225 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.225 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.225 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.226 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:44:36.226 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.226 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.226 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.227 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:44:36.227 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.227 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.227 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.228 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:44:36.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:44:36.230 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:36.230 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.231 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.231 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.231 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:44:36.231 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:44:36.231 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:44:36.232 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:44:36.232 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.232 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.232 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.233 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.233 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.233 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.233 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:44:36.233 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:44:36.234 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.234 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.234 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.234 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.235 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.235 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 28, 2022-07-12T08:44:36.235 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.236 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.236 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:44:36.237 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.238 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.238 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:44:36.238 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.238 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.239 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.239 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.240 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:44:36.240 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.241 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.241 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:44:36.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.242 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.242 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.243 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:44:36.244 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:44:36.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.246 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.246 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.247 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.248 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:44:36.249 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.250 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:44:36.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:36.251 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.251 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.251 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:44:36.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:44:36.252 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:44:36.252 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:44:36.252 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.252 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.253 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.253 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.253 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:44:36.254 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:44:36.254 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.254 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.254 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.255 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.255 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.255 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:36.256 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.256 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.256 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.257 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.257 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.257 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:44:36.257 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.258 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.258 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:44:36.258 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.259 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.259 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.259 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.260 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.260 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:44:36.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.261 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.261 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:44:36.261 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.262 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.262 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.262 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.262 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.263 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:44:36.263 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:44:36.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.265 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.265 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.265 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.266 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.266 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:44:36.266 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.267 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:44:36.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:36.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.268 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.268 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.268 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:44:36.269 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:44:36.269 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:44:36.269 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:44:36.269 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.269 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.270 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.270 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.270 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.270 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.271 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:44:36.271 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:44:36.271 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.271 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.271 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.272 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.272 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.273 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:36.273 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.273 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.273 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.273 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.274 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.274 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:44:36.274 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.275 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.275 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.275 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:44:36.276 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.276 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.276 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.276 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.277 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:44:36.278 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:44:36.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.279 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.280 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.280 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:44:36.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:44:36.282 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.282 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.283 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.283 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.284 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:44:36.284 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:44:36.285 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:36.286 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.286 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.286 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:44:36.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:44:36.287 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:44:36.287 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:44:36.287 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.287 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.288 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.288 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.288 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.289 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:44:36.289 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:44:36.289 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.289 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.290 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.290 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.290 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.291 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:36.291 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.291 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.291 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.292 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.292 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.292 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:44:36.293 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.293 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.293 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.293 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.294 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:44:36.294 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.294 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.294 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.295 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.295 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.295 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.295 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.296 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:44:36.296 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.296 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.297 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:44:36.297 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.297 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.297 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.298 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.298 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.299 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:44:36.299 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:44:36.301 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.301 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.301 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.302 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.302 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.302 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.303 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.303 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:44:36.303 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.304 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:44:36.304 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:36.305 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.305 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.305 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:44:36.305 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:44:36.306 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:44:36.306 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:44:36.306 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.306 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.306 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.307 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.307 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.308 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:44:36.308 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:44:36.308 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.308 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.309 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.309 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.309 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.309 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.309 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.310 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:36.310 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.310 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.310 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.311 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.311 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:44:36.312 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.312 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.312 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:44:36.313 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.313 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.313 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.313 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.314 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.314 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.314 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:44:36.315 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.315 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:44:36.316 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.316 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.316 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.317 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.317 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.317 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.317 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.318 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:44:36.318 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.318 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.318 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.319 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:44:36.319 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.319 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.319 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.319 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.320 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.320 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.320 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.320 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:44:36.321 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.321 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.321 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.322 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:44:36.322 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:36.322 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.322 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.323 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.323 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:44:36.323 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:44:36.323 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:44:36.323 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:44:36.324 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.324 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.324 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.324 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.325 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:44:36.325 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:44:36.325 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:36.326 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:36.326 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:36.326 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:36.326 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:36.326 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:36.327 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:36.327 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 39, 2022-07-12T08:44:36.327 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:36.327 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:36.328 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:36.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.328 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.328 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:44:36.329 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.329 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.330 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:44:36.330 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.330 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.330 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.331 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:36.331 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.331 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.331 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.332 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:44:36.332 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.332 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.332 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:44:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.333 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.334 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.334 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:36.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.334 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.335 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:44:36.335 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.335 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.335 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.336 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.336 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:44:36.336 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.336 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.336 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.337 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.337 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:36.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:36.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.338 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:36.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:44:36.338 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.338 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.338 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.339 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:36.339 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:44:36.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:36.339 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.339 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.340 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:44:36.340 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:44:36.340 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:44:36.341 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:44:36.341 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:36.341 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:36.341 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:36.341 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:36.342 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.342 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:36.342 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:44:36.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.342 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:44:36.343 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.343 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.343 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.343 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.344 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.344 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:44:36.344 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.344 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.345 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:44:36.345 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.345 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.345 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.345 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.346 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.346 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:44:36.346 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.346 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.347 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.347 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:44:36.347 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.347 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.348 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.348 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.348 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.348 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:44:36.349 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.349 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.349 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:44:36.349 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.350 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.350 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.350 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.350 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.351 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:44:36.351 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.351 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.351 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.351 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:44:36.352 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.352 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.352 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.352 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.353 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.353 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:44:36.353 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.353 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.354 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:44:36.354 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.354 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.354 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.355 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.355 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.355 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:44:36.355 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.355 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.356 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.356 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:44:36.356 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.356 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.357 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.357 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.357 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.357 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:44:36.357 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:36.358 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:44:36.358 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:36.358 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:36.359 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:36.359 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:36.359 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:36.359 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:44:36.360 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:36.360 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.360 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:36.360 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:44:36.361 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:44:36.361 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:44:36.361 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:44:36.361 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:44:36.361 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:44:36.362 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:44:36.362 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:44:36.362 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:44:36.362 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:44:36.363 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:44:36.363 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:44:36.363 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:44:36.363 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:36.364 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:36.364 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:44:36.364 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:44:36.364 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:44:36.364 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:44:36.365 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:44:36.366 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:44:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa830053af0 msgr2=0x7fa830055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa830053af0 0x7fa830055fb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa844018000 tx=0x7fa844014000).stop 2022-07-12T08:44:36.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 msgr2=0x7fa850066650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 0x7fa850066650 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fa848006e00 tx=0x7fa8480091d0).stop 2022-07-12T08:44:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 shutdown_connections 2022-07-12T08:44:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa850147000 0x7fa850066650 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa830053af0 0x7fa830055fb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8500b5aa0 0x7fa8500aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 --2- 172.21.15.61:0/1139344852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa85005bfc0 0x7fa850149d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 >> 172.21.15.61:0/1139344852 conn(0x7fa850063520 msgr2=0x7fa8500aad20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 shutdown_connections 2022-07-12T08:44:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.100+0000 7fa855f3a700 1 -- 172.21.15.61:0/1139344852 wait complete. 2022-07-12T08:44:36.374 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:36.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 Processor -- start 2022-07-12T08:44:36.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- start start 2022-07-12T08:44:36.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fef740ab290 0x7fef740ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fef740afa60 0x7fef740afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef740abc70 con 0x7fef7405f140 2022-07-12T08:44:36.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7405f580 con 0x7fef740ab290 2022-07-12T08:44:36.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7405c720 con 0x7fef740afa60 2022-07-12T08:44:36.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fef740ab290 0x7fef740ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef71d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fef7405f140 0x7fef7405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43544/0 (socket says 172.21.15.61:43544) 2022-07-12T08:44:36.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef71d9b700 1 -- 172.21.15.61:0/1614040680 learned_addr learned my addr 172.21.15.61:0/1614040680 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:36.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef72d9d700 1 --2- 172.21.15.61:0/1614040680 >> v2:172.21.15.111:3300/0 conn(0x7fef740afa60 0x7fef740afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7259c700 1 -- 172.21.15.61:0/1614040680 >> v2:172.21.15.111:3300/0 conn(0x7fef740afa60 msgr2=0x7fef740afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:36.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7259c700 1 --2- 172.21.15.61:0/1614040680 >> v2:172.21.15.111:3300/0 conn(0x7fef740afa60 0x7fef740afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7259c700 1 -- 172.21.15.61:0/1614040680 >> v1:172.21.15.61:6789/0 conn(0x7fef7405f140 legacy=0x7fef7405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7159a700 1 -- 172.21.15.61:0/1614040680 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964271138 0 0) 0x7fef740abc70 con 0x7fef7405f140 2022-07-12T08:44:36.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7259c700 1 -- 172.21.15.61:0/1614040680 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7405c9c0 con 0x7fef740ab290 2022-07-12T08:44:36.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7259c700 1 --2- 172.21.15.61:0/1614040680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef740ab290 0x7fef740ab670 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fef6c0032f0 tx=0x7fef6c0091d0).ready entity=mon.0 client_cookie=bc329a134b7b525f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:36.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7159a700 1 -- 172.21.15.61:0/1614040680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6c009ea0 con 0x7fef740ab290 2022-07-12T08:44:36.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7159a700 1 -- 172.21.15.61:0/1614040680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c00b870 con 0x7fef740ab290 2022-07-12T08:44:36.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7159a700 1 -- 172.21.15.61:0/1614040680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6c00b9e0 con 0x7fef740ab290 2022-07-12T08:44:36.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- 172.21.15.61:0/1614040680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef740ab290 msgr2=0x7fef740ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 --2- 172.21.15.61:0/1614040680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef740ab290 0x7fef740ab670 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fef6c0032f0 tx=0x7fef6c0091d0).stop 2022-07-12T08:44:36.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- 172.21.15.61:0/1614040680 shutdown_connections 2022-07-12T08:44:36.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 --2- 172.21.15.61:0/1614040680 >> v2:172.21.15.111:3300/0 conn(0x7fef740afa60 0x7fef740afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 --2- 172.21.15.61:0/1614040680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef740ab290 0x7fef740ab670 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- 172.21.15.61:0/1614040680 >> 172.21.15.61:0/1614040680 conn(0x7fef740b1950 msgr2=0x7fef740b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:36.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- 172.21.15.61:0/1614040680 shutdown_connections 2022-07-12T08:44:36.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.488+0000 7fef7359e700 1 -- 172.21.15.61:0/1614040680 wait complete. 2022-07-12T08:44:36.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 Processor -- start 2022-07-12T08:44:36.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 -- start start 2022-07-12T08:44:36.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 0x7fef74099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fef740ab290 0x7fef74066af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef740afa60 0x7fef741598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef741466e0 con 0x7fef7405f140 2022-07-12T08:44:36.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef74066fe0 con 0x7fef740afa60 2022-07-12T08:44:36.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740657a0 con 0x7fef740ab290 2022-07-12T08:44:36.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 0x7fef74099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 0x7fef74099890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55330/0 (socket says 172.21.15.61:55330) 2022-07-12T08:44:36.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 -- 172.21.15.61:0/1054924080 learned_addr learned my addr 172.21.15.61:0/1054924080 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:36.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef71d9b700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fef740ab290 0x7fef74066af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef72d9d700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef740afa60 0x7fef741598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 -- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fef740ab290 msgr2=0x7fef74066af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fef740ab290 0x7fef74066af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 -- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef740afa60 msgr2=0x7fef741598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef740afa60 0x7fef741598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 -- 172.21.15.61:0/1054924080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c009470 con 0x7fef7405f140 2022-07-12T08:44:36.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7259c700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 0x7fef74099890 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fef6c0032f0 tx=0x7fef6c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:36.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6c009ea0 con 0x7fef7405f140 2022-07-12T08:44:36.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c01f3f0 con 0x7fef7405f140 2022-07-12T08:44:36.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef74065a40 con 0x7fef7405f140 2022-07-12T08:44:36.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef6c01f750 con 0x7fef7405f140 2022-07-12T08:44:36.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.492+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef741592f0 con 0x7fef7405f140 2022-07-12T08:44:36.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.496+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef740667c0 con 0x7fef7405f140 2022-07-12T08:44:36.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.496+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fef6c026030 con 0x7fef7405f140 2022-07-12T08:44:36.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.496+0000 7fef6b7fe700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fef580539b0 0x7fef58055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.496+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fef6c06ca90 con 0x7fef7405f140 2022-07-12T08:44:36.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.496+0000 7fef71d9b700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fef580539b0 0x7fef58055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.496+0000 7fef71d9b700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fef580539b0 0x7fef58055e70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fef5c00d000 tx=0x7fef5c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:36.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.504+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef740667c0 con 0x7fef7405f140 2022-07-12T08:44:36.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.668+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef740667c0 con 0x7fef7405f140 2022-07-12T08:44:36.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.668+0000 7fef6b7fe700 1 -- 172.21.15.61:0/1054924080 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+2141 (secure 0 0 0) 0x7fef740667c0 con 0x7fef7405f140 2022-07-12T08:44:36.675 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:36.675 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:43:16.618859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"a","rank":0,"incarnation":16,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4199688387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4199688387},{"type":"v1","addr":"172.21.15.61:6833","nonce":4199688387}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:44:36.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fef580539b0 msgr2=0x7fef58055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fef580539b0 0x7fef58055e70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fef5c00d000 tx=0x7fef5c009040).stop 2022-07-12T08:44:36.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 msgr2=0x7fef74099890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 0x7fef74099890 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fef6c0032f0 tx=0x7fef6c016000).stop 2022-07-12T08:44:36.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 shutdown_connections 2022-07-12T08:44:36.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fef580539b0 0x7fef58055e70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef740afa60 0x7fef741598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fef740ab290 0x7fef74066af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 --2- 172.21.15.61:0/1054924080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef7405f140 0x7fef74099890 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 >> 172.21.15.61:0/1054924080 conn(0x7fef740b1950 msgr2=0x7fef740b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:36.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 shutdown_connections 2022-07-12T08:44:36.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.672+0000 7fef7359e700 1 -- 172.21.15.61:0/1054924080 wait complete. 2022-07-12T08:44:36.683 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-12T08:44:36.688 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:44:36.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 Processor -- start 2022-07-12T08:44:36.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 -- start start 2022-07-12T08:44:36.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 0x7f43ac09a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43ac05f140 0x7f43ac09a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac067eb0 con 0x7f43ac09ad40 2022-07-12T08:44:36.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ac05f650 con 0x7f43ac05f140 2022-07-12T08:44:36.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43b1694700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0b0530 con 0x7f43ac0b0150 2022-07-12T08:44:36.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 0x7f43ac09a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43ab7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43ac05f140 0x7f43ac09a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 0x7f43ac09a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:37534/0 (socket says 172.21.15.61:37534) 2022-07-12T08:44:36.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 -- 172.21.15.61:0/2195546766 learned_addr learned my addr 172.21.15.61:0/2195546766 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:36.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 -- 172.21.15.61:0/2195546766 >> v2:172.21.15.111:3300/0 conn(0x7f43ac05f140 msgr2=0x7f43ac09a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:36.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 --2- 172.21.15.61:0/2195546766 >> v2:172.21.15.111:3300/0 conn(0x7f43ac05f140 0x7f43ac09a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 -- 172.21.15.61:0/2195546766 >> v1:172.21.15.111:6789/0 conn(0x7f43ac09ad40 legacy=0x7f43ac144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 -- 172.21.15.61:0/2195546766 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac067c10 con 0x7f43ac0b0150 2022-07-12T08:44:36.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.784+0000 7f43abfff700 1 --2- 172.21.15.61:0/2195546766 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 0x7f43ac09a360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f43a40032f0 tx=0x7f43a4009290).ready entity=mon.2 client_cookie=8d84a468c87b5c69 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:36.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43aaffd700 1 -- 172.21.15.61:0/2195546766 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43a400b8a0 con 0x7f43ac0b0150 2022-07-12T08:44:36.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43aaffd700 1 -- 172.21.15.61:0/2195546766 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a400ba00 con 0x7f43ac0b0150 2022-07-12T08:44:36.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43aaffd700 1 -- 172.21.15.61:0/2195546766 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43a400bb70 con 0x7f43ac0b0150 2022-07-12T08:44:36.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 -- 172.21.15.61:0/2195546766 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 msgr2=0x7f43ac09a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 --2- 172.21.15.61:0/2195546766 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 0x7f43ac09a360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f43a40032f0 tx=0x7f43a4009290).stop 2022-07-12T08:44:36.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 -- 172.21.15.61:0/2195546766 shutdown_connections 2022-07-12T08:44:36.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 --2- 172.21.15.61:0/2195546766 >> v2:172.21.15.111:3300/0 conn(0x7f43ac05f140 0x7f43ac09a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 --2- 172.21.15.61:0/2195546766 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac0b0150 0x7f43ac09a360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 -- 172.21.15.61:0/2195546766 >> 172.21.15.61:0/2195546766 conn(0x7f43ac0b1950 msgr2=0x7f43ac0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:36.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 -- 172.21.15.61:0/2195546766 shutdown_connections 2022-07-12T08:44:36.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 -- 172.21.15.61:0/2195546766 wait complete. 2022-07-12T08:44:36.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 Processor -- start 2022-07-12T08:44:36.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.788+0000 7f43b1694700 1 -- start start 2022-07-12T08:44:36.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43ac05f140 0x7f43ac14fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac09ad40 0x7f43ac145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 0x7f43ac066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0adcd0 con 0x7f43ac0b0150 2022-07-12T08:44:36.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0af170 con 0x7f43ac05f140 2022-07-12T08:44:36.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac13e2d0 con 0x7f43ac09ad40 2022-07-12T08:44:36.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 0x7f43ac066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43ac05f140 0x7f43ac14fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43ab7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac09ad40 0x7f43ac145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 0x7f43ac066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55348/0 (socket says 172.21.15.61:55348) 2022-07-12T08:44:36.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43ac05f140 0x7f43ac14fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:33918/0 (socket says 172.21.15.61:33918) 2022-07-12T08:44:36.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 -- 172.21.15.61:0/4260739186 learned_addr learned my addr 172.21.15.61:0/4260739186 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:36.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 -- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac09ad40 msgr2=0x7f43ac145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac09ad40 0x7f43ac145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 -- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43ac05f140 msgr2=0x7f43ac14fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:36.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43ac05f140 0x7f43ac14fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:36.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 -- 172.21.15.61:0/4260739186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a4009530 con 0x7f43ac0b0150 2022-07-12T08:44:36.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b0e93700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 0x7f43ac066cd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f43ac0615d0 tx=0x7f439c007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:36.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f439c012400 con 0x7f43ac0b0150 2022-07-12T08:44:36.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c012560 con 0x7f43ac0b0150 2022-07-12T08:44:36.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac13e570 con 0x7f43ac0b0150 2022-07-12T08:44:36.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f439c0128c0 con 0x7f43ac0b0150 2022-07-12T08:44:36.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.792+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac063860 con 0x7f43ac0b0150 2022-07-12T08:44:36.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.796+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ac057460 con 0x7f43ac0b0150 2022-07-12T08:44:36.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.796+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f439c019030 con 0x7f43ac0b0150 2022-07-12T08:44:36.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.796+0000 7f43a97fa700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43900539b0 0x7f4390055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:36.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.796+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f439c06b1b0 con 0x7f43ac0b0150 2022-07-12T08:44:36.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.796+0000 7f43abfff700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43900539b0 0x7f4390055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:36.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.796+0000 7f43abfff700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43900539b0 0x7f4390055e70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f43a401b000 tx=0x7f43a4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:36.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.800+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f439c042110 con 0x7f43ac0b0150 2022-07-12T08:44:36.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:36.968+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f43ac0609a0 con 0x7f43ac0b0150 2022-07-12T08:44:37.210 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:44:37.208+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:44:37.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.220+0000 7f43a97fa700 1 -- 172.21.15.61:0/4260739186 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7f439c045730 con 0x7f43ac0b0150 2022-07-12T08:44:37.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43900539b0 msgr2=0x7f4390055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43900539b0 0x7f4390055e70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f43a401b000 tx=0x7f43a4016000).stop 2022-07-12T08:44:37.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 msgr2=0x7f43ac066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 0x7f43ac066cd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f43ac0615d0 tx=0x7f439c007bc0).stop 2022-07-12T08:44:37.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 shutdown_connections 2022-07-12T08:44:37.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43ac0b0150 0x7f43ac066cd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43ac09ad40 0x7f43ac145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43900539b0 0x7f4390055e70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 --2- 172.21.15.61:0/4260739186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43ac05f140 0x7f43ac14fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 >> 172.21.15.61:0/4260739186 conn(0x7f43ac0b1950 msgr2=0x7f43ac0ab120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:37.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 shutdown_connections 2022-07-12T08:44:37.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.224+0000 7f43b1694700 1 -- 172.21.15.61:0/4260739186 wait complete. 2022-07-12T08:44:37.249 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:44:37.250 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:37.263 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:44:37.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 Processor -- start 2022-07-12T08:44:37.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 -- start start 2022-07-12T08:44:37.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a881440a0 0x7f1a8805bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a88144480 0x7f1a881413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a880b3010 con 0x7f1a881418d0 2022-07-12T08:44:37.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880624b0 con 0x7f1a88144480 2022-07-12T08:44:37.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a8cc21700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a88144da0 con 0x7f1a881440a0 2022-07-12T08:44:37.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a88144480 0x7f1a881413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a88144480 0x7f1a881413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55364/0 (socket says 172.21.15.61:55364) 2022-07-12T08:44:37.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 -- 172.21.15.61:0/1361439989 learned_addr learned my addr 172.21.15.61:0/1361439989 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:37.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a877fe700 1 --2- 172.21.15.61:0/1361439989 >> v2:172.21.15.111:3300/0 conn(0x7f1a881440a0 0x7f1a8805bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 -- 172.21.15.61:0/1361439989 >> v2:172.21.15.111:3300/0 conn(0x7f1a881440a0 msgr2=0x7f1a8805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:37.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 --2- 172.21.15.61:0/1361439989 >> v2:172.21.15.111:3300/0 conn(0x7f1a881440a0 0x7f1a8805bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 -- 172.21.15.61:0/1361439989 >> v1:172.21.15.111:6789/0 conn(0x7f1a881418d0 legacy=0x7f1a8813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:37.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 -- 172.21.15.61:0/1361439989 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880b2d70 con 0x7f1a88144480 2022-07-12T08:44:37.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a877fe700 1 --2- 172.21.15.61:0/1361439989 >> v2:172.21.15.111:3300/0 conn(0x7f1a881440a0 0x7f1a8805bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:44:37.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.352+0000 7f1a7effd700 1 --2- 172.21.15.61:0/1361439989 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a88144480 0x7f1a881413e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f1a880b7000 tx=0x7f1a70006eb0).ready entity=mon.0 client_cookie=b19129159bfea40b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:37.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a86ffd700 1 -- 172.21.15.61:0/1361439989 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7000d5c0 con 0x7f1a88144480 2022-07-12T08:44:37.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a86ffd700 1 -- 172.21.15.61:0/1361439989 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a70000e90 con 0x7f1a88144480 2022-07-12T08:44:37.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a86ffd700 1 -- 172.21.15.61:0/1361439989 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a7000d910 con 0x7f1a88144480 2022-07-12T08:44:37.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 -- 172.21.15.61:0/1361439989 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a88144480 msgr2=0x7f1a881413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/1361439989 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a88144480 0x7f1a881413e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f1a880b7000 tx=0x7f1a70006eb0).stop 2022-07-12T08:44:37.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 -- 172.21.15.61:0/1361439989 shutdown_connections 2022-07-12T08:44:37.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/1361439989 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a88144480 0x7f1a881413e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/1361439989 >> v2:172.21.15.111:3300/0 conn(0x7f1a881440a0 0x7f1a8805bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 -- 172.21.15.61:0/1361439989 >> 172.21.15.61:0/1361439989 conn(0x7f1a880b70b0 msgr2=0x7f1a880b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:37.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 -- 172.21.15.61:0/1361439989 shutdown_connections 2022-07-12T08:44:37.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 -- 172.21.15.61:0/1361439989 wait complete. 2022-07-12T08:44:37.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 Processor -- start 2022-07-12T08:44:37.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 -- start start 2022-07-12T08:44:37.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 0x7f1a88146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.356+0000 7f1a8cc21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a881440a0 0x7f1a881430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a88144480 0x7f1a880aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88146c80 con 0x7f1a881418d0 2022-07-12T08:44:37.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880ae4c0 con 0x7f1a881440a0 2022-07-12T08:44:37.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880ae760 con 0x7f1a88144480 2022-07-12T08:44:37.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 0x7f1a88146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 0x7f1a88146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55370/0 (socket says 172.21.15.61:55370) 2022-07-12T08:44:37.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 -- 172.21.15.61:0/436943477 learned_addr learned my addr 172.21.15.61:0/436943477 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:37.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a7effd700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a881440a0 0x7f1a881430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a87fff700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a88144480 0x7f1a880aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 -- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a88144480 msgr2=0x7f1a880aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a88144480 0x7f1a880aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 -- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a881440a0 msgr2=0x7f1a881430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a881440a0 0x7f1a881430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 -- 172.21.15.61:0/436943477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7000b1e0 con 0x7f1a881418d0 2022-07-12T08:44:37.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a877fe700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 0x7f1a88146650 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f1a78007000 tx=0x7f1a78008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:37.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a78010580 con 0x7f1a881418d0 2022-07-12T08:44:37.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a780106e0 con 0x7f1a881418d0 2022-07-12T08:44:37.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a880af0f0 con 0x7f1a881418d0 2022-07-12T08:44:37.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a78010a40 con 0x7f1a881418d0 2022-07-12T08:44:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a88146f60 con 0x7f1a881418d0 2022-07-12T08:44:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1a78010ba0 con 0x7f1a881418d0 2022-07-12T08:44:37.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a88057460 con 0x7f1a881418d0 2022-07-12T08:44:37.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.360+0000 7f1a857fa700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a6c053920 0x7f1a6c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.364+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f1a7806bfe0 con 0x7f1a881418d0 2022-07-12T08:44:37.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.364+0000 7f1a7effd700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a6c053920 0x7f1a6c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.364+0000 7f1a7effd700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a6c053920 0x7f1a6c055de0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f1a7001c000 tx=0x7f1a70015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:37.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.368+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a78042de0 con 0x7f1a881418d0 2022-07-12T08:44:37.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.532+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a880b1fb0 con 0x7f1a881418d0 2022-07-12T08:44:37.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a857fa700 1 -- 172.21.15.61:0/436943477 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+1780 (secure 0 0 0) 0x7f1a78046400 con 0x7f1a881418d0 2022-07-12T08:44:37.541 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:37.541 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":18,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15}],"filesystems":[{"mdsmap":{"epoch":18,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:43:15.615120+0000","modified":"2022-07-12T08:44:37.211308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-12T08:44:37.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a6c053920 msgr2=0x7f1a6c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a6c053920 0x7f1a6c055de0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f1a7001c000 tx=0x7f1a70015000).stop 2022-07-12T08:44:37.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 msgr2=0x7f1a88146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 0x7f1a88146650 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f1a78007000 tx=0x7f1a78008040).stop 2022-07-12T08:44:37.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 shutdown_connections 2022-07-12T08:44:37.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a88144480 0x7f1a880aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a6c053920 0x7f1a6c055de0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a881440a0 0x7f1a881430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 --2- 172.21.15.61:0/436943477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a881418d0 0x7f1a88146650 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 >> 172.21.15.61:0/436943477 conn(0x7f1a880b70b0 msgr2=0x7f1a880b63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:37.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 shutdown_connections 2022-07-12T08:44:37.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.536+0000 7f1a8cc21700 1 -- 172.21.15.61:0/436943477 wait complete. 2022-07-12T08:44:37.549 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 18 2022-07-12T08:44:37.556 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:44:37.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 Processor -- start 2022-07-12T08:44:37.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 -- start start 2022-07-12T08:44:37.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b52e0 0x7f2a000b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2a00141e50 0x7f2a001422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a000b56c0 con 0x7f2a000b4050 2022-07-12T08:44:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a00142f00 con 0x7f2a00141e50 2022-07-12T08:44:37.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a0853c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000b1320 con 0x7f2a000b52e0 2022-07-12T08:44:37.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a06ad9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2a00141e50 0x7f2a001422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a05ad7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2a000b4050 0x7f2a000b26a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43596/0 (socket says 172.21.15.61:43596) 2022-07-12T08:44:37.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a05ad7700 1 -- 172.21.15.61:0/1093759056 learned_addr learned my addr 172.21.15.61:0/1093759056 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a062d8700 1 --2- 172.21.15.61:0/1093759056 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b52e0 0x7f2a000b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a052d6700 1 -- 172.21.15.61:0/1093759056 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353241737 0 0) 0x7f2a000b56c0 con 0x7f2a000b4050 2022-07-12T08:44:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a052d6700 1 -- 172.21.15.61:0/1093759056 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003260 con 0x7f2a000b4050 2022-07-12T08:44:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a06ad9700 1 -- 172.21.15.61:0/1093759056 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b52e0 msgr2=0x7f2a000b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a06ad9700 1 --2- 172.21.15.61:0/1093759056 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b52e0 0x7f2a000b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a06ad9700 1 -- 172.21.15.61:0/1093759056 >> v1:172.21.15.61:6789/0 conn(0x7f2a000b4050 legacy=0x7f2a000b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.664+0000 7f2a06ad9700 1 -- 172.21.15.61:0/1093759056 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a000b4bd0 con 0x7f2a00141e50 2022-07-12T08:44:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a06ad9700 1 --2- 172.21.15.61:0/1093759056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a00141e50 0x7f2a001422c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2a000b1460 tx=0x7f29f80091d0).ready entity=mon.0 client_cookie=ffcd5426836e20cd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a052d6700 1 -- 172.21.15.61:0/1093759056 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29f800b600 con 0x7f2a00141e50 2022-07-12T08:44:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a052d6700 1 -- 172.21.15.61:0/1093759056 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f800b760 con 0x7f2a00141e50 2022-07-12T08:44:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a052d6700 1 -- 172.21.15.61:0/1093759056 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29f800b8d0 con 0x7f2a00141e50 2022-07-12T08:44:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 -- 172.21.15.61:0/1093759056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a00141e50 msgr2=0x7f2a001422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 --2- 172.21.15.61:0/1093759056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a00141e50 0x7f2a001422c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2a000b1460 tx=0x7f29f80091d0).stop 2022-07-12T08:44:37.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 -- 172.21.15.61:0/1093759056 shutdown_connections 2022-07-12T08:44:37.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 --2- 172.21.15.61:0/1093759056 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a00141e50 0x7f2a001422c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 --2- 172.21.15.61:0/1093759056 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b52e0 0x7f2a000b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 -- 172.21.15.61:0/1093759056 >> 172.21.15.61:0/1093759056 conn(0x7f2a00067020 msgr2=0x7f2a00067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:37.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 -- 172.21.15.61:0/1093759056 shutdown_connections 2022-07-12T08:44:37.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 -- 172.21.15.61:0/1093759056 wait complete. 2022-07-12T08:44:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 Processor -- start 2022-07-12T08:44:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.668+0000 7f2a0853c700 1 -- start start 2022-07-12T08:44:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b4050 0x7f2a000b8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 0x7f2a00151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a00141e50 0x7f2a00155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001453b0 con 0x7f2a000b52e0 2022-07-12T08:44:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001485a0 con 0x7f2a00141e50 2022-07-12T08:44:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a001555a0 con 0x7f2a000b4050 2022-07-12T08:44:37.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 0x7f2a00151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 0x7f2a00151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55386/0 (socket says 172.21.15.61:55386) 2022-07-12T08:44:37.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 -- 172.21.15.61:0/994410916 learned_addr learned my addr 172.21.15.61:0/994410916 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:37.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a062d8700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b4050 0x7f2a000b8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a06ad9700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a00141e50 0x7f2a00155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 -- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b4050 msgr2=0x7f2a000b8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b4050 0x7f2a000b8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 -- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a00141e50 msgr2=0x7f2a00155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:37.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a00141e50 0x7f2a00155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:37.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 -- 172.21.15.61:0/994410916 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f8009470 con 0x7f2a000b52e0 2022-07-12T08:44:37.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a06ad9700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a00141e50 0x7f2a00155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:37.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a05ad7700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 0x7f2a00151e50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f29f0001110 tx=0x7f29f0007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:37.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29f00072e0 con 0x7f2a000b52e0 2022-07-12T08:44:37.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0007440 con 0x7f2a000b52e0 2022-07-12T08:44:37.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a00152340 con 0x7f2a000b52e0 2022-07-12T08:44:37.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29f0005800 con 0x7f2a000b52e0 2022-07-12T08:44:37.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.672+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a00144af0 con 0x7f2a000b52e0 2022-07-12T08:44:37.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.676+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f29f0010070 con 0x7f2a000b52e0 2022-07-12T08:44:37.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.676+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a001487a0 con 0x7f2a000b52e0 2022-07-12T08:44:37.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.676+0000 7f29ef7fe700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f29e0053af0 0x7f29e0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:37.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.676+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f29f006b5c0 con 0x7f2a000b52e0 2022-07-12T08:44:37.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.676+0000 7f2a062d8700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f29e0053af0 0x7f29e0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:37.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.676+0000 7f2a062d8700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f29e0053af0 0x7f29e0055fb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f29fc00b000 tx=0x7f29fc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:37.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.680+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29f00424d0 con 0x7f2a000b52e0 2022-07-12T08:44:37.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:37.848+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2a00057460 con 0x7f2a000b52e0 2022-07-12T08:44:38.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.224+0000 7f29ef7fe700 1 -- 172.21.15.61:0/994410916 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f29f0045af0 con 0x7f2a000b52e0 2022-07-12T08:44:38.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f29e0053af0 msgr2=0x7f29e0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:38.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f29e0053af0 0x7f29e0055fb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f29fc00b000 tx=0x7f29fc006df0).stop 2022-07-12T08:44:38.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 msgr2=0x7f2a00151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:38.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 0x7f2a00151e50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f29f0001110 tx=0x7f29f0007ad0).stop 2022-07-12T08:44:38.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 shutdown_connections 2022-07-12T08:44:38.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f29e0053af0 0x7f29e0055fb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a00141e50 0x7f2a00155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a000b52e0 0x7f2a00151e50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 --2- 172.21.15.61:0/994410916 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a000b4050 0x7f2a000b8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 >> 172.21.15.61:0/994410916 conn(0x7f2a00067020 msgr2=0x7f2a000b1ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:38.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 shutdown_connections 2022-07-12T08:44:38.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.228+0000 7f2a0853c700 1 -- 172.21.15.61:0/994410916 wait complete. 2022-07-12T08:44:38.248 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:44:38.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.360+0000 7f05d04d4700 1 Processor -- start 2022-07-12T08:44:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.360+0000 7f05d04d4700 1 -- start start 2022-07-12T08:44:38.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05c80b36c0 0x7f05c8143090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:38.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8143960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c8143580 con 0x7f05c8143e50 2022-07-12T08:44:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c805f610 con 0x7f05c80b36c0 2022-07-12T08:44:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c805f8b0 con 0x7f05c80a9b40 2022-07-12T08:44:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05c80b36c0 0x7f05c8143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05c80b36c0 0x7f05c8143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55392/0 (socket says 172.21.15.61:55392) 2022-07-12T08:44:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 -- 172.21.15.61:0/78632458 learned_addr learned my addr 172.21.15.61:0/78632458 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05cda6f700 1 --2- 172.21.15.61:0/78632458 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 -- 172.21.15.61:0/78632458 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 msgr2=0x7f05c8143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 --2- 172.21.15.61:0/78632458 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8143960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 -- 172.21.15.61:0/78632458 >> v1:172.21.15.111:6789/0 conn(0x7f05c8143e50 legacy=0x7f05c80b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 -- 172.21.15.61:0/78632458 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80b1d20 con 0x7f05c80b36c0 2022-07-12T08:44:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05ce270700 1 --2- 172.21.15.61:0/78632458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8143090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f05c4002f70 tx=0x7f05c4006eb0).ready entity=mon.0 client_cookie=5bb2df0009d2daa8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05cd26e700 1 -- 172.21.15.61:0/78632458 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c400d5c0 con 0x7f05c80b36c0 2022-07-12T08:44:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05cd26e700 1 -- 172.21.15.61:0/78632458 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c4000e90 con 0x7f05c80b36c0 2022-07-12T08:44:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05cd26e700 1 -- 172.21.15.61:0/78632458 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05c400d950 con 0x7f05c80b36c0 2022-07-12T08:44:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- 172.21.15.61:0/78632458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 msgr2=0x7f05c8143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 --2- 172.21.15.61:0/78632458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8143090 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f05c4002f70 tx=0x7f05c4006eb0).stop 2022-07-12T08:44:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- 172.21.15.61:0/78632458 shutdown_connections 2022-07-12T08:44:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 --2- 172.21.15.61:0/78632458 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 --2- 172.21.15.61:0/78632458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8143090 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- 172.21.15.61:0/78632458 >> 172.21.15.61:0/78632458 conn(0x7f05c80b6e40 msgr2=0x7f05c80b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- 172.21.15.61:0/78632458 shutdown_connections 2022-07-12T08:44:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.364+0000 7f05d04d4700 1 -- 172.21.15.61:0/78632458 wait complete. 2022-07-12T08:44:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 Processor -- start 2022-07-12T08:44:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 -- start start 2022-07-12T08:44:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c8143e50 0x7f05c8155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8152510 con 0x7f05c80b36c0 2022-07-12T08:44:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c80981c0 con 0x7f05c8143e50 2022-07-12T08:44:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c8098460 con 0x7f05c80a9b40 2022-07-12T08:44:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55400/0 (socket says 172.21.15.61:55400) 2022-07-12T08:44:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 -- 172.21.15.61:0/4212567324 learned_addr learned my addr 172.21.15.61:0/4212567324 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05ce270700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cea71700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c8143e50 0x7f05c8155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 -- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 msgr2=0x7f05c8146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 -- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c8143e50 msgr2=0x7f05c8155b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c8143e50 0x7f05c8155b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 -- 172.21.15.61:0/4212567324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c400b1e0 con 0x7f05c80b36c0 2022-07-12T08:44:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05ce270700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:38.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05cda6f700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8152020 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f05b800a000 tx=0x7f05b8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:38.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b8011780 con 0x7f05c80b36c0 2022-07-12T08:44:38.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b80118e0 con 0x7f05c80b36c0 2022-07-12T08:44:38.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c8155690 con 0x7f05c80b36c0 2022-07-12T08:44:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b8011c40 con 0x7f05c80b36c0 2022-07-12T08:44:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.368+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c8097d20 con 0x7f05c80b36c0 2022-07-12T08:44:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.372+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c8098030 con 0x7f05c80b36c0 2022-07-12T08:44:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.372+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f05b8012420 con 0x7f05c80b36c0 2022-07-12T08:44:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.372+0000 7f05b77fe700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a80539b0 0x7f05a8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:38.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.372+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f05b806bee0 con 0x7f05c80b36c0 2022-07-12T08:44:38.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.372+0000 7f05ce270700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a80539b0 0x7f05a8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:38.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.372+0000 7f05ce270700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a80539b0 0x7f05a8055e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f05c401a000 tx=0x7f05c4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:38.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.380+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05b8042c80 con 0x7f05c80b36c0 2022-07-12T08:44:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:38.512+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f05c80996c0 con 0x7f05c80b36c0 2022-07-12T08:44:39.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.228+0000 7f05b77fe700 1 -- 172.21.15.61:0/4212567324 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 185+0+0 (secure 0 0 0) 0x7f05c80996c0 con 0x7f05c80b36c0 2022-07-12T08:44:39.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a80539b0 msgr2=0x7f05a8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:39.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a80539b0 0x7f05a8055e70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f05c401a000 tx=0x7f05c4015000).stop 2022-07-12T08:44:39.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 msgr2=0x7f05c8152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:39.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8152020 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f05b800a000 tx=0x7f05b8008040).stop 2022-07-12T08:44:39.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 shutdown_connections 2022-07-12T08:44:39.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a80539b0 0x7f05a8055e70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c8143e50 0x7f05c8155b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c80b36c0 0x7f05c8152020 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 --2- 172.21.15.61:0/4212567324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c80a9b40 0x7f05c8146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.232+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 >> 172.21.15.61:0/4212567324 conn(0x7f05c80b6e40 msgr2=0x7f05c8143710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:39.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.236+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 shutdown_connections 2022-07-12T08:44:39.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.236+0000 7f05d04d4700 1 -- 172.21.15.61:0/4212567324 wait complete. 2022-07-12T08:44:39.255 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:44:39.257 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:44:39.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.376+0000 7f94aa53d700 1 Processor -- start 2022-07-12T08:44:39.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.376+0000 7f94aa53d700 1 -- start start 2022-07-12T08:44:39.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f94a405c8e0 0x7f94a40b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:39.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:39.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a4060440 con 0x7f94a4143e70 2022-07-12T08:44:39.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a405ccc0 con 0x7f94a405c8e0 2022-07-12T08:44:39.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40b2da0 con 0x7f94a40b28e0 2022-07-12T08:44:39.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f94a405c8e0 0x7f94a40b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:39.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f94a405c8e0 0x7f94a40b22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55406/0 (socket says 172.21.15.61:55406) 2022-07-12T08:44:39.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 -- 172.21.15.61:0/1377087459 learned_addr learned my addr 172.21.15.61:0/1377087459 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:39.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a8d3a700 1 --2- 172.21.15.61:0/1377087459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:39.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 -- 172.21.15.61:0/1377087459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 msgr2=0x7f94a40afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:39.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 --2- 172.21.15.61:0/1377087459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 -- 172.21.15.61:0/1377087459 >> v1:172.21.15.111:6789/0 conn(0x7f94a4143e70 legacy=0x7f94a40affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:39.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 -- 172.21.15.61:0/1377087459 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40b3040 con 0x7f94a405c8e0 2022-07-12T08:44:39.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94a953b700 1 --2- 172.21.15.61:0/1377087459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a405c8e0 0x7f94a40b22c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f94a0002f70 tx=0x7f94a0006eb0).ready entity=mon.0 client_cookie=d6f4c948e5c5f715 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:39.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f948ffff700 1 -- 172.21.15.61:0/1377087459 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a000d570 con 0x7f94a405c8e0 2022-07-12T08:44:39.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f948ffff700 1 -- 172.21.15.61:0/1377087459 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a0000e90 con 0x7f94a405c8e0 2022-07-12T08:44:39.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f948ffff700 1 -- 172.21.15.61:0/1377087459 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a000d900 con 0x7f94a405c8e0 2022-07-12T08:44:39.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 -- 172.21.15.61:0/1377087459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a405c8e0 msgr2=0x7f94a40b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:39.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.380+0000 7f94aa53d700 1 --2- 172.21.15.61:0/1377087459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a405c8e0 0x7f94a40b22c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f94a0002f70 tx=0x7f94a0006eb0).stop 2022-07-12T08:44:39.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- 172.21.15.61:0/1377087459 shutdown_connections 2022-07-12T08:44:39.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 --2- 172.21.15.61:0/1377087459 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 --2- 172.21.15.61:0/1377087459 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a405c8e0 0x7f94a40b22c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- 172.21.15.61:0/1377087459 >> 172.21.15.61:0/1377087459 conn(0x7f94a4063a60 msgr2=0x7f94a4063e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:39.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- 172.21.15.61:0/1377087459 shutdown_connections 2022-07-12T08:44:39.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- 172.21.15.61:0/1377087459 wait complete. 2022-07-12T08:44:39.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 Processor -- start 2022-07-12T08:44:39.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- start start 2022-07-12T08:44:39.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a405c8e0 0x7f94a4141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:39.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:39.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 0x7f94a4152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:39.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40adeb0 con 0x7f94a4143e70 2022-07-12T08:44:39.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40ae150 con 0x7f94a405c8e0 2022-07-12T08:44:39.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94aa53d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40adba0 con 0x7f94a40b28e0 2022-07-12T08:44:39.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94a9d3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 0x7f94a4152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:39.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94a9d3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 0x7f94a4152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55414/0 (socket says 172.21.15.61:55414) 2022-07-12T08:44:39.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94a9d3c700 1 -- 172.21.15.61:0/843594479 learned_addr learned my addr 172.21.15.61:0/843594479 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:39.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94a953b700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a405c8e0 0x7f94a4141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:39.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.384+0000 7f94a8d3a700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:39.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a9d3c700 1 -- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 msgr2=0x7f94a40aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:39.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a9d3c700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40aece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a9d3c700 1 -- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a405c8e0 msgr2=0x7f94a4141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:39.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a9d3c700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a405c8e0 0x7f94a4141d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:39.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a9d3c700 1 -- 172.21.15.61:0/843594479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a000b1e0 con 0x7f94a4143e70 2022-07-12T08:44:39.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a8d3a700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40aece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:39.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a953b700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a405c8e0 0x7f94a4141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:39.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94a9d3c700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 0x7f94a4152250 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f94a405f030 tx=0x7f949c007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:39.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f949c006aa0 con 0x7f94a4143e70 2022-07-12T08:44:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a4151d40 con 0x7f94a4143e70 2022-07-12T08:44:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949c006c00 con 0x7f94a4143e70 2022-07-12T08:44:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f949c007030 con 0x7f94a4143e70 2022-07-12T08:44:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a41458f0 con 0x7f94a4143e70 2022-07-12T08:44:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a4151f60 con 0x7f94a4143e70 2022-07-12T08:44:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f949c010070 con 0x7f94a4143e70 2022-07-12T08:44:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.388+0000 7f9496ffd700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f94800539b0 0x7f9480055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.392+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f949c06b930 con 0x7f94a4143e70 2022-07-12T08:44:39.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.392+0000 7f94a953b700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f94800539b0 0x7f9480055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:39.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.392+0000 7f94a953b700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f94800539b0 0x7f9480055e70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f94a001a000 tx=0x7f94a0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:39.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.396+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f949c042890 con 0x7f94a4143e70 2022-07-12T08:44:39.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:39.524+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94a4057460 con 0x7f94a4143e70 2022-07-12T08:44:40.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.232+0000 7f9496ffd700 1 -- 172.21.15.61:0/843594479 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 173+0+0 (secure 0 0 0) 0x7f949c001310 con 0x7f94a4143e70 2022-07-12T08:44:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f94800539b0 msgr2=0x7f9480055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f94800539b0 0x7f9480055e70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f94a001a000 tx=0x7f94a0015000).stop 2022-07-12T08:44:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 msgr2=0x7f94a4152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 0x7f94a4152250 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f94a405f030 tx=0x7f949c007f70).stop 2022-07-12T08:44:40.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 shutdown_connections 2022-07-12T08:44:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94a4143e70 0x7f94a4152250 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94a40b28e0 0x7f94a40aece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f94800539b0 0x7f9480055e70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 --2- 172.21.15.61:0/843594479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94a405c8e0 0x7f94a4141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 >> 172.21.15.61:0/843594479 conn(0x7f94a4063a60 msgr2=0x7f94a40b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 shutdown_connections 2022-07-12T08:44:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.236+0000 7f94aa53d700 1 -- 172.21.15.61:0/843594479 wait complete. 2022-07-12T08:44:40.246 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:44:40.255 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:44:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa968051700 1 Processor -- start 2022-07-12T08:44:40.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa968051700 1 -- start start 2022-07-12T08:44:40.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa968051700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa960140310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa968051700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9601408f0 con 0x7fa960067630 2022-07-12T08:44:40.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa968051700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa960140b90 con 0x7fa9600b0ee0 2022-07-12T08:44:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa968051700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9600adcf0 con 0x7fa9600acde0 2022-07-12T08:44:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa9655ec700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa960067630 0x7fa96013fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43636/0 (socket says 172.21.15.61:43636) 2022-07-12T08:44:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa9655ec700 1 -- 172.21.15.61:0/3712473695 learned_addr learned my addr 172.21.15.61:0/3712473695 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa9665ee700 1 --2- 172.21.15.61:0/3712473695 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa960140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa964deb700 1 -- 172.21.15.61:0/3712473695 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515200967 0 0) 0x7fa9601408f0 con 0x7fa960067630 2022-07-12T08:44:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa964deb700 1 -- 172.21.15.61:0/3712473695 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa948003260 con 0x7fa960067630 2022-07-12T08:44:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.356+0000 7fa964deb700 1 -- 172.21.15.61:0/3712473695 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297183702 0 0) 0x7fa960140b90 con 0x7fa9600b0ee0 2022-07-12T08:44:40.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa9665ee700 1 -- 172.21.15.61:0/3712473695 >> v1:172.21.15.111:6789/0 conn(0x7fa9600b0ee0 legacy=0x7fa9600ac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa9665ee700 1 -- 172.21.15.61:0/3712473695 >> v1:172.21.15.61:6789/0 conn(0x7fa960067630 legacy=0x7fa96013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa9665ee700 1 -- 172.21.15.61:0/3712473695 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9600b12c0 con 0x7fa9600acde0 2022-07-12T08:44:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa9665ee700 1 --2- 172.21.15.61:0/3712473695 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa960140310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa9580032f0 tx=0x7fa9580091d0).ready entity=mon.2 client_cookie=c5c73f4fa10156d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:40.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa964deb700 1 -- 172.21.15.61:0/3712473695 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa95800b8b0 con 0x7fa9600acde0 2022-07-12T08:44:40.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa964deb700 1 -- 172.21.15.61:0/3712473695 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95800ba10 con 0x7fa9600acde0 2022-07-12T08:44:40.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa964deb700 1 -- 172.21.15.61:0/3712473695 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa95800bd70 con 0x7fa9600acde0 2022-07-12T08:44:40.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 -- 172.21.15.61:0/3712473695 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 msgr2=0x7fa960140310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 --2- 172.21.15.61:0/3712473695 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa960140310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa9580032f0 tx=0x7fa9580091d0).stop 2022-07-12T08:44:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 -- 172.21.15.61:0/3712473695 shutdown_connections 2022-07-12T08:44:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 --2- 172.21.15.61:0/3712473695 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa960140310 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 -- 172.21.15.61:0/3712473695 >> 172.21.15.61:0/3712473695 conn(0x7fa9600b5290 msgr2=0x7fa9600b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 -- 172.21.15.61:0/3712473695 shutdown_connections 2022-07-12T08:44:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 -- 172.21.15.61:0/3712473695 wait complete. 2022-07-12T08:44:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 Processor -- start 2022-07-12T08:44:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.360+0000 7fa968051700 1 -- start start 2022-07-12T08:44:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 0x7fa960064600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa96013c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9600b0ee0 0x7fa960156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa960063c20 con 0x7fa960067630 2022-07-12T08:44:40.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa960063ec0 con 0x7fa9600b0ee0 2022-07-12T08:44:40.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa960064160 con 0x7fa9600acde0 2022-07-12T08:44:40.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 0x7fa960064600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 0x7fa960064600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55424/0 (socket says 172.21.15.61:55424) 2022-07-12T08:44:40.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 -- 172.21.15.61:0/545234687 learned_addr learned my addr 172.21.15.61:0/545234687 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:40.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa9665ee700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9600b0ee0 0x7fa960156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa9655ec700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa96013c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 -- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 msgr2=0x7fa96013c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa96013c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 -- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9600b0ee0 msgr2=0x7fa960156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9600b0ee0 0x7fa960156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 -- 172.21.15.61:0/545234687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa958009470 con 0x7fa960067630 2022-07-12T08:44:40.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa965ded700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 0x7fa960064600 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa95c000bc0 tx=0x7fa95c007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:40.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa95c001d70 con 0x7fa960067630 2022-07-12T08:44:40.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c007030 con 0x7fa960067630 2022-07-12T08:44:40.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96013fec0 con 0x7fa960067630 2022-07-12T08:44:40.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa95c007390 con 0x7fa960067630 2022-07-12T08:44:40.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa960065d80 con 0x7fa960067630 2022-07-12T08:44:40.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.364+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa960064320 con 0x7fa960067630 2022-07-12T08:44:40.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.368+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa95c010030 con 0x7fa960067630 2022-07-12T08:44:40.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.368+0000 7fa94effd700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9400539b0 0x7fa940055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.368+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa95c06b670 con 0x7fa960067630 2022-07-12T08:44:40.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.368+0000 7fa9655ec700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9400539b0 0x7fa940055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.368+0000 7fa9655ec700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9400539b0 0x7fa940055e70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fa95000d000 tx=0x7fa950009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.372+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa95c0428b0 con 0x7fa960067630 2022-07-12T08:44:40.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.500+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa960057460 con 0x7fa960067630 2022-07-12T08:44:40.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa94effd700 1 -- 172.21.15.61:0/545234687 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa95c045ed0 con 0x7fa960067630 2022-07-12T08:44:40.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9400539b0 msgr2=0x7fa940055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9400539b0 0x7fa940055e70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fa95000d000 tx=0x7fa950009040).stop 2022-07-12T08:44:40.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 msgr2=0x7fa960064600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 0x7fa960064600 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa95c000bc0 tx=0x7fa95c007c30).stop 2022-07-12T08:44:40.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 shutdown_connections 2022-07-12T08:44:40.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9400539b0 0x7fa940055e70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9600b0ee0 0x7fa960156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9600acde0 0x7fa96013c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 --2- 172.21.15.61:0/545234687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa960067630 0x7fa960064600 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 >> 172.21.15.61:0/545234687 conn(0x7fa9600b5290 msgr2=0x7fa9600b48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:40.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 shutdown_connections 2022-07-12T08:44:40.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.508+0000 7fa968051700 1 -- 172.21.15.61:0/545234687 wait complete. 2022-07-12T08:44:40.528 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-12T08:44:40.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf75fd9700 1 Processor -- start 2022-07-12T08:44:40.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf75fd9700 1 -- start start 2022-07-12T08:44:40.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf75fd9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdf700ab290 0x7fdf700ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf75fd9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf700abc70 con 0x7fdf7005f140 2022-07-12T08:44:40.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf75fd9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7005c720 con 0x7fdf700afa60 2022-07-12T08:44:40.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf75fd9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7005f580 con 0x7fdf700ab290 2022-07-12T08:44:40.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf67fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdf7005f140 0x7fdf7005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43654/0 (socket says 172.21.15.61:43654) 2022-07-12T08:44:40.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf67fff700 1 -- 172.21.15.61:0/2383950282 learned_addr learned my addr 172.21.15.61:0/2383950282 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:40.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf74fd7700 1 --2- 172.21.15.61:0/2383950282 >> v2:172.21.15.111:3300/0 conn(0x7fdf700ab290 0x7fdf700ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf677fe700 1 -- 172.21.15.61:0/2383950282 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078660188 0 0) 0x7fdf700abc70 con 0x7fdf7005f140 2022-07-12T08:44:40.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.624+0000 7fdf677fe700 1 -- 172.21.15.61:0/2383950282 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf5c003260 con 0x7fdf7005f140 2022-07-12T08:44:40.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf74fd7700 1 -- 172.21.15.61:0/2383950282 >> v1:172.21.15.111:6789/0 conn(0x7fdf700afa60 legacy=0x7fdf70147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf74fd7700 1 -- 172.21.15.61:0/2383950282 >> v1:172.21.15.61:6789/0 conn(0x7fdf7005f140 legacy=0x7fdf7005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf74fd7700 1 -- 172.21.15.61:0/2383950282 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7005c9c0 con 0x7fdf700ab290 2022-07-12T08:44:40.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf74fd7700 1 --2- 172.21.15.61:0/2383950282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700ab290 0x7fdf700ab670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fdf6c002f70 tx=0x7fdf6c006df0).ready entity=mon.1 client_cookie=6677f32d001e035c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:40.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf677fe700 1 -- 172.21.15.61:0/2383950282 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c00d610 con 0x7fdf700ab290 2022-07-12T08:44:40.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf677fe700 1 -- 172.21.15.61:0/2383950282 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c000e90 con 0x7fdf700ab290 2022-07-12T08:44:40.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf677fe700 1 -- 172.21.15.61:0/2383950282 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c00d960 con 0x7fdf700ab290 2022-07-12T08:44:40.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- 172.21.15.61:0/2383950282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700ab290 msgr2=0x7fdf700ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/2383950282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700ab290 0x7fdf700ab670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fdf6c002f70 tx=0x7fdf6c006df0).stop 2022-07-12T08:44:40.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- 172.21.15.61:0/2383950282 shutdown_connections 2022-07-12T08:44:40.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/2383950282 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700ab290 0x7fdf700ab670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- 172.21.15.61:0/2383950282 >> 172.21.15.61:0/2383950282 conn(0x7fdf700b1950 msgr2=0x7fdf700b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:40.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- 172.21.15.61:0/2383950282 shutdown_connections 2022-07-12T08:44:40.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- 172.21.15.61:0/2383950282 wait complete. 2022-07-12T08:44:40.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 Processor -- start 2022-07-12T08:44:40.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- start start 2022-07-12T08:44:40.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdf7005f140 0x7fdf700ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 0x7fdf70063860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700afa60 0x7fdf70063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70065190 con 0x7fdf700ab290 2022-07-12T08:44:40.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70065430 con 0x7fdf700afa60 2022-07-12T08:44:40.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf75fd9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70067330 con 0x7fdf7005f140 2022-07-12T08:44:40.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 0x7fdf70063860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 0x7fdf70063860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55440/0 (socket says 172.21.15.61:55440) 2022-07-12T08:44:40.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf67fff700 1 -- 172.21.15.61:0/3566399883 learned_addr learned my addr 172.21.15.61:0/3566399883 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:40.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf74fd7700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdf7005f140 0x7fdf700ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.628+0000 7fdf757d8700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700afa60 0x7fdf70063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf74fd7700 1 -- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700afa60 msgr2=0x7fdf70063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf74fd7700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700afa60 0x7fdf70063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf74fd7700 1 -- 172.21.15.61:0/3566399883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 msgr2=0x7fdf70063860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:40.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf74fd7700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 0x7fdf70063860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:40.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf74fd7700 1 -- 172.21.15.61:0/3566399883 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c00b180 con 0x7fdf7005f140 2022-07-12T08:44:40.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf67fff700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 0x7fdf70063860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:40.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf74fd7700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdf7005f140 0x7fdf700ae120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fdf6c002f70 tx=0x7fdf6c014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:40.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c013de0 con 0x7fdf7005f140 2022-07-12T08:44:40.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c00d290 con 0x7fdf7005f140 2022-07-12T08:44:40.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c01e5f0 con 0x7fdf7005f140 2022-07-12T08:44:40.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf70066fb0 con 0x7fdf7005f140 2022-07-12T08:44:40.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf70066030 con 0x7fdf7005f140 2022-07-12T08:44:40.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf700671a0 con 0x7fdf7005f140 2022-07-12T08:44:40.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdf6c01c070 con 0x7fdf7005f140 2022-07-12T08:44:40.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf65ffb700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdf500539d0 0x7fdf50055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:40.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fdf6c06c580 con 0x7fdf7005f140 2022-07-12T08:44:40.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.632+0000 7fdf67fff700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdf500539d0 0x7fdf50055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:40.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.636+0000 7fdf67fff700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdf500539d0 0x7fdf50055e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fdf6000d000 tx=0x7fdf60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:40.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.640+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdf6c0436b0 con 0x7fdf7005f140 2022-07-12T08:44:40.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:40.812+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7fdf7006b2b0 con 0x7fdf7005f140 2022-07-12T08:44:41.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.248+0000 7fdf65ffb700 1 -- 172.21.15.61:0/3566399883 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v163) v1 ==== 144+0+0 (secure 0 0 0) 0x7fdf6c023020 con 0x7fdf7005f140 2022-07-12T08:44:41.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdf500539d0 msgr2=0x7fdf50055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:41.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdf500539d0 0x7fdf50055e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fdf6000d000 tx=0x7fdf60009040).stop 2022-07-12T08:44:41.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdf7005f140 msgr2=0x7fdf700ae120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:41.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdf7005f140 0x7fdf700ae120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fdf6c002f70 tx=0x7fdf6c014000).stop 2022-07-12T08:44:41.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 shutdown_connections 2022-07-12T08:44:41.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdf500539d0 0x7fdf50055e90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdf700afa60 0x7fdf70063d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf700ab290 0x7fdf70063860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 --2- 172.21.15.61:0/3566399883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdf7005f140 0x7fdf700ae120 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 >> 172.21.15.61:0/3566399883 conn(0x7fdf700b1950 msgr2=0x7fdf700b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:41.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 shutdown_connections 2022-07-12T08:44:41.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.252+0000 7fdf75fd9700 1 -- 172.21.15.61:0/3566399883 wait complete. 2022-07-12T08:44:41.271 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-12T08:44:41.272 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-12T08:44:41.272 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-12T08:44:41.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.380+0000 7f10bec78700 1 Processor -- start 2022-07-12T08:44:41.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.380+0000 7f10bec78700 1 -- start start 2022-07-12T08:44:41.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10b80ab670 0x7f10b805f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:41.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b80afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:41.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80abc70 con 0x7f10b80ab290 2022-07-12T08:44:41.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b80aba90 con 0x7f10b80ab670 2022-07-12T08:44:41.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b805c720 con 0x7f10b80afa60 2022-07-12T08:44:41.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10b80ab670 0x7f10b805f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:41.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10b80ab670 0x7f10b805f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55446/0 (socket says 172.21.15.61:55446) 2022-07-12T08:44:41.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bdc76700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f10b80ab290 0x7f10b805ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43664/0 (socket says 172.21.15.61:43664) 2022-07-12T08:44:41.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 -- 172.21.15.61:0/966408345 learned_addr learned my addr 172.21.15.61:0/966408345 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:41.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10be477700 1 --2- 172.21.15.61:0/966408345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b80afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:41.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 -- 172.21.15.61:0/966408345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 msgr2=0x7f10b80afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:41.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 --2- 172.21.15.61:0/966408345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b80afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 -- 172.21.15.61:0/966408345 >> v1:172.21.15.61:6789/0 conn(0x7f10b80ab290 legacy=0x7f10b805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:41.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bcc74700 1 -- 172.21.15.61:0/966408345 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359849484 0 0) 0x7f10b80abc70 con 0x7f10b80ab290 2022-07-12T08:44:41.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 -- 172.21.15.61:0/966408345 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b805c9c0 con 0x7f10b80ab670 2022-07-12T08:44:41.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bd475700 1 --2- 172.21.15.61:0/966408345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b805f140 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f10b4002f70 tx=0x7f10b4006eb0).ready entity=mon.0 client_cookie=8ffff272990f0bd9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:41.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bcc74700 1 -- 172.21.15.61:0/966408345 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10b400d5c0 con 0x7f10b80ab670 2022-07-12T08:44:41.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bcc74700 1 -- 172.21.15.61:0/966408345 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b4000e90 con 0x7f10b80ab670 2022-07-12T08:44:41.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bcc74700 1 -- 172.21.15.61:0/966408345 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10b400d910 con 0x7f10b80ab670 2022-07-12T08:44:41.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- 172.21.15.61:0/966408345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 msgr2=0x7f10b805f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:41.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 --2- 172.21.15.61:0/966408345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b805f140 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f10b4002f70 tx=0x7f10b4006eb0).stop 2022-07-12T08:44:41.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- 172.21.15.61:0/966408345 shutdown_connections 2022-07-12T08:44:41.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 --2- 172.21.15.61:0/966408345 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b80afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 --2- 172.21.15.61:0/966408345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b805f140 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- 172.21.15.61:0/966408345 >> 172.21.15.61:0/966408345 conn(0x7f10b80b1950 msgr2=0x7f10b80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:41.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- 172.21.15.61:0/966408345 shutdown_connections 2022-07-12T08:44:41.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.384+0000 7f10bec78700 1 -- 172.21.15.61:0/966408345 wait complete. 2022-07-12T08:44:41.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 Processor -- start 2022-07-12T08:44:41.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 -- start start 2022-07-12T08:44:41.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b80ab290 0x7f10b8091ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:41.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b808f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:41.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b81598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:41.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b8159490 con 0x7f10b80ab670 2022-07-12T08:44:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b8065fc0 con 0x7f10b80ab290 2022-07-12T08:44:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b8066260 con 0x7f10b80afa60 2022-07-12T08:44:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b808f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b808f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55454/0 (socket says 172.21.15.61:55454) 2022-07-12T08:44:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 -- 172.21.15.61:0/3532501740 learned_addr learned my addr 172.21.15.61:0/3532501740 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bdc76700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b80ab290 0x7f10b8091ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10be477700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b81598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 -- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 msgr2=0x7f10b81598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:41.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b81598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 -- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b80ab290 msgr2=0x7f10b8091ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b80ab290 0x7f10b8091ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 -- 172.21.15.61:0/3532501740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b400b1e0 con 0x7f10b80ab670 2022-07-12T08:44:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bd475700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b808f9d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f10b4002f70 tx=0x7f10b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10b4006b40 con 0x7f10b80ab670 2022-07-12T08:44:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b400de40 con 0x7f10b80ab670 2022-07-12T08:44:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b8097750 con 0x7f10b80ab670 2022-07-12T08:44:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10b4011e20 con 0x7f10b80ab670 2022-07-12T08:44:41.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.388+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b8159db0 con 0x7f10b80ab670 2022-07-12T08:44:41.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.392+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b8159620 con 0x7f10b80ab670 2022-07-12T08:44:41.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.392+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f10b4025030 con 0x7f10b80ab670 2022-07-12T08:44:41.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.392+0000 7f10a6ffd700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1098053ad0 0x7f1098055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:41.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.392+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f10b406cff0 con 0x7f10b80ab670 2022-07-12T08:44:41.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.392+0000 7f10bdc76700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1098053ad0 0x7f1098055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:41.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.392+0000 7f10bdc76700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1098053ad0 0x7f1098055f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f10a800d000 tx=0x7f10a8007a90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:41.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.400+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10b8159620 con 0x7f10b80ab670 2022-07-12T08:44:41.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:41.576+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f10b8159620 con 0x7f10b80ab670 2022-07-12T08:44:42.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.256+0000 7f10a6ffd700 1 -- 172.21.15.61:0/3532501740 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v164) v1 ==== 155+0+0 (secure 0 0 0) 0x7f10b8159620 con 0x7f10b80ab670 2022-07-12T08:44:42.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1098053ad0 msgr2=0x7f1098055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1098053ad0 0x7f1098055f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f10a800d000 tx=0x7f10a8007a90).stop 2022-07-12T08:44:42.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 msgr2=0x7f10b808f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b808f9d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f10b4002f70 tx=0x7f10b4015000).stop 2022-07-12T08:44:42.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 shutdown_connections 2022-07-12T08:44:42.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10b80afa60 0x7f10b81598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10b80ab670 0x7f10b808f9d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1098053ad0 0x7f1098055f90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 --2- 172.21.15.61:0/3532501740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f10b80ab290 0x7f10b8091ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 >> 172.21.15.61:0/3532501740 conn(0x7f10b80b1950 msgr2=0x7f10b80b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 shutdown_connections 2022-07-12T08:44:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.260+0000 7f10bec78700 1 -- 172.21.15.61:0/3532501740 wait complete. 2022-07-12T08:44:42.279 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:44:42.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 Processor -- start 2022-07-12T08:44:42.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 -- start start 2022-07-12T08:44:42.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb100b9db0 0x7ffb100b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb100b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb100b3120 con 0x7ffb100b3ea0 2022-07-12T08:44:42.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1005bd10 con 0x7ffb100b9db0 2022-07-12T08:44:42.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.392+0000 7ffb16512700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100b33c0 con 0x7ffb1005c810 2022-07-12T08:44:42.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb077fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ffb100b3ea0 0x7ffb1005c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43682/0 (socket says 172.21.15.61:43682) 2022-07-12T08:44:42.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb077fe700 1 -- 172.21.15.61:0/1298776338 learned_addr learned my addr 172.21.15.61:0/1298776338 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:42.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/1298776338 >> v2:172.21.15.111:3300/0 conn(0x7ffb100b9db0 0x7ffb100b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb14aaf700 1 --2- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb100b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0f7fe700 1 -- 172.21.15.61:0/1298776338 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010323548 0 0) 0x7ffb100b3120 con 0x7ffb100b3ea0 2022-07-12T08:44:42.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0f7fe700 1 -- 172.21.15.61:0/1298776338 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf0003260 con 0x7ffb100b3ea0 2022-07-12T08:44:42.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0ffff700 1 -- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 msgr2=0x7ffb100b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:42.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb100b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0ffff700 1 -- 172.21.15.61:0/1298776338 >> v1:172.21.15.61:6789/0 conn(0x7ffb100b3ea0 legacy=0x7ffb1005c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0ffff700 1 -- 172.21.15.61:0/1298776338 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1005cc60 con 0x7ffb100b9db0 2022-07-12T08:44:42.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b9db0 0x7ffb100b2a40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ffb100b1230 tx=0x7ffb00006df0).ready entity=mon.1 client_cookie=1683ffef0e1fa0d5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:42.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0f7fe700 1 -- 172.21.15.61:0/1298776338 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb0000d570 con 0x7ffb100b9db0 2022-07-12T08:44:42.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0f7fe700 1 -- 172.21.15.61:0/1298776338 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb00000e90 con 0x7ffb100b9db0 2022-07-12T08:44:42.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb0f7fe700 1 -- 172.21.15.61:0/1298776338 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb0000d8c0 con 0x7ffb100b9db0 2022-07-12T08:44:42.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 -- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b9db0 msgr2=0x7ffb100b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 --2- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b9db0 0x7ffb100b2a40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7ffb100b1230 tx=0x7ffb00006df0).stop 2022-07-12T08:44:42.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 -- 172.21.15.61:0/1298776338 shutdown_connections 2022-07-12T08:44:42.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 --2- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb100b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 --2- 172.21.15.61:0/1298776338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b9db0 0x7ffb100b2a40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 -- 172.21.15.61:0/1298776338 >> 172.21.15.61:0/1298776338 conn(0x7ffb10067960 msgr2=0x7ffb100b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:42.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 -- 172.21.15.61:0/1298776338 shutdown_connections 2022-07-12T08:44:42.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.396+0000 7ffb16512700 1 -- 172.21.15.61:0/1298776338 wait complete. 2022-07-12T08:44:42.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 Processor -- start 2022-07-12T08:44:42.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 -- start start 2022-07-12T08:44:42.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb10116130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b3ea0 0x7ffb101159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 0x7ffb100af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101137e0 con 0x7ffb100b9db0 2022-07-12T08:44:42.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb100ae960 con 0x7ffb100b3ea0 2022-07-12T08:44:42.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.400+0000 7ffb16512700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb101142f0 con 0x7ffb1005c810 2022-07-12T08:44:42.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb14aaf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 0x7ffb100af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb14aaf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 0x7ffb100af390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55472/0 (socket says 172.21.15.61:55472) 2022-07-12T08:44:42.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb14aaf700 1 -- 172.21.15.61:0/2759261730 learned_addr learned my addr 172.21.15.61:0/2759261730 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:42.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb10116130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb077fe700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b3ea0 0x7ffb101159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 -- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b3ea0 msgr2=0x7ffb101159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b3ea0 0x7ffb101159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 -- 172.21.15.61:0/2759261730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 msgr2=0x7ffb100af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 0x7ffb100af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 -- 172.21.15.61:0/2759261730 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb0000b180 con 0x7ffb1005c810 2022-07-12T08:44:42.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb14aaf700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 0x7ffb100af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:42.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0ffff700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb10116130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ffb100b1230 tx=0x7ffb00014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:42.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb00016070 con 0x7ffb1005c810 2022-07-12T08:44:42.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb000138d0 con 0x7ffb1005c810 2022-07-12T08:44:42.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb10113a40 con 0x7ffb1005c810 2022-07-12T08:44:42.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb00013c30 con 0x7ffb1005c810 2022-07-12T08:44:42.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb101525d0 con 0x7ffb1005c810 2022-07-12T08:44:42.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.404+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10057460 con 0x7ffb1005c810 2022-07-12T08:44:42.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.408+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffb00025030 con 0x7ffb1005c810 2022-07-12T08:44:42.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.408+0000 7ffb0dffb700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffaf4053ad0 0x7ffaf4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.408+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ffb0006d140 con 0x7ffb1005c810 2022-07-12T08:44:42.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.408+0000 7ffb077fe700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffaf4053ad0 0x7ffaf4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.408+0000 7ffb077fe700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffaf4053ad0 0x7ffaf4055f90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7ffaf800d000 tx=0x7ffaf8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:42.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.416+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffb00044380 con 0x7ffb1005c810 2022-07-12T08:44:42.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.544+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ffb1006a940 con 0x7ffb1005c810 2022-07-12T08:44:42.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb0dffb700 1 -- 172.21.15.61:0/2759261730 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7ffb000230c0 con 0x7ffb1005c810 2022-07-12T08:44:42.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffaf4053ad0 msgr2=0x7ffaf4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffaf4053ad0 0x7ffaf4055f90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7ffaf800d000 tx=0x7ffaf8009040).stop 2022-07-12T08:44:42.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 msgr2=0x7ffb10116130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb10116130 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ffb100b1230 tx=0x7ffb00014000).stop 2022-07-12T08:44:42.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 shutdown_connections 2022-07-12T08:44:42.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb100b9db0 0x7ffb100af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffaf4053ad0 0x7ffaf4055f90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb100b3ea0 0x7ffb101159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 --2- 172.21.15.61:0/2759261730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb1005c810 0x7ffb10116130 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 >> 172.21.15.61:0/2759261730 conn(0x7ffb10067960 msgr2=0x7ffb100afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:42.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 shutdown_connections 2022-07-12T08:44:42.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.556+0000 7ffb16512700 1 -- 172.21.15.61:0/2759261730 wait complete. 2022-07-12T08:44:42.575 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:44:42.576 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:42.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:42.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:42.651 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:42.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:44:42.695 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:44:42.696 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:42.701 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:42.702 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:42.746 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:44:42.746 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:44:42.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:42.755 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:42.756 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:44:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:42.782 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:44:42.783 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:44:42.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:42.785 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:42.785 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:44:42.794 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:44:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:44:42.797 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:44:42.798 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:44:42.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.908+0000 7f547f7e7700 1 Processor -- start 2022-07-12T08:44:42.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.908+0000 7f547f7e7700 1 -- start start 2022-07-12T08:44:42.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5478066fc0 0x7f54780b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f54780b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54780b2f20 con 0x7f54780b2940 2022-07-12T08:44:42.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54780b3170 con 0x7f5478066fc0 2022-07-12T08:44:42.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54780b4170 con 0x7f54780b45d0 2022-07-12T08:44:42.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5478066fc0 0x7f54780b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5478066fc0 0x7f54780b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55478/0 (socket says 172.21.15.61:55478) 2022-07-12T08:44:42.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 -- 172.21.15.61:0/2558579510 learned_addr learned my addr 172.21.15.61:0/2558579510 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:42.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547cd82700 1 --2- 172.21.15.61:0/2558579510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f54780b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 -- 172.21.15.61:0/2558579510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 msgr2=0x7f54780b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 --2- 172.21.15.61:0/2558579510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f54780b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 -- 172.21.15.61:0/2558579510 >> v1:172.21.15.111:6789/0 conn(0x7f54780b2940 legacy=0x7f547805c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 -- 172.21.15.61:0/2558579510 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780b56e0 con 0x7f5478066fc0 2022-07-12T08:44:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547d583700 1 --2- 172.21.15.61:0/2558579510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f54780b4d00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5468002f70 tx=0x7f5468006eb0).ready entity=mon.0 client_cookie=3bdcb99540c5e66b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f5477fff700 1 -- 172.21.15.61:0/2558579510 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f546800d570 con 0x7f5478066fc0 2022-07-12T08:44:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f5477fff700 1 -- 172.21.15.61:0/2558579510 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468000e90 con 0x7f5478066fc0 2022-07-12T08:44:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f5477fff700 1 -- 172.21.15.61:0/2558579510 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f546800d900 con 0x7f5478066fc0 2022-07-12T08:44:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- 172.21.15.61:0/2558579510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 msgr2=0x7f54780b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 --2- 172.21.15.61:0/2558579510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f54780b4d00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5468002f70 tx=0x7f5468006eb0).stop 2022-07-12T08:44:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- 172.21.15.61:0/2558579510 shutdown_connections 2022-07-12T08:44:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 --2- 172.21.15.61:0/2558579510 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f54780b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 --2- 172.21.15.61:0/2558579510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f54780b4d00 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- 172.21.15.61:0/2558579510 >> 172.21.15.61:0/2558579510 conn(0x7f54780b94d0 msgr2=0x7f54780b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- 172.21.15.61:0/2558579510 shutdown_connections 2022-07-12T08:44:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- 172.21.15.61:0/2558579510 wait complete. 2022-07-12T08:44:42.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 Processor -- start 2022-07-12T08:44:42.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.912+0000 7f547f7e7700 1 -- start start 2022-07-12T08:44:42.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f5478144a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54780b2940 0x7f54781485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f5478148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5478145840 con 0x7f5478066fc0 2022-07-12T08:44:42.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54780af0c0 con 0x7f54780b2940 2022-07-12T08:44:42.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54780af360 con 0x7f54780b45d0 2022-07-12T08:44:42.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f5478144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f5478144a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55482/0 (socket says 172.21.15.61:55482) 2022-07-12T08:44:42.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 -- 172.21.15.61:0/3529990423 learned_addr learned my addr 172.21.15.61:0/3529990423 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:42.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547cd82700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54780b2940 0x7f54781485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547dd84700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f5478148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 -- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 msgr2=0x7f5478148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f5478148ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 -- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54780b2940 msgr2=0x7f54781485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:42.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54780b2940 0x7f54781485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:42.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 -- 172.21.15.61:0/3529990423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546800b1e0 con 0x7f5478066fc0 2022-07-12T08:44:42.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547d583700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f5478144a50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f5468002f70 tx=0x7f5468015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:42.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5468011950 con 0x7f5478066fc0 2022-07-12T08:44:42.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5468011ab0 con 0x7f5478066fc0 2022-07-12T08:44:42.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54780b0220 con 0x7f5478066fc0 2022-07-12T08:44:42.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5468011e10 con 0x7f5478066fc0 2022-07-12T08:44:42.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5478146fb0 con 0x7f5478066fc0 2022-07-12T08:44:42.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.916+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5478147340 con 0x7f5478066fc0 2022-07-12T08:44:42.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.920+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5468025030 con 0x7f5478066fc0 2022-07-12T08:44:42.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.920+0000 7f54767fc700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f54580539b0 0x7f5458055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:42.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.920+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f546806cf70 con 0x7f5478066fc0 2022-07-12T08:44:42.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.920+0000 7f547cd82700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f54580539b0 0x7f5458055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:42.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.920+0000 7f547cd82700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f54580539b0 0x7f5458055e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f546c00b000 tx=0x7f546c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:42.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:42.928+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54680440a0 con 0x7f5478066fc0 2022-07-12T08:44:43.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.048+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f54780b03b0 con 0x7f5478066fc0 2022-07-12T08:44:43.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.052+0000 7f54767fc700 1 -- 172.21.15.61:0/3529990423 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v45) v1 ==== 81+0+28875 (secure 0 0 0) 0x7f5468023070 con 0x7f5478066fc0 2022-07-12T08:44:43.057 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:43.058 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:44:43.058 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 45, 2022-07-12T08:44:43.059 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:44:43.059 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:44:43.060 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:44:40.229539+0000", 2022-07-12T08:44:43.060 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:44:43.060 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:44:43.061 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:44:43.061 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:44:43.062 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:44:43.062 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:44:43.063 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:44:43.064 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:44:43.064 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:44:43.064 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:43.065 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:44:43.065 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:44:43.066 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:44:43.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:44:43.067 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:44:43.068 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 8, 2022-07-12T08:44:43.068 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:44:43.068 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:44:43.069 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:44:43.069 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:44:43.069 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:44:43.070 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.070 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:44:43.070 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:44:43.070 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:44:43.070 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:44:43.071 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:44:43.071 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:44:43.071 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:44:43.071 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:44:43.072 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:44:43.072 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:44:43.072 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:44:43.072 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:44:43.073 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:44:43.073 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:44:43.073 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:44:43.074 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:44:43.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:44:43.074 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:44:43.075 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:44:43.075 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:44:43.075 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:44:43.075 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:44:43.076 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:44:43.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:44:43.077 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:44:43.077 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:44:43.077 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:44:43.077 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:44:43.078 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:44:43.078 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:44:43.078 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:44:43.079 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:44:43.079 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:44:43.079 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:44:43.079 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:44:43.080 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:44:43.080 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:44:43.080 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:44:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:44:43.081 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:44:43.082 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:44:43.082 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:44:43.082 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:44:43.082 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:44:43.083 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:44:43.083 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:44:43.083 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:44:43.083 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:44:43.084 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:44:43.084 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:44:43.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.084 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:44:43.084 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:44:43.085 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:44:43.085 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:44:43.085 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:44:43.085 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:44:43.086 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:44:43.086 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:44:43.086 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:44:43.086 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:44:43.086 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:44:43.087 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:44:43.087 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:44:43.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.087 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:44:43.088 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:44:43.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.088 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.088 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.090 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:44:43.090 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:44:43.091 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:44:43.091 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:44:43.092 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:44:43.092 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:44:43.093 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:44:43.093 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:44:43.094 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:44:43.094 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:44:43.095 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:44:43.095 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:44:43.096 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:44:43.096 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:44:43.097 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:44:43.097 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:44:43.098 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:44:43.098 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:44:43.099 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:44:43.099 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:44:43.099 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:44:43.100 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:44:43.100 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:44:43.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.101 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:44:43.102 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:44:43.102 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:44:43.103 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:44:43.103 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:44:43.103 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:44:43.104 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:44:43.104 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:44:43.105 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:44:43.105 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:44:43.106 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:44:43.106 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:44:43.106 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:44:43.107 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:44:43.107 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:44:43.108 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:44:43.108 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:44:43.109 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:44:43.109 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:44:43.110 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:44:43.111 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:44:43.111 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:44:43.112 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:44:43.112 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:44:43.113 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:44:43.113 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:44:43.114 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:44:43.114 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.115 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:44:43.115 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:44:43.116 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:44:43.116 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:44:43.117 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:44:43.117 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:44:43.118 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:44:43.118 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:44:43.119 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:44:43.119 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:44:43.120 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:44:43.120 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:44:43.121 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:44:43.121 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:44:43.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.123 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:43.123 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:44:43.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.124 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:44:43.125 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:44:43.125 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.126 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.126 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.127 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.127 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.128 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.128 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.129 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:43.130 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.130 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.131 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.131 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.132 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.132 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:44:43.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:44:43.136 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.136 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.136 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.137 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:44:43.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.142 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:44:43.142 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.143 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.144 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.145 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.145 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.146 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:44:43.146 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.147 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.147 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.148 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:44:43.149 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.149 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.149 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.150 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.150 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:44:43.153 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.153 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:44:43.154 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:44:43.155 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.155 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.156 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:44:43.156 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:44:43.156 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:44:43.157 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:44:43.157 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.157 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.158 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.158 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.159 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.159 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.159 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:44:43.160 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:44:43.160 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.160 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.160 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.161 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.161 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.161 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.161 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.162 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:43.162 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.162 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.163 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.163 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.163 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.163 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.163 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:44:43.164 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.164 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.164 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.164 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.165 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:44:43.165 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.165 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.165 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.165 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.166 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.166 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.166 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.166 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:44:43.167 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.167 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.168 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.168 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:44:43.169 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.169 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.169 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.170 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:44:43.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.172 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.173 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:44:43.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.174 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.175 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:44:43.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.176 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:44:43.177 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:44:43.177 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.177 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.178 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:44:43.178 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:44:43.179 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:44:43.179 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:44:43.179 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.179 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.180 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.180 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.181 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:44:43.181 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:44:43.181 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.181 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.181 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.182 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.182 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.182 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.182 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.183 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 28, 2022-07-12T08:44:43.183 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.184 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.184 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.184 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.185 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:44:43.185 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.185 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.186 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.186 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.186 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:44:43.187 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.187 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.187 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.187 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.187 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.188 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:44:43.189 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.189 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.189 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.190 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.190 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:44:43.190 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.191 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.191 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.192 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.192 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.193 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.193 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.193 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:44:43.193 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.193 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.194 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.194 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:44:43.195 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.195 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.195 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.196 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.196 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.196 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:44:43.197 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:44:43.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:44:43.198 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.198 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.199 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:44:43.199 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:44:43.199 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:44:43.199 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:44:43.200 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.200 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.200 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.200 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.200 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.201 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:44:43.201 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:44:43.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.202 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.202 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.202 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.202 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.202 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.203 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.203 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:43.203 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.203 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.204 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.204 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:44:43.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.205 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:44:43.206 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.206 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.206 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.207 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.207 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.207 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:44:43.208 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.208 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.208 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:44:43.209 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.209 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.209 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.210 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.210 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:44:43.211 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.211 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.212 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:44:43.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.212 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.213 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.213 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.214 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:44:43.214 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.214 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:44:43.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:44:43.215 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.216 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.216 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:44:43.216 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:44:43.217 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:44:43.217 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:44:43.217 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.217 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.217 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.218 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.219 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:44:43.219 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:44:43.219 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.220 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.220 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.220 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.220 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.221 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.222 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:43.222 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.222 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.223 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.223 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.223 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:44:43.224 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.225 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.225 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:44:43.225 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.225 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.226 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.226 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.226 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.227 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.227 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:44:43.228 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.228 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:44:43.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.229 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.229 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.230 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:44:43.231 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.232 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:44:43.233 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.233 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.233 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.234 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:44:43.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:44:43.238 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:44:43.238 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.238 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.239 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:44:43.239 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:44:43.240 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:44:43.240 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:44:43.241 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.241 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.242 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.242 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.242 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.243 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:44:43.243 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:44:43.243 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.243 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.244 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.244 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.244 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.245 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.245 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.245 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:43.245 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.245 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.246 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.246 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:44:43.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:44:43.248 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.248 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.248 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.249 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:44:43.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.250 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:44:43.256 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.256 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.257 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.257 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.258 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.258 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.258 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:44:43.259 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:44:43.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.260 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.260 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.261 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.261 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.262 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.262 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:44:43.263 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:44:43.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:44:43.265 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.265 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.265 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:44:43.265 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:44:43.266 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:44:43.266 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:44:43.266 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.267 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.267 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.267 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.268 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.268 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.268 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:44:43.269 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:44:43.269 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.270 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.270 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.271 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.271 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.272 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.273 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-12T08:44:43.273 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.273 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.274 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.274 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.275 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.276 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:44:43.276 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.278 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:44:43.278 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.279 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.279 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.279 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.280 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:44:43.280 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.281 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:44:43.282 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.282 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.282 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.282 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.283 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:44:43.284 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.284 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:44:43.285 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.285 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.285 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.285 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.286 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.286 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:44:43.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.288 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:44:43.289 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:44:43.289 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.289 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.290 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:44:43.290 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:44:43.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:44:43.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:44:43.290 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.291 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.291 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.292 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.293 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:44:43.293 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:44:43.293 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:44:43.293 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:44:43.293 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:44:43.294 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:44:43.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:44:43.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:44:43.295 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:44:43.295 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 39, 2022-07-12T08:44:43.296 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:44:43.296 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:44:43.296 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:44:43.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.297 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.298 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:44:43.298 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.299 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.300 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:44:43.301 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.301 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.301 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.302 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:44:43.303 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.303 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.303 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:44:43.304 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.304 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:44:43.306 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.306 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.307 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.307 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:44:43.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.308 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.308 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:44:43.308 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.309 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.309 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:44:43.310 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.310 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.310 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.311 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:44:43.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:44:43.311 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.311 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:44:43.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:44:43.312 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.312 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:44:43.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:44:43.313 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:44:43.313 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.314 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.314 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.315 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:44:43.315 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:44:43.316 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:44:43.316 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:44:43.317 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:44:43.317 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:44:43.317 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:44:43.318 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:43.318 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.318 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:43.319 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:44:43.319 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.319 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:44:43.320 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.320 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.321 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.321 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.321 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.322 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:44:43.322 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.322 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.323 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.323 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:44:43.324 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.324 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.324 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.325 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.325 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.325 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:44:43.325 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.326 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.326 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.326 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:44:43.326 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.326 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.327 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.327 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.327 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.327 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:44:43.328 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.328 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.328 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:44:43.329 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.329 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.329 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.330 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.330 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.330 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:44:43.330 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.331 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.331 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.331 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:44:43.331 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.332 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.332 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.332 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.332 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.333 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:44:43.333 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.333 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.334 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:44:43.334 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.335 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.335 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.335 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.335 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.336 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:44:43.336 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.336 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.336 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.336 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:44:43.337 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.337 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.337 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.338 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.338 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.339 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:44:43.339 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:43.340 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:44:43.341 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:44:43.341 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:44:43.342 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:44:43.342 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:44:43.342 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:44:43.342 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:44:43.343 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:44:43.343 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.343 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:44:43.343 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:44:43.343 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:44:43.344 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:44:43.344 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:44:43.344 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:44:43.344 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/4199688387": "2022-07-13T08:44:36.973770+0000", 2022-07-12T08:44:43.345 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/4199688387": "2022-07-13T08:44:36.973770+0000" 2022-07-12T08:44:43.345 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.345 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:44:43.345 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:44:43.345 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:44:43.346 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:44:43.346 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:44:43.346 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:44:43.347 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:44:43.347 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:44:43.347 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:43.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:43.348 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:44:43.348 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:44:43.348 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:44:43.349 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:44:43.349 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:44:43.349 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:44:43.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f54580539b0 msgr2=0x7f5458055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f54580539b0 0x7f5458055e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f546c00b000 tx=0x7f546c008040).stop 2022-07-12T08:44:43.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 msgr2=0x7f5478144a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f5478144a50 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f5468002f70 tx=0x7f5468015000).stop 2022-07-12T08:44:43.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 shutdown_connections 2022-07-12T08:44:43.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54780b45d0 0x7f5478148ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f54580539b0 0x7f5458055e70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54780b2940 0x7f54781485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 --2- 172.21.15.61:0/3529990423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5478066fc0 0x7f5478144a50 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 >> 172.21.15.61:0/3529990423 conn(0x7f54780b94d0 msgr2=0x7f54780b76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:43.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 shutdown_connections 2022-07-12T08:44:43.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.056+0000 7f547f7e7700 1 -- 172.21.15.61:0/3529990423 wait complete. 2022-07-12T08:44:43.356 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:43.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 Processor -- start 2022-07-12T08:44:43.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 -- start start 2022-07-12T08:44:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f740b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1f740b47f0 0x7f1f740b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7413d240 con 0x7f1f74142d50 2022-07-12T08:44:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f740b24c0 con 0x7f1f740b47f0 2022-07-12T08:44:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f7b316700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f741431f0 con 0x7f1f7413ce60 2022-07-12T08:44:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f788b1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1f74142d50 0x7f1f740b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43710/0 (socket says 172.21.15.61:43710) 2022-07-12T08:44:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f788b1700 1 -- 172.21.15.61:0/2827986836 learned_addr learned my addr 172.21.15.61:0/2827986836 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f790b2700 1 --2- 172.21.15.61:0/2827986836 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f740b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.464+0000 7f1f798b3700 1 --2- 172.21.15.61:0/2827986836 >> v2:172.21.15.111:3300/0 conn(0x7f1f740b47f0 0x7f1f740b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f67fff700 1 -- 172.21.15.61:0/2827986836 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823494459 0 0) 0x7f1f7413d240 con 0x7f1f74142d50 2022-07-12T08:44:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f67fff700 1 -- 172.21.15.61:0/2827986836 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f58003260 con 0x7f1f74142d50 2022-07-12T08:44:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f790b2700 1 -- 172.21.15.61:0/2827986836 >> v2:172.21.15.111:3300/0 conn(0x7f1f740b47f0 msgr2=0x7f1f740b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:43.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f790b2700 1 --2- 172.21.15.61:0/2827986836 >> v2:172.21.15.111:3300/0 conn(0x7f1f740b47f0 0x7f1f740b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f790b2700 1 -- 172.21.15.61:0/2827986836 >> v1:172.21.15.61:6789/0 conn(0x7f1f74142d50 legacy=0x7f1f740b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f790b2700 1 -- 172.21.15.61:0/2827986836 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f74143490 con 0x7f1f7413ce60 2022-07-12T08:44:43.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f798b3700 1 --2- 172.21.15.61:0/2827986836 >> v2:172.21.15.111:3300/0 conn(0x7f1f740b47f0 0x7f1f740b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:43.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f790b2700 1 --2- 172.21.15.61:0/2827986836 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f740b15a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f70002f70 tx=0x7f1f70006df0).ready entity=mon.2 client_cookie=39d188a0ef7ca025 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:43.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f67fff700 1 -- 172.21.15.61:0/2827986836 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7000d4a0 con 0x7f1f7413ce60 2022-07-12T08:44:43.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f67fff700 1 -- 172.21.15.61:0/2827986836 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f70000e90 con 0x7f1f7413ce60 2022-07-12T08:44:43.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f67fff700 1 -- 172.21.15.61:0/2827986836 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7000d7f0 con 0x7f1f7413ce60 2022-07-12T08:44:43.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 -- 172.21.15.61:0/2827986836 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 msgr2=0x7f1f740b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 --2- 172.21.15.61:0/2827986836 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f740b15a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1f70002f70 tx=0x7f1f70006df0).stop 2022-07-12T08:44:43.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 -- 172.21.15.61:0/2827986836 shutdown_connections 2022-07-12T08:44:43.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 --2- 172.21.15.61:0/2827986836 >> v2:172.21.15.111:3300/0 conn(0x7f1f740b47f0 0x7f1f740b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 --2- 172.21.15.61:0/2827986836 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f740b15a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 -- 172.21.15.61:0/2827986836 >> 172.21.15.61:0/2827986836 conn(0x7f1f7405f9f0 msgr2=0x7f1f74066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:43.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 -- 172.21.15.61:0/2827986836 shutdown_connections 2022-07-12T08:44:43.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.468+0000 7f1f7b316700 1 -- 172.21.15.61:0/2827986836 wait complete. 2022-07-12T08:44:43.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 Processor -- start 2022-07-12T08:44:43.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 -- start start 2022-07-12T08:44:43.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 0x7f1f74149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f7413a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f74142d50 0x7f1f7413a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74145fd0 con 0x7f1f740b47f0 2022-07-12T08:44:43.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74146270 con 0x7f1f74142d50 2022-07-12T08:44:43.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f74146510 con 0x7f1f7413ce60 2022-07-12T08:44:43.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 0x7f1f74149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 0x7f1f74149f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55496/0 (socket says 172.21.15.61:55496) 2022-07-12T08:44:43.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 -- 172.21.15.61:0/1719314245 learned_addr learned my addr 172.21.15.61:0/1719314245 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f788b1700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f7413a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f798b3700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f74142d50 0x7f1f7413a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 -- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 msgr2=0x7f1f7413a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f7413a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 -- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f74142d50 msgr2=0x7f1f7413a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f74142d50 0x7f1f7413a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 -- 172.21.15.61:0/1719314245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c009470 con 0x7f1f740b47f0 2022-07-12T08:44:43.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f798b3700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f74142d50 0x7f1f7413a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:43.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f790b2700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 0x7f1f74149f60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1f70002f70 tx=0x7f1f70014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:43.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f70016070 con 0x7f1f740b47f0 2022-07-12T08:44:43.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f700138d0 con 0x7f1f740b47f0 2022-07-12T08:44:43.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7000b180 con 0x7f1f740b47f0 2022-07-12T08:44:43.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f70013c30 con 0x7f1f740b47f0 2022-07-12T08:44:43.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.472+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f74145b30 con 0x7f1f740b47f0 2022-07-12T08:44:43.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.476+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f741466d0 con 0x7f1f740b47f0 2022-07-12T08:44:43.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.476+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1f70025030 con 0x7f1f740b47f0 2022-07-12T08:44:43.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.476+0000 7f1f667fc700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f5c053ad0 0x7f1f5c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.476+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1f7006ca50 con 0x7f1f740b47f0 2022-07-12T08:44:43.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.476+0000 7f1f788b1700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f5c053ad0 0x7f1f5c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.476+0000 7f1f788b1700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f5c053ad0 0x7f1f5c055f90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f1f6800d000 tx=0x7f1f68009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:43.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.484+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f70043c90 con 0x7f1f740b47f0 2022-07-12T08:44:43.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f74057460 con 0x7f1f740b47f0 2022-07-12T08:44:43.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f667fc700 1 -- 172.21.15.61:0/1719314245 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f1f70023090 con 0x7f1f740b47f0 2022-07-12T08:44:43.661 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:43.662 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15},{"gid":5514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3761489418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3761489418},{"type":"v1","addr":"172.21.15.61:6833","nonce":3761489418}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":20}],"filesystems":[]} 2022-07-12T08:44:43.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f5c053ad0 msgr2=0x7f1f5c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f5c053ad0 0x7f1f5c055f90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f1f6800d000 tx=0x7f1f68009040).stop 2022-07-12T08:44:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 msgr2=0x7f1f74149f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 0x7f1f74149f60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1f70002f70 tx=0x7f1f70014000).stop 2022-07-12T08:44:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 shutdown_connections 2022-07-12T08:44:43.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f5c053ad0 0x7f1f5c055f90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f74142d50 0x7f1f7413a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f7413ce60 0x7f1f7413a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 --2- 172.21.15.61:0/1719314245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f740b47f0 0x7f1f74149f60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 >> 172.21.15.61:0/1719314245 conn(0x7f1f7405f9f0 msgr2=0x7f1f740b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:43.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 shutdown_connections 2022-07-12T08:44:43.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.656+0000 7f1f7b316700 1 -- 172.21.15.61:0/1719314245 wait complete. 2022-07-12T08:44:43.668 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-12T08:44:43.681 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:43.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d7717700 1 Processor -- start 2022-07-12T08:44:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d7717700 1 -- start start 2022-07-12T08:44:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d7717700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d0144ad0 0x7fb5d00ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d7717700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d00ad2f0 con 0x7fb5d00b01f0 2022-07-12T08:44:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d7717700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d005c430 con 0x7fb5d005c050 2022-07-12T08:44:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d7717700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d00b0a40 con 0x7fb5d0144ad0 2022-07-12T08:44:43.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d4cb2700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb5d00b01f0 0x7fb5d01443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43724/0 (socket says 172.21.15.61:43724) 2022-07-12T08:44:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d4cb2700 1 -- 172.21.15.61:0/2722416753 learned_addr learned my addr 172.21.15.61:0/2722416753 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5d5cb4700 1 --2- 172.21.15.61:0/2722416753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d0144ad0 0x7fb5d00ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5cffff700 1 -- 172.21.15.61:0/2722416753 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534926340 0 0) 0x7fb5d00ad2f0 con 0x7fb5d00b01f0 2022-07-12T08:44:43.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.772+0000 7fb5cffff700 1 -- 172.21.15.61:0/2722416753 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5bc003260 con 0x7fb5d00b01f0 2022-07-12T08:44:43.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d5cb4700 1 -- 172.21.15.61:0/2722416753 >> v1:172.21.15.111:6789/0 conn(0x7fb5d005c050 legacy=0x7fb5d005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d5cb4700 1 -- 172.21.15.61:0/2722416753 >> v1:172.21.15.61:6789/0 conn(0x7fb5d00b01f0 legacy=0x7fb5d01443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5cffff700 1 -- 172.21.15.61:0/2722416753 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510171276 0 0) 0x7fb5d005c430 con 0x7fb5d005c050 2022-07-12T08:44:43.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d5cb4700 1 -- 172.21.15.61:0/2722416753 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d00b2420 con 0x7fb5d0144ad0 2022-07-12T08:44:43.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d5cb4700 1 --2- 172.21.15.61:0/2722416753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d0144ad0 0x7fb5d00ac710 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb5c80032f0 tx=0x7fb5c80091d0).ready entity=mon.2 client_cookie=2d4f2bbc9c44cbc3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:43.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5cffff700 1 -- 172.21.15.61:0/2722416753 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5c800b6c0 con 0x7fb5d0144ad0 2022-07-12T08:44:43.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5cffff700 1 -- 172.21.15.61:0/2722416753 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c800b820 con 0x7fb5d0144ad0 2022-07-12T08:44:43.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5cffff700 1 -- 172.21.15.61:0/2722416753 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5c800b990 con 0x7fb5d0144ad0 2022-07-12T08:44:43.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 -- 172.21.15.61:0/2722416753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d0144ad0 msgr2=0x7fb5d00ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 --2- 172.21.15.61:0/2722416753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d0144ad0 0x7fb5d00ac710 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb5c80032f0 tx=0x7fb5c80091d0).stop 2022-07-12T08:44:43.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 -- 172.21.15.61:0/2722416753 shutdown_connections 2022-07-12T08:44:43.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 --2- 172.21.15.61:0/2722416753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d0144ad0 0x7fb5d00ac710 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 -- 172.21.15.61:0/2722416753 >> 172.21.15.61:0/2722416753 conn(0x7fb5d0062c00 msgr2=0x7fb5d0062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 -- 172.21.15.61:0/2722416753 shutdown_connections 2022-07-12T08:44:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 -- 172.21.15.61:0/2722416753 wait complete. 2022-07-12T08:44:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 Processor -- start 2022-07-12T08:44:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.776+0000 7fb5d7717700 1 -- start start 2022-07-12T08:44:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 0x7fb5d013b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d00b01f0 0x7fb5d013f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d0144ad0 0x7fb5d01598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d013f880 con 0x7fb5d0144ad0 2022-07-12T08:44:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d00ad670 con 0x7fb5d00b01f0 2022-07-12T08:44:43.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d00ad910 con 0x7fb5d005c050 2022-07-12T08:44:43.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d5cb4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d0144ad0 0x7fb5d01598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 0x7fb5d013b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d4cb2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d00b01f0 0x7fb5d013f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d5cb4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d0144ad0 0x7fb5d01598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55514/0 (socket says 172.21.15.61:55514) 2022-07-12T08:44:43.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 0x7fb5d013b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:37706/0 (socket says 172.21.15.61:37706) 2022-07-12T08:44:43.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d4cb2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d00b01f0 0x7fb5d013f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:34086/0 (socket says 172.21.15.61:34086) 2022-07-12T08:44:43.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d5cb4700 1 -- 172.21.15.61:0/999957208 learned_addr learned my addr 172.21.15.61:0/999957208 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:43.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 -- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d00b01f0 msgr2=0x7fb5d013f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d00b01f0 0x7fb5d013f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 -- 172.21.15.61:0/999957208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d0144ad0 msgr2=0x7fb5d01598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d0144ad0 0x7fb5d01598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 -- 172.21.15.61:0/999957208 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c8009470 con 0x7fb5d005c050 2022-07-12T08:44:43.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d54b3700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 0x7fb5d013b860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fb5c0000bc0 tx=0x7fb5c0007a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:43.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5c0012400 con 0x7fb5d005c050 2022-07-12T08:44:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c0012560 con 0x7fb5d005c050 2022-07-12T08:44:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d0140870 con 0x7fb5d005c050 2022-07-12T08:44:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5c00128c0 con 0x7fb5d005c050 2022-07-12T08:44:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d0159f60 con 0x7fb5d005c050 2022-07-12T08:44:43.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d00ad4d0 con 0x7fb5d005c050 2022-07-12T08:44:43.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.780+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb5c0019030 con 0x7fb5d005c050 2022-07-12T08:44:43.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.784+0000 7fb5ce7fc700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5b0053a40 0x7fb5b0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:43.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.784+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb5c006afe0 con 0x7fb5d005c050 2022-07-12T08:44:43.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.784+0000 7fb5d4cb2700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5b0053a40 0x7fb5b0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:43.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.784+0000 7fb5d4cb2700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5b0053a40 0x7fb5b0055f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb5c400d000 tx=0x7fb5c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:43.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.792+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5d00ad4d0 con 0x7fb5d005c050 2022-07-12T08:44:43.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.960+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5d00ad4d0 con 0x7fb5d005c050 2022-07-12T08:44:43.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.960+0000 7fb5ce7fc700 1 -- 172.21.15.61:0/999957208 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+1212 (secure 0 0 0) 0x7fb5d00ad4d0 con 0x7fb5d005c050 2022-07-12T08:44:43.966 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:43.966 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4961,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1148967130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1148967130},{"type":"v1","addr":"172.21.15.111:6835","nonce":1148967130}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":15},{"gid":5514,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3761489418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3761489418},{"type":"v1","addr":"172.21.15.61:6833","nonce":3761489418}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":20}],"filesystems":[]} 2022-07-12T08:44:43.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5b0053a40 msgr2=0x7fb5b0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5b0053a40 0x7fb5b0055f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fb5c400d000 tx=0x7fb5c4009040).stop 2022-07-12T08:44:43.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 msgr2=0x7fb5d013b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:43.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 0x7fb5d013b860 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fb5c0000bc0 tx=0x7fb5c0007a40).stop 2022-07-12T08:44:43.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 shutdown_connections 2022-07-12T08:44:43.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d0144ad0 0x7fb5d01598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5b0053a40 0x7fb5b0055f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5d00b01f0 0x7fb5d013f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 --2- 172.21.15.61:0/999957208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5d005c050 0x7fb5d013b860 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:43.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 >> 172.21.15.61:0/999957208 conn(0x7fb5d0062c00 msgr2=0x7fb5d0147370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:43.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 shutdown_connections 2022-07-12T08:44:43.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:43.964+0000 7fb5d7717700 1 -- 172.21.15.61:0/999957208 wait complete. 2022-07-12T08:44:43.975 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-12T08:44:43.985 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:44:43.985 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:44:43.986 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:44:43.987 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:44:43.987 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:44:43.987 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:44:43.988 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:44:43.988 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:44:43.996 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:44:43.997 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:44:44.000 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:44:44.000 DEBUG:teuthology.parallel:result is None 2022-07-12T08:44:44.004 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:44:44.005 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:44:44.052 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:44:44.053 DEBUG:teuthology.parallel:result is None 2022-07-12T08:44:44.053 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:44:44.070 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:44:44.127 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:44:44.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.152+0000 7fa7387cf700 1 Processor -- start 2022-07-12T08:44:44.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- start start 2022-07-12T08:44:44.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa7300b2240 0x7fa730094e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:44.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa730094200 0x7fa73005c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:44.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7301432c0 con 0x7fa730094aa0 2022-07-12T08:44:44.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa730095370 con 0x7fa730094200 2022-07-12T08:44:44.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7300b27c0 con 0x7fa7300b2240 2022-07-12T08:44:44.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa730094200 0x7fa73005c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:44.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa730094200 0x7fa73005c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55532/0 (socket says 172.21.15.61:55532) 2022-07-12T08:44:44.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 -- 172.21.15.61:0/2916310857 learned_addr learned my addr 172.21.15.61:0/2916310857 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:44.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa735d6a700 1 --2- 172.21.15.61:0/2916310857 >> v2:172.21.15.111:3300/0 conn(0x7fa7300b2240 0x7fa730094e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:44.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 -- 172.21.15.61:0/2916310857 >> v2:172.21.15.111:3300/0 conn(0x7fa7300b2240 msgr2=0x7fa730094e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 --2- 172.21.15.61:0/2916310857 >> v2:172.21.15.111:3300/0 conn(0x7fa7300b2240 0x7fa730094e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 -- 172.21.15.61:0/2916310857 >> v1:172.21.15.111:6789/0 conn(0x7fa730094aa0 legacy=0x7fa730060300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 -- 172.21.15.61:0/2916310857 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73005fbc0 con 0x7fa730094200 2022-07-12T08:44:44.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa736d6c700 1 --2- 172.21.15.61:0/2916310857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa73005c5a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa72c002f70 tx=0x7fa72c006eb0).ready entity=mon.0 client_cookie=da7c29df3154e9b5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:44.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa735569700 1 -- 172.21.15.61:0/2916310857 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa72c00d610 con 0x7fa730094200 2022-07-12T08:44:44.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa735569700 1 -- 172.21.15.61:0/2916310857 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72c000e90 con 0x7fa730094200 2022-07-12T08:44:44.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa735569700 1 -- 172.21.15.61:0/2916310857 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa72c00d9a0 con 0x7fa730094200 2022-07-12T08:44:44.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- 172.21.15.61:0/2916310857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 msgr2=0x7fa73005c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:44.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2916310857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa73005c5a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fa72c002f70 tx=0x7fa72c006eb0).stop 2022-07-12T08:44:44.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- 172.21.15.61:0/2916310857 shutdown_connections 2022-07-12T08:44:44.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2916310857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa73005c5a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:44.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2916310857 >> v2:172.21.15.111:3300/0 conn(0x7fa7300b2240 0x7fa730094e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:44.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- 172.21.15.61:0/2916310857 >> 172.21.15.61:0/2916310857 conn(0x7fa730064720 msgr2=0x7fa7300b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:44.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- 172.21.15.61:0/2916310857 shutdown_connections 2022-07-12T08:44:44.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.156+0000 7fa7387cf700 1 -- 172.21.15.61:0/2916310857 wait complete. 2022-07-12T08:44:44.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 Processor -- start 2022-07-12T08:44:44.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 -- start start 2022-07-12T08:44:44.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa730066bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa730094aa0 0x7fa7300aa590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa7300b2240 0x7fa7300ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:44.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7300671f0 con 0x7fa730094200 2022-07-12T08:44:44.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa730067490 con 0x7fa730094aa0 2022-07-12T08:44:44.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa730067730 con 0x7fa7300b2240 2022-07-12T08:44:44.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa730066bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:44.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa730066bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55536/0 (socket says 172.21.15.61:55536) 2022-07-12T08:44:44.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 -- 172.21.15.61:0/2706397107 learned_addr learned my addr 172.21.15.61:0/2706397107 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:44.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa735d6a700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa730094aa0 0x7fa7300aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:44.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa736d6c700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa7300b2240 0x7fa7300ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:44.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 -- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa7300b2240 msgr2=0x7fa7300ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:44.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa7300b2240 0x7fa7300ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:44.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 -- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa730094aa0 msgr2=0x7fa7300aa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:44.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa730094aa0 0x7fa7300aa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:44.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 -- 172.21.15.61:0/2706397107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c00b1e0 con 0x7fa730094200 2022-07-12T08:44:44.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa73656b700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa730066bc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fa73005fa00 tx=0x7fa7280014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:44.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa72800e4a0 con 0x7fa730094200 2022-07-12T08:44:44.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa72800e600 con 0x7fa730094200 2022-07-12T08:44:44.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa730066450 con 0x7fa730094200 2022-07-12T08:44:44.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa72800e960 con 0x7fa730094200 2022-07-12T08:44:44.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.160+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7300ac650 con 0x7fa730094200 2022-07-12T08:44:44.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.164+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7300678f0 con 0x7fa730094200 2022-07-12T08:44:44.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.164+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa728019030 con 0x7fa730094200 2022-07-12T08:44:44.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.164+0000 7fa71f7fe700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa710053940 0x7fa710055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:44.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.164+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa72806b1f0 con 0x7fa730094200 2022-07-12T08:44:44.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.164+0000 7fa735d6a700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa710053940 0x7fa710055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:44.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.164+0000 7fa735d6a700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa710053940 0x7fa710055e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa72000c000 tx=0x7fa720008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:44.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.172+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa728042430 con 0x7fa730094200 2022-07-12T08:44:44.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:44.308+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fa730057460 con 0x7fa730094200 2022-07-12T08:44:45.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.272+0000 7fa71f7fe700 1 -- 172.21.15.61:0/2706397107 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa728045a50 con 0x7fa730094200 2022-07-12T08:44:45.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa710053940 msgr2=0x7fa710055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa710053940 0x7fa710055e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fa72000c000 tx=0x7fa720008040).stop 2022-07-12T08:44:45.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 msgr2=0x7fa730066bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa730066bc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fa73005fa00 tx=0x7fa7280014a0).stop 2022-07-12T08:44:45.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 shutdown_connections 2022-07-12T08:44:45.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa7300b2240 0x7fa7300ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa710053940 0x7fa710055e00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa730094aa0 0x7fa7300aa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 --2- 172.21.15.61:0/2706397107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa730094200 0x7fa730066bc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 >> 172.21.15.61:0/2706397107 conn(0x7fa730064720 msgr2=0x7fa7300615f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:45.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 shutdown_connections 2022-07-12T08:44:45.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.276+0000 7fa7387cf700 1 -- 172.21.15.61:0/2706397107 wait complete. 2022-07-12T08:44:45.289 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:44:45.294 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:44:45.295 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:44:45.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 Processor -- start 2022-07-12T08:44:45.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- start start 2022-07-12T08:44:45.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe53405bfc0 0x7fe5340b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe534147000 0x7fe5341474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53405d3d0 con 0x7fe5340b5aa0 2022-07-12T08:44:45.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53414b590 con 0x7fe53405bfc0 2022-07-12T08:44:45.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5340b5e80 con 0x7fe534147000 2022-07-12T08:44:45.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe532ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe53405bfc0 0x7fe5340b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe5337fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe5340b5aa0 0x7fe53414ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43764/0 (socket says 172.21.15.61:43764) 2022-07-12T08:44:45.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe5337fe700 1 -- 172.21.15.61:0/1343301441 learned_addr learned my addr 172.21.15.61:0/1343301441 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:45.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe533fff700 1 --2- 172.21.15.61:0/1343301441 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe534147000 0x7fe5341474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe532ffd700 1 -- 172.21.15.61:0/1343301441 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe534147000 msgr2=0x7fe5341474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:45.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe532ffd700 1 --2- 172.21.15.61:0/1343301441 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe534147000 0x7fe5341474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe532ffd700 1 -- 172.21.15.61:0/1343301441 >> v1:172.21.15.61:6789/0 conn(0x7fe5340b5aa0 legacy=0x7fe53414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe532ffd700 1 -- 172.21.15.61:0/1343301441 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53414a740 con 0x7fe53405bfc0 2022-07-12T08:44:45.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe532ffd700 1 --2- 172.21.15.61:0/1343301441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe53405bfc0 0x7fe5340b0380 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fe5240032f0 tx=0x7fe5240091d0).ready entity=mon.0 client_cookie=e3d843dd074170bb server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:45.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe5327fc700 1 -- 172.21.15.61:0/1343301441 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe524009ea0 con 0x7fe53405bfc0 2022-07-12T08:44:45.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe5327fc700 1 -- 172.21.15.61:0/1343301441 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52400b8a0 con 0x7fe53405bfc0 2022-07-12T08:44:45.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe5327fc700 1 -- 172.21.15.61:0/1343301441 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe52400ba10 con 0x7fe53405bfc0 2022-07-12T08:44:45.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- 172.21.15.61:0/1343301441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe53405bfc0 msgr2=0x7fe5340b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 --2- 172.21.15.61:0/1343301441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe53405bfc0 0x7fe5340b0380 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7fe5240032f0 tx=0x7fe5240091d0).stop 2022-07-12T08:44:45.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- 172.21.15.61:0/1343301441 shutdown_connections 2022-07-12T08:44:45.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 --2- 172.21.15.61:0/1343301441 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe534147000 0x7fe5341474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 --2- 172.21.15.61:0/1343301441 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe53405bfc0 0x7fe5340b0380 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- 172.21.15.61:0/1343301441 >> 172.21.15.61:0/1343301441 conn(0x7fe534063520 msgr2=0x7fe5340b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:45.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- 172.21.15.61:0/1343301441 shutdown_connections 2022-07-12T08:44:45.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.456+0000 7fe539f13700 1 -- 172.21.15.61:0/1343301441 wait complete. 2022-07-12T08:44:45.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 Processor -- start 2022-07-12T08:44:45.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 -- start start 2022-07-12T08:44:45.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe53405bfc0 0x7fe5340665c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 0x7fe5340ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe534147000 0x7fe534065550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe534156650 con 0x7fe5340b5aa0 2022-07-12T08:44:45.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe534159fd0 con 0x7fe534147000 2022-07-12T08:44:45.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe534156840 con 0x7fe53405bfc0 2022-07-12T08:44:45.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe532ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 0x7fe5340ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe53405bfc0 0x7fe5340665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe532ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 0x7fe5340ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55556/0 (socket says 172.21.15.61:55556) 2022-07-12T08:44:45.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe533fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe534147000 0x7fe534065550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe532ffd700 1 -- 172.21.15.61:0/3384013755 learned_addr learned my addr 172.21.15.61:0/3384013755 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:45.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 -- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe534147000 msgr2=0x7fe534065550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe534147000 0x7fe534065550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 -- 172.21.15.61:0/3384013755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 msgr2=0x7fe5340ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 0x7fe5340ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 -- 172.21.15.61:0/3384013755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009470 con 0x7fe53405bfc0 2022-07-12T08:44:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe532ffd700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 0x7fe5340ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:45.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe533fff700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe534147000 0x7fe534065550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:45.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe5337fe700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe53405bfc0 0x7fe5340665c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe534062360 tx=0x7fe51c007b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:45.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe51c0070d0 con 0x7fe53405bfc0 2022-07-12T08:44:45.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c007230 con 0x7fe53405bfc0 2022-07-12T08:44:45.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe534159d30 con 0x7fe53405bfc0 2022-07-12T08:44:45.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe51c005610 con 0x7fe53405bfc0 2022-07-12T08:44:45.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.460+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5340651e0 con 0x7fe53405bfc0 2022-07-12T08:44:45.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.464+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe534057460 con 0x7fe53405bfc0 2022-07-12T08:44:45.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.464+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe51c010070 con 0x7fe53405bfc0 2022-07-12T08:44:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.464+0000 7fe530ff9700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe50c053a40 0x7fe50c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.464+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe51c06afc0 con 0x7fe53405bfc0 2022-07-12T08:44:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.464+0000 7fe532ffd700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe50c053a40 0x7fe50c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.464+0000 7fe532ffd700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe50c053a40 0x7fe50c055f00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe52401b000 tx=0x7fe524016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:45.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.472+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe51c0421b0 con 0x7fe53405bfc0 2022-07-12T08:44:45.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.640+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe534156980 con 0x7fe53405bfc0 2022-07-12T08:44:45.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.640+0000 7fe530ff9700 1 -- 172.21.15.61:0/3384013755 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fe51c018030 con 0x7fe53405bfc0 2022-07-12T08:44:45.647 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:45.648 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:44:45.648 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:44:45.649 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.649 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:44:45.649 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:44:45.651 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.651 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:44:45.652 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.652 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.653 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.653 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.654 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.654 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.655 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:44:45.656 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:44:45.656 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.657 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:44:45.657 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.658 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.658 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.659 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.659 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.660 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.660 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:44:45.661 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:44:45.662 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.662 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.663 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.663 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.664 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.664 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.665 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.665 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:44:45.665 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:44:45.666 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.666 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.667 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.667 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.668 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.668 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.668 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.669 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:44:45.670 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:44:45.670 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.670 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.671 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.671 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.676 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.676 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.676 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.677 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:44:45.678 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:44:45.678 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.681 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.682 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.682 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.683 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.684 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.685 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.685 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:44:45.686 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:44:45.686 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.686 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.687 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.687 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.688 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.688 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.689 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.689 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:44:45.690 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:44:45.690 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.691 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.691 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.691 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.692 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.692 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.693 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.693 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:44:45.693 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:44:45.695 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.695 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.697 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.697 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:44:45.697 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:44:45.698 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.698 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.698 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.698 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.699 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.699 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.699 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.699 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:44:45.700 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:44:45.700 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.700 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:44:45.700 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:44:45.701 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:44:45.701 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.702 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.702 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:44:45.702 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:44:45.702 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:44:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:44:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:44:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:44:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:44:45.705 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.706 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:44:45.706 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:44:45.706 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:44:45.707 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.707 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.707 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.708 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:44:45.708 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:44:45.708 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.709 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:44:45.709 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:44:45.709 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:44:45.709 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.710 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.710 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.710 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.711 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:44:45.711 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:44:45.711 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.711 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:44:45.712 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:44:45.712 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.712 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.712 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:45.714 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:45.714 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:44:45.714 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:44:45.714 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:44:45.715 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:44:45.716 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:44:45.716 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:44:45.716 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.717 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:45.717 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:44:45.718 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:44:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe50c053a40 msgr2=0x7fe50c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe50c053a40 0x7fe50c055f00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fe52401b000 tx=0x7fe524016000).stop 2022-07-12T08:44:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe53405bfc0 msgr2=0x7fe5340665c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe53405bfc0 0x7fe5340665c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe534062360 tx=0x7fe51c007b50).stop 2022-07-12T08:44:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 shutdown_connections 2022-07-12T08:44:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe50c053a40 0x7fe50c055f00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe534147000 0x7fe534065550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5340b5aa0 0x7fe5340ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 --2- 172.21.15.61:0/3384013755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe53405bfc0 0x7fe5340665c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 >> 172.21.15.61:0/3384013755 conn(0x7fe534063520 msgr2=0x7fe5340ab740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:45.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 shutdown_connections 2022-07-12T08:44:45.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.644+0000 7fe539f13700 1 -- 172.21.15.61:0/3384013755 wait complete. 2022-07-12T08:44:45.727 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:44:45.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.832+0000 7ff651c23700 1 Processor -- start 2022-07-12T08:44:45.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.832+0000 7ff651c23700 1 -- start start 2022-07-12T08:44:45.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c098070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff64c05f340 0x7ff64c142d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64c05d220 con 0x7ff64c0b1210 2022-07-12T08:44:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64c0aaa00 con 0x7ff64c05f340 2022-07-12T08:44:45.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c09b020 con 0x7ff64c05c740 2022-07-12T08:44:45.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff64c0b1210 0x7ff64c142640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43778/0 (socket says 172.21.15.61:43778) 2022-07-12T08:44:45.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64bfff700 1 -- 172.21.15.61:0/1208158012 learned_addr learned my addr 172.21.15.61:0/1208158012 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:45.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651422700 1 --2- 172.21.15.61:0/1208158012 >> v2:172.21.15.61:3300/0 conn(0x7ff64c05f340 0x7ff64c142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff650c21700 1 --2- 172.21.15.61:0/1208158012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c098070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64b7fe700 1 -- 172.21.15.61:0/1208158012 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782379708 0 0) 0x7ff64c05d220 con 0x7ff64c0b1210 2022-07-12T08:44:45.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64b7fe700 1 -- 172.21.15.61:0/1208158012 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638003260 con 0x7ff64c0b1210 2022-07-12T08:44:45.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651422700 1 -- 172.21.15.61:0/1208158012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 msgr2=0x7ff64c098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:45.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651422700 1 --2- 172.21.15.61:0/1208158012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c098070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651422700 1 -- 172.21.15.61:0/1208158012 >> v1:172.21.15.61:6789/0 conn(0x7ff64c0b1210 legacy=0x7ff64c142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651422700 1 -- 172.21.15.61:0/1208158012 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c09aca0 con 0x7ff64c05f340 2022-07-12T08:44:45.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651422700 1 --2- 172.21.15.61:0/1208158012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c05f340 0x7ff64c142d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7ff6440032f0 tx=0x7ff6440091d0).ready entity=mon.0 client_cookie=679d57baf761bf95 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:45.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64b7fe700 1 -- 172.21.15.61:0/1208158012 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64400b600 con 0x7ff64c05f340 2022-07-12T08:44:45.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64b7fe700 1 -- 172.21.15.61:0/1208158012 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64400b760 con 0x7ff64c05f340 2022-07-12T08:44:45.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff64b7fe700 1 -- 172.21.15.61:0/1208158012 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64400b8d0 con 0x7ff64c05f340 2022-07-12T08:44:45.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- 172.21.15.61:0/1208158012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c05f340 msgr2=0x7ff64c142d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 --2- 172.21.15.61:0/1208158012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c05f340 0x7ff64c142d50 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7ff6440032f0 tx=0x7ff6440091d0).stop 2022-07-12T08:44:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- 172.21.15.61:0/1208158012 shutdown_connections 2022-07-12T08:44:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 --2- 172.21.15.61:0/1208158012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c05f340 0x7ff64c142d50 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 --2- 172.21.15.61:0/1208158012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c098070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- 172.21.15.61:0/1208158012 >> 172.21.15.61:0/1208158012 conn(0x7ff64c0b1950 msgr2=0x7ff64c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:45.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- 172.21.15.61:0/1208158012 shutdown_connections 2022-07-12T08:44:45.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.836+0000 7ff651c23700 1 -- 172.21.15.61:0/1208158012 wait complete. 2022-07-12T08:44:45.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 Processor -- start 2022-07-12T08:44:45.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 -- start start 2022-07-12T08:44:45.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff64c05f340 0x7ff64c093aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 0x7ff64c0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0ac640 con 0x7ff64c0b1210 2022-07-12T08:44:45.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0ac3a0 con 0x7ff64c05f340 2022-07-12T08:44:45.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c0ad490 con 0x7ff64c05c740 2022-07-12T08:44:45.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 0x7ff64c0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 0x7ff64c0aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55568/0 (socket says 172.21.15.61:55568) 2022-07-12T08:44:45.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 -- 172.21.15.61:0/352542735 learned_addr learned my addr 172.21.15.61:0/352542735 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:45.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff650c21700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff64bfff700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff64c05f340 0x7ff64c093aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 -- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 msgr2=0x7ff64c138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 -- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff64c05f340 msgr2=0x7ff64c093aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff64c05f340 0x7ff64c093aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 -- 172.21.15.61:0/352542735 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff644009470 con 0x7ff64c0b1210 2022-07-12T08:44:45.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff650c21700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:45.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff64bfff700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff64c05f340 0x7ff64c093aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:45.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651422700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 0x7ff64c0aec50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7ff6440032f0 tx=0x7ff644016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:45.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64400b8d0 con 0x7ff64c0b1210 2022-07-12T08:44:45.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6440107a0 con 0x7ff64c0b1210 2022-07-12T08:44:45.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c141ab0 con 0x7ff64c0b1210 2022-07-12T08:44:45.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64400b600 con 0x7ff64c0b1210 2022-07-12T08:44:45.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.840+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c0af140 con 0x7ff64c0b1210 2022-07-12T08:44:45.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.844+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff644027020 con 0x7ff64c0b1210 2022-07-12T08:44:45.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.844+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c0ad650 con 0x7ff64c0b1210 2022-07-12T08:44:45.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.844+0000 7ff649ffb700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff62c053ad0 0x7ff62c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:45.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.844+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff64406d2c0 con 0x7ff64c0b1210 2022-07-12T08:44:45.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.844+0000 7ff650c21700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff62c053ad0 0x7ff62c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:45.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.844+0000 7ff650c21700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff62c053ad0 0x7ff62c055f90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ff64000b000 tx=0x7ff640008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:45.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.848+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff644044440 con 0x7ff64c0b1210 2022-07-12T08:44:45.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.980+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff64c057460 con 0x7ff64c0b1210 2022-07-12T08:44:45.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.980+0000 7ff649ffb700 1 -- 172.21.15.61:0/352542735 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+13229 (secure 0 0 0) 0x7ff644015090 con 0x7ff64c0b1210 2022-07-12T08:44:45.991 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:45.991 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":48,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:44:45.265142+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:44:45.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff62c053ad0 msgr2=0x7ff62c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff62c053ad0 0x7ff62c055f90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7ff64000b000 tx=0x7ff640008040).stop 2022-07-12T08:44:45.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 msgr2=0x7ff64c0aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:45.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 0x7ff64c0aec50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7ff6440032f0 tx=0x7ff644016000).stop 2022-07-12T08:44:45.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 shutdown_connections 2022-07-12T08:44:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff64c0b1210 0x7ff64c0aec50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff62c053ad0 0x7ff62c055f90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff64c05f340 0x7ff64c093aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 --2- 172.21.15.61:0/352542735 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff64c05c740 0x7ff64c138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:45.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 >> 172.21.15.61:0/352542735 conn(0x7ff64c0b1950 msgr2=0x7ff64c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:45.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 shutdown_connections 2022-07-12T08:44:45.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:45.984+0000 7ff651c23700 1 -- 172.21.15.61:0/352542735 wait complete. 2022-07-12T08:44:46.005 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:44:46.006 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:44:46.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.116+0000 7f024759e700 1 Processor -- start 2022-07-12T08:44:46.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.116+0000 7f024759e700 1 -- start start 2022-07-12T08:44:46.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f024805cd20 0x7f024805d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02480af220 0x7f0248060020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0248067c20 con 0x7f02481460e0 2022-07-12T08:44:46.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02480b00d0 con 0x7f024805cd20 2022-07-12T08:44:46.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0248147040 con 0x7f02480af220 2022-07-12T08:44:46.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f0245d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f02481460e0 0x7f024805f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:57392/0 (socket says 172.21.15.61:57392) 2022-07-12T08:44:46.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f0245d9b700 1 -- 172.21.15.61:0/3647620714 learned_addr learned my addr 172.21.15.61:0/3647620714 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f0246d9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02480af220 0x7f0248060020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f024805cd20 0x7f024805d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024559a700 1 -- 172.21.15.61:0/3647620714 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080755789 0 0) 0x7f023c001110 con 0x7f02481460e0 2022-07-12T08:44:46.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024559a700 1 -- 172.21.15.61:0/3647620714 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022c003260 con 0x7f02481460e0 2022-07-12T08:44:46.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024659c700 1 -- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02480af220 msgr2=0x7f0248060020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024659c700 1 --2- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02480af220 0x7f0248060020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024659c700 1 -- 172.21.15.61:0/3647620714 >> v1:172.21.15.111:6789/0 conn(0x7f02481460e0 legacy=0x7f024805f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024659c700 1 -- 172.21.15.61:0/3647620714 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0248146660 con 0x7f024805cd20 2022-07-12T08:44:46.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f0246d9d700 1 --2- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02480af220 0x7f0248060020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:46.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024659c700 1 --2- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024805cd20 0x7f024805d100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f0248147180 tx=0x7f0238006df0).ready entity=mon.1 client_cookie=94e72ea0d7b241d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024559a700 1 -- 172.21.15.61:0/3647620714 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023800d570 con 0x7f024805cd20 2022-07-12T08:44:46.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024559a700 1 -- 172.21.15.61:0/3647620714 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0238000e90 con 0x7f024805cd20 2022-07-12T08:44:46.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024559a700 1 -- 172.21.15.61:0/3647620714 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023800d8c0 con 0x7f024805cd20 2022-07-12T08:44:46.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024805cd20 msgr2=0x7f024805d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 --2- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024805cd20 0x7f024805d100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f0248147180 tx=0x7f0238006df0).stop 2022-07-12T08:44:46.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- 172.21.15.61:0/3647620714 shutdown_connections 2022-07-12T08:44:46.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 --2- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02480af220 0x7f0248060020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 --2- 172.21.15.61:0/3647620714 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f024805cd20 0x7f024805d100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- 172.21.15.61:0/3647620714 >> 172.21.15.61:0/3647620714 conn(0x7f02480b6820 msgr2=0x7f0248066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- 172.21.15.61:0/3647620714 shutdown_connections 2022-07-12T08:44:46.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.120+0000 7f024759e700 1 -- 172.21.15.61:0/3647620714 wait complete. 2022-07-12T08:44:46.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 Processor -- start 2022-07-12T08:44:46.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 -- start start 2022-07-12T08:44:46.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f024805cd20 0x7f02480b84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 0x7f02480b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02481460e0 0x7f02480b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02480b0440 con 0x7f02480af220 2022-07-12T08:44:46.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0248152440 con 0x7f02481460e0 2022-07-12T08:44:46.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02481526d0 con 0x7f024805cd20 2022-07-12T08:44:46.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f0245d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 0x7f02480b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f0245d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 0x7f02480b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55578/0 (socket says 172.21.15.61:55578) 2022-07-12T08:44:46.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f0245d9b700 1 -- 172.21.15.61:0/1626852001 learned_addr learned my addr 172.21.15.61:0/1626852001 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:46.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f024659c700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f024805cd20 0x7f02480b84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.124+0000 7f0246d9d700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02481460e0 0x7f02480b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f0245d9b700 1 -- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f024805cd20 msgr2=0x7f02480b84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f0245d9b700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f024805cd20 0x7f02480b84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f0245d9b700 1 -- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02481460e0 msgr2=0x7f02480b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f0245d9b700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02481460e0 0x7f02480b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f0245d9b700 1 -- 172.21.15.61:0/1626852001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240009470 con 0x7f02480af220 2022-07-12T08:44:46.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f0245d9b700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 0x7f02480b0870 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f023c0012a0 tx=0x7f023c007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023c00ce80 con 0x7f02480af220 2022-07-12T08:44:46.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c0072e0 con 0x7f02480af220 2022-07-12T08:44:46.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023800b180 con 0x7f02480af220 2022-07-12T08:44:46.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f023c005800 con 0x7f02480af220 2022-07-12T08:44:46.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02480604d0 con 0x7f02480af220 2022-07-12T08:44:46.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0248057460 con 0x7f02480af220 2022-07-12T08:44:46.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f023c010070 con 0x7f02480af220 2022-07-12T08:44:46.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f02377fe700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02240539b0 0x7f0224055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f023c06b140 con 0x7f02480af220 2022-07-12T08:44:46.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.128+0000 7f024659c700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02240539b0 0x7f0224055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.132+0000 7f024659c700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02240539b0 0x7f0224055e70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0238018000 tx=0x7f0238014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.140+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f023c0423d0 con 0x7f02480af220 2022-07-12T08:44:46.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.268+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f02480b05d0 con 0x7f02480af220 2022-07-12T08:44:46.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.268+0000 7f02377fe700 1 -- 172.21.15.61:0/1626852001 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v48) v1 ==== 110+0+10 (secure 0 0 0) 0x7f023c0459f0 con 0x7f02480af220 2022-07-12T08:44:46.273 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:44:46.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02240539b0 msgr2=0x7f0224055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02240539b0 0x7f0224055e70 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0238018000 tx=0x7f0238014000).stop 2022-07-12T08:44:46.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 msgr2=0x7f02480b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 0x7f02480b0870 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f023c0012a0 tx=0x7f023c007a00).stop 2022-07-12T08:44:46.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 shutdown_connections 2022-07-12T08:44:46.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02240539b0 0x7f0224055e70 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02481460e0 0x7f02480b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02480af220 0x7f02480b0870 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 --2- 172.21.15.61:0/1626852001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f024805cd20 0x7f02480b84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 >> 172.21.15.61:0/1626852001 conn(0x7f02480b6820 msgr2=0x7f0248141900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 shutdown_connections 2022-07-12T08:44:46.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.272+0000 7f024759e700 1 -- 172.21.15.61:0/1626852001 wait complete. 2022-07-12T08:44:46.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:44:46.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.400+0000 7fc094e08700 1 Processor -- start 2022-07-12T08:44:46.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.400+0000 7fc094e08700 1 -- start start 2022-07-12T08:44:46.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.400+0000 7fc094e08700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc0900b5920 0x7fc0900b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.400+0000 7fc094e08700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc0900b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0900b00a0 con 0x7fc090067ad0 2022-07-12T08:44:46.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0900b11b0 con 0x7fc0900b5920 2022-07-12T08:44:46.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0900b5d00 con 0x7fc0900afa80 2022-07-12T08:44:46.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc0900b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc0900b5920 0x7fc0900b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc0900b0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:37786/0 (socket says 172.21.15.61:37786) 2022-07-12T08:44:46.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 -- 172.21.15.61:0/3199975255 learned_addr learned my addr 172.21.15.61:0/3199975255 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:46.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08effd700 1 --1- 172.21.15.61:0/3199975255 >> v1:172.21.15.111:6789/0 conn(0x7fc090067ad0 0x7fc09005bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:57408/0 (socket says 172.21.15.61:57408) 2022-07-12T08:44:46.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 -- 172.21.15.61:0/3199975255 >> v2:172.21.15.111:3300/0 conn(0x7fc0900b5920 msgr2=0x7fc0900b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 --2- 172.21.15.61:0/3199975255 >> v2:172.21.15.111:3300/0 conn(0x7fc0900b5920 0x7fc0900b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 -- 172.21.15.61:0/3199975255 >> v1:172.21.15.111:6789/0 conn(0x7fc090067ad0 legacy=0x7fc09005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 -- 172.21.15.61:0/3199975255 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc090098190 con 0x7fc0900afa80 2022-07-12T08:44:46.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08ffff700 1 --2- 172.21.15.61:0/3199975255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc0900b0cc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc088002f70 tx=0x7fc088006eb0).ready entity=mon.2 client_cookie=41cd3a2d0a19221a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08e7fc700 1 -- 172.21.15.61:0/3199975255 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc08800d570 con 0x7fc0900afa80 2022-07-12T08:44:46.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08e7fc700 1 -- 172.21.15.61:0/3199975255 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc088000e90 con 0x7fc0900afa80 2022-07-12T08:44:46.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc08e7fc700 1 -- 172.21.15.61:0/3199975255 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc08800d900 con 0x7fc0900afa80 2022-07-12T08:44:46.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- 172.21.15.61:0/3199975255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 msgr2=0x7fc0900b0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 --2- 172.21.15.61:0/3199975255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc0900b0cc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc088002f70 tx=0x7fc088006eb0).stop 2022-07-12T08:44:46.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- 172.21.15.61:0/3199975255 shutdown_connections 2022-07-12T08:44:46.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 --2- 172.21.15.61:0/3199975255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc0900b0cc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 --2- 172.21.15.61:0/3199975255 >> v2:172.21.15.111:3300/0 conn(0x7fc0900b5920 0x7fc0900b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- 172.21.15.61:0/3199975255 >> 172.21.15.61:0/3199975255 conn(0x7fc090063340 msgr2=0x7fc0900b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- 172.21.15.61:0/3199975255 shutdown_connections 2022-07-12T08:44:46.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.404+0000 7fc094e08700 1 -- 172.21.15.61:0/3199975255 wait complete. 2022-07-12T08:44:46.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 Processor -- start 2022-07-12T08:44:46.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 -- start start 2022-07-12T08:44:46.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc090067ad0 0x7fc090151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc090152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0900b5920 0x7fc090141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090064c60 con 0x7fc0900b5920 2022-07-12T08:44:46.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc090064f00 con 0x7fc090067ad0 2022-07-12T08:44:46.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc094e08700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc090155270 con 0x7fc0900afa80 2022-07-12T08:44:46.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc08ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0900b5920 0x7fc090141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc08f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc090067ad0 0x7fc090151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc08ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0900b5920 0x7fc090141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55596/0 (socket says 172.21.15.61:55596) 2022-07-12T08:44:46.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc08ffff700 1 -- 172.21.15.61:0/1708001195 learned_addr learned my addr 172.21.15.61:0/1708001195 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:46.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.408+0000 7fc08effd700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc090152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08f7fe700 1 -- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 msgr2=0x7fc090152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08f7fe700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc090152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08f7fe700 1 -- 172.21.15.61:0/1708001195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0900b5920 msgr2=0x7fc090141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08f7fe700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0900b5920 0x7fc090141020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08f7fe700 1 -- 172.21.15.61:0/1708001195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08800b1e0 con 0x7fc090067ad0 2022-07-12T08:44:46.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08f7fe700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc090067ad0 0x7fc090151ca0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc07800c000 tx=0x7fc078006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc078010780 con 0x7fc090067ad0 2022-07-12T08:44:46.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0780108e0 con 0x7fc090067ad0 2022-07-12T08:44:46.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc090152b80 con 0x7fc090067ad0 2022-07-12T08:44:46.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc078010c40 con 0x7fc090067ad0 2022-07-12T08:44:46.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0900ac520 con 0x7fc090067ad0 2022-07-12T08:44:46.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc090057460 con 0x7fc090067ad0 2022-07-12T08:44:46.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.412+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc07801c460 con 0x7fc090067ad0 2022-07-12T08:44:46.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.416+0000 7fc08cff9700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc074053920 0x7fc074055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.416+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc07806bdf0 con 0x7fc090067ad0 2022-07-12T08:44:46.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.416+0000 7fc08effd700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc074053920 0x7fc074055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.416+0000 7fc08effd700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc074053920 0x7fc074055de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc08000d000 tx=0x7fc080000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.424+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc078042f10 con 0x7fc090067ad0 2022-07-12T08:44:46.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.548+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc09006b2b0 con 0x7fc090067ad0 2022-07-12T08:44:46.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc08cff9700 1 -- 172.21.15.61:0/1708001195 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v48) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc078046530 con 0x7fc090067ad0 2022-07-12T08:44:46.560 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:44:46.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc074053920 msgr2=0x7fc074055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc074053920 0x7fc074055de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fc08000d000 tx=0x7fc080000f40).stop 2022-07-12T08:44:46.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc090067ad0 msgr2=0x7fc090151ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc090067ad0 0x7fc090151ca0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc07800c000 tx=0x7fc078006df0).stop 2022-07-12T08:44:46.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 shutdown_connections 2022-07-12T08:44:46.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0900b5920 0x7fc090141020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0900afa80 0x7fc090152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc074053920 0x7fc074055de0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 --2- 172.21.15.61:0/1708001195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc090067ad0 0x7fc090151ca0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 >> 172.21.15.61:0/1708001195 conn(0x7fc090063340 msgr2=0x7fc090098710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 shutdown_connections 2022-07-12T08:44:46.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.552+0000 7fc094e08700 1 -- 172.21.15.61:0/1708001195 wait complete. 2022-07-12T08:44:46.570 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:44:46.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.672+0000 7f3eee9ea700 1 Processor -- start 2022-07-12T08:44:46.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- start start 2022-07-12T08:44:46.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3ee80b06b0 0x7f3ee805c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ee8140150 0x7f3ee81405c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee8141f90 con 0x7f3ee8064850 2022-07-12T08:44:46.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee8064150 con 0x7f3ee8140150 2022-07-12T08:44:46.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee805c580 con 0x7f3ee80b06b0 2022-07-12T08:44:46.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ee8140150 0x7f3ee81405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ee8140150 0x7f3ee81405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55604/0 (socket says 172.21.15.61:55604) 2022-07-12T08:44:46.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 -- 172.21.15.61:0/3185612976 learned_addr learned my addr 172.21.15.61:0/3185612976 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:46.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3ee7fff700 1 --2- 172.21.15.61:0/3185612976 >> v2:172.21.15.111:3300/0 conn(0x7f3ee80b06b0 0x7f3ee805c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 -- 172.21.15.61:0/3185612976 >> v2:172.21.15.111:3300/0 conn(0x7f3ee80b06b0 msgr2=0x7f3ee805c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 --2- 172.21.15.61:0/3185612976 >> v2:172.21.15.111:3300/0 conn(0x7f3ee80b06b0 0x7f3ee805c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 -- 172.21.15.61:0/3185612976 >> v1:172.21.15.111:6789/0 conn(0x7f3ee8064850 legacy=0x7f3ee8144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3ee6ffd700 1 -- 172.21.15.61:0/3185612976 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963201988 0 0) 0x7f3ee8141f90 con 0x7f3ee8064850 2022-07-12T08:44:46.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 -- 172.21.15.61:0/3185612976 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80643b0 con 0x7f3ee8140150 2022-07-12T08:44:46.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eecf87700 1 --2- 172.21.15.61:0/3185612976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8140150 0x7f3ee81405c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f3ee00032f0 tx=0x7f3ee0009290).ready entity=mon.0 client_cookie=4fe493950f99409e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3ee6ffd700 1 -- 172.21.15.61:0/3185612976 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee000b870 con 0x7f3ee8140150 2022-07-12T08:44:46.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3ee6ffd700 1 -- 172.21.15.61:0/3185612976 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee000b9d0 con 0x7f3ee8140150 2022-07-12T08:44:46.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3ee6ffd700 1 -- 172.21.15.61:0/3185612976 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee000bb40 con 0x7f3ee8140150 2022-07-12T08:44:46.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3185612976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8140150 msgr2=0x7f3ee81405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3185612976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8140150 0x7f3ee81405c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f3ee00032f0 tx=0x7f3ee0009290).stop 2022-07-12T08:44:46.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3185612976 shutdown_connections 2022-07-12T08:44:46.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3185612976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8140150 0x7f3ee81405c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3185612976 >> v2:172.21.15.111:3300/0 conn(0x7f3ee80b06b0 0x7f3ee805c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3185612976 >> 172.21.15.61:0/3185612976 conn(0x7f3ee8062c00 msgr2=0x7f3ee8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3185612976 shutdown_connections 2022-07-12T08:44:46.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.676+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3185612976 wait complete. 2022-07-12T08:44:46.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 Processor -- start 2022-07-12T08:44:46.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 -- start start 2022-07-12T08:44:46.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 0x7f3ee8141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ee80b06b0 0x7f3ee8153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3ee8140150 0x7f3ee80a25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee80a2190 con 0x7f3ee8064850 2022-07-12T08:44:46.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee809fae0 con 0x7f3ee80b06b0 2022-07-12T08:44:46.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eee9ea700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee809e6d0 con 0x7f3ee8140150 2022-07-12T08:44:46.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 0x7f3ee8141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 0x7f3ee8141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55606/0 (socket says 172.21.15.61:55606) 2022-07-12T08:44:46.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 -- 172.21.15.61:0/3356743351 learned_addr learned my addr 172.21.15.61:0/3356743351 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:46.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee77fe700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ee80b06b0 0x7f3ee8153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3eecf87700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3ee8140150 0x7f3ee80a25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 -- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3ee8140150 msgr2=0x7f3ee80a25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3ee8140150 0x7f3ee80a25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 -- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ee80b06b0 msgr2=0x7f3ee8153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ee80b06b0 0x7f3ee8153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 -- 172.21.15.61:0/3356743351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee0009530 con 0x7f3ee8064850 2022-07-12T08:44:46.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee7fff700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 0x7f3ee8141a20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f3ed800c000 tx=0x7f3ed8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.680+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ed8010780 con 0x7f3ee8064850 2022-07-12T08:44:46.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed800b3b0 con 0x7f3ee8064850 2022-07-12T08:44:46.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee809fd80 con 0x7f3ee8064850 2022-07-12T08:44:46.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ed8010ad0 con 0x7f3ee8064850 2022-07-12T08:44:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee809ef30 con 0x7f3ee8064850 2022-07-12T08:44:46.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee809ec40 con 0x7f3ee8064850 2022-07-12T08:44:46.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3ed8010c30 con 0x7f3ee8064850 2022-07-12T08:44:46.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3ee57fa700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ec8053ad0 0x7f3ec8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3ed806bed0 con 0x7f3ee8064850 2022-07-12T08:44:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.684+0000 7f3ee77fe700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ec8053ad0 0x7f3ec8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.688+0000 7f3ee77fe700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ec8053ad0 0x7f3ec8055f90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3edc00d000 tx=0x7f3edc007ab0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.692+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ed8043060 con 0x7f3ee8064850 2022-07-12T08:44:46.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.816+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3ee8057460 con 0x7f3ee8064850 2022-07-12T08:44:46.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3ee57fa700 1 -- 172.21.15.61:0/3356743351 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v48) v1 ==== 84+0+143 (secure 0 0 0) 0x7f3ed8046680 con 0x7f3ee8064850 2022-07-12T08:44:46.824 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:46.824 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:44:46.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:46.825 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:44:46.825 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:44:46.825 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:44:46.825 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:44:46.826 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:44:46.826 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:44:46.826 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:44:46.828 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:44:46.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ec8053ad0 msgr2=0x7f3ec8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ec8053ad0 0x7f3ec8055f90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3edc00d000 tx=0x7f3edc007ab0).stop 2022-07-12T08:44:46.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 msgr2=0x7f3ee8141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 0x7f3ee8141a20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f3ed800c000 tx=0x7f3ed8006df0).stop 2022-07-12T08:44:46.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 shutdown_connections 2022-07-12T08:44:46.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3ee8140150 0x7f3ee80a25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ec8053ad0 0x7f3ec8055f90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ee80b06b0 0x7f3ee8153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 --2- 172.21.15.61:0/3356743351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ee8064850 0x7f3ee8141a20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 >> 172.21.15.61:0/3356743351 conn(0x7f3ee8062c00 msgr2=0x7f3ee80b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 shutdown_connections 2022-07-12T08:44:46.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.820+0000 7f3eee9ea700 1 -- 172.21.15.61:0/3356743351 wait complete. 2022-07-12T08:44:46.850 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:44:46.851 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:44:46.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 Processor -- start 2022-07-12T08:44:46.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 -- start start 2022-07-12T08:44:46.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc2240b05f0 0x7fc2240b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc2240b3730 0x7fc224142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2240b3560 con 0x7fc2240b5480 2022-07-12T08:44:46.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240b5b80 con 0x7fc2240b05f0 2022-07-12T08:44:46.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc22a9a3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240b5e20 con 0x7fc2240b3730 2022-07-12T08:44:46.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc223fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc2240b5480 0x7fc2240afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43832/0 (socket says 172.21.15.61:43832) 2022-07-12T08:44:46.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc223fff700 1 -- 172.21.15.61:0/1399657068 learned_addr learned my addr 172.21.15.61:0/1399657068 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:46.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.960+0000 7fc2237fe700 1 --2- 172.21.15.61:0/1399657068 >> v2:172.21.15.61:3300/0 conn(0x7fc2240b05f0 0x7fc2240b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc228f40700 1 --2- 172.21.15.61:0/1399657068 >> v2:172.21.15.111:3300/0 conn(0x7fc2240b3730 0x7fc224142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847778473 0 0) 0x7fc2240b3560 con 0x7fc2240b5480 2022-07-12T08:44:46.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204003260 con 0x7fc2240b5480 2022-07-12T08:44:46.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368121013 0 0) 0x7fc204003260 con 0x7fc2240b5480 2022-07-12T08:44:46.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 >> v2:172.21.15.111:3300/0 conn(0x7fc2240b3730 msgr2=0x7fc224142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 --2- 172.21.15.61:0/1399657068 >> v2:172.21.15.111:3300/0 conn(0x7fc2240b3730 0x7fc224142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 >> v2:172.21.15.61:3300/0 conn(0x7fc2240b05f0 msgr2=0x7fc2240b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 --2- 172.21.15.61:0/1399657068 >> v2:172.21.15.61:3300/0 conn(0x7fc2240b05f0 0x7fc2240b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240b5860 con 0x7fc2240b5480 2022-07-12T08:44:46.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fc21c0019e0 con 0x7fc2240b5480 2022-07-12T08:44:46.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 >> v1:172.21.15.61:6789/0 conn(0x7fc2240b5480 legacy=0x7fc2240afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc204003750 0x7fc204003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc204004500 0x7fc204004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204004e70 0x7fc20400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240b5b80 con 0x7fc204004500 2022-07-12T08:44:46.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240b5e20 con 0x7fc204003750 2022-07-12T08:44:46.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20400bc40 con 0x7fc204004e70 2022-07-12T08:44:46.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc204003750 0x7fc204003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc228f40700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204004e70 0x7fc20400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc2237fe700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc204004500 0x7fc204004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:46.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 -- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204004e70 msgr2=0x7fc20400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204004e70 0x7fc20400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 -- 172.21.15.61:0/1399657068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc204004500 msgr2=0x7fc204004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:46.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc204004500 0x7fc204004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 -- 172.21.15.61:0/1399657068 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc204003260 con 0x7fc204003750 2022-07-12T08:44:46.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc228f40700 1 -- 172.21.15.61:0/1399657068 reap_dead start 2022-07-12T08:44:46.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc2237fe700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc204004500 0x7fc204004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:46.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc223fff700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc204003750 0x7fc204003b70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fc21c0016f0 tx=0x7fc21c0021d0).ready entity=mon.1 client_cookie=45c8f7b702db8e14 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:46.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21c00cba0 con 0x7fc204003750 2022-07-12T08:44:46.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c00cd00 con 0x7fc204003750 2022-07-12T08:44:46.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc222ffd700 1 -- 172.21.15.61:0/1399657068 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21c005b80 con 0x7fc204003750 2022-07-12T08:44:46.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 -- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc204003750 msgr2=0x7fc204003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:46.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc204003750 0x7fc204003b70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fc21c0016f0 tx=0x7fc21c0021d0).stop 2022-07-12T08:44:46.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 -- 172.21.15.61:0/1399657068 shutdown_connections 2022-07-12T08:44:46.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/1399657068 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc204003750 0x7fc204003b70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:46.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 -- 172.21.15.61:0/1399657068 >> 172.21.15.61:0/1399657068 conn(0x7fc2240b96b0 msgr2=0x7fc2240b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:46.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 -- 172.21.15.61:0/1399657068 shutdown_connections 2022-07-12T08:44:46.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 -- 172.21.15.61:0/1399657068 wait complete. 2022-07-12T08:44:46.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 Processor -- start 2022-07-12T08:44:46.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.964+0000 7fc22a9a3700 1 -- start start 2022-07-12T08:44:46.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204003750 0x7fc224141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 0x7fc22413e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc22413e500 0x7fc2240aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:46.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240acce0 con 0x7fc2240ad420 2022-07-12T08:44:46.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240acf80 con 0x7fc22413e500 2022-07-12T08:44:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc224147fa0 con 0x7fc204003750 2022-07-12T08:44:47.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 0x7fc22413e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 0x7fc22413e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55628/0 (socket says 172.21.15.61:55628) 2022-07-12T08:44:47.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 -- 172.21.15.61:0/108810691 learned_addr learned my addr 172.21.15.61:0/108810691 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:47.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc223fff700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204003750 0x7fc224141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc228f40700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc22413e500 0x7fc2240aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 -- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204003750 msgr2=0x7fc224141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204003750 0x7fc224141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 -- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc22413e500 msgr2=0x7fc2240aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc22413e500 0x7fc2240aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 -- 172.21.15.61:0/108810691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240b01e0 con 0x7fc2240ad420 2022-07-12T08:44:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2237fe700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 0x7fc22413e010 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fc20c00b000 tx=0x7fc20c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:47.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc20c00f830 con 0x7fc2240ad420 2022-07-12T08:44:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c00f990 con 0x7fc2240ad420 2022-07-12T08:44:47.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22413e980 con 0x7fc2240ad420 2022-07-12T08:44:47.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc20c00fcf0 con 0x7fc2240ad420 2022-07-12T08:44:47.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc224142290 con 0x7fc2240ad420 2022-07-12T08:44:47.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc20c01e460 con 0x7fc2240ad420 2022-07-12T08:44:47.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.968+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2240ad140 con 0x7fc2240ad420 2022-07-12T08:44:47.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.972+0000 7fc2217fa700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc208053ad0 0x7fc208055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.972+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc20c06c5d0 con 0x7fc2240ad420 2022-07-12T08:44:47.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.972+0000 7fc223fff700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc208053ad0 0x7fc208055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.972+0000 7fc223fff700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc208053ad0 0x7fc208055f90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fc21c01a000 tx=0x7fc21c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:47.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:46.976+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc20c043780 con 0x7fc2240ad420 2022-07-12T08:44:47.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.104+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fc2240b03d0 con 0x7fc2240ad420 2022-07-12T08:44:47.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.288+0000 7fc2217fa700 1 -- 172.21.15.61:0/108810691 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v49) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc20c046da0 con 0x7fc2240ad420 2022-07-12T08:44:47.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc208053ad0 msgr2=0x7fc208055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc208053ad0 0x7fc208055f90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fc21c01a000 tx=0x7fc21c015000).stop 2022-07-12T08:44:47.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 msgr2=0x7fc22413e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 0x7fc22413e010 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7fc20c00b000 tx=0x7fc20c006df0).stop 2022-07-12T08:44:47.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 shutdown_connections 2022-07-12T08:44:47.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc208053ad0 0x7fc208055f90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc22413e500 0x7fc2240aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2240ad420 0x7fc22413e010 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 --2- 172.21.15.61:0/108810691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc204003750 0x7fc224141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 >> 172.21.15.61:0/108810691 conn(0x7fc2240b96b0 msgr2=0x7fc22405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:47.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 shutdown_connections 2022-07-12T08:44:47.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.296+0000 7fc22a9a3700 1 -- 172.21.15.61:0/108810691 wait complete. 2022-07-12T08:44:47.303 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:44:47.317 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:44:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 Processor -- start 2022-07-12T08:44:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 -- start start 2022-07-12T08:44:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f12b80b5920 0x7f12b80b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f12b8098c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b8097c80 con 0x7f12b8143580 2022-07-12T08:44:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b80b5d00 con 0x7f12b80b5920 2022-07-12T08:44:47.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b805c7f0 con 0x7f12b80987f0 2022-07-12T08:44:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b6d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f12b8143580 0x7f12b805faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43854/0 (socket says 172.21.15.61:43854) 2022-07-12T08:44:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.428+0000 7f12b6d9d700 1 -- 172.21.15.61:0/2478429397 learned_addr learned my addr 172.21.15.61:0/2478429397 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b659c700 1 --2- 172.21.15.61:0/2478429397 >> v2:172.21.15.111:3300/0 conn(0x7f12b80b5920 0x7f12b80b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f12b8098c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248031375 0 0) 0x7f12b8097c80 con 0x7f12b8143580 2022-07-12T08:44:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c003260 con 0x7f12b8143580 2022-07-12T08:44:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255096814 0 0) 0x7f129c003260 con 0x7f12b8143580 2022-07-12T08:44:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 msgr2=0x7f12b8098c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f12b8098c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 >> v2:172.21.15.111:3300/0 conn(0x7f12b80b5920 msgr2=0x7f12b80b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:47.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 --2- 172.21.15.61:0/2478429397 >> v2:172.21.15.111:3300/0 conn(0x7f12b80b5920 0x7f12b80b01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b805cbf0 con 0x7f12b8143580 2022-07-12T08:44:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f12b00019e0 con 0x7f12b8143580 2022-07-12T08:44:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f12b8098c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:47.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 >> v1:172.21.15.61:6789/0 conn(0x7f12b8143580 legacy=0x7f12b805faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f129c003750 0x7f129c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c004500 0x7f129c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f129c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b80b5d00 con 0x7f129c004500 2022-07-12T08:44:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b805c7f0 con 0x7f129c003750 2022-07-12T08:44:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c00b750 con 0x7f12b80987f0 2022-07-12T08:44:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b659c700 1 --2- 172.21.15.61:0/2478429397 >> v2:172.21.15.111:3300/0 conn(0x7f12b80b5920 0x7f12b80b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:44:47.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c004500 0x7f129c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b6d9d700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f129c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b659c700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f129c003750 0x7f129c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 msgr2=0x7f129c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:47.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f12b80987f0 0x7f129c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f129c003750 msgr2=0x7f129c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:47.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f129c003750 0x7f129c003b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2478429397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a800b180 con 0x7f129c004500 2022-07-12T08:44:47.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c004500 0x7f129c004980 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f12a800c000 tx=0x7f12a8006df0).ready entity=mon.0 client_cookie=ce410b94e7bdfc20 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12a800e830 con 0x7f129c004500 2022-07-12T08:44:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a800e990 con 0x7f129c004500 2022-07-12T08:44:47.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b559a700 1 -- 172.21.15.61:0/2478429397 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12a800ecf0 con 0x7f129c004500 2022-07-12T08:44:47.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b759e700 1 -- 172.21.15.61:0/2478429397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c004500 msgr2=0x7f129c004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b759e700 1 --2- 172.21.15.61:0/2478429397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c004500 0x7f129c004980 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f12a800c000 tx=0x7f12a8006df0).stop 2022-07-12T08:44:47.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b6d9d700 1 -- 172.21.15.61:0/2478429397 reap_dead start 2022-07-12T08:44:47.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b759e700 1 -- 172.21.15.61:0/2478429397 shutdown_connections 2022-07-12T08:44:47.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.432+0000 7f12b759e700 1 -- 172.21.15.61:0/2478429397 >> 172.21.15.61:0/2478429397 conn(0x7f12b8063340 msgr2=0x7f12b80b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:47.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 -- 172.21.15.61:0/2478429397 shutdown_connections 2022-07-12T08:44:47.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 -- 172.21.15.61:0/2478429397 wait complete. 2022-07-12T08:44:47.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 Processor -- start 2022-07-12T08:44:47.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 -- start start 2022-07-12T08:44:47.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f129c004500 0x7f12b813e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 0x7f12b813d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b80aebf0 0x7f12b813c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b813f340 con 0x7f12b813fa80 2022-07-12T08:44:47.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b813f5e0 con 0x7f12b80aebf0 2022-07-12T08:44:47.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b813cb00 con 0x7f129c004500 2022-07-12T08:44:47.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.436+0000 7f12b5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 0x7f12b813d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 0x7f12b813d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55648/0 (socket says 172.21.15.61:55648) 2022-07-12T08:44:47.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2925939371 learned_addr learned my addr 172.21.15.61:0/2925939371 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:47.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b6d9d700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b80aebf0 0x7f12b813c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b659c700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f129c004500 0x7f12b813e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f129c004500 msgr2=0x7f12b813e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f129c004500 0x7f12b813e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b80aebf0 msgr2=0x7f12b813c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:47.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b80aebf0 0x7f12b813c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:47.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 -- 172.21.15.61:0/2925939371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b813ccc0 con 0x7f12b813fa80 2022-07-12T08:44:47.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b5d9b700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 0x7f12b813d780 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f12a800c000 tx=0x7f12a8019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:47.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12a801b610 con 0x7f12b813fa80 2022-07-12T08:44:47.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8000e90 con 0x7f12b813fa80 2022-07-12T08:44:47.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b813ce50 con 0x7f12b813fa80 2022-07-12T08:44:47.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12a801b960 con 0x7f12b813fa80 2022-07-12T08:44:47.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b80add30 con 0x7f12b813fa80 2022-07-12T08:44:47.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b813eb90 con 0x7f12b813fa80 2022-07-12T08:44:47.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.440+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f12a801bac0 con 0x7f12b813fa80 2022-07-12T08:44:47.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.444+0000 7f12a77fe700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1294053ad0 0x7f1294055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:47.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.444+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f12a806d340 con 0x7f12b813fa80 2022-07-12T08:44:47.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.444+0000 7f12b659c700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1294053ad0 0x7f1294055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:47.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.444+0000 7f12b659c700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1294053ad0 0x7f1294055f90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f12ac00b000 tx=0x7f12ac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:47.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.452+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12a8044a20 con 0x7f12b813fa80 2022-07-12T08:44:47.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:47.580+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f12b813f7a0 con 0x7f12b813fa80 2022-07-12T08:44:48.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.296+0000 7f12a77fe700 1 -- 172.21.15.61:0/2925939371 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v50) v1 ==== 125+0+0 (secure 0 0 0) 0x7f12a8047a90 con 0x7f12b813fa80 2022-07-12T08:44:48.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1294053ad0 msgr2=0x7f1294055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1294053ad0 0x7f1294055f90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f12ac00b000 tx=0x7f12ac008040).stop 2022-07-12T08:44:48.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 msgr2=0x7f12b813d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 0x7f12b813d780 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f12a800c000 tx=0x7f12a8019000).stop 2022-07-12T08:44:48.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 shutdown_connections 2022-07-12T08:44:48.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1294053ad0 0x7f1294055f90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f12b80aebf0 0x7f12b813c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12b813fa80 0x7f12b813d780 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 --2- 172.21.15.61:0/2925939371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f129c004500 0x7f12b813e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 >> 172.21.15.61:0/2925939371 conn(0x7f12b8063340 msgr2=0x7f12b8099aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:48.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 shutdown_connections 2022-07-12T08:44:48.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.304+0000 7f12b759e700 1 -- 172.21.15.61:0/2925939371 wait complete. 2022-07-12T08:44:48.315 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:44:48.322 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:44:48.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.436+0000 7ff6cc898700 1 Processor -- start 2022-07-12T08:44:48.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.436+0000 7ff6cc898700 1 -- start start 2022-07-12T08:44:48.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b5ea0 0x7ff6c8141f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff6c80b1f50 0x7ff6c80b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c80b5500 con 0x7ff6c80b3cf0 2022-07-12T08:44:48.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c80b34d0 con 0x7ff6c80b1f50 2022-07-12T08:44:48.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c80b3970 con 0x7ff6c80b5ea0 2022-07-12T08:44:48.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6beffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff6c80b3cf0 0x7ff6c8142480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43874/0 (socket says 172.21.15.61:43874) 2022-07-12T08:44:48.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6beffd700 1 -- 172.21.15.61:0/4064466194 learned_addr learned my addr 172.21.15.61:0/4064466194 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:48.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c77fe700 1 --2- 172.21.15.61:0/4064466194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b5ea0 0x7ff6c8141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c7fff700 1 --2- 172.21.15.61:0/4064466194 >> v2:172.21.15.111:3300/0 conn(0x7ff6c80b1f50 0x7ff6c80b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c6ffd700 1 -- 172.21.15.61:0/4064466194 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084998909 0 0) 0x7ff6c80b5500 con 0x7ff6c80b3cf0 2022-07-12T08:44:48.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c6ffd700 1 -- 172.21.15.61:0/4064466194 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a8003260 con 0x7ff6c80b3cf0 2022-07-12T08:44:48.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c77fe700 1 -- 172.21.15.61:0/4064466194 >> v2:172.21.15.111:3300/0 conn(0x7ff6c80b1f50 msgr2=0x7ff6c80b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c77fe700 1 --2- 172.21.15.61:0/4064466194 >> v2:172.21.15.111:3300/0 conn(0x7ff6c80b1f50 0x7ff6c80b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c77fe700 1 -- 172.21.15.61:0/4064466194 >> v1:172.21.15.61:6789/0 conn(0x7ff6c80b3cf0 legacy=0x7ff6c8142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c77fe700 1 -- 172.21.15.61:0/4064466194 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c80b57a0 con 0x7ff6c80b5ea0 2022-07-12T08:44:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c77fe700 1 --2- 172.21.15.61:0/4064466194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b5ea0 0x7ff6c8141f90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff6b8002f70 tx=0x7ff6b8006df0).ready entity=mon.2 client_cookie=69f388de3f70df18 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c6ffd700 1 -- 172.21.15.61:0/4064466194 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b800d4a0 con 0x7ff6c80b5ea0 2022-07-12T08:44:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c6ffd700 1 -- 172.21.15.61:0/4064466194 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b8000e90 con 0x7ff6c80b5ea0 2022-07-12T08:44:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6c6ffd700 1 -- 172.21.15.61:0/4064466194 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b800d7f0 con 0x7ff6c80b5ea0 2022-07-12T08:44:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- 172.21.15.61:0/4064466194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b5ea0 msgr2=0x7ff6c8141f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 --2- 172.21.15.61:0/4064466194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b5ea0 0x7ff6c8141f90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff6b8002f70 tx=0x7ff6b8006df0).stop 2022-07-12T08:44:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- 172.21.15.61:0/4064466194 shutdown_connections 2022-07-12T08:44:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 --2- 172.21.15.61:0/4064466194 >> v2:172.21.15.111:3300/0 conn(0x7ff6c80b1f50 0x7ff6c80b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 --2- 172.21.15.61:0/4064466194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b5ea0 0x7ff6c8141f90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- 172.21.15.61:0/4064466194 >> 172.21.15.61:0/4064466194 conn(0x7ff6c8066d70 msgr2=0x7ff6c8067150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- 172.21.15.61:0/4064466194 shutdown_connections 2022-07-12T08:44:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.440+0000 7ff6cc898700 1 -- 172.21.15.61:0/4064466194 wait complete. 2022-07-12T08:44:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 Processor -- start 2022-07-12T08:44:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 -- start start 2022-07-12T08:44:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 0x7ff6c80b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b3cf0 0x7ff6c8145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6c80b5ea0 0x7ff6c8145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8140c80 con 0x7ff6c80b1f50 2022-07-12T08:44:48.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8147ce0 con 0x7ff6c80b5ea0 2022-07-12T08:44:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6cc898700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8147f80 con 0x7ff6c80b3cf0 2022-07-12T08:44:48.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 0x7ff6c80b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 0x7ff6c80b8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55660/0 (socket says 172.21.15.61:55660) 2022-07-12T08:44:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6c77fe700 1 -- 172.21.15.61:0/2197622786 learned_addr learned my addr 172.21.15.61:0/2197622786 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6beffd700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b3cf0 0x7ff6c8145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.444+0000 7ff6c7fff700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6c80b5ea0 0x7ff6c8145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c77fe700 1 -- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b3cf0 msgr2=0x7ff6c8145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c77fe700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b3cf0 0x7ff6c8145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c77fe700 1 -- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6c80b5ea0 msgr2=0x7ff6c8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c77fe700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6c80b5ea0 0x7ff6c8145770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c77fe700 1 -- 172.21.15.61:0/2197622786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b800b180 con 0x7ff6c80b1f50 2022-07-12T08:44:48.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c7fff700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6c80b5ea0 0x7ff6c8145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:48.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c77fe700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 0x7ff6c80b8b40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff6b8002f70 tx=0x7ff6b8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:48.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b8016070 con 0x7ff6c80b1f50 2022-07-12T08:44:48.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b80138d0 con 0x7ff6c80b1f50 2022-07-12T08:44:48.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c8140900 con 0x7ff6c80b1f50 2022-07-12T08:44:48.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b8013c30 con 0x7ff6c80b1f50 2022-07-12T08:44:48.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c8145c60 con 0x7ff6c80b1f50 2022-07-12T08:44:48.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c8140a90 con 0x7ff6c80b1f50 2022-07-12T08:44:48.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.448+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff6b8025030 con 0x7ff6c80b1f50 2022-07-12T08:44:48.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.452+0000 7ff6c57fa700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff6ac053ad0 0x7ff6ac055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.452+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff6b806ce00 con 0x7ff6c80b1f50 2022-07-12T08:44:48.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.452+0000 7ff6beffd700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff6ac053ad0 0x7ff6ac055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.452+0000 7ff6beffd700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff6ac053ad0 0x7ff6ac055f90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff6b000d000 tx=0x7ff6b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:48.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.456+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff6b8043dd0 con 0x7ff6c80b1f50 2022-07-12T08:44:48.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.620+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff6c8057460 con 0x7ff6c80b1f50 2022-07-12T08:44:48.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.632+0000 7ff6c57fa700 1 -- 172.21.15.61:0/2197622786 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 9 and data pool 10 v22) v1 ==== 188+0+0 (secure 0 0 0) 0x7ff6b8023070 con 0x7ff6c80b1f50 2022-07-12T08:44:48.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff6ac053ad0 msgr2=0x7ff6ac055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff6ac053ad0 0x7ff6ac055f90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7ff6b000d000 tx=0x7ff6b0009040).stop 2022-07-12T08:44:48.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 msgr2=0x7ff6c80b8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 0x7ff6c80b8b40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7ff6b8002f70 tx=0x7ff6b8014000).stop 2022-07-12T08:44:48.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 shutdown_connections 2022-07-12T08:44:48.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff6ac053ad0 0x7ff6ac055f90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6c80b5ea0 0x7ff6c8145770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff6c80b3cf0 0x7ff6c8145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 --2- 172.21.15.61:0/2197622786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6c80b1f50 0x7ff6c80b8b40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 >> 172.21.15.61:0/2197622786 conn(0x7ff6c8066d70 msgr2=0x7ff6c8142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:48.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 shutdown_connections 2022-07-12T08:44:48.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.636+0000 7ff6cc898700 1 -- 172.21.15.61:0/2197622786 wait complete. 2022-07-12T08:44:48.653 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 9 and data pool 10 2022-07-12T08:44:48.654 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:44:48.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 Processor -- start 2022-07-12T08:44:48.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- start start 2022-07-12T08:44:48.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f393c05c050 0x7f393c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f393c0b01f0 0x7f393c14a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05cb00 0x7f393c05cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c0b0a40 con 0x7f393c05c050 2022-07-12T08:44:48.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c05d470 con 0x7f393c0b01f0 2022-07-12T08:44:48.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0ad2f0 con 0x7f393c05cb00 2022-07-12T08:44:48.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f393c05c050 0x7f393c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f393c05c050 0x7f393c0ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55678/0 (socket says 172.21.15.61:55678) 2022-07-12T08:44:48.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 -- 172.21.15.61:0/669563499 learned_addr learned my addr 172.21.15.61:0/669563499 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:48.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393a7fc700 1 --2- 172.21.15.61:0/669563499 >> v2:172.21.15.111:3300/0 conn(0x7f393c0b01f0 0x7f393c14a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393b7fe700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05cb00 0x7f393c05cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 -- 172.21.15.61:0/669563499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05cb00 msgr2=0x7f393c05cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05cb00 0x7f393c05cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 -- 172.21.15.61:0/669563499 >> v2:172.21.15.111:3300/0 conn(0x7f393c0b01f0 msgr2=0x7f393c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 --2- 172.21.15.61:0/669563499 >> v2:172.21.15.111:3300/0 conn(0x7f393c0b01f0 0x7f393c14a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 -- 172.21.15.61:0/669563499 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c05c430 con 0x7f393c05c050 2022-07-12T08:44:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393a7fc700 1 --2- 172.21.15.61:0/669563499 >> v2:172.21.15.111:3300/0 conn(0x7f393c0b01f0 0x7f393c14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393b7fe700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05cb00 0x7f393c05cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:44:48.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f393affd700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c05c050 0x7f393c0ac710 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f393c0b1940 tx=0x7f3924006eb0).ready entity=mon.0 client_cookie=4f553e5a0d8d6bfd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3939ffb700 1 -- 172.21.15.61:0/669563499 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f392400d570 con 0x7f393c05c050 2022-07-12T08:44:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3939ffb700 1 -- 172.21.15.61:0/669563499 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924000e90 con 0x7f393c05c050 2022-07-12T08:44:48.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3939ffb700 1 -- 172.21.15.61:0/669563499 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f392400d8c0 con 0x7f393c05c050 2022-07-12T08:44:48.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- 172.21.15.61:0/669563499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c05c050 msgr2=0x7f393c0ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c05c050 0x7f393c0ac710 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f393c0b1940 tx=0x7f3924006eb0).stop 2022-07-12T08:44:48.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- 172.21.15.61:0/669563499 shutdown_connections 2022-07-12T08:44:48.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05cb00 0x7f393c05cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- 172.21.15.61:0/669563499 >> v2:172.21.15.111:3300/0 conn(0x7f393c0b01f0 0x7f393c14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 --2- 172.21.15.61:0/669563499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c05c050 0x7f393c0ac710 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- 172.21.15.61:0/669563499 >> 172.21.15.61:0/669563499 conn(0x7f393c062c00 msgr2=0x7f393c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:48.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.748+0000 7f3941a04700 1 -- 172.21.15.61:0/669563499 shutdown_connections 2022-07-12T08:44:48.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 -- 172.21.15.61:0/669563499 wait complete. 2022-07-12T08:44:48.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 Processor -- start 2022-07-12T08:44:48.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 -- start start 2022-07-12T08:44:48.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05c050 0x7f393c143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f393c05cb00 0x7f393c064ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 0x7f393c0653b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c15a2f0 con 0x7f393c0b01f0 2022-07-12T08:44:48.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c148f40 con 0x7f393c05cb00 2022-07-12T08:44:48.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f3941a04700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c149370 con 0x7f393c05c050 2022-07-12T08:44:48.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f393b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 0x7f393c0653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f393b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 0x7f393c0653b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55686/0 (socket says 172.21.15.61:55686) 2022-07-12T08:44:48.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f393b7fe700 1 -- 172.21.15.61:0/3513977930 learned_addr learned my addr 172.21.15.61:0/3513977930 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:48.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f393a7fc700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f393c05cb00 0x7f393c064ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.752+0000 7f393affd700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05c050 0x7f393c143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393b7fe700 1 -- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05c050 msgr2=0x7f393c143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393b7fe700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05c050 0x7f393c143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393b7fe700 1 -- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f393c05cb00 msgr2=0x7f393c064ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393b7fe700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f393c05cb00 0x7f393c064ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393b7fe700 1 -- 172.21.15.61:0/3513977930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392400b1e0 con 0x7f393c0b01f0 2022-07-12T08:44:48.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393affd700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05c050 0x7f393c143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:48.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f393b7fe700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 0x7f393c0653b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f393c064a70 tx=0x7f392c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:48.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f392c015070 con 0x7f393c0b01f0 2022-07-12T08:44:48.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392c010580 con 0x7f393c0b01f0 2022-07-12T08:44:48.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c15a4f0 con 0x7f393c0b01f0 2022-07-12T08:44:48.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f392c0108e0 con 0x7f393c0b01f0 2022-07-12T08:44:48.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c14a030 con 0x7f393c0b01f0 2022-07-12T08:44:48.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c057460 con 0x7f393c0b01f0 2022-07-12T08:44:48.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f392c010a40 con 0x7f393c0b01f0 2022-07-12T08:44:48.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.756+0000 7f3933fff700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3920053a60 0x7f3920055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:48.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.760+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f392c06c830 con 0x7f393c0b01f0 2022-07-12T08:44:48.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.760+0000 7f393affd700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3920053a60 0x7f3920055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:48.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.760+0000 7f393affd700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3920053a60 0x7f3920055f20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3924019000 tx=0x7f3924015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:48.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.764+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f392c043680 con 0x7f393c0b01f0 2022-07-12T08:44:48.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.888+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f393c149080 con 0x7f393c0b01f0 2022-07-12T08:44:48.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3933fff700 1 -- 172.21.15.61:0/3513977930 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16114 (secure 0 0 0) 0x7f392c046ca0 con 0x7f393c0b01f0 2022-07-12T08:44:48.900 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:48.900 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:44:48.629524+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:44:47.163319+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-12T08:44:47.639688+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:44:48.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3920053a60 msgr2=0x7f3920055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3920053a60 0x7f3920055f20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f3924019000 tx=0x7f3924015000).stop 2022-07-12T08:44:48.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 msgr2=0x7f393c0653b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:48.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 0x7f393c0653b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f393c064a70 tx=0x7f392c008040).stop 2022-07-12T08:44:48.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 shutdown_connections 2022-07-12T08:44:48.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f393c0b01f0 0x7f393c0653b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3920053a60 0x7f3920055f20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f393c05cb00 0x7f393c064ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 --2- 172.21.15.61:0/3513977930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f393c05c050 0x7f393c143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:48.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 >> 172.21.15.61:0/3513977930 conn(0x7f393c062c00 msgr2=0x7f393c147b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:48.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 shutdown_connections 2022-07-12T08:44:48.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:48.892+0000 7f3941a04700 1 -- 172.21.15.61:0/3513977930 wait complete. 2022-07-12T08:44:48.913 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:44:49.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.012+0000 7fbc4bdc0700 1 Processor -- start 2022-07-12T08:44:49.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.012+0000 7fbc4bdc0700 1 -- start start 2022-07-12T08:44:49.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc4405c050 0x7fbc440ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbc4414a8f0 0x7fbc441443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440ad2f0 con 0x7fbc440b01f0 2022-07-12T08:44:49.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc440b0a40 con 0x7fbc4405c050 2022-07-12T08:44:49.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4405c430 con 0x7fbc4414a8f0 2022-07-12T08:44:49.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4935b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbc440b01f0 0x7fbc4405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43910/0 (socket says 172.21.15.61:43910) 2022-07-12T08:44:49.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4935b700 1 -- 172.21.15.61:0/4165966095 learned_addr learned my addr 172.21.15.61:0/4165966095 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:49.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/4165966095 >> v2:172.21.15.61:3300/0 conn(0x7fbc4405c050 0x7fbc440ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4a35d700 1 --2- 172.21.15.61:0/4165966095 >> v2:172.21.15.111:3300/0 conn(0x7fbc4414a8f0 0x7fbc441443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698849868 0 0) 0x7fbc440ad2f0 con 0x7fbc440b01f0 2022-07-12T08:44:49.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c003260 con 0x7fbc440b01f0 2022-07-12T08:44:49.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3032895977 0 0) 0x7fbc2c003260 con 0x7fbc440b01f0 2022-07-12T08:44:49.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 >> v2:172.21.15.111:3300/0 conn(0x7fbc4414a8f0 msgr2=0x7fbc441443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 --2- 172.21.15.61:0/4165966095 >> v2:172.21.15.111:3300/0 conn(0x7fbc4414a8f0 0x7fbc441443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 >> v2:172.21.15.61:3300/0 conn(0x7fbc4405c050 msgr2=0x7fbc440ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 --2- 172.21.15.61:0/4165966095 >> v2:172.21.15.61:3300/0 conn(0x7fbc4405c050 0x7fbc440ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440b2420 con 0x7fbc440b01f0 2022-07-12T08:44:49.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fbc38001660 con 0x7fbc440b01f0 2022-07-12T08:44:49.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 >> v1:172.21.15.61:6789/0 conn(0x7fbc440b01f0 legacy=0x7fbc4405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc2c004500 0x7fbc2c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440b0a40 con 0x7fbc2c003750 2022-07-12T08:44:49.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4405c430 con 0x7fbc2c004500 2022-07-12T08:44:49.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c00bc40 con 0x7fbc2c004e70 2022-07-12T08:44:49.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4935b700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc2c004500 0x7fbc2c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4a35d700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 -- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc2c004e70 msgr2=0x7fbc2c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc2c004e70 0x7fbc2c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 -- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc2c004500 msgr2=0x7fbc2c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc2c004500 0x7fbc2c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4a35d700 1 -- 172.21.15.61:0/4165966095 reap_dead start 2022-07-12T08:44:49.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 -- 172.21.15.61:0/4165966095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c003260 con 0x7fbc2c003750 2022-07-12T08:44:49.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fbc4000c000 tx=0x7fbc40006df0).ready entity=mon.0 client_cookie=e35a7e2506b9210a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:49.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4000e7d0 con 0x7fbc2c003750 2022-07-12T08:44:49.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4000e930 con 0x7fbc2c003750 2022-07-12T08:44:49.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc48b5a700 1 -- 172.21.15.61:0/4165966095 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4000ec90 con 0x7fbc2c003750 2022-07-12T08:44:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/4165966095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 msgr2=0x7fbc2c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7fbc4000c000 tx=0x7fbc40006df0).stop 2022-07-12T08:44:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/4165966095 shutdown_connections 2022-07-12T08:44:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/4165966095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc2c003b70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/4165966095 >> 172.21.15.61:0/4165966095 conn(0x7fbc44062c00 msgr2=0x7fbc44062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.016+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/4165966095 shutdown_connections 2022-07-12T08:44:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.020+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/4165966095 wait complete. 2022-07-12T08:44:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.020+0000 7fbc4bdc0700 1 Processor -- start 2022-07-12T08:44:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.020+0000 7fbc4bdc0700 1 -- start start 2022-07-12T08:44:49.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc44159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440ad4f0 0x7fbc441463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc440ada40 0x7fbc440aded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44147130 con 0x7fbc2c003750 2022-07-12T08:44:49.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc441425f0 con 0x7fbc440ad4f0 2022-07-12T08:44:49.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc44064540 con 0x7fbc440ada40 2022-07-12T08:44:49.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc44159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc44159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55704/0 (socket says 172.21.15.61:55704) 2022-07-12T08:44:49.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 -- 172.21.15.61:0/2955741198 learned_addr learned my addr 172.21.15.61:0/2955741198 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:49.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4a35d700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc440ada40 0x7fbc440aded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4935b700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440ad4f0 0x7fbc441463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 -- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc440ada40 msgr2=0x7fbc440aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc440ada40 0x7fbc440aded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 -- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440ad4f0 msgr2=0x7fbc441463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440ad4f0 0x7fbc441463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 -- 172.21.15.61:0/2955741198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440647b0 con 0x7fbc2c003750 2022-07-12T08:44:49.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc49b5c700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc44159bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fbc441477c0 tx=0x7fbc40019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:49.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4001c9d0 con 0x7fbc2c003750 2022-07-12T08:44:49.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc40000e90 con 0x7fbc2c003750 2022-07-12T08:44:49.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc44146890 con 0x7fbc2c003750 2022-07-12T08:44:49.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc4001cd20 con 0x7fbc2c003750 2022-07-12T08:44:49.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc44147360 con 0x7fbc2c003750 2022-07-12T08:44:49.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.024+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44057460 con 0x7fbc2c003750 2022-07-12T08:44:49.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.028+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbc4002c070 con 0x7fbc2c003750 2022-07-12T08:44:49.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.028+0000 7fbc36ffd700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc24053940 0x7fbc24055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.028+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc4006d830 con 0x7fbc2c003750 2022-07-12T08:44:49.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.028+0000 7fbc4935b700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc24053940 0x7fbc24055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.028+0000 7fbc4935b700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc24053940 0x7fbc24055e00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fbc3800d000 tx=0x7fbc38009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:49.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.036+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc400480b0 con 0x7fbc2c003750 2022-07-12T08:44:49.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.164+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc440447a0 con 0x7fbc2c003750 2022-07-12T08:44:49.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.164+0000 7fbc36ffd700 1 -- 172.21.15.61:0/2955741198 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16114 (secure 0 0 0) 0x7fbc4002a090 con 0x7fbc2c003750 2022-07-12T08:44:49.176 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:49.176 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:44:48.629524+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:44:47.163319+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-12T08:44:47.639688+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:44:49.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc24053940 msgr2=0x7fbc24055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc24053940 0x7fbc24055e00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fbc3800d000 tx=0x7fbc38009040).stop 2022-07-12T08:44:49.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 msgr2=0x7fbc44159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc44159bd0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7fbc441477c0 tx=0x7fbc40019000).stop 2022-07-12T08:44:49.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 shutdown_connections 2022-07-12T08:44:49.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc440ada40 0x7fbc440aded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc24053940 0x7fbc24055e00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc440ad4f0 0x7fbc441463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 --2- 172.21.15.61:0/2955741198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc2c003750 0x7fbc44159bd0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 >> 172.21.15.61:0/2955741198 conn(0x7fbc44062c00 msgr2=0x7fbc440b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:49.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 shutdown_connections 2022-07-12T08:44:49.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.168+0000 7fbc4bdc0700 1 -- 172.21.15.61:0/2955741198 wait complete. 2022-07-12T08:44:49.187 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:44:49.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.292+0000 7ffa25bab700 1 Processor -- start 2022-07-12T08:44:49.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.292+0000 7ffa25bab700 1 -- start start 2022-07-12T08:44:49.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa25bab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffa200acfe0 0x7ffa200b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa25bab700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200b8110 con 0x7ffa20066580 2022-07-12T08:44:49.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa25bab700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200b1d20 con 0x7ffa200b7d30 2022-07-12T08:44:49.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa25bab700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa200b1fc0 con 0x7ffa200acfe0 2022-07-12T08:44:49.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ffa20066580 0x7ffa2013fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43932/0 (socket says 172.21.15.61:43932) 2022-07-12T08:44:49.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 -- 172.21.15.61:0/3691104235 learned_addr learned my addr 172.21.15.61:0/3691104235 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:49.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa16ffd700 1 --2- 172.21.15.61:0/3691104235 >> v2:172.21.15.111:3300/0 conn(0x7ffa200acfe0 0x7ffa200b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599520216 0 0) 0x7ffa200b8110 con 0x7ffa20066580 2022-07-12T08:44:49.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa04003260 con 0x7ffa20066580 2022-07-12T08:44:49.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 651568430 0 0) 0x7ffa04003260 con 0x7ffa20066580 2022-07-12T08:44:49.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 >> v2:172.21.15.111:3300/0 conn(0x7ffa200acfe0 msgr2=0x7ffa200b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 --2- 172.21.15.61:0/3691104235 >> v2:172.21.15.111:3300/0 conn(0x7ffa200acfe0 0x7ffa200b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 >> v1:172.21.15.111:6789/0 conn(0x7ffa200b7d30 legacy=0x7ffa20065e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200b2260 con 0x7ffa20066580 2022-07-12T08:44:49.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7ffa18001a00 con 0x7ffa20066580 2022-07-12T08:44:49.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 >> v1:172.21.15.61:6789/0 conn(0x7ffa20066580 legacy=0x7ffa2013fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa04003750 0x7ffa04003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa04004500 0x7ffa04004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa0400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200b1fc0 con 0x7ffa04004e70 2022-07-12T08:44:49.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400bd20 con 0x7ffa04003750 2022-07-12T08:44:49.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0400bee0 con 0x7ffa04004500 2022-07-12T08:44:49.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa0400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa16ffd700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa04004500 0x7ffa04004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1f7fe700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa04003750 0x7ffa04003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 -- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa04004500 msgr2=0x7ffa04004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa04004500 0x7ffa04004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 -- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa04003750 msgr2=0x7ffa04003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:49.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa04003750 0x7ffa04003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 -- 172.21.15.61:0/3691104235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa04003260 con 0x7ffa04004e70 2022-07-12T08:44:49.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 -- 172.21.15.61:0/3691104235 reap_dead start 2022-07-12T08:44:49.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.296+0000 7ffa1ffff700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa0400b750 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7ffa18001710 tx=0x7ffa20140f00).ready entity=mon.0 client_cookie=6c0d110beb81c88c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:49.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa1800cc70 con 0x7ffa04004e70 2022-07-12T08:44:49.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1800cdd0 con 0x7ffa04004e70 2022-07-12T08:44:49.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1effd700 1 -- 172.21.15.61:0/3691104235 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa18004b50 con 0x7ffa04004e70 2022-07-12T08:44:49.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/3691104235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 msgr2=0x7ffa0400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa0400b750 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7ffa18001710 tx=0x7ffa20140f00).stop 2022-07-12T08:44:49.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/3691104235 shutdown_connections 2022-07-12T08:44:49.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 --2- 172.21.15.61:0/3691104235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa0400b750 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/3691104235 >> 172.21.15.61:0/3691104235 conn(0x7ffa20065730 msgr2=0x7ffa200b72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:49.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/3691104235 shutdown_connections 2022-07-12T08:44:49.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/3691104235 wait complete. 2022-07-12T08:44:49.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 Processor -- start 2022-07-12T08:44:49.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- start start 2022-07-12T08:44:49.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa201479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa20142d60 0x7ffa20138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa20138e50 0x7ffa200ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200b1490 con 0x7ffa04004e70 2022-07-12T08:44:49.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200b0280 con 0x7ffa20138e50 2022-07-12T08:44:49.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa200b04b0 con 0x7ffa20142d60 2022-07-12T08:44:49.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa201479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa201479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55726/0 (socket says 172.21.15.61:55726) 2022-07-12T08:44:49.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 -- 172.21.15.61:0/1402181169 learned_addr learned my addr 172.21.15.61:0/1402181169 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:49.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa16ffd700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa20142d60 0x7ffa20138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1ffff700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa20138e50 0x7ffa200ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 -- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa20142d60 msgr2=0x7ffa20138960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa20142d60 0x7ffa20138960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 -- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa20138e50 msgr2=0x7ffa200ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:49.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa20138e50 0x7ffa200ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:49.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 -- 172.21.15.61:0/1402181169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200b0d20 con 0x7ffa04004e70 2022-07-12T08:44:49.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1ffff700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa20138e50 0x7ffa200ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:49.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1f7fe700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa201479a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7ffa200b1c20 tx=0x7ffa10009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:49.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa100155f0 con 0x7ffa04004e70 2022-07-12T08:44:49.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa10010040 con 0x7ffa04004e70 2022-07-12T08:44:49.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa10015940 con 0x7ffa04004e70 2022-07-12T08:44:49.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa20148080 con 0x7ffa04004e70 2022-07-12T08:44:49.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.300+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa200afde0 con 0x7ffa04004e70 2022-07-12T08:44:49.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.304+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa20057460 con 0x7ffa04004e70 2022-07-12T08:44:49.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.304+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffa10015aa0 con 0x7ffa04004e70 2022-07-12T08:44:49.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.304+0000 7ffa1d7fa700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9f8053a40 0x7ff9f8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:49.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.304+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffa1006bfd0 con 0x7ffa04004e70 2022-07-12T08:44:49.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.304+0000 7ffa16ffd700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9f8053a40 0x7ff9f8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:49.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.308+0000 7ffa16ffd700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9f8053a40 0x7ff9f8055f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ffa08006000 tx=0x7ffa08006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:49.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.316+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffa10042ed0 con 0x7ffa04004e70 2022-07-12T08:44:49.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:49.480+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ffa2005bd10 con 0x7ffa04004e70 2022-07-12T08:44:50.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.320+0000 7ffa1d7fa700 1 -- 172.21.15.61:0/1402181169 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffa100464f0 con 0x7ffa04004e70 2022-07-12T08:44:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9f8053a40 msgr2=0x7ff9f8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9f8053a40 0x7ff9f8055f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ffa08006000 tx=0x7ffa08006d90).stop 2022-07-12T08:44:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 msgr2=0x7ffa201479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa201479a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7ffa200b1c20 tx=0x7ffa10009040).stop 2022-07-12T08:44:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 shutdown_connections 2022-07-12T08:44:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9f8053a40 0x7ff9f8055f00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffa20138e50 0x7ffa200ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffa20142d60 0x7ffa20138960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 --2- 172.21.15.61:0/1402181169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa04004e70 0x7ffa201479a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 >> 172.21.15.61:0/1402181169 conn(0x7ffa20065730 msgr2=0x7ffa200b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 shutdown_connections 2022-07-12T08:44:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.324+0000 7ffa25bab700 1 -- 172.21.15.61:0/1402181169 wait complete. 2022-07-12T08:44:50.342 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:50.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.456+0000 7fc499657700 1 Processor -- start 2022-07-12T08:44:50.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.456+0000 7fc499657700 1 -- start start 2022-07-12T08:44:50.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 0x7fc4940ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc49414a8f0 0x7fc4941443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4940ad2f0 con 0x7fc49405c050 2022-07-12T08:44:50.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4940b0a40 con 0x7fc49414a8f0 2022-07-12T08:44:50.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49405c430 con 0x7fc4940b01f0 2022-07-12T08:44:50.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 0x7fc4940ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4937fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc49414a8f0 0x7fc4941443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 0x7fc4940ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:37928/0 (socket says 172.21.15.61:37928) 2022-07-12T08:44:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 -- 172.21.15.61:0/1300884915 learned_addr learned my addr 172.21.15.61:0/1300884915 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc492ffd700 1 --1- 172.21.15.61:0/1300884915 >> v1:172.21.15.111:6789/0 conn(0x7fc49405c050 0x7fc49405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:57552/0 (socket says 172.21.15.61:57552) 2022-07-12T08:44:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 -- 172.21.15.61:0/1300884915 >> v2:172.21.15.111:3300/0 conn(0x7fc49414a8f0 msgr2=0x7fc4941443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:50.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 --2- 172.21.15.61:0/1300884915 >> v2:172.21.15.111:3300/0 conn(0x7fc49414a8f0 0x7fc4941443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 -- 172.21.15.61:0/1300884915 >> v1:172.21.15.111:6789/0 conn(0x7fc49405c050 legacy=0x7fc49405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 -- 172.21.15.61:0/1300884915 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4940b2420 con 0x7fc4940b01f0 2022-07-12T08:44:50.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc4927fc700 1 --2- 172.21.15.61:0/1300884915 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 0x7fc4940ac710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc4840032f0 tx=0x7fc484009290).ready entity=mon.2 client_cookie=b671ac7fe8242555 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:50.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc491ffb700 1 -- 172.21.15.61:0/1300884915 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc48400b8a0 con 0x7fc4940b01f0 2022-07-12T08:44:50.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc491ffb700 1 -- 172.21.15.61:0/1300884915 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48400ba00 con 0x7fc4940b01f0 2022-07-12T08:44:50.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc491ffb700 1 -- 172.21.15.61:0/1300884915 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc48400bb70 con 0x7fc4940b01f0 2022-07-12T08:44:50.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- 172.21.15.61:0/1300884915 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 msgr2=0x7fc4940ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 --2- 172.21.15.61:0/1300884915 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 0x7fc4940ac710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc4840032f0 tx=0x7fc484009290).stop 2022-07-12T08:44:50.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- 172.21.15.61:0/1300884915 shutdown_connections 2022-07-12T08:44:50.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 --2- 172.21.15.61:0/1300884915 >> v2:172.21.15.111:3300/0 conn(0x7fc49414a8f0 0x7fc4941443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 --2- 172.21.15.61:0/1300884915 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4940b01f0 0x7fc4940ac710 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- 172.21.15.61:0/1300884915 >> 172.21.15.61:0/1300884915 conn(0x7fc494062c00 msgr2=0x7fc494062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:50.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- 172.21.15.61:0/1300884915 shutdown_connections 2022-07-12T08:44:50.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.460+0000 7fc499657700 1 -- 172.21.15.61:0/1300884915 wait complete. 2022-07-12T08:44:50.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 Processor -- start 2022-07-12T08:44:50.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 -- start start 2022-07-12T08:44:50.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc49405c050 0x7fc4940a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4940b01f0 0x7fc49409ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc49414a8f0 0x7fc49409e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4941452d0 con 0x7fc49414a8f0 2022-07-12T08:44:50.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4940a1f70 con 0x7fc4940b01f0 2022-07-12T08:44:50.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.464+0000 7fc499657700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc494067190 con 0x7fc49405c050 2022-07-12T08:44:50.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc49414a8f0 0x7fc49409e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc49414a8f0 0x7fc49409e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55742/0 (socket says 172.21.15.61:55742) 2022-07-12T08:44:50.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4937fe700 1 -- 172.21.15.61:0/1485797533 learned_addr learned my addr 172.21.15.61:0/1485797533 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:50.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4940b01f0 0x7fc49409ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc492ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc49405c050 0x7fc4940a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 -- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc49405c050 msgr2=0x7fc4940a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc49405c050 0x7fc4940a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 -- 172.21.15.61:0/1485797533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc49414a8f0 msgr2=0x7fc49409e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc49414a8f0 0x7fc49409e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 -- 172.21.15.61:0/1485797533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc484009530 con 0x7fc4940b01f0 2022-07-12T08:44:50.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc4927fc700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4940b01f0 0x7fc49409ecd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc4840032f0 tx=0x7fc484016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:50.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc48400b8a0 con 0x7fc4940b01f0 2022-07-12T08:44:50.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc484021670 con 0x7fc4940b01f0 2022-07-12T08:44:50.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4840219d0 con 0x7fc4940b01f0 2022-07-12T08:44:50.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49414a230 con 0x7fc4940b01f0 2022-07-12T08:44:50.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49409f1c0 con 0x7fc4940b01f0 2022-07-12T08:44:50.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.468+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49409e790 con 0x7fc4940b01f0 2022-07-12T08:44:50.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.472+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc48400f070 con 0x7fc4940b01f0 2022-07-12T08:44:50.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.472+0000 7fc483fff700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4700539b0 0x7fc470055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.472+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc48406d080 con 0x7fc4940b01f0 2022-07-12T08:44:50.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.472+0000 7fc492ffd700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4700539b0 0x7fc470055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.472+0000 7fc492ffd700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4700539b0 0x7fc470055e70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc48800d000 tx=0x7fc488007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:50.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.480+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc484044020 con 0x7fc4940b01f0 2022-07-12T08:44:50.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.644+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc494066ff0 con 0x7fc4940b01f0 2022-07-12T08:44:50.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc483fff700 1 -- 172.21.15.61:0/1485797533 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fc494066ff0 con 0x7fc4940b01f0 2022-07-12T08:44:50.656 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:50.656 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:44:50.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4700539b0 msgr2=0x7fc470055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4700539b0 0x7fc470055e70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc48800d000 tx=0x7fc488007b40).stop 2022-07-12T08:44:50.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4940b01f0 msgr2=0x7fc49409ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4940b01f0 0x7fc49409ecd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc4840032f0 tx=0x7fc484016000).stop 2022-07-12T08:44:50.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 shutdown_connections 2022-07-12T08:44:50.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc49414a8f0 0x7fc49409e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4700539b0 0x7fc470055e70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4940b01f0 0x7fc49409ecd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 --2- 172.21.15.61:0/1485797533 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc49405c050 0x7fc4940a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 >> 172.21.15.61:0/1485797533 conn(0x7fc494062c00 msgr2=0x7fc494062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:50.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 shutdown_connections 2022-07-12T08:44:50.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.648+0000 7fc499657700 1 -- 172.21.15.61:0/1485797533 wait complete. 2022-07-12T08:44:50.661 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:44:50.664 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:44:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 Processor -- start 2022-07-12T08:44:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 -- start start 2022-07-12T08:44:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 0x7f56600ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5660142f40 0x7f56601433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56600b26a0 con 0x7f56600628e0 2022-07-12T08:44:50.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56600ae2e0 con 0x7f5660142f40 2022-07-12T08:44:50.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5667472700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56600b7ee0 con 0x7f56600b7b00 2022-07-12T08:44:50.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.756+0000 7f5665c6f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f56600628e0 0x7f56600b1f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43964/0 (socket says 172.21.15.61:43964) 2022-07-12T08:44:50.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 -- 172.21.15.61:0/1610927347 learned_addr learned my addr 172.21.15.61:0/1610927347 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:50.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5666470700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 0x7f56600ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5666c71700 1 --2- 172.21.15.61:0/1610927347 >> v2:172.21.15.111:3300/0 conn(0x7f5660142f40 0x7f56601433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250001485 0 0) 0x7f56600b26a0 con 0x7f56600628e0 2022-07-12T08:44:50.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5648003260 con 0x7f56600628e0 2022-07-12T08:44:50.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857275979 0 0) 0x7f5648003260 con 0x7f56600628e0 2022-07-12T08:44:50.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 msgr2=0x7f56600ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:50.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 0x7f56600ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 >> v2:172.21.15.111:3300/0 conn(0x7f5660142f40 msgr2=0x7f56601433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:50.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 --2- 172.21.15.61:0/1610927347 >> v2:172.21.15.111:3300/0 conn(0x7f5660142f40 0x7f56601433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56600b2d30 con 0x7f56600628e0 2022-07-12T08:44:50.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f565c001660 con 0x7f56600628e0 2022-07-12T08:44:50.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 >> v1:172.21.15.61:6789/0 conn(0x7f56600628e0 legacy=0x7f56600b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 0x7f5648003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f5648004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56480049f0 0x7f5648004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56600ae2e0 con 0x7f5648003c80 2022-07-12T08:44:50.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56600b7ee0 con 0x7f56480049f0 2022-07-12T08:44:50.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564800b750 con 0x7f56600b7b00 2022-07-12T08:44:50.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f5648004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5666470700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 0x7f5648003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5666c71700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56480049f0 0x7f5648004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 -- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 msgr2=0x7f5648003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:50.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600b7b00 0x7f5648003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 -- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56480049f0 msgr2=0x7f5648004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:50.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56480049f0 0x7f5648004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 -- 172.21.15.61:0/1610927347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5648003260 con 0x7f5648003c80 2022-07-12T08:44:50.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5665c6f700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f5648004500 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f56480054d0 tx=0x7f56600ac6e0).ready entity=mon.0 client_cookie=4a6cc9d42ebb2638 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:50.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f565c0070d0 con 0x7f5648003c80 2022-07-12T08:44:50.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565c007230 con 0x7f5648003c80 2022-07-12T08:44:50.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f566546e700 1 -- 172.21.15.61:0/1610927347 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f565c00c980 con 0x7f5648003c80 2022-07-12T08:44:50.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5667472700 1 -- 172.21.15.61:0/1610927347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 msgr2=0x7f5648004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5667472700 1 --2- 172.21.15.61:0/1610927347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f5648004500 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f56480054d0 tx=0x7f56600ac6e0).stop 2022-07-12T08:44:50.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5666c71700 1 -- 172.21.15.61:0/1610927347 reap_dead start 2022-07-12T08:44:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5667472700 1 -- 172.21.15.61:0/1610927347 shutdown_connections 2022-07-12T08:44:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5667472700 1 -- 172.21.15.61:0/1610927347 >> 172.21.15.61:0/1610927347 conn(0x7f5660065550 msgr2=0x7f56600b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5667472700 1 -- 172.21.15.61:0/1610927347 shutdown_connections 2022-07-12T08:44:50.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.760+0000 7f5667472700 1 -- 172.21.15.61:0/1610927347 wait complete. 2022-07-12T08:44:50.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 Processor -- start 2022-07-12T08:44:50.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 -- start start 2022-07-12T08:44:50.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f566009bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5660065a90 0x7f5660066d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600ae550 0x7f56601443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56600671f0 con 0x7f5648003c80 2022-07-12T08:44:50.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5660065fa0 con 0x7f5660065a90 2022-07-12T08:44:50.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56600676d0 con 0x7f56600ae550 2022-07-12T08:44:50.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f566009bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f566009bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55758/0 (socket says 172.21.15.61:55758) 2022-07-12T08:44:50.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 -- 172.21.15.61:0/2662570291 learned_addr learned my addr 172.21.15.61:0/2662570291 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:50.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666c71700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600ae550 0x7f56601443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5665c6f700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5660065a90 0x7f5660066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 -- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600ae550 msgr2=0x7f56601443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600ae550 0x7f56601443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 -- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5660065a90 msgr2=0x7f5660066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5660065a90 0x7f5660066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 -- 172.21.15.61:0/2662570291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5660067970 con 0x7f5648003c80 2022-07-12T08:44:50.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5666470700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f566009bf30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f5660067c80 tx=0x7f5650006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:50.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f565000f760 con 0x7f5648003c80 2022-07-12T08:44:50.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565000f8c0 con 0x7f5648003c80 2022-07-12T08:44:50.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5660146620 con 0x7f5648003c80 2022-07-12T08:44:50.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f565000fa30 con 0x7f5648003c80 2022-07-12T08:44:50.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.764+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56601448a0 con 0x7f5648003c80 2022-07-12T08:44:50.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.768+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5660057460 con 0x7f5648003c80 2022-07-12T08:44:50.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.768+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5650024070 con 0x7f5648003c80 2022-07-12T08:44:50.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.768+0000 7f564f7fe700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56400539d0 0x7f5640055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:50.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.768+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f565006c560 con 0x7f5648003c80 2022-07-12T08:44:50.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.768+0000 7f5665c6f700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56400539d0 0x7f5640055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:50.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.768+0000 7f5665c6f700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56400539d0 0x7f5640055e90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f565c01b000 tx=0x7f565c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:50.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.772+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56500434c0 con 0x7f5648003c80 2022-07-12T08:44:50.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.900+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f566006b2b0 con 0x7f5648003c80 2022-07-12T08:44:50.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.904+0000 7f564f7fe700 1 -- 172.21.15.61:0/2662570291 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16088 (secure 0 0 0) 0x7f5650046ae0 con 0x7f5648003c80 2022-07-12T08:44:50.910 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:50.910 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":53,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:44:50.635406+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:44:47.163319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-12T08:44:47.639688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:44:50.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56400539d0 msgr2=0x7f5640055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56400539d0 0x7f5640055e90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f565c01b000 tx=0x7f565c015000).stop 2022-07-12T08:44:50.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 msgr2=0x7f566009bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:50.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f566009bf30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f5660067c80 tx=0x7f5650006df0).stop 2022-07-12T08:44:50.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 shutdown_connections 2022-07-12T08:44:50.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56600ae550 0x7f56601443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56400539d0 0x7f5640055e90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5660065a90 0x7f5660066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 --2- 172.21.15.61:0/2662570291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5648003c80 0x7f566009bf30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:50.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 >> 172.21.15.61:0/2662570291 conn(0x7f5660065550 msgr2=0x7f56600b5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:50.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 shutdown_connections 2022-07-12T08:44:50.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:50.908+0000 7f5667472700 1 -- 172.21.15.61:0/2662570291 wait complete. 2022-07-12T08:44:50.928 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:44:51.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 Processor -- start 2022-07-12T08:44:51.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 -- start start 2022-07-12T08:44:51.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad7c0b3580 0x7fad7c096100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c0965f0 0x7fad7c096a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c05fc90 con 0x7fad7c05c5a0 2022-07-12T08:44:51.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad7c0b3960 con 0x7fad7c0b3580 2022-07-12T08:44:51.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c05f9f0 con 0x7fad7c0965f0 2022-07-12T08:44:51.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7ad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fad7c05c5a0 0x7fad7c05c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43988/0 (socket says 172.21.15.61:43988) 2022-07-12T08:44:51.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7ad9d700 1 -- 172.21.15.61:0/1478696587 learned_addr learned my addr 172.21.15.61:0/1478696587 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:51.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7a59c700 1 --2- 172.21.15.61:0/1478696587 >> v2:172.21.15.111:3300/0 conn(0x7fad7c0b3580 0x7fad7c096100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad79d9b700 1 --2- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c0965f0 0x7fad7c096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7959a700 1 -- 172.21.15.61:0/1478696587 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588864958 0 0) 0x7fad7c05fc90 con 0x7fad7c05c5a0 2022-07-12T08:44:51.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7959a700 1 -- 172.21.15.61:0/1478696587 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad60003260 con 0x7fad7c05c5a0 2022-07-12T08:44:51.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7a59c700 1 -- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c0965f0 msgr2=0x7fad7c096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:51.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7a59c700 1 --2- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c0965f0 0x7fad7c096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7a59c700 1 -- 172.21.15.61:0/1478696587 >> v1:172.21.15.61:6789/0 conn(0x7fad7c05c5a0 legacy=0x7fad7c05c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7a59c700 1 -- 172.21.15.61:0/1478696587 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c060550 con 0x7fad7c0b3580 2022-07-12T08:44:51.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7a59c700 1 --2- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c096100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fad6c002f70 tx=0x7fad6c006df0).ready entity=mon.1 client_cookie=c1924a55d059667 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7959a700 1 -- 172.21.15.61:0/1478696587 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6c00d4a0 con 0x7fad7c0b3580 2022-07-12T08:44:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7959a700 1 -- 172.21.15.61:0/1478696587 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c000e90 con 0x7fad7c0b3580 2022-07-12T08:44:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.044+0000 7fad7959a700 1 -- 172.21.15.61:0/1478696587 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad6c00d7f0 con 0x7fad7c0b3580 2022-07-12T08:44:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 msgr2=0x7fad7c096100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 --2- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c096100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fad6c002f70 tx=0x7fad6c006df0).stop 2022-07-12T08:44:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/1478696587 shutdown_connections 2022-07-12T08:44:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 --2- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c0965f0 0x7fad7c096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 --2- 172.21.15.61:0/1478696587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c096100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/1478696587 >> 172.21.15.61:0/1478696587 conn(0x7fad7c0b6b40 msgr2=0x7fad7c0b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:51.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/1478696587 shutdown_connections 2022-07-12T08:44:51.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/1478696587 wait complete. 2022-07-12T08:44:51.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 Processor -- start 2022-07-12T08:44:51.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- start start 2022-07-12T08:44:51.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c05c5a0 0x7fad7c066e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 0x7fad7c0b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c14f7b0 con 0x7fad7c0965f0 2022-07-12T08:44:51.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c099280 con 0x7fad7c0b3580 2022-07-12T08:44:51.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c0994e0 con 0x7fad7c05c5a0 2022-07-12T08:44:51.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 0x7fad7c0b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 0x7fad7c0b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55776/0 (socket says 172.21.15.61:55776) 2022-07-12T08:44:51.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 -- 172.21.15.61:0/3157566492 learned_addr learned my addr 172.21.15.61:0/3157566492 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:51.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7a59c700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c05c5a0 0x7fad7c066e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7ad9d700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 -- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c05c5a0 msgr2=0x7fad7c066e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c05c5a0 0x7fad7c066e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 -- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 msgr2=0x7fad7c152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 -- 172.21.15.61:0/3157566492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c00b180 con 0x7fad7c0965f0 2022-07-12T08:44:51.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad79d9b700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 0x7fad7c0b41e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7fad6400c000 tx=0x7fad64006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad64010780 con 0x7fad7c0965f0 2022-07-12T08:44:51.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad640108e0 con 0x7fad7c0965f0 2022-07-12T08:44:51.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c099780 con 0x7fad7c0965f0 2022-07-12T08:44:51.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad64010c40 con 0x7fad7c0965f0 2022-07-12T08:44:51.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.048+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c09bd30 con 0x7fad7c0965f0 2022-07-12T08:44:51.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.052+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c099910 con 0x7fad7c0965f0 2022-07-12T08:44:51.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.052+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fad6401c460 con 0x7fad7c0965f0 2022-07-12T08:44:51.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.052+0000 7fad737fe700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad540539b0 0x7fad54055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.052+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fad6406c390 con 0x7fad7c0965f0 2022-07-12T08:44:51.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.052+0000 7fad7a59c700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad540539b0 0x7fad54055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.052+0000 7fad7a59c700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad540539b0 0x7fad54055e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fad6c01a000 tx=0x7fad6c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.060+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad64043180 con 0x7fad7c0965f0 2022-07-12T08:44:51.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.228+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fad7c057460 con 0x7fad7c0965f0 2022-07-12T08:44:51.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad737fe700 1 -- 172.21.15.61:0/3157566492 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7fad640467a0 con 0x7fad7c0965f0 2022-07-12T08:44:51.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad540539b0 msgr2=0x7fad54055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad540539b0 0x7fad54055e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fad6c01a000 tx=0x7fad6c014000).stop 2022-07-12T08:44:51.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 msgr2=0x7fad7c0b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 0x7fad7c0b41e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7fad6400c000 tx=0x7fad64006df0).stop 2022-07-12T08:44:51.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 shutdown_connections 2022-07-12T08:44:51.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad540539b0 0x7fad54055e70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad7c0b3580 0x7fad7c152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad7c0965f0 0x7fad7c0b41e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 --2- 172.21.15.61:0/3157566492 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad7c05c5a0 0x7fad7c066e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.236+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 >> 172.21.15.61:0/3157566492 conn(0x7fad7c0b6b40 msgr2=0x7fad7c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:51.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.240+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 shutdown_connections 2022-07-12T08:44:51.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.240+0000 7fad7b59e700 1 -- 172.21.15.61:0/3157566492 wait complete. 2022-07-12T08:44:51.250 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:44:51.264 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:44:51.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.372+0000 7f130d765700 1 Processor -- start 2022-07-12T08:44:51.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.372+0000 7f130d765700 1 -- start start 2022-07-12T08:44:51.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f130d765700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f13080b52e0 0x7f13080b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f130d765700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080b56c0 con 0x7f1308141e50 2022-07-12T08:44:51.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f130d765700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13080b4bd0 con 0x7f13080b4050 2022-07-12T08:44:51.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f130d765700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1308142f00 con 0x7f13080b52e0 2022-07-12T08:44:51.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f13077fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1308141e50 0x7f13081422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44002/0 (socket says 172.21.15.61:44002) 2022-07-12T08:44:51.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f13077fe700 1 -- 172.21.15.61:0/1767898686 learned_addr learned my addr 172.21.15.61:0/1767898686 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:51.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1306ffd700 1 --2- 172.21.15.61:0/1767898686 >> v2:172.21.15.111:3300/0 conn(0x7f13080b52e0 0x7f13080b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512144336 0 0) 0x7f13080b56c0 con 0x7f1308141e50 2022-07-12T08:44:51.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12ec003260 con 0x7f1308141e50 2022-07-12T08:44:51.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124663783 0 0) 0x7f13080b4bd0 con 0x7f13080b4050 2022-07-12T08:44:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13080b56c0 con 0x7f13080b4050 2022-07-12T08:44:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1306ffd700 1 -- 172.21.15.61:0/1767898686 >> v1:172.21.15.111:6789/0 conn(0x7f13080b4050 legacy=0x7f13080b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1306ffd700 1 -- 172.21.15.61:0/1767898686 >> v1:172.21.15.61:6789/0 conn(0x7f1308141e50 legacy=0x7f13081422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1306ffd700 1 -- 172.21.15.61:0/1767898686 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13080b0e80 con 0x7f13080b52e0 2022-07-12T08:44:51.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1306ffd700 1 --2- 172.21.15.61:0/1767898686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13080b52e0 0x7f13080b21b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f12fc002f70 tx=0x7f12fc006df0).ready entity=mon.1 client_cookie=c10695e62a9f5cf8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12fc00d540 con 0x7f13080b52e0 2022-07-12T08:44:51.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12fc000e90 con 0x7f13080b52e0 2022-07-12T08:44:51.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.376+0000 7f1305ffb700 1 -- 172.21.15.61:0/1767898686 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12fc00d890 con 0x7f13080b52e0 2022-07-12T08:44:51.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 -- 172.21.15.61:0/1767898686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13080b52e0 msgr2=0x7f13080b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 --2- 172.21.15.61:0/1767898686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13080b52e0 0x7f13080b21b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f12fc002f70 tx=0x7f12fc006df0).stop 2022-07-12T08:44:51.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 -- 172.21.15.61:0/1767898686 shutdown_connections 2022-07-12T08:44:51.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 --2- 172.21.15.61:0/1767898686 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13080b52e0 0x7f13080b21b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 -- 172.21.15.61:0/1767898686 >> 172.21.15.61:0/1767898686 conn(0x7f1308067020 msgr2=0x7f1308067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:51.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 -- 172.21.15.61:0/1767898686 shutdown_connections 2022-07-12T08:44:51.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 -- 172.21.15.61:0/1767898686 wait complete. 2022-07-12T08:44:51.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 Processor -- start 2022-07-12T08:44:51.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.380+0000 7f130d765700 1 -- start start 2022-07-12T08:44:51.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f13080b4050 0x7f13080b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 0x7f13081469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1308141e50 0x7f1308151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1308138cc0 con 0x7f13080b52e0 2022-07-12T08:44:51.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1308145860 con 0x7f1308141e50 2022-07-12T08:44:51.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1308138e60 con 0x7f13080b4050 2022-07-12T08:44:51.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 0x7f13081469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 0x7f13081469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55788/0 (socket says 172.21.15.61:55788) 2022-07-12T08:44:51.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 -- 172.21.15.61:0/1427499742 learned_addr learned my addr 172.21.15.61:0/1427499742 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:51.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13077fe700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1308141e50 0x7f1308151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f1306ffd700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f13080b4050 0x7f13080b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 -- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f13080b4050 msgr2=0x7f13080b86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f13080b4050 0x7f13080b86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 -- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1308141e50 msgr2=0x7f1308151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1308141e50 0x7f1308151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 -- 172.21.15.61:0/1427499742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc00b180 con 0x7f13080b52e0 2022-07-12T08:44:51.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13077fe700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1308141e50 0x7f1308151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:51.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f13067fc700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 0x7f13081469b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f12f8001110 tx=0x7f12f8007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f800f470 con 0x7f13080b52e0 2022-07-12T08:44:51.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f800f5d0 con 0x7f13080b52e0 2022-07-12T08:44:51.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1308139100 con 0x7f13080b52e0 2022-07-12T08:44:51.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f800f930 con 0x7f13080b52e0 2022-07-12T08:44:51.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1308144400 con 0x7f13080b52e0 2022-07-12T08:44:51.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.384+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1308057460 con 0x7f13080b52e0 2022-07-12T08:44:51.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.388+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f12f8019030 con 0x7f13080b52e0 2022-07-12T08:44:51.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.388+0000 7f12f7fff700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12e4053ad0 0x7f12e4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.388+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f12f806b3c0 con 0x7f13080b52e0 2022-07-12T08:44:51.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.388+0000 7f1306ffd700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12e4053ad0 0x7f12e4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.388+0000 7f1306ffd700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12e4053ad0 0x7f12e4055f90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f12fc018000 tx=0x7f12fc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.396+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12f8042320 con 0x7f13080b52e0 2022-07-12T08:44:51.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.560+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f13080b6ee0 con 0x7f13080b52e0 2022-07-12T08:44:51.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.568+0000 7f12f7fff700 1 -- 172.21.15.61:0/1427499742 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7f12f8045940 con 0x7f13080b52e0 2022-07-12T08:44:51.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12e4053ad0 msgr2=0x7f12e4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12e4053ad0 0x7f12e4055f90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f12fc018000 tx=0x7f12fc014000).stop 2022-07-12T08:44:51.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 msgr2=0x7f13081469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 0x7f13081469b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f12f8001110 tx=0x7f12f8007ae0).stop 2022-07-12T08:44:51.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 shutdown_connections 2022-07-12T08:44:51.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12e4053ad0 0x7f12e4055f90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1308141e50 0x7f1308151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13080b52e0 0x7f13081469b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 --2- 172.21.15.61:0/1427499742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f13080b4050 0x7f13080b86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 >> 172.21.15.61:0/1427499742 conn(0x7f1308067020 msgr2=0x7f13080b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:51.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 shutdown_connections 2022-07-12T08:44:51.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.572+0000 7f130d765700 1 -- 172.21.15.61:0/1427499742 wait complete. 2022-07-12T08:44:51.581 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:44:51.594 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:44:51.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 Processor -- start 2022-07-12T08:44:51.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 -- start start 2022-07-12T08:44:51.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c3805cd20 0x7f3c380b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c3805d100 0x7f3c38146da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c380afe60 con 0x7f3c38098930 2022-07-12T08:44:51.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c38143f40 con 0x7f3c3805cd20 2022-07-12T08:44:51.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c3e0fb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c380b0090 con 0x7f3c3805d100 2022-07-12T08:44:51.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c3805cd20 0x7f3c380b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c3805cd20 0x7f3c380b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55796/0 (socket says 172.21.15.61:55796) 2022-07-12T08:44:51.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 -- 172.21.15.61:0/1495573801 learned_addr learned my addr 172.21.15.61:0/1495573801 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:51.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c37fff700 1 --2- 172.21.15.61:0/1495573801 >> v2:172.21.15.111:3300/0 conn(0x7f3c3805d100 0x7f3c38146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 -- 172.21.15.61:0/1495573801 >> v2:172.21.15.111:3300/0 conn(0x7f3c3805d100 msgr2=0x7f3c38146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:51.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 --2- 172.21.15.61:0/1495573801 >> v2:172.21.15.111:3300/0 conn(0x7f3c3805d100 0x7f3c38146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 -- 172.21.15.61:0/1495573801 >> v1:172.21.15.111:6789/0 conn(0x7f3c38098930 legacy=0x7f3c38146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.704+0000 7f3c377fe700 1 -- 172.21.15.61:0/1495573801 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38146260 con 0x7f3c3805cd20 2022-07-12T08:44:51.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c377fe700 1 --2- 172.21.15.61:0/1495573801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c3805cd20 0x7f3c380b5870 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3c2c002f70 tx=0x7f3c2c006eb0).ready entity=mon.0 client_cookie=e72f142b6af1d659 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c367fc700 1 -- 172.21.15.61:0/1495573801 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c2c00d5c0 con 0x7f3c3805cd20 2022-07-12T08:44:51.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c367fc700 1 -- 172.21.15.61:0/1495573801 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c000e90 con 0x7f3c3805cd20 2022-07-12T08:44:51.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c367fc700 1 -- 172.21.15.61:0/1495573801 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c2c00d950 con 0x7f3c3805cd20 2022-07-12T08:44:51.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/1495573801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c3805cd20 msgr2=0x7f3c380b5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/1495573801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c3805cd20 0x7f3c380b5870 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f3c2c002f70 tx=0x7f3c2c006eb0).stop 2022-07-12T08:44:51.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/1495573801 shutdown_connections 2022-07-12T08:44:51.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/1495573801 >> v2:172.21.15.111:3300/0 conn(0x7f3c3805d100 0x7f3c38146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/1495573801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c3805cd20 0x7f3c380b5870 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/1495573801 >> 172.21.15.61:0/1495573801 conn(0x7f3c380b69d0 msgr2=0x7f3c38066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:51.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/1495573801 shutdown_connections 2022-07-12T08:44:51.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/1495573801 wait complete. 2022-07-12T08:44:51.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 Processor -- start 2022-07-12T08:44:51.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- start start 2022-07-12T08:44:51.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3805cd20 0x7f3c3805fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c3805d100 0x7f3c381402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 0x7f3c38140790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c38151dc0 con 0x7f3c38098930 2022-07-12T08:44:51.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3809d340 con 0x7f3c3805cd20 2022-07-12T08:44:51.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c3e0fb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3809d0a0 con 0x7f3c3805d100 2022-07-12T08:44:51.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 0x7f3c38140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 0x7f3c38140790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55802/0 (socket says 172.21.15.61:55802) 2022-07-12T08:44:51.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 -- 172.21.15.61:0/4279996978 learned_addr learned my addr 172.21.15.61:0/4279996978 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:51.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c377fe700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3805cd20 0x7f3c3805fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c36ffd700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c3805d100 0x7f3c381402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 -- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c3805d100 msgr2=0x7f3c381402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c3805d100 0x7f3c381402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 -- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3805cd20 msgr2=0x7f3c3805fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3805cd20 0x7f3c3805fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 -- 172.21.15.61:0/4279996978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c00b1e0 con 0x7f3c38098930 2022-07-12T08:44:51.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c377fe700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3805cd20 0x7f3c3805fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:51.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.708+0000 7f3c37fff700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 0x7f3c38140790 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f3c3809d4d0 tx=0x7f3c300091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c3000c850 con 0x7f3c38098930 2022-07-12T08:44:51.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3000c9b0 con 0x7f3c38098930 2022-07-12T08:44:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c380b2710 con 0x7f3c38098930 2022-07-12T08:44:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c3000cd10 con 0x7f3c38098930 2022-07-12T08:44:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c38060340 con 0x7f3c38098930 2022-07-12T08:44:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c3809cee0 con 0x7f3c38098930 2022-07-12T08:44:51.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3c30022070 con 0x7f3c38098930 2022-07-12T08:44:51.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c34ff9700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c18053a40 0x7f3c18055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:51.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3c3006c690 con 0x7f3c38098930 2022-07-12T08:44:51.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c377fe700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c18053a40 0x7f3c18055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:51.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.712+0000 7f3c377fe700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c18053a40 0x7f3c18055f00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3c2c01a000 tx=0x7f3c2c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:51.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.716+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c30043550 con 0x7f3c38098930 2022-07-12T08:44:51.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.884+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f3c380b28a0 con 0x7f3c38098930 2022-07-12T08:44:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.892+0000 7f3c34ff9700 1 -- 172.21.15.61:0/4279996978 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3c3006ae60 con 0x7f3c38098930 2022-07-12T08:44:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c18053a40 msgr2=0x7f3c18055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c18053a40 0x7f3c18055f00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3c2c01a000 tx=0x7f3c2c015000).stop 2022-07-12T08:44:51.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 msgr2=0x7f3c38140790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:51.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 0x7f3c38140790 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f3c3809d4d0 tx=0x7f3c300091d0).stop 2022-07-12T08:44:51.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 shutdown_connections 2022-07-12T08:44:51.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c38098930 0x7f3c38140790 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c3805d100 0x7f3c381402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c18053a40 0x7f3c18055f00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 --2- 172.21.15.61:0/4279996978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c3805cd20 0x7f3c3805fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:51.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 >> 172.21.15.61:0/4279996978 conn(0x7f3c380b69d0 msgr2=0x7f3c381421d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:51.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 shutdown_connections 2022-07-12T08:44:51.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:51.896+0000 7f3c3e0fb700 1 -- 172.21.15.61:0/4279996978 wait complete. 2022-07-12T08:44:51.909 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:44:51.914 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:52.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecffd27700 1 Processor -- start 2022-07-12T08:44:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecffd27700 1 -- start start 2022-07-12T08:44:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecffd27700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fecf80ad1c0 0x7fecf8140760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecffd27700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf8140c50 con 0x7fecf80b4250 2022-07-12T08:44:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecffd27700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80b4630 con 0x7fecf80b2480 2022-07-12T08:44:52.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecffd27700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf80acce0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecfe2c4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fecf80ad1c0 0x7fecf8140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecfdac3700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fecf80b4250 0x7fecf8144350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44028/0 (socket says 172.21.15.61:44028) 2022-07-12T08:44:52.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecfe2c4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fecf80ad1c0 0x7fecf8140760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55812/0 (socket says 172.21.15.61:55812) 2022-07-12T08:44:52.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.012+0000 7fecfdac3700 1 -- 172.21.15.61:0/256860918 learned_addr learned my addr 172.21.15.61:0/256860918 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:52.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfe2c4700 1 -- 172.21.15.61:0/256860918 >> v1:172.21.15.111:6789/0 conn(0x7fecf80b2480 legacy=0x7fecf8140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfe2c4700 1 -- 172.21.15.61:0/256860918 >> v1:172.21.15.61:6789/0 conn(0x7fecf80b4250 legacy=0x7fecf8144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfcac1700 1 -- 172.21.15.61:0/256860918 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809287671 0 0) 0x7fecf4001110 con 0x7fecf80b4250 2022-07-12T08:44:52.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfe2c4700 1 -- 172.21.15.61:0/256860918 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf8065fa0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfe2c4700 1 --2- 172.21.15.61:0/256860918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8140760 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fecf00032f0 tx=0x7fecf00091d0).ready entity=mon.0 client_cookie=bc027c429e510269 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfcac1700 1 -- 172.21.15.61:0/256860918 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecf0009ea0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfcac1700 1 -- 172.21.15.61:0/256860918 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf000b8a0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecfcac1700 1 -- 172.21.15.61:0/256860918 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecf000ba10 con 0x7fecf80ad1c0 2022-07-12T08:44:52.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 -- 172.21.15.61:0/256860918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 msgr2=0x7fecf8140760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 --2- 172.21.15.61:0/256860918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8140760 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fecf00032f0 tx=0x7fecf00091d0).stop 2022-07-12T08:44:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 -- 172.21.15.61:0/256860918 shutdown_connections 2022-07-12T08:44:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 --2- 172.21.15.61:0/256860918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8140760 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 -- 172.21.15.61:0/256860918 >> 172.21.15.61:0/256860918 conn(0x7fecf8064e10 msgr2=0x7fecf8065220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 -- 172.21.15.61:0/256860918 shutdown_connections 2022-07-12T08:44:52.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.016+0000 7fecffd27700 1 -- 172.21.15.61:0/256860918 wait complete. 2022-07-12T08:44:52.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 Processor -- start 2022-07-12T08:44:52.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 -- start start 2022-07-12T08:44:52.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecf80b2480 0x7fecf8137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf80b4250 0x7fecf8143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf81544e0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf8144c80 con 0x7fecf80b4250 2022-07-12T08:44:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf813d270 con 0x7fecf80b2480 2022-07-12T08:44:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8153550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55814/0 (socket says 172.21.15.61:55814) 2022-07-12T08:44:52.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 -- 172.21.15.61:0/1323879121 learned_addr learned my addr 172.21.15.61:0/1323879121 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:52.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfd2c2700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecf80b2480 0x7fecf8137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfe2c4700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf80b4250 0x7fecf8143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 -- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecf80b2480 msgr2=0x7fecf8137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecf80b2480 0x7fecf8137f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 -- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf80b4250 msgr2=0x7fecf8143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf80b4250 0x7fecf8143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 -- 172.21.15.61:0/1323879121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf0009470 con 0x7fecf80ad1c0 2022-07-12T08:44:52.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecfdac3700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8153550 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fecf4000bc0 tx=0x7fecf4007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecf40070d0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf4007230 con 0x7fecf80ad1c0 2022-07-12T08:44:52.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf81448f0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecf4005610 con 0x7fecf80ad1c0 2022-07-12T08:44:52.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.020+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf8138450 con 0x7fecf80ad1c0 2022-07-12T08:44:52.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.024+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf80b8390 con 0x7fecf80ad1c0 2022-07-12T08:44:52.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.024+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fecf401a060 con 0x7fecf80ad1c0 2022-07-12T08:44:52.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.024+0000 7fece6ffd700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecd8053ad0 0x7fecd8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.024+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fecf406b3c0 con 0x7fecf80ad1c0 2022-07-12T08:44:52.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.024+0000 7fecfd2c2700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecd8053ad0 0x7fecd8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.024+0000 7fecfd2c2700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecd8053ad0 0x7fecd8055f90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fece800d000 tx=0x7fece8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.032+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fecf4042320 con 0x7fecf80ad1c0 2022-07-12T08:44:52.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf8140c50 con 0x7fecf80ad1c0 2022-07-12T08:44:52.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fece6ffd700 1 -- 172.21.15.61:0/1323879121 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fecf8140c50 con 0x7fecf80ad1c0 2022-07-12T08:44:52.201 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:52.201 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:44:52.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecd8053ad0 msgr2=0x7fecd8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecd8053ad0 0x7fecd8055f90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fece800d000 tx=0x7fece8007b90).stop 2022-07-12T08:44:52.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 msgr2=0x7fecf8153550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8153550 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 rx=0x7fecf4000bc0 tx=0x7fecf4007bf0).stop 2022-07-12T08:44:52.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 shutdown_connections 2022-07-12T08:44:52.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecd8053ad0 0x7fecd8055f90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecf80b4250 0x7fecf8143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecf80b2480 0x7fecf8137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 --2- 172.21.15.61:0/1323879121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecf80ad1c0 0x7fecf8153550 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 >> 172.21.15.61:0/1323879121 conn(0x7fecf8064e10 msgr2=0x7fecf805bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:52.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 shutdown_connections 2022-07-12T08:44:52.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.196+0000 7fecffd27700 1 -- 172.21.15.61:0/1323879121 wait complete. 2022-07-12T08:44:52.207 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:44:52.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:44:48.629894+0000', 'modified': '2022-07-12T08:44:50.312792+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5595': {'gid': 5595, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/4155742423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 4155742423}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 4155742423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:44:52.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:44:52.418 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:44:52.537 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5595,"want_state":"up:active","state":"up:active","rank_uptime":3.8950324209999998,"mdsmap_epoch":24,"osdmap_epoch":53,"osdmap_epoch_barrier":52,"uptime":8.4535445320000004} 2022-07-12T08:44:52.539 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5595, "mdsmap_epoch": 24, "osdmap_epoch": 53, "osdmap_epoch_barrier": 52, "rank_uptime": 3.895032421, "state": "up:active", "uptime": 8.453544532, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:44:52.540 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:44:52.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 Processor -- start 2022-07-12T08:44:52.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- start start 2022-07-12T08:44:52.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f131c0b4090 0x7f131c0b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c142f30 0x7f131c0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131c0b5740 con 0x7f131c0b1080 2022-07-12T08:44:52.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f131c05cc00 con 0x7f131c0b4090 2022-07-12T08:44:52.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0b1460 con 0x7f131c142f30 2022-07-12T08:44:52.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f131c0b4090 0x7f131c0b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1320ad0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f131c0b1080 0x7f131c05c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44040/0 (socket says 172.21.15.61:44040) 2022-07-12T08:44:52.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1320ad0700 1 -- 172.21.15.61:0/1735370509 learned_addr learned my addr 172.21.15.61:0/1735370509 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:52.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f13212d1700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c142f30 0x7f131c0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131bfff700 1 -- 172.21.15.61:0/1735370509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c142f30 msgr2=0x7f131c0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:52.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131bfff700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c142f30 0x7f131c0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131bfff700 1 -- 172.21.15.61:0/1735370509 >> v1:172.21.15.61:6789/0 conn(0x7f131c0b1080 legacy=0x7f131c05c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131b7fe700 1 -- 172.21.15.61:0/1735370509 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470263145 0 0) 0x7f131c0b5740 con 0x7f131c0b1080 2022-07-12T08:44:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131bfff700 1 -- 172.21.15.61:0/1735370509 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c0b6880 con 0x7f131c0b4090 2022-07-12T08:44:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f13212d1700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c142f30 0x7f131c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:44:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131bfff700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b4090 0x7f131c0b62a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f1308002f70 tx=0x7f1308006df0).ready entity=mon.0 client_cookie=aaa497af0d3521d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131b7fe700 1 -- 172.21.15.61:0/1735370509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f130800d570 con 0x7f131c0b4090 2022-07-12T08:44:52.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131b7fe700 1 -- 172.21.15.61:0/1735370509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1308000e90 con 0x7f131c0b4090 2022-07-12T08:44:52.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f131b7fe700 1 -- 172.21.15.61:0/1735370509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f130800d8c0 con 0x7f131c0b4090 2022-07-12T08:44:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- 172.21.15.61:0/1735370509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b4090 msgr2=0x7f131c0b62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b4090 0x7f131c0b62a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f1308002f70 tx=0x7f1308006df0).stop 2022-07-12T08:44:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- 172.21.15.61:0/1735370509 shutdown_connections 2022-07-12T08:44:52.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c142f30 0x7f131c0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 --2- 172.21.15.61:0/1735370509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b4090 0x7f131c0b62a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- 172.21.15.61:0/1735370509 >> 172.21.15.61:0/1735370509 conn(0x7f131c067020 msgr2=0x7f131c067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:52.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- 172.21.15.61:0/1735370509 shutdown_connections 2022-07-12T08:44:52.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.644+0000 7f1322d34700 1 -- 172.21.15.61:0/1735370509 wait complete. 2022-07-12T08:44:52.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 Processor -- start 2022-07-12T08:44:52.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 -- start start 2022-07-12T08:44:52.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 0x7f131c0b7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c0b4090 0x7f131c0ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f131c142f30 0x7f131c0afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0acb60 con 0x7f131c0b1080 2022-07-12T08:44:52.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0ace00 con 0x7f131c142f30 2022-07-12T08:44:52.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f131c0ad0a0 con 0x7f131c0b4090 2022-07-12T08:44:52.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 0x7f131c0b7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 0x7f131c0b7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55826/0 (socket says 172.21.15.61:55826) 2022-07-12T08:44:52.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 -- 172.21.15.61:0/529620957 learned_addr learned my addr 172.21.15.61:0/529620957 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:52.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f131bfff700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c0b4090 0x7f131c0ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f13212d1700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f131c142f30 0x7f131c0afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 -- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c0b4090 msgr2=0x7f131c0ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c0b4090 0x7f131c0ae990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 -- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f131c142f30 msgr2=0x7f131c0afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f131c142f30 0x7f131c0afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 -- 172.21.15.61:0/529620957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130800b180 con 0x7f131c0b1080 2022-07-12T08:44:52.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1320ad0700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 0x7f131c0b7ba0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f1310000fe0 tx=0x7f1310007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13100070d0 con 0x7f131c0b1080 2022-07-12T08:44:52.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1310007230 con 0x7f131c0b1080 2022-07-12T08:44:52.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1310005610 con 0x7f131c0b1080 2022-07-12T08:44:52.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c0ad340 con 0x7f131c0b1080 2022-07-12T08:44:52.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.648+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c145a30 con 0x7f131c0b1080 2022-07-12T08:44:52.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.652+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c1455d0 con 0x7f131c0b1080 2022-07-12T08:44:52.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.652+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1310010070 con 0x7f131c0b1080 2022-07-12T08:44:52.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.652+0000 7f1319ffb700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12fc053a40 0x7f12fc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.652+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f131006b5b0 con 0x7f131c0b1080 2022-07-12T08:44:52.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.652+0000 7f131bfff700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12fc053a40 0x7f12fc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.652+0000 7f131bfff700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12fc053a40 0x7f12fc055f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f130801c000 tx=0x7f1308015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.660+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f131c1455d0 con 0x7f131c0b1080 2022-07-12T08:44:52.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.788+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f131c1455d0 con 0x7f131c0b1080 2022-07-12T08:44:52.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1319ffb700 1 -- 172.21.15.61:0/529620957 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16088 (secure 0 0 0) 0x7f131c1455d0 con 0x7f131c0b1080 2022-07-12T08:44:52.799 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:52.799 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":53,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:44:50.635406+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:44:47.163319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-12T08:44:47.639688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:44:52.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12fc053a40 msgr2=0x7f12fc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12fc053a40 0x7f12fc055f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f130801c000 tx=0x7f1308015000).stop 2022-07-12T08:44:52.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 msgr2=0x7f131c0b7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 0x7f131c0b7ba0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f1310000fe0 tx=0x7f1310007b50).stop 2022-07-12T08:44:52.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 shutdown_connections 2022-07-12T08:44:52.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f12fc053a40 0x7f12fc055f00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f131c142f30 0x7f131c0afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f131c0b4090 0x7f131c0ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 --2- 172.21.15.61:0/529620957 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f131c0b1080 0x7f131c0b7ba0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 >> 172.21.15.61:0/529620957 conn(0x7f131c067020 msgr2=0x7f131c1433a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:52.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 shutdown_connections 2022-07-12T08:44:52.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.792+0000 7f1322d34700 1 -- 172.21.15.61:0/529620957 wait complete. 2022-07-12T08:44:52.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:44:47.163319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:44:47.639688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:44:52.812 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:44:52.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.920+0000 7f8a119c7700 1 Processor -- start 2022-07-12T08:44:52.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.920+0000 7f8a119c7700 1 -- start start 2022-07-12T08:44:52.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a0c05d060 0x7f8a0c0b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c0afb50 0x7f8a0c143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0c05d440 con 0x7f8a0c142ab0 2022-07-12T08:44:52.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a0c143d80 con 0x7f8a0c05d060 2022-07-12T08:44:52.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0674b0 con 0x7f8a0c0afb50 2022-07-12T08:44:52.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8a0c142ab0 0x7f8a0c142f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44052/0 (socket says 172.21.15.61:44052) 2022-07-12T08:44:52.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0bfff700 1 -- 172.21.15.61:0/2550475754 learned_addr learned my addr 172.21.15.61:0/2550475754 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:52.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a111c6700 1 --2- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c0afb50 0x7f8a0c143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a109c5700 1 --2- 172.21.15.61:0/2550475754 >> v2:172.21.15.111:3300/0 conn(0x7f8a0c05d060 0x7f8a0c0b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0b7fe700 1 -- 172.21.15.61:0/2550475754 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855677861 0 0) 0x7f8a0c05d440 con 0x7f8a0c142ab0 2022-07-12T08:44:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0b7fe700 1 -- 172.21.15.61:0/2550475754 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ec003260 con 0x7f8a0c142ab0 2022-07-12T08:44:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a109c5700 1 -- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c0afb50 msgr2=0x7f8a0c143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a109c5700 1 --2- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c0afb50 0x7f8a0c143620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a109c5700 1 -- 172.21.15.61:0/2550475754 >> v1:172.21.15.61:6789/0 conn(0x7f8a0c142ab0 legacy=0x7f8a0c142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a109c5700 1 -- 172.21.15.61:0/2550475754 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0c067750 con 0x7f8a0c05d060 2022-07-12T08:44:52.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a109c5700 1 --2- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c0b0130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f89fc002f70 tx=0x7f89fc006df0).ready entity=mon.1 client_cookie=9a1ae5164f03259f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0b7fe700 1 -- 172.21.15.61:0/2550475754 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89fc00d4a0 con 0x7f8a0c05d060 2022-07-12T08:44:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0b7fe700 1 -- 172.21.15.61:0/2550475754 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89fc000e90 con 0x7f8a0c05d060 2022-07-12T08:44:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a0b7fe700 1 -- 172.21.15.61:0/2550475754 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89fc00d7f0 con 0x7f8a0c05d060 2022-07-12T08:44:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 msgr2=0x7f8a0c0b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 --2- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c0b0130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f89fc002f70 tx=0x7f89fc006df0).stop 2022-07-12T08:44:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- 172.21.15.61:0/2550475754 shutdown_connections 2022-07-12T08:44:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 --2- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c0afb50 0x7f8a0c143620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 --2- 172.21.15.61:0/2550475754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c0b0130 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- 172.21.15.61:0/2550475754 >> 172.21.15.61:0/2550475754 conn(0x7f8a0c0b6900 msgr2=0x7f8a0c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:52.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- 172.21.15.61:0/2550475754 shutdown_connections 2022-07-12T08:44:52.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.924+0000 7f8a119c7700 1 -- 172.21.15.61:0/2550475754 wait complete. 2022-07-12T08:44:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 Processor -- start 2022-07-12T08:44:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 -- start start 2022-07-12T08:44:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 0x7f8a0c151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c09c420 0x7f8a0c155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0b3820 con 0x7f8a0c0afb50 2022-07-12T08:44:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c142820 con 0x7f8a0c05d060 2022-07-12T08:44:52.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c0b1ae0 con 0x7f8a0c09c420 2022-07-12T08:44:52.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 0x7f8a0c151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 0x7f8a0c151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55842/0 (socket says 172.21.15.61:55842) 2022-07-12T08:44:52.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 -- 172.21.15.61:0/3618208640 learned_addr learned my addr 172.21.15.61:0/3618208640 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:52.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a109c5700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a111c6700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c09c420 0x7f8a0c155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 -- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c09c420 msgr2=0x7f8a0c155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c09c420 0x7f8a0c155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 -- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 msgr2=0x7f8a0c145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:52.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:52.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 -- 172.21.15.61:0/3618208640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc00b180 con 0x7f8a0c0afb50 2022-07-12T08:44:52.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a0bfff700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 0x7f8a0c151cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f89f4001110 tx=0x7f89f4007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89f40070d0 con 0x7f8a0c0afb50 2022-07-12T08:44:52.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f4007230 con 0x7f8a0c0afb50 2022-07-12T08:44:52.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c155490 con 0x7f8a0c0afb50 2022-07-12T08:44:52.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89f4005610 con 0x7f8a0c0afb50 2022-07-12T08:44:52.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.928+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c09e2a0 con 0x7f8a0c0afb50 2022-07-12T08:44:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.932+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f89f4010070 con 0x7f8a0c0afb50 2022-07-12T08:44:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.932+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c155620 con 0x7f8a0c0afb50 2022-07-12T08:44:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.932+0000 7f8a09ffb700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89f0053940 0x7f89f0055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.932+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f89f406b650 con 0x7f8a0c0afb50 2022-07-12T08:44:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.932+0000 7f8a109c5700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89f0053940 0x7f89f0055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.932+0000 7f8a109c5700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89f0053940 0x7f89f0055e00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f89fc01a000 tx=0x7f89fc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:52.940+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89f4042510 con 0x7f8a0c0afb50 2022-07-12T08:44:53.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8a0c057460 con 0x7f8a0c0afb50 2022-07-12T08:44:53.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a09ffb700 1 -- 172.21.15.61:0/3618208640 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v53) v1 ==== 110+0+10 (secure 0 0 0) 0x7f89f4045b30 con 0x7f8a0c0afb50 2022-07-12T08:44:53.070 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:44:53.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89f0053940 msgr2=0x7f89f0055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89f0053940 0x7f89f0055e00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f89fc01a000 tx=0x7f89fc014000).stop 2022-07-12T08:44:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 msgr2=0x7f8a0c151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 0x7f8a0c151cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f89f4001110 tx=0x7f89f4007b20).stop 2022-07-12T08:44:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 shutdown_connections 2022-07-12T08:44:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a0c09c420 0x7f8a0c155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a0c0afb50 0x7f8a0c151cf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89f0053940 0x7f89f0055e00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 --2- 172.21.15.61:0/3618208640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a0c05d060 0x7f8a0c145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 >> 172.21.15.61:0/3618208640 conn(0x7f8a0c0b6900 msgr2=0x7f8a0c05f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 shutdown_connections 2022-07-12T08:44:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.064+0000 7f8a119c7700 1 -- 172.21.15.61:0/3618208640 wait complete. 2022-07-12T08:44:53.082 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:44:53.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 Processor -- start 2022-07-12T08:44:53.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- start start 2022-07-12T08:44:53.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f6005bd10 0x7f9f60146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600ab170 0x7f9f60147280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600b18b0 con 0x7f9f600aad90 2022-07-12T08:44:53.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f60146e80 con 0x7f9f6005bd10 2022-07-12T08:44:53.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600b3110 con 0x7f9f600ab170 2022-07-12T08:44:53.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f6005bd10 0x7f9f60146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f6005bd10 0x7f9f60146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55850/0 (socket says 172.21.15.61:55850) 2022-07-12T08:44:53.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 -- 172.21.15.61:0/1635537265 learned_addr learned my addr 172.21.15.61:0/1635537265 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:53.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66743700 1 --2- 172.21.15.61:0/1635537265 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600ab170 0x7f9f60147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 -- 172.21.15.61:0/1635537265 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600ab170 msgr2=0x7f9f60147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:53.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 --2- 172.21.15.61:0/1635537265 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600ab170 0x7f9f60147280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 -- 172.21.15.61:0/1635537265 >> v1:172.21.15.111:6789/0 conn(0x7f9f600aad90 legacy=0x7f9f60146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:53.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 -- 172.21.15.61:0/1635537265 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600b16d0 con 0x7f9f6005bd10 2022-07-12T08:44:53.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f65741700 1 --2- 172.21.15.61:0/1635537265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f6005bd10 0x7f9f60146990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f9f5c002f70 tx=0x7f9f5c006eb0).ready entity=mon.0 client_cookie=6ec4bb16309c1039 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f64f40700 1 -- 172.21.15.61:0/1635537265 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5c00d5c0 con 0x7f9f6005bd10 2022-07-12T08:44:53.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f64f40700 1 -- 172.21.15.61:0/1635537265 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c000e90 con 0x7f9f6005bd10 2022-07-12T08:44:53.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f64f40700 1 -- 172.21.15.61:0/1635537265 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5c00d950 con 0x7f9f6005bd10 2022-07-12T08:44:53.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- 172.21.15.61:0/1635537265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f6005bd10 msgr2=0x7f9f60146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1635537265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f6005bd10 0x7f9f60146990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f9f5c002f70 tx=0x7f9f5c006eb0).stop 2022-07-12T08:44:53.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- 172.21.15.61:0/1635537265 shutdown_connections 2022-07-12T08:44:53.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1635537265 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600ab170 0x7f9f60147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1635537265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f6005bd10 0x7f9f60146990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- 172.21.15.61:0/1635537265 >> 172.21.15.61:0/1635537265 conn(0x7f9f600b57c0 msgr2=0x7f9f600b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- 172.21.15.61:0/1635537265 shutdown_connections 2022-07-12T08:44:53.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.192+0000 7f9f66f44700 1 -- 172.21.15.61:0/1635537265 wait complete. 2022-07-12T08:44:53.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 Processor -- start 2022-07-12T08:44:53.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 -- start start 2022-07-12T08:44:53.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f6005bd10 0x7f9f600ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600aad90 0x7f9f600adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 0x7f9f60142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600ae730 con 0x7f9f600ab170 2022-07-12T08:44:53.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f60066a40 con 0x7f9f6005bd10 2022-07-12T08:44:53.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f60066c90 con 0x7f9f600aad90 2022-07-12T08:44:53.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 0x7f9f60142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 0x7f9f60142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55856/0 (socket says 172.21.15.61:55856) 2022-07-12T08:44:53.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 -- 172.21.15.61:0/1606808715 learned_addr learned my addr 172.21.15.61:0/1606808715 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:53.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f65741700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600aad90 0x7f9f600adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f65f42700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f6005bd10 0x7f9f600ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 -- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600aad90 msgr2=0x7f9f600adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600aad90 0x7f9f600adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 -- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f6005bd10 msgr2=0x7f9f600ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f6005bd10 0x7f9f600ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 -- 172.21.15.61:0/1606808715 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c00b1e0 con 0x7f9f600ab170 2022-07-12T08:44:53.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66743700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 0x7f9f60142df0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f9f5800a000 tx=0x7f9f58008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f58013620 con 0x7f9f600ab170 2022-07-12T08:44:53.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58013780 con 0x7f9f600ab170 2022-07-12T08:44:53.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f600ae490 con 0x7f9f600ab170 2022-07-12T08:44:53.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f58013ae0 con 0x7f9f600ab170 2022-07-12T08:44:53.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.196+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f600acc00 con 0x7f9f600ab170 2022-07-12T08:44:53.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.200+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f60057460 con 0x7f9f600ab170 2022-07-12T08:44:53.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.200+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f58013c40 con 0x7f9f600ab170 2022-07-12T08:44:53.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.200+0000 7f9f56ffd700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f480539b0 0x7f9f48055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.200+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f5806bca0 con 0x7f9f600ab170 2022-07-12T08:44:53.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.200+0000 7f9f65f42700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f480539b0 0x7f9f48055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.200+0000 7f9f65f42700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f480539b0 0x7f9f48055e70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f9f5000b000 tx=0x7f9f50009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.204+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f58042af0 con 0x7f9f600ab170 2022-07-12T08:44:53.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9f60068470 con 0x7f9f600ab170 2022-07-12T08:44:53.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f56ffd700 1 -- 172.21.15.61:0/1606808715 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v53) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9f5800f040 con 0x7f9f600ab170 2022-07-12T08:44:53.338 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:44:53.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f480539b0 msgr2=0x7f9f48055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f480539b0 0x7f9f48055e70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f9f5000b000 tx=0x7f9f50009040).stop 2022-07-12T08:44:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 msgr2=0x7f9f60142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 0x7f9f60142df0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f9f5800a000 tx=0x7f9f58008040).stop 2022-07-12T08:44:53.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 shutdown_connections 2022-07-12T08:44:53.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600ab170 0x7f9f60142df0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f600aad90 0x7f9f600adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f480539b0 0x7f9f48055e70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 --2- 172.21.15.61:0/1606808715 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f6005bd10 0x7f9f600ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 >> 172.21.15.61:0/1606808715 conn(0x7f9f600b57c0 msgr2=0x7f9f60145460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 shutdown_connections 2022-07-12T08:44:53.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.332+0000 7f9f66f44700 1 -- 172.21.15.61:0/1606808715 wait complete. 2022-07-12T08:44:53.350 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:44:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 Processor -- start 2022-07-12T08:44:53.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- start start 2022-07-12T08:44:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b5480 0x7f72dc0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72dc0b3730 0x7f72dc0afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc0b3560 con 0x7f72dc0b03d0 2022-07-12T08:44:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72dc0b5b80 con 0x7f72dc0b3730 2022-07-12T08:44:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc0b5e20 con 0x7f72dc0b5480 2022-07-12T08:44:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72dbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72dc0b3730 0x7f72dc0afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e110c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f72dc0b03d0 0x7f72dc142f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44084/0 (socket says 172.21.15.61:44084) 2022-07-12T08:44:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e110c700 1 -- 172.21.15.61:0/2339248505 learned_addr learned my addr 172.21.15.61:0/2339248505 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e090b700 1 --2- 172.21.15.61:0/2339248505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b5480 0x7f72dc0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72dbfff700 1 -- 172.21.15.61:0/2339248505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b5480 msgr2=0x7f72dc0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72dbfff700 1 --2- 172.21.15.61:0/2339248505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b5480 0x7f72dc0b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72dbfff700 1 -- 172.21.15.61:0/2339248505 >> v1:172.21.15.61:6789/0 conn(0x7f72dc0b03d0 legacy=0x7f72dc142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72dbfff700 1 -- 172.21.15.61:0/2339248505 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc0b0820 con 0x7f72dc0b3730 2022-07-12T08:44:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72db7fe700 1 -- 172.21.15.61:0/2339248505 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870480180 0 0) 0x7f72dc0b3560 con 0x7f72dc0b03d0 2022-07-12T08:44:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72dbfff700 1 --2- 172.21.15.61:0/2339248505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b3730 0x7f72dc0afee0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f72c4002f70 tx=0x7f72c4006df0).ready entity=mon.0 client_cookie=a74111b124df68ea server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72db7fe700 1 -- 172.21.15.61:0/2339248505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72c400d570 con 0x7f72dc0b3730 2022-07-12T08:44:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72db7fe700 1 -- 172.21.15.61:0/2339248505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c4000e90 con 0x7f72dc0b3730 2022-07-12T08:44:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72db7fe700 1 -- 172.21.15.61:0/2339248505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72c400d8c0 con 0x7f72dc0b3730 2022-07-12T08:44:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- 172.21.15.61:0/2339248505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b3730 msgr2=0x7f72dc0afee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/2339248505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b3730 0x7f72dc0afee0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f72c4002f70 tx=0x7f72c4006df0).stop 2022-07-12T08:44:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- 172.21.15.61:0/2339248505 shutdown_connections 2022-07-12T08:44:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/2339248505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b3730 0x7f72dc0afee0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/2339248505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b5480 0x7f72dc0b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- 172.21.15.61:0/2339248505 >> 172.21.15.61:0/2339248505 conn(0x7f72dc0b96b0 msgr2=0x7f72dc0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- 172.21.15.61:0/2339248505 shutdown_connections 2022-07-12T08:44:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.452+0000 7f72e2b6f700 1 -- 172.21.15.61:0/2339248505 wait complete. 2022-07-12T08:44:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 Processor -- start 2022-07-12T08:44:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 -- start start 2022-07-12T08:44:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 0x7f72dc0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b3730 0x7f72dc147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc0b5480 0x7f72dc140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc0ad6c0 con 0x7f72dc0b03d0 2022-07-12T08:44:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc141d00 con 0x7f72dc0b5480 2022-07-12T08:44:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc1403c0 con 0x7f72dc0b3730 2022-07-12T08:44:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 0x7f72dc0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 0x7f72dc0ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55868/0 (socket says 172.21.15.61:55868) 2022-07-12T08:44:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 -- 172.21.15.61:0/1210885378 learned_addr learned my addr 172.21.15.61:0/1210885378 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72dbfff700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b3730 0x7f72dc147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e110c700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc0b5480 0x7f72dc140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 -- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b3730 msgr2=0x7f72dc147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b3730 0x7f72dc147650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 -- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc0b5480 msgr2=0x7f72dc140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc0b5480 0x7f72dc140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 -- 172.21.15.61:0/1210885378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c400b180 con 0x7f72dc0b03d0 2022-07-12T08:44:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e110c700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc0b5480 0x7f72dc140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:44:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e090b700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 0x7f72dc0ad1d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f72cc003430 tx=0x7f72cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72cc011780 con 0x7f72dc0b03d0 2022-07-12T08:44:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc0118e0 con 0x7f72dc0b03d0 2022-07-12T08:44:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72cc011c40 con 0x7f72dc0b03d0 2022-07-12T08:44:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc0b0140 con 0x7f72dc0b03d0 2022-07-12T08:44:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.456+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc141370 con 0x7f72dc0b03d0 2022-07-12T08:44:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.460+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f72cc012420 con 0x7f72dc0b03d0 2022-07-12T08:44:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.460+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc141f00 con 0x7f72dc0b03d0 2022-07-12T08:44:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.460+0000 7f72d9ffb700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72c0053a40 0x7f72c0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.460+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f72cc06c060 con 0x7f72dc0b03d0 2022-07-12T08:44:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.460+0000 7f72dbfff700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72c0053a40 0x7f72c0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.460+0000 7f72dbfff700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72c0053a40 0x7f72c0055f00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f72c401c000 tx=0x7f72c4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.468+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72cc042e50 con 0x7f72dc0b03d0 2022-07-12T08:44:53.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.592+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f72dc057460 con 0x7f72dc0b03d0 2022-07-12T08:44:53.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.592+0000 7f72d9ffb700 1 -- 172.21.15.61:0/1210885378 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+10 (secure 0 0 0) 0x7f72cc046470 con 0x7f72dc0b03d0 2022-07-12T08:44:53.605 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:44:53.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.592+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72c0053a40 msgr2=0x7f72c0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.592+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72c0053a40 0x7f72c0055f00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f72c401c000 tx=0x7f72c4015000).stop 2022-07-12T08:44:53.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.592+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 msgr2=0x7f72dc0ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.592+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 0x7f72dc0ad1d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7f72cc003430 tx=0x7f72cc008040).stop 2022-07-12T08:44:53.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 shutdown_connections 2022-07-12T08:44:53.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72c0053a40 0x7f72c0055f00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72dc0b5480 0x7f72dc140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72dc0b3730 0x7f72dc147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 --2- 172.21.15.61:0/1210885378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72dc0b03d0 0x7f72dc0ad1d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 >> 172.21.15.61:0/1210885378 conn(0x7f72dc0b96b0 msgr2=0x7f72dc0b8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 shutdown_connections 2022-07-12T08:44:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.596+0000 7f72e2b6f700 1 -- 172.21.15.61:0/1210885378 wait complete. 2022-07-12T08:44:53.611 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:44:53.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 Processor -- start 2022-07-12T08:44:53.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- start start 2022-07-12T08:44:53.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff47c142eb0 0x7ff47c05c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff47c0b6130 0x7ff47c0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff47c143500 con 0x7ff47c05d130 2022-07-12T08:44:53.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47c0b65c0 con 0x7ff47c0b6130 2022-07-12T08:44:53.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47c143290 con 0x7ff47c142eb0 2022-07-12T08:44:53.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff480ca8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff47c05d130 0x7ff47c0b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44096/0 (socket says 172.21.15.61:44096) 2022-07-12T08:44:53.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff480ca8700 1 -- 172.21.15.61:0/2993805851 learned_addr learned my addr 172.21.15.61:0/2993805851 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:44:53.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff481caa700 1 --2- 172.21.15.61:0/2993805851 >> v2:172.21.15.61:3300/0 conn(0x7ff47c0b6130 0x7ff47c0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4814a9700 1 --2- 172.21.15.61:0/2993805851 >> v2:172.21.15.111:3300/0 conn(0x7ff47c142eb0 0x7ff47c05c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff473fff700 1 -- 172.21.15.61:0/2993805851 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084423028 0 0) 0x7ff47c143500 con 0x7ff47c05d130 2022-07-12T08:44:53.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff473fff700 1 -- 172.21.15.61:0/2993805851 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff460003260 con 0x7ff47c05d130 2022-07-12T08:44:53.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff481caa700 1 -- 172.21.15.61:0/2993805851 >> v2:172.21.15.111:3300/0 conn(0x7ff47c142eb0 msgr2=0x7ff47c05c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:53.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff481caa700 1 --2- 172.21.15.61:0/2993805851 >> v2:172.21.15.111:3300/0 conn(0x7ff47c142eb0 0x7ff47c05c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff481caa700 1 -- 172.21.15.61:0/2993805851 >> v1:172.21.15.61:6789/0 conn(0x7ff47c05d130 legacy=0x7ff47c0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff481caa700 1 -- 172.21.15.61:0/2993805851 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c143760 con 0x7ff47c0b6130 2022-07-12T08:44:53.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff481caa700 1 --2- 172.21.15.61:0/2993805851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c0b6130 0x7ff47c0b1ad0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7ff47c143c30 tx=0x7ff4740091d0).ready entity=mon.0 client_cookie=e6453f837f8864d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff473fff700 1 -- 172.21.15.61:0/2993805851 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff47400b5a0 con 0x7ff47c0b6130 2022-07-12T08:44:53.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff473fff700 1 -- 172.21.15.61:0/2993805851 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47400b700 con 0x7ff47c0b6130 2022-07-12T08:44:53.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff473fff700 1 -- 172.21.15.61:0/2993805851 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff47400b870 con 0x7ff47c0b6130 2022-07-12T08:44:53.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- 172.21.15.61:0/2993805851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c0b6130 msgr2=0x7ff47c0b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2993805851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c0b6130 0x7ff47c0b1ad0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7ff47c143c30 tx=0x7ff4740091d0).stop 2022-07-12T08:44:53.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- 172.21.15.61:0/2993805851 shutdown_connections 2022-07-12T08:44:53.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2993805851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c0b6130 0x7ff47c0b1ad0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2993805851 >> v2:172.21.15.111:3300/0 conn(0x7ff47c142eb0 0x7ff47c05c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- 172.21.15.61:0/2993805851 >> 172.21.15.61:0/2993805851 conn(0x7ff47c066d70 msgr2=0x7ff47c067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- 172.21.15.61:0/2993805851 shutdown_connections 2022-07-12T08:44:53.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.716+0000 7ff4824ab700 1 -- 172.21.15.61:0/2993805851 wait complete. 2022-07-12T08:44:53.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.720+0000 7ff4824ab700 1 Processor -- start 2022-07-12T08:44:53.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.720+0000 7ff4824ab700 1 -- start start 2022-07-12T08:44:53.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff47c05d130 0x7ff47c148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff47c0b6130 0x7ff47c151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 0x7ff47c1523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c151ac0 con 0x7ff47c142eb0 2022-07-12T08:44:53.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c147d20 con 0x7ff47c05d130 2022-07-12T08:44:53.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c144cd0 con 0x7ff47c0b6130 2022-07-12T08:44:53.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 0x7ff47c1523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4814a9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff47c05d130 0x7ff47c148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 0x7ff47c1523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55886/0 (socket says 172.21.15.61:55886) 2022-07-12T08:44:53.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 -- 172.21.15.61:0/2315368377 learned_addr learned my addr 172.21.15.61:0/2315368377 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:53.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff480ca8700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff47c0b6130 0x7ff47c151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 -- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff47c0b6130 msgr2=0x7ff47c151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff47c0b6130 0x7ff47c151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 -- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff47c05d130 msgr2=0x7ff47c148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff47c05d130 0x7ff47c148a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 -- 172.21.15.61:0/2315368377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff474009470 con 0x7ff47c142eb0 2022-07-12T08:44:53.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff481caa700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 0x7ff47c1523d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7ff47c143c30 tx=0x7ff474016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff47400b870 con 0x7ff47c142eb0 2022-07-12T08:44:53.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4740032f0 con 0x7ff47c142eb0 2022-07-12T08:44:53.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff47400b5a0 con 0x7ff47c142eb0 2022-07-12T08:44:53.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c147f80 con 0x7ff47c142eb0 2022-07-12T08:44:53.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c148350 con 0x7ff47c142eb0 2022-07-12T08:44:53.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.724+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47c057460 con 0x7ff47c142eb0 2022-07-12T08:44:53.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.728+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff474026030 con 0x7ff47c142eb0 2022-07-12T08:44:53.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.728+0000 7ff4727fc700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff464053ad0 0x7ff464055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.728+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff47406c9b0 con 0x7ff47c142eb0 2022-07-12T08:44:53.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.728+0000 7ff4814a9700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff464053ad0 0x7ff464055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.728+0000 7ff4814a9700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff464053ad0 0x7ff464055f90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7ff47800b000 tx=0x7ff478006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.736+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4740472e0 con 0x7ff47c142eb0 2022-07-12T08:44:53.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff47c148110 con 0x7ff47c142eb0 2022-07-12T08:44:53.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4727fc700 1 -- 172.21.15.61:0/2315368377 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff474024020 con 0x7ff47c142eb0 2022-07-12T08:44:53.866 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:44:53.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff464053ad0 msgr2=0x7ff464055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff464053ad0 0x7ff464055f90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7ff47800b000 tx=0x7ff478006df0).stop 2022-07-12T08:44:53.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 msgr2=0x7ff47c1523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 0x7ff47c1523d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7ff47c143c30 tx=0x7ff474016000).stop 2022-07-12T08:44:53.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 shutdown_connections 2022-07-12T08:44:53.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff47c142eb0 0x7ff47c1523d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff47c0b6130 0x7ff47c151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff464053ad0 0x7ff464055f90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 --2- 172.21.15.61:0/2315368377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff47c05d130 0x7ff47c148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 >> 172.21.15.61:0/2315368377 conn(0x7ff47c066d70 msgr2=0x7ff47c1492e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 shutdown_connections 2022-07-12T08:44:53.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.860+0000 7ff4824ab700 1 -- 172.21.15.61:0/2315368377 wait complete. 2022-07-12T08:44:53.879 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:44:53.881 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:44:53.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 Processor -- start 2022-07-12T08:44:53.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 -- start start 2022-07-12T08:44:53.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc0628e0 0x7f87dc0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f87dc142f40 0x7f87dc1433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87dc0b7ee0 con 0x7f87dc0b7b00 2022-07-12T08:44:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87dc0acf80 con 0x7f87dc142f40 2022-07-12T08:44:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e1ac5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0ae2e0 con 0x7f87dc0628e0 2022-07-12T08:44:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f87dc142f40 0x7f87dc1433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f87dc142f40 0x7f87dc1433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55894/0 (socket says 172.21.15.61:55894) 2022-07-12T08:44:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 -- 172.21.15.61:0/3264655902 learned_addr learned my addr 172.21.15.61:0/3264655902 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e0ac3700 1 --1- 172.21.15.61:0/3264655902 >> v1:172.21.15.61:6789/0 conn(0x7f87dc0b7b00 0x7f87dc0ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44108/0 (socket says 172.21.15.61:44108) 2022-07-12T08:44:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87dbfff700 1 --2- 172.21.15.61:0/3264655902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc0628e0 0x7f87dc0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:53.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 -- 172.21.15.61:0/3264655902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc0628e0 msgr2=0x7f87dc0b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:44:53.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 --2- 172.21.15.61:0/3264655902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc0628e0 0x7f87dc0b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 -- 172.21.15.61:0/3264655902 >> v1:172.21.15.61:6789/0 conn(0x7f87dc0b7b00 legacy=0x7f87dc0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 -- 172.21.15.61:0/3264655902 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc0b2d30 con 0x7f87dc142f40 2022-07-12T08:44:53.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87e12c4700 1 --2- 172.21.15.61:0/3264655902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc142f40 0x7f87dc1433d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f87d40032f0 tx=0x7f87d4009290).ready entity=mon.0 client_cookie=92866b8723e38bdd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:53.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87db7fe700 1 -- 172.21.15.61:0/3264655902 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d400b8a0 con 0x7f87dc142f40 2022-07-12T08:44:53.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87db7fe700 1 -- 172.21.15.61:0/3264655902 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d400ba00 con 0x7f87dc142f40 2022-07-12T08:44:53.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.976+0000 7f87db7fe700 1 -- 172.21.15.61:0/3264655902 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87d400bb70 con 0x7f87dc142f40 2022-07-12T08:44:53.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- 172.21.15.61:0/3264655902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc142f40 msgr2=0x7f87dc1433d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:53.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/3264655902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc142f40 0x7f87dc1433d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f87d40032f0 tx=0x7f87d4009290).stop 2022-07-12T08:44:53.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- 172.21.15.61:0/3264655902 shutdown_connections 2022-07-12T08:44:53.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/3264655902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc142f40 0x7f87dc1433d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/3264655902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc0628e0 0x7f87dc0b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:53.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- 172.21.15.61:0/3264655902 >> 172.21.15.61:0/3264655902 conn(0x7f87dc065550 msgr2=0x7f87dc0b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:53.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- 172.21.15.61:0/3264655902 shutdown_connections 2022-07-12T08:44:53.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- 172.21.15.61:0/3264655902 wait complete. 2022-07-12T08:44:53.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 Processor -- start 2022-07-12T08:44:53.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- start start 2022-07-12T08:44:53.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87dc0628e0 0x7f87dc0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 0x7f87dc13f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc142f40 0x7f87dc13fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:53.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc0604b0 con 0x7f87dc0b7b00 2022-07-12T08:44:53.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc060750 con 0x7f87dc0628e0 2022-07-12T08:44:53.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e1ac5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87dc095440 con 0x7f87dc142f40 2022-07-12T08:44:53.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 0x7f87dc13f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:54.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e12c4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc142f40 0x7f87dc13fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:54.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 0x7f87dc13f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55900/0 (socket says 172.21.15.61:55900) 2022-07-12T08:44:54.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87dbfff700 1 -- 172.21.15.61:0/2476037668 learned_addr learned my addr 172.21.15.61:0/2476037668 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:44:54.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.980+0000 7f87e0ac3700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87dc0628e0 0x7f87dc0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:54.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87dbfff700 1 -- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc142f40 msgr2=0x7f87dc13fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:54.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87dbfff700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc142f40 0x7f87dc13fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:54.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87dbfff700 1 -- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87dc0628e0 msgr2=0x7f87dc0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:54.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87dbfff700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87dc0628e0 0x7f87dc0b4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:54.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87dbfff700 1 -- 172.21.15.61:0/2476037668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4009530 con 0x7f87dc0b7b00 2022-07-12T08:44:54.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87dbfff700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 0x7f87dc13f790 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f87cc0030b0 tx=0x7f87cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:54.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87cc011780 con 0x7f87dc0b7b00 2022-07-12T08:44:54.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc0118e0 con 0x7f87dc0b7b00 2022-07-12T08:44:54.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87dc05fb10 con 0x7f87dc0b7b00 2022-07-12T08:44:54.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87cc011c40 con 0x7f87dc0b7b00 2022-07-12T08:44:54.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87dc0b74c0 con 0x7f87dc0b7b00 2022-07-12T08:44:54.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f87cc012420 con 0x7f87dc0b7b00 2022-07-12T08:44:54.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87dc057460 con 0x7f87dc0b7b00 2022-07-12T08:44:54.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87d9ffb700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87bc053a40 0x7f87bc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:44:54.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f87cc06bd30 con 0x7f87dc0b7b00 2022-07-12T08:44:54.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.984+0000 7f87e0ac3700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87bc053a40 0x7f87bc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:44:54.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.988+0000 7f87e0ac3700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87bc053a40 0x7f87bc055f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f87d000d000 tx=0x7f87d0007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:44:54.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:53.988+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f87cc042b20 con 0x7f87dc0b7b00 2022-07-12T08:44:54.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.156+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87dc06b2b0 con 0x7f87dc0b7b00 2022-07-12T08:44:54.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.156+0000 7f87d9ffb700 1 -- 172.21.15.61:0/2476037668 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f87cc046140 con 0x7f87dc0b7b00 2022-07-12T08:44:54.162 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:44:54.162 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:44:54.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87bc053a40 msgr2=0x7f87bc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:54.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87bc053a40 0x7f87bc055f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f87d000d000 tx=0x7f87d0007b40).stop 2022-07-12T08:44:54.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 msgr2=0x7f87dc13f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:44:54.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 0x7f87dc13f790 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f87cc0030b0 tx=0x7f87cc008040).stop 2022-07-12T08:44:54.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 shutdown_connections 2022-07-12T08:44:54.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87dc142f40 0x7f87dc13fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:54.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87dc0b7b00 0x7f87dc13f790 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:54.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f87bc053a40 0x7f87bc055f00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:54.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 --2- 172.21.15.61:0/2476037668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87dc0628e0 0x7f87dc0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:44:54.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 >> 172.21.15.61:0/2476037668 conn(0x7f87dc065550 msgr2=0x7f87dc066da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:44:54.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 shutdown_connections 2022-07-12T08:44:54.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:44:54.160+0000 7f87e1ac5700 1 -- 172.21.15.61:0/2476037668 wait complete. 2022-07-12T08:44:54.169 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:44:54.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:44:48.629894+0000', 'modified': '2022-07-12T08:44:50.312792+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5595': {'gid': 5595, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/4155742423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 4155742423}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 4155742423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:44:54.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:44:54.384 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:44:54.502 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5595,"want_state":"up:active","state":"up:active","rank_uptime":5.8605360769999999,"mdsmap_epoch":24,"osdmap_epoch":53,"osdmap_epoch_barrier":52,"uptime":10.419045105} 2022-07-12T08:44:54.503 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5595, "mdsmap_epoch": 24, "osdmap_epoch": 53, "osdmap_epoch_barrier": 52, "rank_uptime": 5.860536077, "state": "up:active", "uptime": 10.419045105, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:44:54.503 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:44:54.504 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:44:54.504 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:44:54.504 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.505 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:54.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:44:54.534 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:44:54.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:44:54.547 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:44:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:44:54.549 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.549 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:44:54.600 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:44:54.600 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:54.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:44:54.623 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:44:54.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:44:54.679 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:44:54.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:44:54.678+0000 7fe5b81c2080 -1 init, newargv = 0x55980617a370 newargc=9 2022-07-12T08:44:54.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21328]: starting ceph client 2022-07-12T08:44:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:44:54.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.685 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:44:54.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21328]: starting fuse 2022-07-12T08:44:54.691 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:44:54.692 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:44:54.693 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.693 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:44:54.694 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:44:54.694 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:44:54.695 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:44:54.695 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:44:54.695 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.696 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:44:54.696 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:44:54.697 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:44:54.697 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.698 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:44:54.698 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:44:54.699 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:44:54.699 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:44:54.700 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.700 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:44:54.701 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:44:54.701 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:44:54.701 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:44:54.702 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:44:54.702 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.703 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:44:54.703 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:44:54.779 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21328.asok 2022-07-12T08:44:54.783 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21328.asok 2022-07-12T08:44:54.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.784 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21328.asok status 2022-07-12T08:44:54.875 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:44:54.876 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:44:54.876 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:44:54.877 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:44:54.877 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:44:54.877 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:44:54.878 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:44:54.878 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21328", 2022-07-12T08:44:54.879 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:44:54.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:44:54.879 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:44:54.880 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:44:54.880 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5772, 2022-07-12T08:44:54.881 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:44:54.881 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:44:54.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:44:54.883 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5772 2022-07-12T08:44:54.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:44:54.884 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:44:54.884 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:44:54.885 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:44:54.885 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3365326284 2022-07-12T08:44:54.886 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:44:54.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:44:54.887 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:44:54.888 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:44:54.888 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:44:54.889 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3365326284 2022-07-12T08:44:54.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:44:54.890 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5772 172.21.15.121:0/3365326284", 2022-07-12T08:44:54.891 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/3365326284", 2022-07-12T08:44:54.892 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:44:54.892 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 24, 2022-07-12T08:44:54.892 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 53, 2022-07-12T08:44:54.893 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:44:54.893 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:44:54.893 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:44:54.893 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:44:54.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.896 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:54.901 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:54.915 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:44:54.916 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:44:54.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.919 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:54.979 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:44:54.980 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:44:54.981 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:44:54.990 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:44:54.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:44:54.993 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:44:54.994 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-07-12T08:44:54.994 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-07-12T08:44:54.995 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:44:54.995 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.996 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-07-12T08:44:54.996 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-12T08:44:54.996 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.997 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-12T08:44:54.997 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-07-12T08:44:54.998 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:44:54.998 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-07-12T08:44:54.998 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-07-12T08:44:54.999 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-07-12T08:44:55.000 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-07-12T08:44:55.001 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-07-12T08:44:55.001 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-07-12T08:44:55.001 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-07-12T08:44:55.002 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:45:02.537 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:02.542 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:02.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:02.561 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:02.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:04.323 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:45:04.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.432+0000 7f0ff7559700 1 Processor -- start 2022-07-12T08:45:04.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.432+0000 7f0ff7559700 1 -- start start 2022-07-12T08:45:04.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0ff009bce0 0x7f0ff00af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ff00b3f60 0x7f0ff00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ff00aecb0 con 0x7f0ff00af280 2022-07-12T08:45:04.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff0143b00 con 0x7f0ff00b3f60 2022-07-12T08:45:04.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ff01435c0 con 0x7f0ff009bce0 2022-07-12T08:45:04.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6557700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0ff00af280 0x7f0ff0060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44122/0 (socket says 172.21.15.61:44122) 2022-07-12T08:45:04.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6d58700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ff00b3f60 0x7f0ff00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6557700 1 -- 172.21.15.61:0/3429364892 learned_addr learned my addr 172.21.15.61:0/3429364892 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:04.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff5d56700 1 --2- 172.21.15.61:0/3429364892 >> v2:172.21.15.111:3300/0 conn(0x7f0ff009bce0 0x7f0ff00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6d58700 1 -- 172.21.15.61:0/3429364892 >> v2:172.21.15.111:3300/0 conn(0x7f0ff009bce0 msgr2=0x7f0ff00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:04.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6d58700 1 --2- 172.21.15.61:0/3429364892 >> v2:172.21.15.111:3300/0 conn(0x7f0ff009bce0 0x7f0ff00af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6d58700 1 -- 172.21.15.61:0/3429364892 >> v1:172.21.15.61:6789/0 conn(0x7f0ff00af280 legacy=0x7f0ff0060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6d58700 1 -- 172.21.15.61:0/3429364892 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0143860 con 0x7f0ff00b3f60 2022-07-12T08:45:04.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff6d58700 1 --2- 172.21.15.61:0/3429364892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00b3f60 0x7f0ff00b4810 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f0fe80032f0 tx=0x7f0fe80091d0).ready entity=mon.0 client_cookie=a7ecf00e4957477c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:04.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff5555700 1 -- 172.21.15.61:0/3429364892 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe800b680 con 0x7f0ff00b3f60 2022-07-12T08:45:04.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff5555700 1 -- 172.21.15.61:0/3429364892 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe800b7e0 con 0x7f0ff00b3f60 2022-07-12T08:45:04.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff5555700 1 -- 172.21.15.61:0/3429364892 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fe800b950 con 0x7f0ff00b3f60 2022-07-12T08:45:04.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- 172.21.15.61:0/3429364892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00b3f60 msgr2=0x7f0ff00b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3429364892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00b3f60 0x7f0ff00b4810 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f0fe80032f0 tx=0x7f0fe80091d0).stop 2022-07-12T08:45:04.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- 172.21.15.61:0/3429364892 shutdown_connections 2022-07-12T08:45:04.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3429364892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00b3f60 0x7f0ff00b4810 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3429364892 >> v2:172.21.15.111:3300/0 conn(0x7f0ff009bce0 0x7f0ff00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- 172.21.15.61:0/3429364892 >> 172.21.15.61:0/3429364892 conn(0x7f0ff00b92c0 msgr2=0x7f0ff00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:04.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- 172.21.15.61:0/3429364892 shutdown_connections 2022-07-12T08:45:04.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.436+0000 7f0ff7559700 1 -- 172.21.15.61:0/3429364892 wait complete. 2022-07-12T08:45:04.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 Processor -- start 2022-07-12T08:45:04.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 -- start start 2022-07-12T08:45:04.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ff009bce0 0x7f0ff01554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 0x7f0ff0142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ff00b3f60 0x7f0ff0144660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff009e370 con 0x7f0ff00af280 2022-07-12T08:45:04.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff009e610 con 0x7f0ff009bce0 2022-07-12T08:45:04.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff7559700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff009e8b0 con 0x7f0ff00b3f60 2022-07-12T08:45:04.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 0x7f0ff0142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 0x7f0ff0142b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55912/0 (socket says 172.21.15.61:55912) 2022-07-12T08:45:04.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 -- 172.21.15.61:0/3271879428 learned_addr learned my addr 172.21.15.61:0/3271879428 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:04.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff6557700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ff009bce0 0x7f0ff01554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff6d58700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ff00b3f60 0x7f0ff0144660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 -- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ff00b3f60 msgr2=0x7f0ff0144660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ff00b3f60 0x7f0ff0144660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 -- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ff009bce0 msgr2=0x7f0ff01554d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ff009bce0 0x7f0ff01554d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 -- 172.21.15.61:0/3271879428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe8009470 con 0x7f0ff00af280 2022-07-12T08:45:04.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0ff5d56700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 0x7f0ff0142b40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f0fec00c000 tx=0x7f0fec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:04.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.440+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fec010780 con 0x7f0ff00af280 2022-07-12T08:45:04.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fec0108e0 con 0x7f0ff00af280 2022-07-12T08:45:04.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff009f0d0 con 0x7f0ff00af280 2022-07-12T08:45:04.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0fec010c40 con 0x7f0ff00af280 2022-07-12T08:45:04.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff009e070 con 0x7f0ff00af280 2022-07-12T08:45:04.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff00b0a40 con 0x7f0ff00af280 2022-07-12T08:45:04.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0fec01c460 con 0x7f0ff00af280 2022-07-12T08:45:04.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0fdf7fe700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0fd0053a00 0x7f0fd0055ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.444+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0fec06c240 con 0x7f0ff00af280 2022-07-12T08:45:04.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.448+0000 7f0ff6557700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0fd0053a00 0x7f0fd0055ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.448+0000 7f0ff6557700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0fd0053a00 0x7f0fd0055ec0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f0fe000d000 tx=0x7f0fe0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:04.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.452+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0ff00b0a40 con 0x7f0ff00af280 2022-07-12T08:45:04.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.608+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ff00b0a40 con 0x7f0ff00af280 2022-07-12T08:45:04.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0fdf7fe700 1 -- 172.21.15.61:0/3271879428 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f0ff00b0a40 con 0x7f0ff00af280 2022-07-12T08:45:04.616 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:45:04.616 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:45:04.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0fd0053a00 msgr2=0x7f0fd0055ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0fd0053a00 0x7f0fd0055ec0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f0fe000d000 tx=0x7f0fe0009040).stop 2022-07-12T08:45:04.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 msgr2=0x7f0ff0142b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 0x7f0ff0142b40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f0fec00c000 tx=0x7f0fec006df0).stop 2022-07-12T08:45:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 shutdown_connections 2022-07-12T08:45:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ff00b3f60 0x7f0ff0144660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ff00af280 0x7f0ff0142b40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0fd0053a00 0x7f0fd0055ec0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 --2- 172.21.15.61:0/3271879428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ff009bce0 0x7f0ff01554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 >> 172.21.15.61:0/3271879428 conn(0x7f0ff00b92c0 msgr2=0x7f0ff00602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:04.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 shutdown_connections 2022-07-12T08:45:04.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.612+0000 7f0ff7559700 1 -- 172.21.15.61:0/3271879428 wait complete. 2022-07-12T08:45:04.631 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:45:04.634 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-12T08:45:04.740 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-12T08:45:04.741 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-12T08:45:04.742 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:45:04.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.832+0000 7fa60359e700 1 Processor -- start 2022-07-12T08:45:04.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.832+0000 7fa60359e700 1 -- start start 2022-07-12T08:45:04.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa60405f140 0x7fa604151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6040abc70 con 0x7fa6040ab670 2022-07-12T08:45:04.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60405c720 con 0x7fa6040ab290 2022-07-12T08:45:04.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6040aba90 con 0x7fa60405f140 2022-07-12T08:45:04.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa601d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa6040ab670 0x7fa6040afa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44138/0 (socket says 172.21.15.61:44138) 2022-07-12T08:45:04.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa601d9b700 1 -- 172.21.15.61:0/2011232433 learned_addr learned my addr 172.21.15.61:0/2011232433 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:04.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa602d9d700 1 --2- 172.21.15.61:0/2011232433 >> v2:172.21.15.111:3300/0 conn(0x7fa60405f140 0x7fa604151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60159a700 1 -- 172.21.15.61:0/2011232433 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530247881 0 0) 0x7fa6040abc70 con 0x7fa6040ab670 2022-07-12T08:45:04.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60159a700 1 -- 172.21.15.61:0/2011232433 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e8003260 con 0x7fa6040ab670 2022-07-12T08:45:04.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa602d9d700 1 -- 172.21.15.61:0/2011232433 >> v1:172.21.15.111:6789/0 conn(0x7fa6040ab290 legacy=0x7fa60405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa602d9d700 1 -- 172.21.15.61:0/2011232433 >> v1:172.21.15.61:6789/0 conn(0x7fa6040ab670 legacy=0x7fa6040afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa602d9d700 1 -- 172.21.15.61:0/2011232433 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60405c9c0 con 0x7fa60405f140 2022-07-12T08:45:04.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa602d9d700 1 --2- 172.21.15.61:0/2011232433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa60405f140 0x7fa604151bf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fa5fc0032f0 tx=0x7fa5fc0091d0).ready entity=mon.1 client_cookie=9e87775f2540ce43 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:04.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60159a700 1 -- 172.21.15.61:0/2011232433 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5fc00b6c0 con 0x7fa60405f140 2022-07-12T08:45:04.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60159a700 1 -- 172.21.15.61:0/2011232433 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc00b820 con 0x7fa60405f140 2022-07-12T08:45:04.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60159a700 1 -- 172.21.15.61:0/2011232433 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5fc00b990 con 0x7fa60405f140 2022-07-12T08:45:04.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- 172.21.15.61:0/2011232433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa60405f140 msgr2=0x7fa604151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 --2- 172.21.15.61:0/2011232433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa60405f140 0x7fa604151bf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fa5fc0032f0 tx=0x7fa5fc0091d0).stop 2022-07-12T08:45:04.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- 172.21.15.61:0/2011232433 shutdown_connections 2022-07-12T08:45:04.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 --2- 172.21.15.61:0/2011232433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa60405f140 0x7fa604151bf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- 172.21.15.61:0/2011232433 >> 172.21.15.61:0/2011232433 conn(0x7fa6040b1950 msgr2=0x7fa6040b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- 172.21.15.61:0/2011232433 shutdown_connections 2022-07-12T08:45:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 -- 172.21.15.61:0/2011232433 wait complete. 2022-07-12T08:45:04.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.836+0000 7fa60359e700 1 Processor -- start 2022-07-12T08:45:04.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 -- start start 2022-07-12T08:45:04.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa60405f140 0x7fa604063c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 0x7fa60414ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6040ab670 0x7fa60414b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604066c50 con 0x7fa6040ab290 2022-07-12T08:45:04.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604067460 con 0x7fa6040ab670 2022-07-12T08:45:04.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa604066510 con 0x7fa60405f140 2022-07-12T08:45:04.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 0x7fa60414ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 0x7fa60414ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55928/0 (socket says 172.21.15.61:55928) 2022-07-12T08:45:04.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa602d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6040ab670 0x7fa60414b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60259c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa60405f140 0x7fa604063c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 -- 172.21.15.61:0/455235931 learned_addr learned my addr 172.21.15.61:0/455235931 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:04.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 -- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa60405f140 msgr2=0x7fa604063c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa60405f140 0x7fa604063c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 -- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6040ab670 msgr2=0x7fa60414b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:04.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6040ab670 0x7fa60414b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:04.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 -- 172.21.15.61:0/455235931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc009470 con 0x7fa6040ab290 2022-07-12T08:45:04.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa601d9b700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 0x7fa60414ad30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fa5f8001110 tx=0x7fa5f8007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:04.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5f80070d0 con 0x7fa6040ab290 2022-07-12T08:45:04.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f8007230 con 0x7fa6040ab290 2022-07-12T08:45:04.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5f8005610 con 0x7fa6040ab290 2022-07-12T08:45:04.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6040667b0 con 0x7fa6040ab290 2022-07-12T08:45:04.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.840+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa604067030 con 0x7fa6040ab290 2022-07-12T08:45:04.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.844+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa604148950 con 0x7fa6040ab290 2022-07-12T08:45:04.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.844+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa5f801a070 con 0x7fa6040ab290 2022-07-12T08:45:04.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.844+0000 7fa5f37fe700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5e0053ad0 0x7fa5e0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:04.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.844+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5f806b150 con 0x7fa6040ab290 2022-07-12T08:45:04.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.844+0000 7fa60259c700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5e0053ad0 0x7fa5e0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:04.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.844+0000 7fa60259c700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5e0053ad0 0x7fa5e0055f90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa5f400d000 tx=0x7fa5f4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:04.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:04.848+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa604148950 con 0x7fa6040ab290 2022-07-12T08:45:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa604148950 con 0x7fa6040ab290 2022-07-12T08:45:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa5f37fe700 1 -- 172.21.15.61:0/455235931 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fa604148950 con 0x7fa6040ab290 2022-07-12T08:45:05.016 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:45:05.017 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:45:05.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5e0053ad0 msgr2=0x7fa5e0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5e0053ad0 0x7fa5e0055f90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fa5f400d000 tx=0x7fa5f4007b90).stop 2022-07-12T08:45:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 msgr2=0x7fa60414ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 0x7fa60414ad30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fa5f8001110 tx=0x7fa5f8007b20).stop 2022-07-12T08:45:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 shutdown_connections 2022-07-12T08:45:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5e0053ad0 0x7fa5e0055f90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6040ab670 0x7fa60414b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6040ab290 0x7fa60414ad30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 --2- 172.21.15.61:0/455235931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa60405f140 0x7fa604063c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 >> 172.21.15.61:0/455235931 conn(0x7fa6040b1950 msgr2=0x7fa6040b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 shutdown_connections 2022-07-12T08:45:05.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.012+0000 7fa60359e700 1 -- 172.21.15.61:0/455235931 wait complete. 2022-07-12T08:45:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:45:05.031 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-12T08:45:05.140 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-12T08:45:05.141 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-12T08:45:05.142 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:45:05.162 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:45:05.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f467f59e700 1 Processor -- start 2022-07-12T08:45:05.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f467f59e700 1 -- start start 2022-07-12T08:45:05.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f467f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 0x7f4680143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f467f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46800afac0 con 0x7f4680060c30 2022-07-12T08:45:05.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f467f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46800611b0 con 0x7f468013cb80 2022-07-12T08:45:05.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f467f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4680143970 con 0x7f46800af6e0 2022-07-12T08:45:05.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f4675d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4680060c30 0x7f468013c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44154/0 (socket says 172.21.15.61:44154) 2022-07-12T08:45:05.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.264+0000 7f4675d9b700 1 -- 172.21.15.61:0/3913660948 learned_addr learned my addr 172.21.15.61:0/3913660948 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:05.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467e59c700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 0x7f4680143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258338771 0 0) 0x7f46800afac0 con 0x7f4680060c30 2022-07-12T08:45:05.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660003260 con 0x7f4680060c30 2022-07-12T08:45:05.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925573224 0 0) 0x7f46800611b0 con 0x7f468013cb80 2022-07-12T08:45:05.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46800afac0 con 0x7f468013cb80 2022-07-12T08:45:05.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365197634 0 0) 0x7f4660003260 con 0x7f4680060c30 2022-07-12T08:45:05.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 msgr2=0x7f4680143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 0x7f4680143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 >> v1:172.21.15.111:6789/0 conn(0x7f468013cb80 legacy=0x7f46800b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680061450 con 0x7f4680060c30 2022-07-12T08:45:05.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f4668001660 con 0x7f4680060c30 2022-07-12T08:45:05.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 >> v1:172.21.15.61:6789/0 conn(0x7f4680060c30 legacy=0x7f468013c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 0x7f4660003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4660003e70 0x7f4660004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f4660004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680143970 con 0x7f46600049f0 2022-07-12T08:45:05.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466000b830 con 0x7f4660003e70 2022-07-12T08:45:05.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466000b9f0 con 0x7f46800af6e0 2022-07-12T08:45:05.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f4660004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467e59c700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 0x7f4660003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f4675d9b700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4660003e70 0x7f4660004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 -- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 msgr2=0x7f4660003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46800af6e0 0x7f4660003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 -- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4660003e70 msgr2=0x7f4660004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4660003e70 0x7f4660004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 -- 172.21.15.61:0/3913660948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4660003260 con 0x7f46600049f0 2022-07-12T08:45:05.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f4660004e70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f4678001490 tx=0x7f4678007d80).ready entity=mon.0 client_cookie=ee9c65fa816870cd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4678006890 con 0x7f46600049f0 2022-07-12T08:45:05.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46780069f0 con 0x7f46600049f0 2022-07-12T08:45:05.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467dd9b700 1 -- 172.21.15.61:0/3913660948 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4678006d50 con 0x7f46600049f0 2022-07-12T08:45:05.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467f59e700 1 -- 172.21.15.61:0/3913660948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 msgr2=0x7f4660004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467f59e700 1 --2- 172.21.15.61:0/3913660948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f4660004e70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f4678001490 tx=0x7f4678007d80).stop 2022-07-12T08:45:05.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467ed9d700 1 -- 172.21.15.61:0/3913660948 reap_dead start 2022-07-12T08:45:05.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467f59e700 1 -- 172.21.15.61:0/3913660948 shutdown_connections 2022-07-12T08:45:05.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467f59e700 1 -- 172.21.15.61:0/3913660948 >> 172.21.15.61:0/3913660948 conn(0x7f46800b5780 msgr2=0x7f46800b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467f59e700 1 -- 172.21.15.61:0/3913660948 shutdown_connections 2022-07-12T08:45:05.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.268+0000 7f467f59e700 1 -- 172.21.15.61:0/3913660948 wait complete. 2022-07-12T08:45:05.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 Processor -- start 2022-07-12T08:45:05.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 -- start start 2022-07-12T08:45:05.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f46800ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4680151f00 0x7f4680152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4680152850 0x7f4680065160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680154f80 con 0x7f46600049f0 2022-07-12T08:45:05.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4680067010 con 0x7f4680151f00 2022-07-12T08:45:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46801551b0 con 0x7f4680152850 2022-07-12T08:45:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f46800ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f46800ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55944/0 (socket says 172.21.15.61:55944) 2022-07-12T08:45:05.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 -- 172.21.15.61:0/615086402 learned_addr learned my addr 172.21.15.61:0/615086402 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:05.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467ed9d700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4680152850 0x7f4680065160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f4675d9b700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4680151f00 0x7f4680152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 -- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4680152850 msgr2=0x7f4680065160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4680152850 0x7f4680065160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 -- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4680151f00 msgr2=0x7f4680152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4680151f00 0x7f4680152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 -- 172.21.15.61:0/615086402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680143400 con 0x7f46600049f0 2022-07-12T08:45:05.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467e59c700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f46800ae500 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f467000b000 tx=0x7f4670006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f467000f760 con 0x7f46600049f0 2022-07-12T08:45:05.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467000f8c0 con 0x7f46600049f0 2022-07-12T08:45:05.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46801519e0 con 0x7f46600049f0 2022-07-12T08:45:05.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f467000fc60 con 0x7f46600049f0 2022-07-12T08:45:05.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.272+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468013cf90 con 0x7f46600049f0 2022-07-12T08:45:05.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.276+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4680057460 con 0x7f46600049f0 2022-07-12T08:45:05.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.276+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4670014440 con 0x7f46600049f0 2022-07-12T08:45:05.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.276+0000 7f4677fff700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46640539d0 0x7f4664055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.276+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f467006c8b0 con 0x7f46600049f0 2022-07-12T08:45:05.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.276+0000 7f4675d9b700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46640539d0 0x7f4664055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.276+0000 7f4675d9b700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46640539d0 0x7f4664055e90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f4668005000 tx=0x7f46800b1d70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.284+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f46700436a0 con 0x7f46600049f0 2022-07-12T08:45:05.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4680068400 con 0x7f46600049f0 2022-07-12T08:45:05.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f4677fff700 1 -- 172.21.15.61:0/615086402 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f4670018020 con 0x7f46600049f0 2022-07-12T08:45:05.454 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:45:05.454 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:45:05.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46640539d0 msgr2=0x7f4664055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46640539d0 0x7f4664055e90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f4668005000 tx=0x7f46800b1d70).stop 2022-07-12T08:45:05.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 msgr2=0x7f46800ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f46800ae500 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7f467000b000 tx=0x7f4670006df0).stop 2022-07-12T08:45:05.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 shutdown_connections 2022-07-12T08:45:05.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4680152850 0x7f4680065160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46640539d0 0x7f4664055e90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4680151f00 0x7f4680152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 --2- 172.21.15.61:0/615086402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46600049f0 0x7f46800ae500 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 >> 172.21.15.61:0/615086402 conn(0x7f46800b5780 msgr2=0x7f46800ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.448+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 shutdown_connections 2022-07-12T08:45:05.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.452+0000 7f467f59e700 1 -- 172.21.15.61:0/615086402 wait complete. 2022-07-12T08:45:05.459 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:45:05.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 5 2022-07-12T08:45:05.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.580+0000 7fc94ed94700 1 Processor -- start 2022-07-12T08:45:05.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.580+0000 7fc94ed94700 1 -- start start 2022-07-12T08:45:05.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc9480afa40 0x7fc94805f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc94805c5a0 0x7fc9481403a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9480afe20 con 0x7fc9480b2370 2022-07-12T08:45:05.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b1990 con 0x7fc9480afa40 2022-07-12T08:45:05.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9480b1c30 con 0x7fc94805c5a0 2022-07-12T08:45:05.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94dd92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc9480afa40 0x7fc94805f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94d591700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc9480b2370 0x7fc94813fc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44170/0 (socket says 172.21.15.61:44170) 2022-07-12T08:45:05.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94d591700 1 -- 172.21.15.61:0/2599770647 learned_addr learned my addr 172.21.15.61:0/2599770647 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:05.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94e593700 1 --2- 172.21.15.61:0/2599770647 >> v2:172.21.15.111:3300/0 conn(0x7fc94805c5a0 0x7fc9481403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94dd92700 1 -- 172.21.15.61:0/2599770647 >> v2:172.21.15.111:3300/0 conn(0x7fc94805c5a0 msgr2=0x7fc9481403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94dd92700 1 --2- 172.21.15.61:0/2599770647 >> v2:172.21.15.111:3300/0 conn(0x7fc94805c5a0 0x7fc9481403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94dd92700 1 -- 172.21.15.61:0/2599770647 >> v1:172.21.15.61:6789/0 conn(0x7fc9480b2370 legacy=0x7fc94813fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94dd92700 1 -- 172.21.15.61:0/2599770647 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9480b1ed0 con 0x7fc9480afa40 2022-07-12T08:45:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94e593700 1 --2- 172.21.15.61:0/2599770647 >> v2:172.21.15.111:3300/0 conn(0x7fc94805c5a0 0x7fc9481403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:45:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94dd92700 1 --2- 172.21.15.61:0/2599770647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480afa40 0x7fc94805f540 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fc93c002f70 tx=0x7fc93c006df0).ready entity=mon.0 client_cookie=6f044c6b4a5ad3b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94cd90700 1 -- 172.21.15.61:0/2599770647 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc93c00d5c0 con 0x7fc9480afa40 2022-07-12T08:45:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94cd90700 1 -- 172.21.15.61:0/2599770647 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c000e90 con 0x7fc9480afa40 2022-07-12T08:45:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94cd90700 1 -- 172.21.15.61:0/2599770647 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc93c00d910 con 0x7fc9480afa40 2022-07-12T08:45:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- 172.21.15.61:0/2599770647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480afa40 msgr2=0x7fc94805f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 --2- 172.21.15.61:0/2599770647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480afa40 0x7fc94805f540 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fc93c002f70 tx=0x7fc93c006df0).stop 2022-07-12T08:45:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- 172.21.15.61:0/2599770647 shutdown_connections 2022-07-12T08:45:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 --2- 172.21.15.61:0/2599770647 >> v2:172.21.15.111:3300/0 conn(0x7fc94805c5a0 0x7fc9481403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 --2- 172.21.15.61:0/2599770647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480afa40 0x7fc94805f540 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- 172.21.15.61:0/2599770647 >> 172.21.15.61:0/2599770647 conn(0x7fc948061f50 msgr2=0x7fc948063a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- 172.21.15.61:0/2599770647 shutdown_connections 2022-07-12T08:45:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.584+0000 7fc94ed94700 1 -- 172.21.15.61:0/2599770647 wait complete. 2022-07-12T08:45:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 Processor -- start 2022-07-12T08:45:05.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 -- start start 2022-07-12T08:45:05.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc94805c5a0 0x7fc948140ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9480afa40 0x7fc94813b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 0x7fc94813ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948144ef0 con 0x7fc9480b2370 2022-07-12T08:45:05.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94813bf40 con 0x7fc9480afa40 2022-07-12T08:45:05.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94ed94700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc948150d80 con 0x7fc94805c5a0 2022-07-12T08:45:05.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 0x7fc94813ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 0x7fc94813ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55962/0 (socket says 172.21.15.61:55962) 2022-07-12T08:45:05.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 -- 172.21.15.61:0/54857816 learned_addr learned my addr 172.21.15.61:0/54857816 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:05.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94d591700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9480afa40 0x7fc94813b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94dd92700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc94805c5a0 0x7fc948140ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 -- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc94805c5a0 msgr2=0x7fc948140ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc94805c5a0 0x7fc948140ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 -- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9480afa40 msgr2=0x7fc94813b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9480afa40 0x7fc94813b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 -- 172.21.15.61:0/54857816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c00b180 con 0x7fc9480b2370 2022-07-12T08:45:05.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94dd92700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc94805c5a0 0x7fc948140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:45:05.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc94e593700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 0x7fc94813ba50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fc940006cc0 tx=0x7fc940008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.588+0000 7fc93affd700 1 -- 172.21.15.61:0/54857816 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc94000c770 con 0x7fc9480b2370 2022-07-12T08:45:05.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc93affd700 1 -- 172.21.15.61:0/54857816 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94000c8d0 con 0x7fc9480b2370 2022-07-12T08:45:05.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc93affd700 1 -- 172.21.15.61:0/54857816 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc94000cc30 con 0x7fc9480b2370 2022-07-12T08:45:05.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9481418a0 con 0x7fc9480b2370 2022-07-12T08:45:05.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9480675a0 con 0x7fc9480b2370 2022-07-12T08:45:05.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc93affd700 1 -- 172.21.15.61:0/54857816 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc940013430 con 0x7fc9480b2370 2022-07-12T08:45:05.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc93affd700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc928053a60 0x7fc928055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc93affd700 1 -- 172.21.15.61:0/54857816 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc94006bdd0 con 0x7fc9480b2370 2022-07-12T08:45:05.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc94dd92700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc928053a60 0x7fc928055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 Processor -- start 2022-07-12T08:45:05.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 -- start start 2022-07-12T08:45:05.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1762d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x1760e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x16d4130 con 0x1763230 2022-07-12T08:45:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x16b4430 con 0x179d0c0 2022-07-12T08:45:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.592+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x175e980 con 0x17609b0 2022-07-12T08:45:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17636b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55966/0 (socket says 172.21.15.61:55966) 2022-07-12T08:45:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 -- 172.21.15.61:0/588091105 learned_addr learned my addr 172.21.15.61:0/588091105 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94dd92700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc928053a60 0x7fc928055f20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc93c015000 tx=0x7fc93c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94e593700 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x1760e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 -- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 msgr2=0x1760e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x1760e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 -- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 msgr2=0x1762d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1762d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-12T08:45:05.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 -- 172.21.15.61:0/588091105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x170bbf0 con 0x1763230 2022-07-12T08:45:05.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17636b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7fc94400b000 tx=0x7fc944009040).ready entity=mon.0 client_cookie=42a9754e7c045f2b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94ed94700 1 -- 172.21.15.61:0/588091105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9440072c0 con 0x1763230 2022-07-12T08:45:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94ed94700 1 -- 172.21.15.61:0/588091105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc944007420 con 0x1763230 2022-07-12T08:45:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94ed94700 1 -- 172.21.15.61:0/588091105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc944005780 con 0x1763230 2022-07-12T08:45:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- 172.21.15.61:0/588091105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 msgr2=0x17636b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17636b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7fc94400b000 tx=0x7fc944009040).stop 2022-07-12T08:45:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- 172.21.15.61:0/588091105 shutdown_connections 2022-07-12T08:45:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x1760e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17636b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- 172.21.15.61:0/588091105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1762d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-12T08:45:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- 172.21.15.61:0/588091105 >> 172.21.15.61:0/588091105 conn(0x175f9b0 msgr2=0x179c800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- 172.21.15.61:0/588091105 shutdown_connections 2022-07-12T08:45:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- 172.21.15.61:0/588091105 wait complete. 2022-07-12T08:45:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- start start 2022-07-12T08:45:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 Processor -- start 2022-07-12T08:45:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x17609b0 0x176a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x17a8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x176a860 con 0x1763230 2022-07-12T08:45:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x176aa40 con 0x17609b0 2022-07-12T08:45:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x176ace0 con 0x179d0c0 2022-07-12T08:45:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc95a24d740 10 client.0 ll_register_callbacks cb 0x171b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:45:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17a7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55974/0 (socket says 172.21.15.61:55974) 2022-07-12T08:45:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.596+0000 7fc94d591700 1 -- 172.21.15.61:0/2352636007 learned_addr learned my addr 172.21.15.61:0/2352636007 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94e593700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x17a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94dd92700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x17609b0 0x176a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94d591700 1 -- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 msgr2=0x17a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94d591700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x17a8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94d591700 1 -- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x17609b0 msgr2=0x176a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94d591700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x17609b0 0x176a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94d591700 1 -- 172.21.15.61:0/2352636007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94400f040 con 0x1763230 2022-07-12T08:45:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94e593700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x17a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:45:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94d591700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17a7d50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fc94400b000 tx=0x7fc944009040).ready entity=mon.0 client_cookie=c285dd8b3b138333 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:45:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 1 -- 172.21.15.61:0/2352636007 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc944005b50 con 0x1763230 2022-07-12T08:45:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 1 -- 172.21.15.61:0/2352636007 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc944005cb0 con 0x1763230 2022-07-12T08:45:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x16ca1a0 con 0x1763230 2022-07-12T08:45:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 1 -- 172.21.15.61:0/2352636007 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc944005e70 con 0x1763230 2022-07-12T08:45:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 10 client.5808 fetch_fsmap learned FSMap version 24 2022-07-12T08:45:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x16b9110 con 0x1763230 2022-07-12T08:45:05.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 1 -- 172.21.15.61:0/2352636007 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x16ca1a0 con 0x1763230 2022-07-12T08:45:05.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 1 -- 172.21.15.61:0/2352636007 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 24) v1 ==== 1247+0+0 (secure 0 0 0) 0x7fc9440209d0 con 0x1763230 2022-07-12T08:45:05.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 10 client.5808 fetch_fsmap finished waiting for FSMap version 24 2022-07-12T08:45:05.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 10 client.5808 resolve_mds: resolved ID 'a' to GID 5595 2022-07-12T08:45:05.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 20 client.5808 populate_metadata read hostname 'smithi061' 2022-07-12T08:45:05.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1768290 0x1768670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 4 client.5808 mds_command: new command op to 5595 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-12T08:45:05.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 --> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x16c2220 con 0x1768290 2022-07-12T08:45:05.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94dd92700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1768290 0x1768670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 10 client.5808 ms_handle_connect on v2:172.21.15.61:6832/4155742423 2022-07-12T08:45:05.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc94dd92700 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1768290 0x1768670 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=7483389cd3e87424 server_cookie=588680c16ac59dd6 in_seq=0 out_seq=0 2022-07-12T08:45:05.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 1 -- 172.21.15.61:0/2352636007 <== mds.0 v2:172.21.15.61:6832/4155742423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x16c2220 con 0x1768290 2022-07-12T08:45:05.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 10 client.5808 handle_command_reply: tid=0 2022-07-12T08:45:05.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 client.5808 shutdown 2022-07-12T08:45:05.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 msgr2=0x17a7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17a7d50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7fc94400b000 tx=0x7fc944009040).stop 2022-07-12T08:45:05.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 shutdown_connections 2022-07-12T08:45:05.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1768290 0x1768670 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x17a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x17a7d50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc924ff9700 0 client.5808 ms_handle_reset on v2:172.21.15.61:6832/4155742423 2022-07-12T08:45:05.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 --2- 172.21.15.61:0/2352636007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x17609b0 0x176a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 >> 172.21.15.61:0/2352636007 conn(0x175f9b0 msgr2=0x179c710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 shutdown_connections 2022-07-12T08:45:05.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.600+0000 7fc95a24d740 1 -- 172.21.15.61:0/2352636007 wait complete. 2022-07-12T08:45:05.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.604+0000 7fc95a24d740 20 client.5808 trim_cache size 0 max 16384 2022-07-12T08:45:05.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 Processor -- start 2022-07-12T08:45:05.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 -- start start 2022-07-12T08:45:05.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 0x1768670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1763230 0x17a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x1761ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x1767840 con 0x17609b0 2022-07-12T08:45:05.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x1769f00 con 0x1763230 2022-07-12T08:45:05.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x175efe0 con 0x179d0c0 2022-07-12T08:45:05.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 0x1768670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 0x1768670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55980/0 (socket says 172.21.15.61:55980) 2022-07-12T08:45:05.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 -- 172.21.15.61:0/3742616495 learned_addr learned my addr 172.21.15.61:0/3742616495 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:05.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94e593700 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x1761ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94d591700 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1763230 0x17a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 -- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 msgr2=0x1761ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x1761ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 -- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1763230 msgr2=0x17a6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1763230 0x17a6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 -- 172.21.15.61:0/3742616495 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x16c3790 con 0x17609b0 2022-07-12T08:45:05.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.616+0000 7fc94dd92700 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 0x1768670 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x1799e40 tx=0x7fc93c012000).ready entity=mon.0 client_cookie=e3fc01445b81e5a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc926ffd700 1 -- 172.21.15.61:0/3742616495 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc93c02c4f0 con 0x17609b0 2022-07-12T08:45:05.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc926ffd700 1 -- 172.21.15.61:0/3742616495 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c02c650 con 0x17609b0 2022-07-12T08:45:05.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc926ffd700 1 -- 172.21.15.61:0/3742616495 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc93c02c9b0 con 0x17609b0 2022-07-12T08:45:05.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- 172.21.15.61:0/3742616495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 msgr2=0x1768670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:05.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 0x1768670 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x1799e40 tx=0x7fc93c012000).stop 2022-07-12T08:45:05.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- 172.21.15.61:0/3742616495 shutdown_connections 2022-07-12T08:45:05.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x179d0c0 0x1761ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x1763230 0x17a6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3742616495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x17609b0 0x1768670 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- 172.21.15.61:0/3742616495 >> 172.21.15.61:0/3742616495 conn(0x175f9b0 msgr2=0x179c710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:05.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- 172.21.15.61:0/3742616495 shutdown_connections 2022-07-12T08:45:05.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- 172.21.15.61:0/3742616495 wait complete. 2022-07-12T08:45:05.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- start start 2022-07-12T08:45:05.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 Processor -- start 2022-07-12T08:45:05.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x17956a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x1802c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1803110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x1803600 con 0x1763230 2022-07-12T08:45:05.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x18027c0 con 0x179d0c0 2022-07-12T08:45:05.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x17667b0 con 0x17609b0 2022-07-12T08:45:05.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc95a24d740 10 client.0 ll_register_callbacks cb 0x1769e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:45:05.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x1802c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94dd92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x17956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x1802c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:55990/0 (socket says 172.21.15.61:55990) 2022-07-12T08:45:05.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 -- 172.21.15.61:0/3295312972 learned_addr learned my addr 172.21.15.61:0/3295312972 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:05.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94e593700 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1803110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 -- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 msgr2=0x17956a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x17956a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 -- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 msgr2=0x1803110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:05.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1803110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:05.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.620+0000 7fc94d591700 1 -- 172.21.15.61:0/3295312972 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c026040 con 0x1763230 2022-07-12T08:45:05.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc94d591700 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x1802c20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fc94402f000 tx=0x7fc94401e000).ready entity=mon.0 client_cookie=3089ef821d6f76aa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:05.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:45:05.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 1 -- 172.21.15.61:0/3295312972 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc944005960 con 0x1763230 2022-07-12T08:45:05.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 1 -- 172.21.15.61:0/3295312972 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc944005ac0 con 0x1763230 2022-07-12T08:45:05.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc944007420 con 0x1763230 2022-07-12T08:45:05.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 1 -- 172.21.15.61:0/3295312972 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc944005c80 con 0x1763230 2022-07-12T08:45:05.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 10 client.5814 fetch_fsmap learned FSMap version 24 2022-07-12T08:45:05.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x16cc380 con 0x1763230 2022-07-12T08:45:05.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 1 -- 172.21.15.61:0/3295312972 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc944007420 con 0x1763230 2022-07-12T08:45:05.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 1 -- 172.21.15.61:0/3295312972 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 24) v1 ==== 1247+0+0 (secure 0 0 0) 0x7fc944020480 con 0x1763230 2022-07-12T08:45:05.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 10 client.5814 fetch_fsmap finished waiting for FSMap version 24 2022-07-12T08:45:05.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 10 client.5814 resolve_mds: resolved ID 'a' to GID 5595 2022-07-12T08:45:05.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 20 client.5814 populate_metadata read hostname 'smithi061' 2022-07-12T08:45:05.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1766c00 0x17a3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:05.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 4 client.5814 mds_command: new command op to 5595 tid=0[{"prefix": "cache drop", "timeout": 5}] 2022-07-12T08:45:05.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 --> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] -- command(tid 0: {"prefix": "cache drop", "timeout": 5}) v1 -- 0x15b75e0 con 0x1766c00 2022-07-12T08:45:05.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc94dd92700 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1766c00 0x17a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:05.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc94dd92700 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1766c00 0x17a3760 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=f99dee2a788a69f7 server_cookie=5cee7a93e5aa90b4 in_seq=0 out_seq=0 2022-07-12T08:45:05.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:05.624+0000 7fc925ffb700 10 client.5814 ms_handle_connect on v2:172.21.15.61:6832/4155742423 2022-07-12T08:45:10.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc925ffb700 1 -- 172.21.15.61:0/3295312972 <== mds.0 v2:172.21.15.61:6832/4155742423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+404 (crc 0 0 0) 0x15b75e0 con 0x1766c00 2022-07-12T08:45:10.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc925ffb700 10 client.5814 handle_command_reply: tid=0 2022-07-12T08:45:10.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 client.5814 shutdown 2022-07-12T08:45:10.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 msgr2=0x1802c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:10.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x1802c20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fc94402f000 tx=0x7fc94401e000).stop 2022-07-12T08:45:10.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 shutdown_connections 2022-07-12T08:45:10.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:6832/4155742423,v1:172.21.15.61:6833/4155742423] conn(0x1766c00 0x17a3760 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x179d0c0 0x1803110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1763230 0x1802c20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 --2- 172.21.15.61:0/3295312972 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x17609b0 0x17956a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 >> 172.21.15.61:0/3295312972 conn(0x175f9b0 msgr2=0x1768670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:10.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc925ffb700 0 client.5814 ms_handle_reset on v2:172.21.15.61:6832/4155742423 2022-07-12T08:45:10.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 shutdown_connections 2022-07-12T08:45:10.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 1 -- 172.21.15.61:0/3295312972 wait complete. 2022-07-12T08:45:10.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc95a24d740 20 client.5814 trim_cache size 0 max 16384 2022-07-12T08:45:10.861 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:45:10.862 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-12T08:45:10.862 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": -110, 2022-07-12T08:45:10.862 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(110) Connection timed out", 2022-07-12T08:45:10.863 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 120 2022-07-12T08:45:10.863 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:45:10.864 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-12T08:45:10.864 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-12T08:45:10.865 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-12T08:45:10.865 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:45:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-12T08:45:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 0 2022-07-12T08:45:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:45:10.867 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-12T08:45:10.867 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-12T08:45:10.868 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 20069, 2022-07-12T08:45:10.868 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 3104736 2022-07-12T08:45:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:45:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:45:10.870 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 5.1898514430000002 2022-07-12T08:45:10.870 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:45:10.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc928053a60 msgr2=0x7fc928055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:10.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc928053a60 0x7fc928055f20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fc93c015000 tx=0x7fc93c014000).stop 2022-07-12T08:45:10.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 msgr2=0x7fc94813ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:10.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 0x7fc94813ba50 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fc940006cc0 tx=0x7fc940008040).stop 2022-07-12T08:45:10.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 shutdown_connections 2022-07-12T08:45:10.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9480b2370 0x7fc94813ba50 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc928053a60 0x7fc928055f20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9480afa40 0x7fc94813b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 --2- 172.21.15.61:0/54857816 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc94805c5a0 0x7fc948140ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:10.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 >> 172.21.15.61:0/54857816 conn(0x7fc948061f50 msgr2=0x7fc948063920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:10.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 shutdown_connections 2022-07-12T08:45:10.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:10.816+0000 7fc94ed94700 1 -- 172.21.15.61:0/54857816 wait complete. 2022-07-12T08:45:10.880 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-12T08:45:10.881 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:10.881 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:45:10.882 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:45:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-12T08:45:10.898 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-12T08:45:10.899 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:45:10.900 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:45:10.900 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:10.960 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:10.961 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:45:10.974 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:10.983 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:45:10.991 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:45:10.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:45:10.998 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:11.008 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:45:11.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 Processor -- start 2022-07-12T08:45:11.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- start start 2022-07-12T08:45:11.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8e780b29a0 0x7f8e78066430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e780b3270 0x7f8e7805bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e781434a0 0x7f8e78143930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e78066920 con 0x7f8e780b29a0 2022-07-12T08:45:11.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e78066b70 con 0x7f8e780b3270 2022-07-12T08:45:11.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e780b2d80 con 0x7f8e781434a0 2022-07-12T08:45:11.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8e780b29a0 0x7f8e78066430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8e780b29a0 0x7f8e78066430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56000/0 (socket says 172.21.15.61:56000) 2022-07-12T08:45:11.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 -- 172.21.15.61:0/710488988 learned_addr learned my addr 172.21.15.61:0/710488988 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:11.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7ea61700 1 --2- 172.21.15.61:0/710488988 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e781434a0 0x7f8e78143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7da5f700 1 --2- 172.21.15.61:0/710488988 >> v2:172.21.15.111:3300/0 conn(0x7f8e780b3270 0x7f8e7805bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 -- 172.21.15.61:0/710488988 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e781434a0 msgr2=0x7f8e78143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 --2- 172.21.15.61:0/710488988 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e781434a0 0x7f8e78143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 -- 172.21.15.61:0/710488988 >> v2:172.21.15.111:3300/0 conn(0x7f8e780b3270 msgr2=0x7f8e7805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 --2- 172.21.15.61:0/710488988 >> v2:172.21.15.111:3300/0 conn(0x7f8e780b3270 0x7f8e7805bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 -- 172.21.15.61:0/710488988 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78142ac0 con 0x7f8e780b29a0 2022-07-12T08:45:11.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7e260700 1 --2- 172.21.15.61:0/710488988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e780b29a0 0x7f8e78066430 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f8e780633c0 tx=0x7f8e70009290).ready entity=mon.0 client_cookie=d1537182b537fe6d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7d25e700 1 -- 172.21.15.61:0/710488988 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e7000b860 con 0x7f8e780b29a0 2022-07-12T08:45:11.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7d25e700 1 -- 172.21.15.61:0/710488988 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7000b9c0 con 0x7f8e780b29a0 2022-07-12T08:45:11.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7d25e700 1 -- 172.21.15.61:0/710488988 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e7000bb30 con 0x7f8e780b29a0 2022-07-12T08:45:11.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- 172.21.15.61:0/710488988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e780b29a0 msgr2=0x7f8e78066430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- 172.21.15.61:0/710488988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e780b29a0 0x7f8e78066430 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f8e780633c0 tx=0x7f8e70009290).stop 2022-07-12T08:45:11.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- 172.21.15.61:0/710488988 shutdown_connections 2022-07-12T08:45:11.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- 172.21.15.61:0/710488988 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e781434a0 0x7f8e78143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- 172.21.15.61:0/710488988 >> v2:172.21.15.111:3300/0 conn(0x7f8e780b3270 0x7f8e7805bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 --2- 172.21.15.61:0/710488988 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e780b29a0 0x7f8e78066430 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- 172.21.15.61:0/710488988 >> 172.21.15.61:0/710488988 conn(0x7f8e780b70b0 msgr2=0x7f8e780b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:11.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.128+0000 7f8e7f262700 1 -- 172.21.15.61:0/710488988 shutdown_connections 2022-07-12T08:45:11.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- 172.21.15.61:0/710488988 wait complete. 2022-07-12T08:45:11.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 Processor -- start 2022-07-12T08:45:11.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- start start 2022-07-12T08:45:11.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e780b29a0 0x7f8e7809b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e780b3270 0x7f8e78140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 0x7f8e7813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7809aaf0 con 0x7f8e781434a0 2022-07-12T08:45:11.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7809ad60 con 0x7f8e780b29a0 2022-07-12T08:45:11.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7809b000 con 0x7f8e780b3270 2022-07-12T08:45:11.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 0x7f8e7813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 0x7f8e7813b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56010/0 (socket says 172.21.15.61:56010) 2022-07-12T08:45:11.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 -- 172.21.15.61:0/343879055 learned_addr learned my addr 172.21.15.61:0/343879055 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:11.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7e260700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e780b29a0 0x7f8e7809b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7da5f700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e780b3270 0x7f8e78140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 -- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e780b3270 msgr2=0x7f8e78140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e780b3270 0x7f8e78140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 -- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e780b29a0 msgr2=0x7f8e7809b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e780b29a0 0x7f8e7809b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 -- 172.21.15.61:0/343879055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70009530 con 0x7f8e781434a0 2022-07-12T08:45:11.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7e260700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e780b29a0 0x7f8e7809b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:45:11.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7da5f700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e780b3270 0x7f8e78140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:45:11.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7ea61700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 0x7f8e7813b220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f8e7400a000 tx=0x7f8e74008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e74011780 con 0x7f8e781434a0 2022-07-12T08:45:11.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e740118e0 con 0x7f8e781434a0 2022-07-12T08:45:11.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7813b710 con 0x7f8e781434a0 2022-07-12T08:45:11.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e74011c40 con 0x7f8e781434a0 2022-07-12T08:45:11.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.132+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e780ad8c0 con 0x7f8e781434a0 2022-07-12T08:45:11.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.136+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8e74012420 con 0x7f8e781434a0 2022-07-12T08:45:11.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.136+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e78057460 con 0x7f8e781434a0 2022-07-12T08:45:11.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.136+0000 7f8e677fe700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e58053940 0x7f8e58055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.136+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e7406bca0 con 0x7f8e781434a0 2022-07-12T08:45:11.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.136+0000 7f8e7e260700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e58053940 0x7f8e58055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.136+0000 7f8e7e260700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e58053940 0x7f8e58055e00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8e70019000 tx=0x7f8e70016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.140+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8e74070070 con 0x7f8e781434a0 2022-07-12T08:45:11.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.272+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e780b6870 con 0x7f8e781434a0 2022-07-12T08:45:11.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.272+0000 7f8e677fe700 1 -- 172.21.15.61:0/343879055 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16088 (secure 0 0 0) 0x7f8e74042ba0 con 0x7f8e781434a0 2022-07-12T08:45:11.280 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:45:11.280 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":53,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:44:50.635406+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:44:47.163319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-12T08:44:47.639688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:45:11.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e58053940 msgr2=0x7f8e58055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e58053940 0x7f8e58055e00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8e70019000 tx=0x7f8e70016000).stop 2022-07-12T08:45:11.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 msgr2=0x7f8e7813b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 0x7f8e7813b220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f8e7400a000 tx=0x7f8e74008040).stop 2022-07-12T08:45:11.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 shutdown_connections 2022-07-12T08:45:11.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e781434a0 0x7f8e7813b220 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e780b3270 0x7f8e78140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e58053940 0x7f8e58055e00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 --2- 172.21.15.61:0/343879055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e780b29a0 0x7f8e7809b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 >> 172.21.15.61:0/343879055 conn(0x7f8e780b70b0 msgr2=0x7f8e7805bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:11.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 shutdown_connections 2022-07-12T08:45:11.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.276+0000 7f8e7f262700 1 -- 172.21.15.61:0/343879055 wait complete. 2022-07-12T08:45:11.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:44:47.163319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:44:47.639688+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:45:11.305 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:45:11.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.420+0000 7fca69da5700 1 Processor -- start 2022-07-12T08:45:11.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.420+0000 7fca69da5700 1 -- start start 2022-07-12T08:45:11.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca640ac7d0 0x7fca640acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca640b1600 0x7fca640b0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca64142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca640af430 con 0x7fca640b1600 2022-07-12T08:45:11.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca640ad0a0 con 0x7fca640ac7d0 2022-07-12T08:45:11.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca640b0810 con 0x7fca641426f0 2022-07-12T08:45:11.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca640b1600 0x7fca640b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca640b1600 0x7fca640b0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56018/0 (socket says 172.21.15.61:56018) 2022-07-12T08:45:11.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 -- 172.21.15.61:0/649369437 learned_addr learned my addr 172.21.15.61:0/649369437 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:11.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca63fff700 1 --2- 172.21.15.61:0/649369437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca64142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca637fe700 1 --2- 172.21.15.61:0/649369437 >> v2:172.21.15.111:3300/0 conn(0x7fca640ac7d0 0x7fca640acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 -- 172.21.15.61:0/649369437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 msgr2=0x7fca64142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 --2- 172.21.15.61:0/649369437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca64142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 -- 172.21.15.61:0/649369437 >> v2:172.21.15.111:3300/0 conn(0x7fca640ac7d0 msgr2=0x7fca640acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 --2- 172.21.15.61:0/649369437 >> v2:172.21.15.111:3300/0 conn(0x7fca640ac7d0 0x7fca640acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 -- 172.21.15.61:0/649369437 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640af190 con 0x7fca640b1600 2022-07-12T08:45:11.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca62ffd700 1 --2- 172.21.15.61:0/649369437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca640b0320 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7fca54002f70 tx=0x7fca54006eb0).ready entity=mon.0 client_cookie=682623f61f6efa2f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca627fc700 1 -- 172.21.15.61:0/649369437 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5400d570 con 0x7fca640b1600 2022-07-12T08:45:11.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca627fc700 1 -- 172.21.15.61:0/649369437 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54000e90 con 0x7fca640b1600 2022-07-12T08:45:11.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca627fc700 1 -- 172.21.15.61:0/649369437 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca5400d900 con 0x7fca640b1600 2022-07-12T08:45:11.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- 172.21.15.61:0/649369437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 msgr2=0x7fca640b0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- 172.21.15.61:0/649369437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca640b0320 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7fca54002f70 tx=0x7fca54006eb0).stop 2022-07-12T08:45:11.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- 172.21.15.61:0/649369437 shutdown_connections 2022-07-12T08:45:11.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- 172.21.15.61:0/649369437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca64142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- 172.21.15.61:0/649369437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca640b0320 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 --2- 172.21.15.61:0/649369437 >> v2:172.21.15.111:3300/0 conn(0x7fca640ac7d0 0x7fca640acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- 172.21.15.61:0/649369437 >> 172.21.15.61:0/649369437 conn(0x7fca640634d0 msgr2=0x7fca640b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:11.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- 172.21.15.61:0/649369437 shutdown_connections 2022-07-12T08:45:11.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.424+0000 7fca69da5700 1 -- 172.21.15.61:0/649369437 wait complete. 2022-07-12T08:45:11.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 Processor -- start 2022-07-12T08:45:11.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 -- start start 2022-07-12T08:45:11.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca640ac7d0 0x7fca64149400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca64066430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca641554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca64147050 con 0x7fca640b1600 2022-07-12T08:45:11.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641487b0 con 0x7fca640ac7d0 2022-07-12T08:45:11.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca641484f0 con 0x7fca641426f0 2022-07-12T08:45:11.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca64066430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca637fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca640ac7d0 0x7fca64149400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca64066430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56024/0 (socket says 172.21.15.61:56024) 2022-07-12T08:45:11.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 -- 172.21.15.61:0/2912704639 learned_addr learned my addr 172.21.15.61:0/2912704639 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:11.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca63fff700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca641554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 -- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 msgr2=0x7fca641554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca641554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 -- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca640ac7d0 msgr2=0x7fca64149400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca640ac7d0 0x7fca64149400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 -- 172.21.15.61:0/2912704639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5400b1e0 con 0x7fca640b1600 2022-07-12T08:45:11.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca62ffd700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca64066430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fca54002f70 tx=0x7fca54015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca540116e0 con 0x7fca640b1600 2022-07-12T08:45:11.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54011840 con 0x7fca640b1600 2022-07-12T08:45:11.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca64148a50 con 0x7fca640b1600 2022-07-12T08:45:11.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca54011ba0 con 0x7fca640b1600 2022-07-12T08:45:11.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.428+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca640a1110 con 0x7fca640b1600 2022-07-12T08:45:11.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.432+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca64057460 con 0x7fca640b1600 2022-07-12T08:45:11.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.432+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fca5401a040 con 0x7fca640b1600 2022-07-12T08:45:11.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.432+0000 7fca60ff9700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca440539d0 0x7fca44055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.432+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fca5406cb10 con 0x7fca640b1600 2022-07-12T08:45:11.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.432+0000 7fca637fe700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca440539d0 0x7fca44055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.432+0000 7fca637fe700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca440539d0 0x7fca44055e90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fca58006000 tx=0x7fca58006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.436+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca54043a70 con 0x7fca640b1600 2022-07-12T08:45:11.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.564+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fca640b29e0 con 0x7fca640b1600 2022-07-12T08:45:11.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.564+0000 7fca60ff9700 1 -- 172.21.15.61:0/2912704639 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v53) v1 ==== 110+0+10 (secure 0 0 0) 0x7fca54023020 con 0x7fca640b1600 2022-07-12T08:45:11.572 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:45:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca440539d0 msgr2=0x7fca44055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca440539d0 0x7fca44055e90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fca58006000 tx=0x7fca58006d90).stop 2022-07-12T08:45:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 msgr2=0x7fca64066430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca64066430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fca54002f70 tx=0x7fca54015000).stop 2022-07-12T08:45:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 shutdown_connections 2022-07-12T08:45:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca641426f0 0x7fca641554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca640b1600 0x7fca64066430 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca440539d0 0x7fca44055e90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 --2- 172.21.15.61:0/2912704639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca640ac7d0 0x7fca64149400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 >> 172.21.15.61:0/2912704639 conn(0x7fca640634d0 msgr2=0x7fca6413f400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:11.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 shutdown_connections 2022-07-12T08:45:11.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.568+0000 7fca69da5700 1 -- 172.21.15.61:0/2912704639 wait complete. 2022-07-12T08:45:11.587 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:45:11.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270759e700 1 Processor -- start 2022-07-12T08:45:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270759e700 1 -- start start 2022-07-12T08:45:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 0x7f270814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f270805c3e0 con 0x7f27080b5920 2022-07-12T08:45:11.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270759e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27080b5d00 con 0x7f27080aa830 2022-07-12T08:45:11.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270805c610 con 0x7f27080b0280 2022-07-12T08:45:11.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270659c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f27080b5920 0x7f270814a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44246/0 (socket says 172.21.15.61:44246) 2022-07-12T08:45:11.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270659c700 1 -- 172.21.15.61:0/393576080 learned_addr learned my addr 172.21.15.61:0/393576080 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:11.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 0x7f270814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949906400 0 0) 0x7f270805c3e0 con 0x7f27080b5920 2022-07-12T08:45:11.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e8003260 con 0x7f27080b5920 2022-07-12T08:45:11.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2586539799 0 0) 0x7f26e8003260 con 0x7f27080b5920 2022-07-12T08:45:11.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 msgr2=0x7f270814b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 0x7f270814b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 >> v1:172.21.15.111:6789/0 conn(0x7f27080aa830 legacy=0x7f27080aac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270805c8b0 con 0x7f27080b5920 2022-07-12T08:45:11.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f26f8001660 con 0x7f27080b5920 2022-07-12T08:45:11.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 >> v1:172.21.15.61:6789/0 conn(0x7f27080b5920 legacy=0x7f270814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8003780 0x7f26e8003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26e8004500 0x7f26e8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 0x7f26e8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270805c610 con 0x7f26e8004500 2022-07-12T08:45:11.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e800b750 con 0x7f26e8003780 2022-07-12T08:45:11.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e800b8c0 con 0x7f27080b0280 2022-07-12T08:45:11.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26e8004500 0x7f26e8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270659c700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8003780 0x7f26e8003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2706d9d700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 0x7f26e8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 -- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 msgr2=0x7f26e8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f27080b0280 0x7f26e8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 -- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8003780 msgr2=0x7f26e8003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:11.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8003780 0x7f26e8003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 -- 172.21.15.61:0/393576080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8003260 con 0x7f26e8004500 2022-07-12T08:45:11.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f2705d9b700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26e8004500 0x7f26e8004980 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f26f000c000 tx=0x7f26f0006df0).ready entity=mon.0 client_cookie=dc647e4f21b2a40a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.700+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26f000e7d0 con 0x7f26e8004500 2022-07-12T08:45:11.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f000e930 con 0x7f26e8004500 2022-07-12T08:45:11.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270559a700 1 -- 172.21.15.61:0/393576080 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f26f000ec90 con 0x7f26e8004500 2022-07-12T08:45:11.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 -- 172.21.15.61:0/393576080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26e8004500 msgr2=0x7f26e8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 --2- 172.21.15.61:0/393576080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f26e8004500 0x7f26e8004980 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f26f000c000 tx=0x7f26f0006df0).stop 2022-07-12T08:45:11.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f2706d9d700 1 -- 172.21.15.61:0/393576080 reap_dead start 2022-07-12T08:45:11.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 -- 172.21.15.61:0/393576080 shutdown_connections 2022-07-12T08:45:11.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 -- 172.21.15.61:0/393576080 >> 172.21.15.61:0/393576080 conn(0x7f2708063340 msgr2=0x7f27080b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:11.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 -- 172.21.15.61:0/393576080 shutdown_connections 2022-07-12T08:45:11.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 -- 172.21.15.61:0/393576080 wait complete. 2022-07-12T08:45:11.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 Processor -- start 2022-07-12T08:45:11.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.704+0000 7f270759e700 1 -- start start 2022-07-12T08:45:11.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8004500 0x7f27080998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2708065ac0 0x7f27081593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 0x7f2708091130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2708090880 con 0x7f2708090cb0 2022-07-12T08:45:11.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2708091620 con 0x7f26e8004500 2022-07-12T08:45:11.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2708067960 con 0x7f2708065ac0 2022-07-12T08:45:11.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 0x7f2708091130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 0x7f2708091130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56044/0 (socket says 172.21.15.61:56044) 2022-07-12T08:45:11.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 -- 172.21.15.61:0/1691307509 learned_addr learned my addr 172.21.15.61:0/1691307509 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:11.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270659c700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8004500 0x7f27080998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2705d9b700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2708065ac0 0x7f27081593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 -- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2708065ac0 msgr2=0x7f27081593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2708065ac0 0x7f27081593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 -- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8004500 msgr2=0x7f27080998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8004500 0x7f27080998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 -- 172.21.15.61:0/1691307509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2708067c00 con 0x7f2708090cb0 2022-07-12T08:45:11.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f2706d9d700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 0x7f2708091130 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f2700001210 tx=0x7f2700009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2700003220 con 0x7f2708090cb0 2022-07-12T08:45:11.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27080657a0 con 0x7f2708090cb0 2022-07-12T08:45:11.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2700003380 con 0x7f2708090cb0 2022-07-12T08:45:11.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2700001830 con 0x7f2708090cb0 2022-07-12T08:45:11.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2708067590 con 0x7f2708090cb0 2022-07-12T08:45:11.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27080677c0 con 0x7f2708090cb0 2022-07-12T08:45:11.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.708+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f270001c060 con 0x7f2708090cb0 2022-07-12T08:45:11.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.712+0000 7f26ff7fe700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f26ec0539d0 0x7f26ec055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:11.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.712+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f270006b5f0 con 0x7f2708090cb0 2022-07-12T08:45:11.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.712+0000 7f270659c700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f26ec0539d0 0x7f26ec055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:11.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.712+0000 7f270659c700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f26ec0539d0 0x7f26ec055e90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f26f800d000 tx=0x7f26f8007040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:11.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.716+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f27080677c0 con 0x7f2708090cb0 2022-07-12T08:45:11.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.840+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f27080677c0 con 0x7f2708090cb0 2022-07-12T08:45:11.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.840+0000 7f26ff7fe700 1 -- 172.21.15.61:0/1691307509 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v53) v1 ==== 92+0+10 (secure 0 0 0) 0x7f27080677c0 con 0x7f2708090cb0 2022-07-12T08:45:11.856 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:45:11.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.840+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f26ec0539d0 msgr2=0x7f26ec055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.840+0000 7f270759e700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f26ec0539d0 0x7f26ec055e90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f26f800d000 tx=0x7f26f8007040).stop 2022-07-12T08:45:11.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.840+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 msgr2=0x7f2708091130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:11.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.840+0000 7f270759e700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 0x7f2708091130 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f2700001210 tx=0x7f2700009040).stop 2022-07-12T08:45:11.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 shutdown_connections 2022-07-12T08:45:11.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2708090cb0 0x7f2708091130 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2708065ac0 0x7f27081593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f26ec0539d0 0x7f26ec055e90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 --2- 172.21.15.61:0/1691307509 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26e8004500 0x7f27080998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:11.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 >> 172.21.15.61:0/1691307509 conn(0x7f2708063340 msgr2=0x7f27080b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:11.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 shutdown_connections 2022-07-12T08:45:11.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:11.844+0000 7f270759e700 1 -- 172.21.15.61:0/1691307509 wait complete. 2022-07-12T08:45:11.887 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:45:12.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 Processor -- start 2022-07-12T08:45:12.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 -- start start 2022-07-12T08:45:12.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c142eb0 0x7fe84c05c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe84c05d130 0x7fe84c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe84c0b5f10 0x7fe84c146090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c143500 con 0x7fe84c05d130 2022-07-12T08:45:12.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c143290 con 0x7fe84c0b5f10 2022-07-12T08:45:12.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe850a38700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c143760 con 0x7fe84c142eb0 2022-07-12T08:45:12.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe84c05d130 0x7fe84c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe84c05d130 0x7fe84c0b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56050/0 (socket says 172.21.15.61:56050) 2022-07-12T08:45:12.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 -- 172.21.15.61:0/3736008646 learned_addr learned my addr 172.21.15.61:0/3736008646 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:12.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84ad9d700 1 --2- 172.21.15.61:0/3736008646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c142eb0 0x7fe84c05c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84b59e700 1 --2- 172.21.15.61:0/3736008646 >> v2:172.21.15.111:3300/0 conn(0x7fe84c0b5f10 0x7fe84c146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 -- 172.21.15.61:0/3736008646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c142eb0 msgr2=0x7fe84c05c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:12.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 --2- 172.21.15.61:0/3736008646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c142eb0 0x7fe84c05c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 -- 172.21.15.61:0/3736008646 >> v2:172.21.15.111:3300/0 conn(0x7fe84c0b5f10 msgr2=0x7fe84c146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:12.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 --2- 172.21.15.61:0/3736008646 >> v2:172.21.15.111:3300/0 conn(0x7fe84c0b5f10 0x7fe84c146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 -- 172.21.15.61:0/3736008646 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0b4030 con 0x7fe84c05d130 2022-07-12T08:45:12.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe84a59c700 1 --2- 172.21.15.61:0/3736008646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c05d130 0x7fe84c0b5a20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fe840002f70 tx=0x7fe840006eb0).ready entity=mon.0 client_cookie=5ffa93e8782023c1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe849d9b700 1 -- 172.21.15.61:0/3736008646 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84000d570 con 0x7fe84c05d130 2022-07-12T08:45:12.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe849d9b700 1 -- 172.21.15.61:0/3736008646 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe840000e90 con 0x7fe84c05d130 2022-07-12T08:45:12.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.004+0000 7fe849d9b700 1 -- 172.21.15.61:0/3736008646 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84000d900 con 0x7fe84c05d130 2022-07-12T08:45:12.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 -- 172.21.15.61:0/3736008646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c05d130 msgr2=0x7fe84c0b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 --2- 172.21.15.61:0/3736008646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c05d130 0x7fe84c0b5a20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fe840002f70 tx=0x7fe840006eb0).stop 2022-07-12T08:45:12.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 -- 172.21.15.61:0/3736008646 shutdown_connections 2022-07-12T08:45:12.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 --2- 172.21.15.61:0/3736008646 >> v2:172.21.15.111:3300/0 conn(0x7fe84c0b5f10 0x7fe84c146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 --2- 172.21.15.61:0/3736008646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c05d130 0x7fe84c0b5a20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 --2- 172.21.15.61:0/3736008646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c142eb0 0x7fe84c05c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 -- 172.21.15.61:0/3736008646 >> 172.21.15.61:0/3736008646 conn(0x7fe84c066d70 msgr2=0x7fe84c067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:12.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 -- 172.21.15.61:0/3736008646 shutdown_connections 2022-07-12T08:45:12.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 -- 172.21.15.61:0/3736008646 wait complete. 2022-07-12T08:45:12.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 Processor -- start 2022-07-12T08:45:12.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.008+0000 7fe850a38700 1 -- start start 2022-07-12T08:45:12.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c05d130 0x7fe84c13ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe84c0b5f10 0x7fe84c144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 0x7fe84c151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c13f990 con 0x7fe84c142eb0 2022-07-12T08:45:12.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c146e00 con 0x7fe84c0b5f10 2022-07-12T08:45:12.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c13e6e0 con 0x7fe84c05d130 2022-07-12T08:45:12.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 0x7fe84c151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 0x7fe84c151970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56058/0 (socket says 172.21.15.61:56058) 2022-07-12T08:45:12.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 -- 172.21.15.61:0/752005842 learned_addr learned my addr 172.21.15.61:0/752005842 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:12.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84ad9d700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c05d130 0x7fe84c13ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84a59c700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe84c0b5f10 0x7fe84c144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 -- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c05d130 msgr2=0x7fe84c13ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c05d130 0x7fe84c13ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 -- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe84c0b5f10 msgr2=0x7fe84c144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe84c0b5f10 0x7fe84c144670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 -- 172.21.15.61:0/752005842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84000b1e0 con 0x7fe84c142eb0 2022-07-12T08:45:12.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe84b59e700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 0x7fe84c151970 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fe844006e00 tx=0x7fe8440091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84400eeb0 con 0x7fe84c142eb0 2022-07-12T08:45:12.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84400c850 con 0x7fe84c142eb0 2022-07-12T08:45:12.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c143f90 con 0x7fe84c142eb0 2022-07-12T08:45:12.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84400cbb0 con 0x7fe84c142eb0 2022-07-12T08:45:12.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.012+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c144d00 con 0x7fe84c142eb0 2022-07-12T08:45:12.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.016+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c146fc0 con 0x7fe84c142eb0 2022-07-12T08:45:12.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.016+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe844022070 con 0x7fe84c142eb0 2022-07-12T08:45:12.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.016+0000 7fe83bfff700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe828053a40 0x7fe828055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.016+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe84406c5a0 con 0x7fe84c142eb0 2022-07-12T08:45:12.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.016+0000 7fe84ad9d700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe828053a40 0x7fe828055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.016+0000 7fe84ad9d700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe828053a40 0x7fe828055f00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fe83c00b000 tx=0x7fe83c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.024+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe844043460 con 0x7fe84c142eb0 2022-07-12T08:45:12.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.152+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe84c057460 con 0x7fe84c142eb0 2022-07-12T08:45:12.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.152+0000 7fe83bfff700 1 -- 172.21.15.61:0/752005842 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe844046a80 con 0x7fe84c142eb0 2022-07-12T08:45:12.157 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:45:12.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe828053a40 msgr2=0x7fe828055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe828053a40 0x7fe828055f00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fe83c00b000 tx=0x7fe83c006df0).stop 2022-07-12T08:45:12.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 msgr2=0x7fe84c151970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 0x7fe84c151970 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fe844006e00 tx=0x7fe8440091d0).stop 2022-07-12T08:45:12.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 shutdown_connections 2022-07-12T08:45:12.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c142eb0 0x7fe84c151970 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe828053a40 0x7fe828055f00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe84c0b5f10 0x7fe84c144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 --2- 172.21.15.61:0/752005842 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe84c05d130 0x7fe84c13ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 >> 172.21.15.61:0/752005842 conn(0x7fe84c066d70 msgr2=0x7fe84c0b9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:12.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 shutdown_connections 2022-07-12T08:45:12.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.156+0000 7fe850a38700 1 -- 172.21.15.61:0/752005842 wait complete. 2022-07-12T08:45:12.171 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:45:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 Processor -- start 2022-07-12T08:45:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 -- start start 2022-07-12T08:45:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa14c13fba0 0x7fa14c13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa14c0b6710 con 0x7fa14c065990 2022-07-12T08:45:12.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa14c0655a0 con 0x7fa14c13fba0 2022-07-12T08:45:12.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151b38700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c1431d0 con 0x7fa14c0b6330 2022-07-12T08:45:12.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa14bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa14c065990 0x7fa14c0b1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44280/0 (socket says 172.21.15.61:44280) 2022-07-12T08:45:12.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa14bfff700 1 -- 172.21.15.61:0/3076232797 learned_addr learned my addr 172.21.15.61:0/3076232797 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:12.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151337700 1 --2- 172.21.15.61:0/3076232797 >> v2:172.21.15.111:3300/0 conn(0x7fa14c13fba0 0x7fa14c13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa150b36700 1 --2- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa14b7fe700 1 -- 172.21.15.61:0/3076232797 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058781779 0 0) 0x7fa14c0b6710 con 0x7fa14c065990 2022-07-12T08:45:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa14b7fe700 1 -- 172.21.15.61:0/3076232797 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa12c003260 con 0x7fa14c065990 2022-07-12T08:45:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151337700 1 -- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 msgr2=0x7fa14c0b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:45:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151337700 1 --2- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151337700 1 -- 172.21.15.61:0/3076232797 >> v1:172.21.15.61:6789/0 conn(0x7fa14c065990 legacy=0x7fa14c0b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151337700 1 -- 172.21.15.61:0/3076232797 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c143470 con 0x7fa14c13fba0 2022-07-12T08:45:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.280+0000 7fa151337700 1 --2- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c13fba0 0x7fa14c13fff0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fa144002f70 tx=0x7fa144006df0).ready entity=mon.1 client_cookie=f344d6e08bdeca10 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa14b7fe700 1 -- 172.21.15.61:0/3076232797 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa144014070 con 0x7fa14c13fba0 2022-07-12T08:45:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa14b7fe700 1 -- 172.21.15.61:0/3076232797 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa144000e90 con 0x7fa14c13fba0 2022-07-12T08:45:12.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa14b7fe700 1 -- 172.21.15.61:0/3076232797 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14400fa90 con 0x7fa14c13fba0 2022-07-12T08:45:12.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 -- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c13fba0 msgr2=0x7fa14c13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 --2- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c13fba0 0x7fa14c13fff0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fa144002f70 tx=0x7fa144006df0).stop 2022-07-12T08:45:12.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 -- 172.21.15.61:0/3076232797 shutdown_connections 2022-07-12T08:45:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 --2- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c13fba0 0x7fa14c13fff0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 --2- 172.21.15.61:0/3076232797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 -- 172.21.15.61:0/3076232797 >> 172.21.15.61:0/3076232797 conn(0x7fa14c063ad0 msgr2=0x7fa14c063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 -- 172.21.15.61:0/3076232797 shutdown_connections 2022-07-12T08:45:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 -- 172.21.15.61:0/3076232797 wait complete. 2022-07-12T08:45:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.284+0000 7fa151b38700 1 Processor -- start 2022-07-12T08:45:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 -- start start 2022-07-12T08:45:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c065990 0x7fa14c0ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 0x7fa14c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c140ea0 con 0x7fa14c13fba0 2022-07-12T08:45:12.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c151520 con 0x7fa14c065990 2022-07-12T08:45:12.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa14c151750 con 0x7fa14c0b6330 2022-07-12T08:45:12.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 0x7fa14c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 0x7fa14c0b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56072/0 (socket says 172.21.15.61:56072) 2022-07-12T08:45:12.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 -- 172.21.15.61:0/3786725383 learned_addr learned my addr 172.21.15.61:0/3786725383 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:12.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa14bfff700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa150b36700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c065990 0x7fa14c0ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 -- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 msgr2=0x7fa14c0b3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 -- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c065990 msgr2=0x7fa14c0ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c065990 0x7fa14c0ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 -- 172.21.15.61:0/3786725383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14400b180 con 0x7fa14c13fba0 2022-07-12T08:45:12.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151337700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 0x7fa14c0b43b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fa144002f70 tx=0x7fa144016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa144010de0 con 0x7fa14c13fba0 2022-07-12T08:45:12.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14401d450 con 0x7fa14c13fba0 2022-07-12T08:45:12.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa14c0aebe0 con 0x7fa14c13fba0 2022-07-12T08:45:12.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa14401d7b0 con 0x7fa14c13fba0 2022-07-12T08:45:12.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.288+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa14c0b5c90 con 0x7fa14c13fba0 2022-07-12T08:45:12.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.292+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa14c0aea20 con 0x7fa14c13fba0 2022-07-12T08:45:12.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.292+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa14401d910 con 0x7fa14c13fba0 2022-07-12T08:45:12.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.292+0000 7fa149ffb700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa130053ad0 0x7fa130055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.292+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa14406fd40 con 0x7fa14c13fba0 2022-07-12T08:45:12.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.292+0000 7fa150b36700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa130053ad0 0x7fa130055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.292+0000 7fa150b36700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa130053ad0 0x7fa130055f90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fa13c00b000 tx=0x7fa13c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.296+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa144046ca0 con 0x7fa14c13fba0 2022-07-12T08:45:12.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.424+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa14c151910 con 0x7fa14c13fba0 2022-07-12T08:45:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa149ffb700 1 -- 172.21.15.61:0/3786725383 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa14404a2c0 con 0x7fa14c13fba0 2022-07-12T08:45:12.434 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:45:12.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa130053ad0 msgr2=0x7fa130055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa130053ad0 0x7fa130055f90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fa13c00b000 tx=0x7fa13c0091d0).stop 2022-07-12T08:45:12.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 msgr2=0x7fa14c0b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 0x7fa14c0b43b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fa144002f70 tx=0x7fa144016000).stop 2022-07-12T08:45:12.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 shutdown_connections 2022-07-12T08:45:12.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa14c13fba0 0x7fa14c0b43b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa14c0b6330 0x7fa14c0b3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa130053ad0 0x7fa130055f90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 --2- 172.21.15.61:0/3786725383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa14c065990 0x7fa14c0ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 >> 172.21.15.61:0/3786725383 conn(0x7fa14c063ad0 msgr2=0x7fa14c13ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 shutdown_connections 2022-07-12T08:45:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.428+0000 7fa151b38700 1 -- 172.21.15.61:0/3786725383 wait complete. 2022-07-12T08:45:12.457 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:45:12.458 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:45:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1bfff700 1 Processor -- start 2022-07-12T08:45:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1bfff700 1 -- start start 2022-07-12T08:45:12.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c0abc70 con 0x7eff1c0afa60 2022-07-12T08:45:12.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff1c05c720 con 0x7eff1c05f140 2022-07-12T08:45:12.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c05f580 con 0x7eff1c0ab290 2022-07-12T08:45:12.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7eff1c0afa60 0x7eff1c147530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44294/0 (socket says 172.21.15.61:44294) 2022-07-12T08:45:12.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1b7fe700 1 -- 172.21.15.61:0/4076058628 learned_addr learned my addr 172.21.15.61:0/4076058628 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:45:12.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff1affd700 1 --2- 172.21.15.61:0/4076058628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff19ffb700 1 -- 172.21.15.61:0/4076058628 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125752156 0 0) 0x7eff1c0abc70 con 0x7eff1c0afa60 2022-07-12T08:45:12.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.560+0000 7eff19ffb700 1 -- 172.21.15.61:0/4076058628 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff00003260 con 0x7eff1c0afa60 2022-07-12T08:45:12.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1affd700 1 -- 172.21.15.61:0/4076058628 >> v1:172.21.15.111:6789/0 conn(0x7eff1c05f140 legacy=0x7eff1c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff19ffb700 1 -- 172.21.15.61:0/4076058628 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311110279 0 0) 0x7eff1c05c720 con 0x7eff1c05f140 2022-07-12T08:45:12.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1affd700 1 -- 172.21.15.61:0/4076058628 >> v1:172.21.15.61:6789/0 conn(0x7eff1c0afa60 legacy=0x7eff1c147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1affd700 1 -- 172.21.15.61:0/4076058628 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1c05c9c0 con 0x7eff1c0ab290 2022-07-12T08:45:12.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1affd700 1 --2- 172.21.15.61:0/4076058628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c0ab670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7eff140032f0 tx=0x7eff140091d0).ready entity=mon.2 client_cookie=a783e06467f50392 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff19ffb700 1 -- 172.21.15.61:0/4076058628 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff1400b6c0 con 0x7eff1c0ab290 2022-07-12T08:45:12.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff19ffb700 1 -- 172.21.15.61:0/4076058628 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1400b820 con 0x7eff1c0ab290 2022-07-12T08:45:12.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff19ffb700 1 -- 172.21.15.61:0/4076058628 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff1400b990 con 0x7eff1c0ab290 2022-07-12T08:45:12.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 -- 172.21.15.61:0/4076058628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 msgr2=0x7eff1c0ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 --2- 172.21.15.61:0/4076058628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c0ab670 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7eff140032f0 tx=0x7eff140091d0).stop 2022-07-12T08:45:12.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 -- 172.21.15.61:0/4076058628 shutdown_connections 2022-07-12T08:45:12.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 --2- 172.21.15.61:0/4076058628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c0ab670 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 -- 172.21.15.61:0/4076058628 >> 172.21.15.61:0/4076058628 conn(0x7eff1c0b1950 msgr2=0x7eff1c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:12.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 -- 172.21.15.61:0/4076058628 shutdown_connections 2022-07-12T08:45:12.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 -- 172.21.15.61:0/4076058628 wait complete. 2022-07-12T08:45:12.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 Processor -- start 2022-07-12T08:45:12.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.564+0000 7eff1bfff700 1 -- start start 2022-07-12T08:45:12.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff1c05f140 0x7eff1c064090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff1c0afa60 0x7eff1c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c1490b0 con 0x7eff1c05f140 2022-07-12T08:45:12.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c149350 con 0x7eff1c0afa60 2022-07-12T08:45:12.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1c1495f0 con 0x7eff1c0ab290 2022-07-12T08:45:12.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff1c05f140 0x7eff1c064090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c1597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38278/0 (socket says 172.21.15.61:38278) 2022-07-12T08:45:12.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff1c0afa60 0x7eff1c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 -- 172.21.15.61:0/3237243219 learned_addr learned my addr 172.21.15.61:0/3237243219 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:45:12.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff1c05f140 0x7eff1c064090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56080/0 (socket says 172.21.15.61:56080) 2022-07-12T08:45:12.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 -- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff1c0afa60 msgr2=0x7eff1c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff1c0afa60 0x7eff1c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 -- 172.21.15.61:0/3237243219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff1c05f140 msgr2=0x7eff1c064090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff1c05f140 0x7eff1c064090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 -- 172.21.15.61:0/3237243219 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14009470 con 0x7eff1c0ab290 2022-07-12T08:45:12.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1a7fc700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c1597a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7eff04000bc0 tx=0x7eff04007af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff04012400 con 0x7eff1c0ab290 2022-07-12T08:45:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff04012560 con 0x7eff1c0ab290 2022-07-12T08:45:12.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1c149890 con 0x7eff1c0ab290 2022-07-12T08:45:12.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff040128c0 con 0x7eff1c0ab290 2022-07-12T08:45:12.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff1c15a320 con 0x7eff1c0ab290 2022-07-12T08:45:12.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.568+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1c149a20 con 0x7eff1c0ab290 2022-07-12T08:45:12.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.572+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7eff04019030 con 0x7eff1c0ab290 2022-07-12T08:45:12.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.572+0000 7eff13fff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efef4053ad0 0x7efef4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:45:12.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.572+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7eff0406b180 con 0x7eff1c0ab290 2022-07-12T08:45:12.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.572+0000 7eff1affd700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efef4053ad0 0x7efef4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:45:12.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.572+0000 7eff1affd700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efef4053ad0 0x7efef4055f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7eff14019000 tx=0x7eff14016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:45:12.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.580+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7eff04042190 con 0x7eff1c0ab290 2022-07-12T08:45:12.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.744+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff1c057460 con 0x7eff1c0ab290 2022-07-12T08:45:12.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.744+0000 7eff13fff700 1 -- 172.21.15.61:0/3237243219 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7eff040457b0 con 0x7eff1c0ab290 2022-07-12T08:45:12.757 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:45:12.757 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:45:12.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efef4053ad0 msgr2=0x7efef4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efef4053ad0 0x7efef4055f90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7eff14019000 tx=0x7eff14016000).stop 2022-07-12T08:45:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 msgr2=0x7eff1c1597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:45:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c1597a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7eff04000bc0 tx=0x7eff04007af0).stop 2022-07-12T08:45:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 shutdown_connections 2022-07-12T08:45:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efef4053ad0 0x7efef4055f90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7eff1c0afa60 0x7eff1c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7eff1c0ab290 0x7eff1c1597a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 --2- 172.21.15.61:0/3237243219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff1c05f140 0x7eff1c064090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:45:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 >> 172.21.15.61:0/3237243219 conn(0x7eff1c0b1950 msgr2=0x7eff1c0b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:45:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 shutdown_connections 2022-07-12T08:45:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:45:12.748+0000 7eff1bfff700 1 -- 172.21.15.61:0/3237243219 wait complete. 2022-07-12T08:45:12.765 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:45:12.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:44:48.629894+0000', 'modified': '2022-07-12T08:44:50.312792+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5595}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5595': {'gid': 5595, 'name': 'a', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/4155742423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 4155742423}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 4155742423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:45:12.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:45:12.971 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:45:13.089 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5595,"want_state":"up:active","state":"up:active","rank_uptime":24.448022985000001,"mdsmap_epoch":24,"osdmap_epoch":53,"osdmap_epoch_barrier":52,"uptime":29.006525320000001} 2022-07-12T08:45:13.090 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5595, "mdsmap_epoch": 24, "osdmap_epoch": 53, "osdmap_epoch_barrier": 52, "rank_uptime": 24.448022985, "state": "up:active", "uptime": 29.00652532, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:45:13.090 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:45:13.091 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:45:13.092 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:45:13.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.094 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:13.108 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:45:13.134 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:45:13.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:45:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:45:13.202 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:45:13.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.203 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:45:13.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:45:13.215 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:13.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:45:13.237 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:45:13.238 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:45:13.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:45:13.294+0000 7f45afa46080 -1 init, newargv = 0x5614133fb390 newargc=9 2022-07-12T08:45:13.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21418]: starting ceph client 2022-07-12T08:45:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:45:13.303 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:45:13.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.304 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:45:13.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21418]: starting fuse 2022-07-12T08:45:13.314 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:45:13.315 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:45:13.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:45:13.316 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:45:13.317 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:45:13.317 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:45:13.318 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:45:13.318 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:45:13.319 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:45:13.319 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:45:13.319 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:45:13.320 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:45:13.320 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:45:13.321 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:45:13.321 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:45:13.322 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:45:13.322 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:45:13.322 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:45:13.323 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:45:13.323 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:45:13.324 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:45:13.324 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:45:13.325 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:45:13.326 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:45:13.326 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:45:13.399 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21418.asok 2022-07-12T08:45:13.405 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21418.asok 2022-07-12T08:45:13.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.407 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21418.asok status 2022-07-12T08:45:13.504 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:45:13.504 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:45:13.505 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:45:13.505 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:45:13.505 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:45:13.505 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:45:13.506 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:45:13.506 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21418", 2022-07-12T08:45:13.506 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:45:13.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:45:13.507 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:45:13.507 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:45:13.507 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 5856, 2022-07-12T08:45:13.507 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:45:13.508 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:45:13.508 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:45:13.508 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 5856 2022-07-12T08:45:13.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:45:13.509 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:45:13.509 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:45:13.510 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:45:13.510 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 364245264 2022-07-12T08:45:13.510 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:45:13.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:45:13.511 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:45:13.511 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:45:13.511 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:45:13.511 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 364245264 2022-07-12T08:45:13.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:45:13.512 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.5856 172.21.15.121:0/364245264", 2022-07-12T08:45:13.512 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/364245264", 2022-07-12T08:45:13.513 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:45:13.513 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 24, 2022-07-12T08:45:13.513 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 53, 2022-07-12T08:45:13.513 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:45:13.513 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:45:13.514 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:45:13.514 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:45:13.516 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.516 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:45:13.517 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:13.526 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:45:13.531 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:45:13.532 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:45:13.533 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:45:32.660 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:32.707 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:32.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:32.720 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:45:32.727 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:02.820 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:02.831 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:02.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:02.841 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:02.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:09.080 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:46:09.081 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:09.102 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:09.125 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:09.149 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:09.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:09.199 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:46:09.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:09.200 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:09.200 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:09.219 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:46:09.220 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:46:09.221 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:46:09.222 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:46:09.223 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:09.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21418]: fuse finished with error 0 and tester_r 0 2022-07-12T08:46:09.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:09.296 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:09.296 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:09.304 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:46:09.305 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:46:09.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:46:09.367 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:09.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:09.377 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:09.377 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:46:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:09.388 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:46:09.389 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:46:09.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:46:09.391 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:46:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:09.440 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:46:09.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:09.441 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:09.441 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:46:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:09.452 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:46:09.453 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:46:09.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:46:09.455 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:46:09.463 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:46:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:09.465 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:46:09.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.576+0000 7f75ad4fb700 1 Processor -- start 2022-07-12T08:46:09.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.576+0000 7f75ad4fb700 1 -- start start 2022-07-12T08:46:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f75a80b5780 0x7f75a8146540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75a80abdf0 0x7f75a8146a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a80ab3f0 con 0x7f75a805c730 2022-07-12T08:46:09.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a80ab950 con 0x7f75a80b5780 2022-07-12T08:46:09.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a80ab690 con 0x7f75a80abdf0 2022-07-12T08:46:09.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f75a805c730 0x7f75a805faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44308/0 (socket says 172.21.15.61:44308) 2022-07-12T08:46:09.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 -- 172.21.15.61:0/975141573 learned_addr learned my addr 172.21.15.61:0/975141573 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:09.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a7fff700 1 --2- 172.21.15.61:0/975141573 >> v2:172.21.15.61:3300/0 conn(0x7f75a80b5780 0x7f75a8146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379041558 0 0) 0x7f75a80ab3f0 con 0x7f75a805c730 2022-07-12T08:46:09.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a77fe700 1 --2- 172.21.15.61:0/975141573 >> v2:172.21.15.111:3300/0 conn(0x7f75a80abdf0 0x7f75a8146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7594003260 con 0x7f75a805c730 2022-07-12T08:46:09.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372882866 0 0) 0x7f7594003260 con 0x7f75a805c730 2022-07-12T08:46:09.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 >> v2:172.21.15.111:3300/0 conn(0x7f75a80abdf0 msgr2=0x7f75a8146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:09.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 --2- 172.21.15.61:0/975141573 >> v2:172.21.15.111:3300/0 conn(0x7f75a80abdf0 0x7f75a8146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 >> v2:172.21.15.61:3300/0 conn(0x7f75a80b5780 msgr2=0x7f75a8146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:09.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 --2- 172.21.15.61:0/975141573 >> v2:172.21.15.61:3300/0 conn(0x7f75a80b5780 0x7f75a8146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a80b5b60 con 0x7f75a805c730 2022-07-12T08:46:09.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f75a00019e0 con 0x7f75a805c730 2022-07-12T08:46:09.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 >> v1:172.21.15.61:6789/0 conn(0x7f75a805c730 legacy=0x7f75a805faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:09.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7594003750 0x7f7594003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004500 0x7f7594004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7594004e70 0x7f759400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80ab950 con 0x7f7594004e70 2022-07-12T08:46:09.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80ab690 con 0x7f7594004500 2022-07-12T08:46:09.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759400bcc0 con 0x7f7594003750 2022-07-12T08:46:09.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a7fff700 1 --2- 172.21.15.61:0/975141573 >> v2:172.21.15.61:3300/0 conn(0x7f75a80b5780 0x7f75a8146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:09.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7594004e70 0x7f759400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a77fe700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004500 0x7f7594004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a7fff700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7594003750 0x7f7594003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 -- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7594003750 msgr2=0x7f7594003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:09.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7594003750 0x7f7594003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 -- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004500 msgr2=0x7f7594004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:09.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004500 0x7f7594004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 -- 172.21.15.61:0/975141573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7594003260 con 0x7f7594004e70 2022-07-12T08:46:09.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a7fff700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7594003750 0x7f7594003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:46:09.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 -- 172.21.15.61:0/975141573 reap_dead start 2022-07-12T08:46:09.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75accfa700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7594004e70 0x7f759400b750 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f75a80b18b0 tx=0x7f75a0002190).ready entity=mon.0 client_cookie=e2cb6cc42779776d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a000cd10 con 0x7f7594004e70 2022-07-12T08:46:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a000ce70 con 0x7f7594004e70 2022-07-12T08:46:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75a6ffd700 1 -- 172.21.15.61:0/975141573 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75a0005b80 con 0x7f7594004e70 2022-07-12T08:46:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- 172.21.15.61:0/975141573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7594004e70 msgr2=0x7f759400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7594004e70 0x7f759400b750 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f75a80b18b0 tx=0x7f75a0002190).stop 2022-07-12T08:46:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- 172.21.15.61:0/975141573 shutdown_connections 2022-07-12T08:46:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/975141573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7594004e70 0x7f759400b750 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- 172.21.15.61:0/975141573 >> 172.21.15.61:0/975141573 conn(0x7f75a8063380 msgr2=0x7f75a80b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- 172.21.15.61:0/975141573 shutdown_connections 2022-07-12T08:46:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.580+0000 7f75ad4fb700 1 -- 172.21.15.61:0/975141573 wait complete. 2022-07-12T08:46:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.584+0000 7f75ad4fb700 1 Processor -- start 2022-07-12T08:46:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.584+0000 7f75ad4fb700 1 -- start start 2022-07-12T08:46:09.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004e70 0x7f75a8099e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 0x7f75a80671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75a80ae860 0x7f75a80676d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a8064980 con 0x7f75a8063a90 2022-07-12T08:46:09.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a80aef80 con 0x7f7594004e70 2022-07-12T08:46:09.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a8064bb0 con 0x7f75a80ae860 2022-07-12T08:46:09.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 0x7f75a80671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 0x7f75a80671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56102/0 (socket says 172.21.15.61:56102) 2022-07-12T08:46:09.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 -- 172.21.15.61:0/4072452111 learned_addr learned my addr 172.21.15.61:0/4072452111 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:09.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a7fff700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004e70 0x7f75a8099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75accfa700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75a80ae860 0x7f75a80676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 -- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75a80ae860 msgr2=0x7f75a80676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:09.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75a80ae860 0x7f75a80676d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 -- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004e70 msgr2=0x7f75a8099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:09.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004e70 0x7f75a8099e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:09.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 -- 172.21.15.61:0/4072452111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a80af220 con 0x7f75a8063a90 2022-07-12T08:46:09.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a77fe700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 0x7f75a80671e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f759800c000 tx=0x7f7598008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:09.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7598017070 con 0x7f75a8063a90 2022-07-12T08:46:09.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598000e90 con 0x7f75a8063a90 2022-07-12T08:46:09.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a8155d30 con 0x7f75a8063a90 2022-07-12T08:46:09.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75980137e0 con 0x7f75a8063a90 2022-07-12T08:46:09.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a8067d60 con 0x7f75a8063a90 2022-07-12T08:46:09.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a8159300 con 0x7f75a8063a90 2022-07-12T08:46:09.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.588+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7598013940 con 0x7f75a8063a90 2022-07-12T08:46:09.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.592+0000 7f75a57fa700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7588053a40 0x7f7588055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:09.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.592+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f759806bba0 con 0x7f75a8063a90 2022-07-12T08:46:09.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.592+0000 7f75a7fff700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7588053a40 0x7f7588055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:09.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.592+0000 7f75a7fff700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7588053a40 0x7f7588055f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f759c00b000 tx=0x7f759c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:09.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.600+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7598042b00 con 0x7f75a8063a90 2022-07-12T08:46:09.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.724+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f75a806b2b0 con 0x7f75a8063a90 2022-07-12T08:46:09.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75a57fa700 1 -- 172.21.15.61:0/4072452111 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v53) v1 ==== 81+0+33695 (secure 0 0 0) 0x7f7598046120 con 0x7f75a8063a90 2022-07-12T08:46:09.730 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:09.730 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:46:09.731 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 53, 2022-07-12T08:46:09.731 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:46:09.731 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:46:09.731 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:44:50.635406+0000", 2022-07-12T08:46:09.732 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:09.732 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:09.732 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:46:09.732 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:46:09.733 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:46:09.733 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:46:09.733 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:46:09.733 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:46:09.734 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:46:09.734 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:09.734 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:46:09.734 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:46:09.735 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:46:09.735 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:46:09.735 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:46:09.735 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 10, 2022-07-12T08:46:09.736 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:46:09.736 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:46:09.736 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:46:09.736 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:46:09.737 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:46:09.737 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.738 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:46:09.738 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:46:09.738 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:46:09.738 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:09.739 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:09.739 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:09.739 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:09.740 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:09.740 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:09.740 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:09.741 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:09.741 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:46:09.742 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:46:09.742 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:46:09.743 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:46:09.743 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:46:09.744 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:09.744 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:09.744 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:09.745 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:09.745 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:09.746 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:09.746 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:09.747 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.747 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:46:09.748 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:09.748 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:09.749 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:09.749 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:09.750 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:09.750 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:09.751 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:09.751 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:09.751 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:09.752 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:09.752 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:09.753 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:09.753 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:09.753 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:09.754 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:09.754 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:09.755 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:09.755 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:09.756 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:09.775 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:09.776 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:09.776 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:09.777 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:09.777 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:09.777 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:09.778 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:09.778 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.779 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:09.779 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:09.779 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:09.780 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:09.780 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:09.780 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:09.781 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:09.781 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:09.781 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:09.781 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:09.782 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:09.782 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:46:09.782 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:46:09.783 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.784 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:09.785 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:46:09.785 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.786 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.787 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:46:09.787 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:46:09.788 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:46:09.788 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:09.789 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:09.789 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:09.790 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:09.790 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:09.791 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:09.791 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:09.792 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:09.792 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:09.793 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:09.793 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:09.794 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:09.794 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:09.794 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:09.796 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:09.796 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:09.797 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:09.797 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:09.797 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:09.798 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:09.798 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.799 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:46:09.799 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:09.800 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:09.800 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:09.800 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:09.801 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:09.801 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:09.802 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:09.802 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:09.803 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:09.803 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:09.804 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:09.804 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:09.805 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:09.805 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:09.806 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:09.806 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:09.806 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:09.807 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:09.807 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:09.807 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:09.808 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:09.808 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:09.809 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:09.809 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:09.810 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:09.810 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:09.811 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.811 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:09.812 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:09.812 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:09.812 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:09.813 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:09.813 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:09.814 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:09.814 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:09.815 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:09.815 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:09.816 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:09.816 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:46:09.817 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:09.817 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:46:09.817 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.818 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.818 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.819 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 9, 2022-07-12T08:46:09.819 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:46:09.819 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:44:47.163319+0000", 2022-07-12T08:46:09.820 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:09.820 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:09.821 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:09.822 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:09.822 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:09.822 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:09.823 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:09.823 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:09.823 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:09.824 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:09.824 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:09.825 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:09.825 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:09.826 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:09.826 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:09.826 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:09.827 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:09.827 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:09.828 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:09.828 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:09.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.829 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "52", 2022-07-12T08:46:09.829 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:09.831 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:09.832 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:09.832 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:09.833 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:09.833 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:09.833 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:09.834 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:09.835 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:09.835 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:09.836 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:09.836 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:09.836 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:09.837 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:09.837 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:09.839 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:09.839 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:09.839 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:09.840 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:09.840 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:09.840 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:09.840 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:09.841 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:09.841 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:09.841 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:09.842 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:09.842 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.842 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:09.843 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:09.843 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:09.844 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:09.844 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:09.844 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:09.845 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:09.845 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:09.846 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:09.847 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:09.848 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:09.848 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:46:09.848 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:46:09.849 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:46:09.849 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:46:09.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.850 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:09.850 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:46:09.851 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:46:09.851 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.851 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.852 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.852 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 10, 2022-07-12T08:46:09.852 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:46:09.852 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:44:47.639688+0000", 2022-07-12T08:46:09.853 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:09.853 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:09.853 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:09.853 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:09.853 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:09.854 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:09.854 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:09.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:09.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:09.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:09.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:09.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:09.856 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:09.856 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:09.856 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:09.857 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:09.857 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:09.857 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:09.857 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:09.857 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:09.858 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.858 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "53", 2022-07-12T08:46:09.858 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:09.858 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:09.859 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:09.859 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:09.859 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:09.860 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:09.860 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:09.861 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:09.861 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:09.861 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:09.862 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:09.862 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:09.863 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:09.863 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:09.864 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:09.864 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:09.865 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:09.865 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:09.866 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:09.866 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:09.867 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:09.867 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:09.868 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:09.868 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:09.869 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:09.869 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:09.869 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.871 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:09.871 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:09.872 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:09.872 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:09.873 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:09.873 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:09.874 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:09.874 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:09.875 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:09.875 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:09.875 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:09.876 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:46:09.876 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:09.876 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:46:09.877 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:46:09.877 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.878 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.878 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.879 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:09.879 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:46:09.879 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.880 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:09.881 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:46:09.881 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:09.881 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:09.882 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:09.882 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:09.883 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:09.883 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:09.884 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:09.884 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:09.884 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:09.885 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:09.885 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:09.886 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.886 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.887 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.887 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:46:09.887 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.888 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.888 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.889 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.889 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:46:09.890 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.890 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.890 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.891 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.891 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:09.892 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.893 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.893 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:46:09.893 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.894 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.894 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.894 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.894 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:46:09.895 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.895 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.895 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.895 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.896 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:09.896 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.896 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.896 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:46:09.897 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.897 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.897 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.897 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.898 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:46:09.898 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.898 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.898 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.899 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:46:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.900 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:46:09.901 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.902 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:46:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:46:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:46:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:46:09.903 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:09.904 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:09.904 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:09.904 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.904 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.904 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.905 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:09.905 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:46:09.906 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:09.906 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:09.906 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:09.906 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:09.907 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:09.907 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:09.908 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:09.908 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:09.908 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:09.908 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:09.909 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:09.909 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.909 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.909 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:46:09.910 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.910 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.910 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.911 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.911 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:46:09.911 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.911 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.912 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.913 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.913 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:09.914 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.914 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.915 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.915 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:46:09.915 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.916 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.916 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.917 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.917 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:46:09.918 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.918 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.919 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.920 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:09.920 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.921 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.921 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.922 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:46:09.922 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.923 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.923 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.924 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:46:09.924 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.925 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.925 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.926 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.926 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:09.927 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.927 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.927 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.928 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:46:09.928 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.929 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.929 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.930 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.930 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:46:09.931 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:09.931 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.932 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.932 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.932 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:46:09.933 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:46:09.933 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:46:09.934 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:46:09.935 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:09.935 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:09.935 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:09.936 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.936 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.937 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.937 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:09.938 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:46:09.938 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:09.938 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:09.939 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:09.939 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:09.940 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:09.940 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:09.941 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:09.941 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 51, 2022-07-12T08:46:09.941 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:09.942 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:09.942 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:09.943 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.943 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.944 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.944 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:46:09.945 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.945 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.945 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.946 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.946 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:46:09.947 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.947 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.949 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.950 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.950 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:09.951 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.951 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.952 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.952 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:46:09.953 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.953 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.954 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.954 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.955 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:46:09.955 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.956 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.956 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.956 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.957 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:09.957 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.957 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.958 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:46:09.959 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.959 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.959 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.960 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.960 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:46:09.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.961 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.961 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.962 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.962 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:09.962 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.963 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.963 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.963 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:46:09.964 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.965 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.965 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.965 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.965 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:46:09.966 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:09.966 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.966 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.966 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.967 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:46:09.967 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:46:09.967 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:46:09.968 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:46:09.968 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:09.968 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:09.969 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:09.969 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.969 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.969 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.970 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:09.970 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:46:09.970 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:09.971 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:09.971 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:09.971 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:09.972 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:09.972 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:09.973 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:09.973 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-12T08:46:09.973 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:09.974 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:09.974 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:09.974 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.975 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.976 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:46:09.977 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.977 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.978 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.978 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.979 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:46:09.979 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.979 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.980 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.980 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.981 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:09.981 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.982 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.982 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.982 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:46:09.983 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.983 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.983 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.984 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.984 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:46:09.985 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.985 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.986 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.986 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.986 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:09.987 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.987 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.988 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.988 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:46:09.988 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.989 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.989 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.990 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.990 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:46:09.990 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.991 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.991 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.992 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.992 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:09.992 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:09.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.993 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:09.994 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:46:09.994 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.994 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.995 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:09.995 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:09.996 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:46:09.996 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:09.997 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:09.997 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:09.998 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:09.998 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:46:09.998 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:46:09.999 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:46:09.999 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:46:09.999 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:10.000 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:10.000 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:10.001 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.002 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.002 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:10.003 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:46:10.003 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:10.003 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:10.004 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:10.004 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:10.005 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:10.005 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:10.006 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:10.006 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:10.007 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:10.007 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:10.008 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:10.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.011 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:46:10.011 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.013 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.013 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.013 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:46:10.014 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.014 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.015 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.015 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:10.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.017 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.017 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:46:10.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.018 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.018 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.019 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.019 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:46:10.020 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.020 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.020 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.021 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.021 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:10.021 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.022 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.022 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:46:10.023 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.024 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.024 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.024 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:46:10.024 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.025 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.025 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.025 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.025 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:10.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.026 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.026 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:46:10.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.028 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.028 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:46:10.029 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:10.029 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.029 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.029 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:46:10.030 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:46:10.030 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:46:10.030 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:46:10.030 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:10.031 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:10.031 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:10.031 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.032 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:10.032 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:46:10.033 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:10.033 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:10.033 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:10.034 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:10.034 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:10.034 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:10.034 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:10.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:10.035 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:10.036 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:10.036 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:10.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.037 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:46:10.038 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.039 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.039 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:46:10.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.041 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.042 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.042 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.043 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:10.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:46:10.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.044 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:46:10.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.046 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.046 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:10.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.047 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:46:10.048 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.049 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:46:10.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.049 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.049 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.050 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:10.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.051 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.051 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:46:10.051 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.052 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:46:10.053 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:10.053 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.053 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.054 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:46:10.054 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:46:10.054 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:46:10.054 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:46:10.055 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:10.055 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:10.055 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:10.055 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.055 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.056 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:10.056 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:46:10.056 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:10.056 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:10.057 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:10.057 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:10.057 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:10.057 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:10.058 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:10.058 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 51, 2022-07-12T08:46:10.058 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:10.058 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:10.059 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:10.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:46:10.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.060 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:46:10.061 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.061 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.061 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.062 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:10.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.062 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.062 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:46:10.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.063 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.064 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.064 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:46:10.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.064 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:10.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:46:10.066 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.066 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.067 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:46:10.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.067 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.068 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.068 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:10.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:46:10.070 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.070 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.071 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:46:10.071 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:10.071 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.072 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.072 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:46:10.072 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:46:10.073 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:46:10.073 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:46:10.073 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:10.073 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:10.073 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:10.074 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.074 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:10.075 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:46:10.075 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:10.075 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:10.075 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:10.075 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:10.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:10.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:10.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:10.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-12T08:46:10.077 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:10.077 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:10.077 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:10.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.077 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.078 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:46:10.078 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.078 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.079 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.079 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:46:10.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.080 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.080 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.080 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:10.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.081 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.081 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:46:10.081 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.082 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.082 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:46:10.083 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.083 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.083 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.084 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:10.084 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.084 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.084 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:46:10.085 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.085 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.085 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.086 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:46:10.086 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.086 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.087 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.087 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:10.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:10.088 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.088 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:10.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:46:10.089 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.089 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:10.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:46:10.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:10.090 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.090 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.090 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.091 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:46:10.091 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:46:10.091 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:46:10.091 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:46:10.091 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:10.092 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:10.092 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:10.092 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:10.092 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.093 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:10.093 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:46:10.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.093 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:10.093 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.094 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.094 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.094 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.095 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.095 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:46:10.095 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.095 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.096 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:10.096 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.096 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.097 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.097 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.097 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.097 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:46:10.097 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.098 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.098 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.099 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:10.099 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.099 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.099 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.099 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.100 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.100 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:46:10.100 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.100 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.101 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:10.101 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.101 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.102 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.102 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.102 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.102 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:46:10.103 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.103 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.103 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.103 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:10.104 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.104 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.104 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.104 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.104 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.105 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:46:10.105 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.106 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:10.106 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.106 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.107 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.107 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.107 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.108 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:46:10.108 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.109 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:10.109 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.109 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.109 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.109 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.110 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.110 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:46:10.110 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:10.111 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:10.111 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:10.112 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:10.112 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:10.112 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:10.112 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:10.113 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:46:10.113 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:10.113 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.114 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:10.114 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:46:10.114 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:46:10.114 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:46:10.115 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:46:10.115 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:46:10.115 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:46:10.115 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:46:10.115 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:46:10.116 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:46:10.116 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:46:10.116 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:46:10.116 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:46:10.117 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:46:10.117 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:10.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:10.117 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:46:10.117 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:46:10.118 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:46:10.118 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:46:10.118 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:46:10.118 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:46:10.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7588053a40 msgr2=0x7f7588055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7588053a40 0x7f7588055f00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f759c00b000 tx=0x7f759c008040).stop 2022-07-12T08:46:10.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 msgr2=0x7f75a80671e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 0x7f75a80671e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f759800c000 tx=0x7f7598008040).stop 2022-07-12T08:46:10.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 shutdown_connections 2022-07-12T08:46:10.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75a80ae860 0x7f75a80676d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75a8063a90 0x7f75a80671e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7588053a40 0x7f7588055f00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 --2- 172.21.15.61:0/4072452111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7594004e70 0x7f75a8099e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 >> 172.21.15.61:0/4072452111 conn(0x7f75a8063380 msgr2=0x7f75a80af870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:10.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 shutdown_connections 2022-07-12T08:46:10.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:09.728+0000 7f75ad4fb700 1 -- 172.21.15.61:0/4072452111 wait complete. 2022-07-12T08:46:10.125 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:10.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.244+0000 7fc8fedd5700 1 Processor -- start 2022-07-12T08:46:10.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.244+0000 7fc8fedd5700 1 -- start start 2022-07-12T08:46:10.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8f805d040 0x7fc8f813a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f80607e0 con 0x7fc8f805caf0 2022-07-12T08:46:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f80b6ea0 con 0x7fc8f80b6ac0 2022-07-12T08:46:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f8060340 con 0x7fc8f805d040 2022-07-12T08:46:10.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fd5d2700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc8f805caf0 0x7fc8f80b1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44328/0 (socket says 172.21.15.61:44328) 2022-07-12T08:46:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fd5d2700 1 -- 172.21.15.61:0/2024430994 learned_addr learned my addr 172.21.15.61:0/2024430994 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fe5d4700 1 --2- 172.21.15.61:0/2024430994 >> v2:172.21.15.61:3300/0 conn(0x7fc8f805d040 0x7fc8f813a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fcdd1700 1 -- 172.21.15.61:0/2024430994 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175886606 0 0) 0x7fc8f80607e0 con 0x7fc8f805caf0 2022-07-12T08:46:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fcdd1700 1 -- 172.21.15.61:0/2024430994 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e0003260 con 0x7fc8f805caf0 2022-07-12T08:46:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fe5d4700 1 -- 172.21.15.61:0/2024430994 >> v1:172.21.15.111:6789/0 conn(0x7fc8f80b6ac0 legacy=0x7fc8f8143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fe5d4700 1 -- 172.21.15.61:0/2024430994 >> v1:172.21.15.61:6789/0 conn(0x7fc8f805caf0 legacy=0x7fc8f80b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fe5d4700 1 -- 172.21.15.61:0/2024430994 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8060540 con 0x7fc8f805d040 2022-07-12T08:46:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fe5d4700 1 --2- 172.21.15.61:0/2024430994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813a510 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fc8f00032f0 tx=0x7fc8f00091d0).ready entity=mon.0 client_cookie=728737a2cc903bd7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fcdd1700 1 -- 172.21.15.61:0/2024430994 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f000b6c0 con 0x7fc8f805d040 2022-07-12T08:46:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fcdd1700 1 -- 172.21.15.61:0/2024430994 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f000b820 con 0x7fc8f805d040 2022-07-12T08:46:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fcdd1700 1 -- 172.21.15.61:0/2024430994 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f000b990 con 0x7fc8f805d040 2022-07-12T08:46:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- 172.21.15.61:0/2024430994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 msgr2=0x7fc8f813a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/2024430994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813a510 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fc8f00032f0 tx=0x7fc8f00091d0).stop 2022-07-12T08:46:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- 172.21.15.61:0/2024430994 shutdown_connections 2022-07-12T08:46:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/2024430994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813a510 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- 172.21.15.61:0/2024430994 >> 172.21.15.61:0/2024430994 conn(0x7fc8f8064570 msgr2=0x7fc8f80b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.248+0000 7fc8fedd5700 1 -- 172.21.15.61:0/2024430994 shutdown_connections 2022-07-12T08:46:10.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.252+0000 7fc8fedd5700 1 -- 172.21.15.61:0/2024430994 wait complete. 2022-07-12T08:46:10.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.252+0000 7fc8fedd5700 1 Processor -- start 2022-07-12T08:46:10.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.252+0000 7fc8fedd5700 1 -- start start 2022-07-12T08:46:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8f805caf0 0x7fc8f813efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f80b6ac0 0x7fc8f8140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8147f70 con 0x7fc8f805d040 2022-07-12T08:46:10.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f8150a30 con 0x7fc8f80b6ac0 2022-07-12T08:46:10.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f80ada70 con 0x7fc8f805caf0 2022-07-12T08:46:10.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56116/0 (socket says 172.21.15.61:56116) 2022-07-12T08:46:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 -- 172.21.15.61:0/1990563063 learned_addr learned my addr 172.21.15.61:0/1990563063 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fddd3700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8f805caf0 0x7fc8f813efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fe5d4700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f80b6ac0 0x7fc8f8140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 -- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8f805caf0 msgr2=0x7fc8f813efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8f805caf0 0x7fc8f813efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 -- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f80b6ac0 msgr2=0x7fc8f8140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f80b6ac0 0x7fc8f8140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 -- 172.21.15.61:0/1990563063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0009470 con 0x7fc8f805d040 2022-07-12T08:46:10.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fd5d2700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813f490 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fc8f4001110 tx=0x7fc8f4007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:10.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f40070d0 con 0x7fc8f805d040 2022-07-12T08:46:10.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4007230 con 0x7fc8f805d040 2022-07-12T08:46:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f8150590 con 0x7fc8f805d040 2022-07-12T08:46:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f4005610 con 0x7fc8f805d040 2022-07-12T08:46:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f80ad4f0 con 0x7fc8f805d040 2022-07-12T08:46:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.256+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f8150720 con 0x7fc8f805d040 2022-07-12T08:46:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.260+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc8f401a060 con 0x7fc8f805d040 2022-07-12T08:46:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.260+0000 7fc8e6ffd700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc8d80539d0 0x7fc8d8055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.260+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc8f406b1e0 con 0x7fc8f805d040 2022-07-12T08:46:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.260+0000 7fc8fddd3700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc8d80539d0 0x7fc8d8055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.260+0000 7fc8fddd3700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc8d80539d0 0x7fc8d8055e90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc8e800d000 tx=0x7fc8e8007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.268+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc8f4042140 con 0x7fc8f805d040 2022-07-12T08:46:10.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.428+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8f8057460 con 0x7fc8f805d040 2022-07-12T08:46:10.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.428+0000 7fc8e6ffd700 1 -- 172.21.15.61:0/1990563063 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fc8f4018020 con 0x7fc8f805d040 2022-07-12T08:46:10.433 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:10.433 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":24,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:44:50.312792+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5595},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5595":{"gid":5595,"name":"a","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/4155742423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":4155742423},{"type":"v1","addr":"172.21.15.61:6833","nonce":4155742423}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:46:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc8d80539d0 msgr2=0x7fc8d8055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc8d80539d0 0x7fc8d8055e90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc8e800d000 tx=0x7fc8e8007b70).stop 2022-07-12T08:46:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 msgr2=0x7fc8f813f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813f490 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fc8f4001110 tx=0x7fc8f4007b50).stop 2022-07-12T08:46:10.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 shutdown_connections 2022-07-12T08:46:10.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc8d80539d0 0x7fc8d8055e90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8f80b6ac0 0x7fc8f8140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8f805d040 0x7fc8f813f490 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 --2- 172.21.15.61:0/1990563063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8f805caf0 0x7fc8f813efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 >> 172.21.15.61:0/1990563063 conn(0x7fc8f8064570 msgr2=0x7fc8f80b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:10.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 shutdown_connections 2022-07-12T08:46:10.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.432+0000 7fc8fedd5700 1 -- 172.21.15.61:0/1990563063 wait complete. 2022-07-12T08:46:10.443 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 24 2022-07-12T08:46:10.451 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:46:10.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cd686700 1 Processor -- start 2022-07-12T08:46:10.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cd686700 1 -- start start 2022-07-12T08:46:10.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cd686700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc1c80ab290 0x7fc1c80ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cd686700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80abc70 con 0x7fc1c80afa60 2022-07-12T08:46:10.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cd686700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c805c720 con 0x7fc1c805f140 2022-07-12T08:46:10.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cd686700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c805f580 con 0x7fc1c80ab290 2022-07-12T08:46:10.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc1c80ab290 0x7fc1c80ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc1c80ab290 0x7fc1c80ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56122/0 (socket says 172.21.15.61:56122) 2022-07-12T08:46:10.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 -- 172.21.15.61:0/3466137260 learned_addr learned my addr 172.21.15.61:0/3466137260 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:10.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1cce85700 1 --1- 172.21.15.61:0/3466137260 >> v1:172.21.15.61:6789/0 conn(0x7fc1c80afa60 0x7fc1c8147530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44344/0 (socket says 172.21.15.61:44344) 2022-07-12T08:46:10.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 -- 172.21.15.61:0/3466137260 >> v1:172.21.15.111:6789/0 conn(0x7fc1c805f140 legacy=0x7fc1c805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:10.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 -- 172.21.15.61:0/3466137260 >> v1:172.21.15.61:6789/0 conn(0x7fc1c80afa60 legacy=0x7fc1c8147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 -- 172.21.15.61:0/3466137260 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c805c9c0 con 0x7fc1c80ab290 2022-07-12T08:46:10.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c7fff700 1 --2- 172.21.15.61:0/3466137260 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c80ab670 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fc1c80af510 tx=0x7fc1c0009290).ready entity=mon.0 client_cookie=f4188deb9fbc5517 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:10.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c6ffd700 1 -- 172.21.15.61:0/3466137260 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1c000b860 con 0x7fc1c80ab290 2022-07-12T08:46:10.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c6ffd700 1 -- 172.21.15.61:0/3466137260 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c000b9c0 con 0x7fc1c80ab290 2022-07-12T08:46:10.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.552+0000 7fc1c6ffd700 1 -- 172.21.15.61:0/3466137260 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1c000bb30 con 0x7fc1c80ab290 2022-07-12T08:46:10.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 -- 172.21.15.61:0/3466137260 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 msgr2=0x7fc1c80ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 --2- 172.21.15.61:0/3466137260 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c80ab670 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7fc1c80af510 tx=0x7fc1c0009290).stop 2022-07-12T08:46:10.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 -- 172.21.15.61:0/3466137260 shutdown_connections 2022-07-12T08:46:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 --2- 172.21.15.61:0/3466137260 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c80ab670 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 -- 172.21.15.61:0/3466137260 >> 172.21.15.61:0/3466137260 conn(0x7fc1c80b1950 msgr2=0x7fc1c80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 -- 172.21.15.61:0/3466137260 shutdown_connections 2022-07-12T08:46:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 -- 172.21.15.61:0/3466137260 wait complete. 2022-07-12T08:46:10.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 Processor -- start 2022-07-12T08:46:10.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.556+0000 7fc1cd686700 1 -- start start 2022-07-12T08:46:10.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1c805f140 0x7fc1c80ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c8063860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc1c80afa60 0x7fc1c8156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81562d0 con 0x7fc1c80ab290 2022-07-12T08:46:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8156570 con 0x7fc1c805f140 2022-07-12T08:46:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c8067860 con 0x7fc1c80afa60 2022-07-12T08:46:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c8063860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c8063860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56130/0 (socket says 172.21.15.61:56130) 2022-07-12T08:46:10.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 -- 172.21.15.61:0/1199276066 learned_addr learned my addr 172.21.15.61:0/1199276066 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:10.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c7fff700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1c805f140 0x7fc1c80ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cce85700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc1c80afa60 0x7fc1c8156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 -- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc1c80afa60 msgr2=0x7fc1c8156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc1c80afa60 0x7fc1c8156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 -- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1c805f140 msgr2=0x7fc1c80ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1c805f140 0x7fc1c80ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 -- 172.21.15.61:0/1199276066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0009530 con 0x7fc1c80ab290 2022-07-12T08:46:10.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c7fff700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1c805f140 0x7fc1c80ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:10.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cce85700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc1c80afa60 0x7fc1c8156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:10.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c77fe700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c8063860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fc1bc001fa0 tx=0x7fc1bc007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:10.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1bc00ce80 con 0x7fc1c80ab290 2022-07-12T08:46:10.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc007240 con 0x7fc1c80ab290 2022-07-12T08:46:10.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c80aeb30 con 0x7fc1c80ab290 2022-07-12T08:46:10.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc1bc006510 con 0x7fc1c80ab290 2022-07-12T08:46:10.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.560+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c8063d50 con 0x7fc1c80ab290 2022-07-12T08:46:10.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.564+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc1bc010030 con 0x7fc1c80ab290 2022-07-12T08:46:10.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.564+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c8057460 con 0x7fc1c80ab290 2022-07-12T08:46:10.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.564+0000 7fc1c57fa700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc1a8053920 0x7fc1a8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:10.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.564+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc1bc06bc60 con 0x7fc1c80ab290 2022-07-12T08:46:10.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.564+0000 7fc1c7fff700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc1a8053920 0x7fc1a8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:10.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.564+0000 7fc1c7fff700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc1a8053920 0x7fc1a8055de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fc1c0019000 tx=0x7fc1c0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:10.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.568+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc1bc042ab0 con 0x7fc1c80ab290 2022-07-12T08:46:10.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:10.740+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc1c806b2b0 con 0x7fc1c80ab290 2022-07-12T08:46:11.142 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:46:11.140+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:46:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.152+0000 7fc1c57fa700 1 -- 172.21.15.61:0/1199276066 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc1bc0460d0 con 0x7fc1c80ab290 2022-07-12T08:46:11.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc1a8053920 msgr2=0x7fc1a8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc1a8053920 0x7fc1a8055de0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fc1c0019000 tx=0x7fc1c0016000).stop 2022-07-12T08:46:11.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 msgr2=0x7fc1c8063860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c8063860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fc1bc001fa0 tx=0x7fc1bc007c30).stop 2022-07-12T08:46:11.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 shutdown_connections 2022-07-12T08:46:11.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc1c80afa60 0x7fc1c8156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1c80ab290 0x7fc1c8063860 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc1a8053920 0x7fc1a8055de0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 --2- 172.21.15.61:0/1199276066 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1c805f140 0x7fc1c80ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.156+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 >> 172.21.15.61:0/1199276066 conn(0x7fc1c80b1950 msgr2=0x7fc1c80afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:11.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.160+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 shutdown_connections 2022-07-12T08:46:11.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.160+0000 7fc1cd686700 1 -- 172.21.15.61:0/1199276066 wait complete. 2022-07-12T08:46:11.185 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:46:11.188 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:11.193 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:46:11.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.284+0000 7fb54d493700 1 Processor -- start 2022-07-12T08:46:11.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.284+0000 7fb54d493700 1 -- start start 2022-07-12T08:46:11.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5480b1210 0x7fb54813f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb54805f340 0x7fb54813f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54805d220 con 0x7fb54805c740 2022-07-12T08:46:11.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb548098780 con 0x7fb5480b1210 2022-07-12T08:46:11.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5480aaa00 con 0x7fb54805f340 2022-07-12T08:46:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb547fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb54805c740 0x7fb548098070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44366/0 (socket says 172.21.15.61:44366) 2022-07-12T08:46:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb547fff700 1 -- 172.21.15.61:0/1988600480 learned_addr learned my addr 172.21.15.61:0/1988600480 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1988600480 >> v2:172.21.15.61:3300/0 conn(0x7fb5480b1210 0x7fb54813f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848411683 0 0) 0x7fb54805d220 con 0x7fb54805c740 2022-07-12T08:46:11.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb534004370 con 0x7fb54805c740 2022-07-12T08:46:11.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54cc92700 1 --2- 172.21.15.61:0/1988600480 >> v2:172.21.15.111:3300/0 conn(0x7fb54805f340 0x7fb54813f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2293815932 0 0) 0x7fb534004370 con 0x7fb54805c740 2022-07-12T08:46:11.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 >> v2:172.21.15.111:3300/0 conn(0x7fb54805f340 msgr2=0x7fb54813f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:11.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 --2- 172.21.15.61:0/1988600480 >> v2:172.21.15.111:3300/0 conn(0x7fb54805f340 0x7fb54813f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 >> v2:172.21.15.61:3300/0 conn(0x7fb5480b1210 msgr2=0x7fb54813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:11.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 --2- 172.21.15.61:0/1988600480 >> v2:172.21.15.61:3300/0 conn(0x7fb5480b1210 0x7fb54813f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54809aca0 con 0x7fb54805c740 2022-07-12T08:46:11.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fb538001660 con 0x7fb54805c740 2022-07-12T08:46:11.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 >> v1:172.21.15.61:6789/0 conn(0x7fb54805c740 legacy=0x7fb548098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534004860 0x7fb534004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb534005610 0x7fb534005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb534005f80 0x7fb53400c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb548098780 con 0x7fb534005610 2022-07-12T08:46:11.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480aaa00 con 0x7fb534004860 2022-07-12T08:46:11.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53400cdd0 con 0x7fb534005f80 2022-07-12T08:46:11.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb534005610 0x7fb534005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb547fff700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534004860 0x7fb534004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54cc92700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb534005f80 0x7fb53400c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 -- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb534005f80 msgr2=0x7fb53400c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:11.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb534005f80 0x7fb53400c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 -- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534004860 msgr2=0x7fb534004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:11.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534004860 0x7fb534004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 -- 172.21.15.61:0/1988600480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534004370 con 0x7fb534005610 2022-07-12T08:46:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54cc92700 1 -- 172.21.15.61:0/1988600480 reap_dead start 2022-07-12T08:46:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb534005610 0x7fb534005a90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fb53c00c000 tx=0x7fb53c006df0).ready entity=mon.0 client_cookie=a117e5854b5e19e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53c00e800 con 0x7fb534005610 2022-07-12T08:46:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c00e960 con 0x7fb534005610 2022-07-12T08:46:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb546ffd700 1 -- 172.21.15.61:0/1988600480 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53c00ecc0 con 0x7fb534005610 2022-07-12T08:46:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 -- 172.21.15.61:0/1988600480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb534005610 msgr2=0x7fb534005a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.288+0000 7fb54d493700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb534005610 0x7fb534005a90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fb53c00c000 tx=0x7fb53c006df0).stop 2022-07-12T08:46:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- 172.21.15.61:0/1988600480 shutdown_connections 2022-07-12T08:46:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 --2- 172.21.15.61:0/1988600480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb534005610 0x7fb534005a90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- 172.21.15.61:0/1988600480 >> 172.21.15.61:0/1988600480 conn(0x7fb5480b1950 msgr2=0x7fb5480b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- 172.21.15.61:0/1988600480 shutdown_connections 2022-07-12T08:46:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- 172.21.15.61:0/1988600480 wait complete. 2022-07-12T08:46:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 Processor -- start 2022-07-12T08:46:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- start start 2022-07-12T08:46:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 0x7fb548150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5480ac5a0 0x7fb54813f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5480639b0 0x7fb5480ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480af300 con 0x7fb5480ac5a0 2022-07-12T08:46:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb548063e10 con 0x7fb534005610 2022-07-12T08:46:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54d493700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5480aedc0 con 0x7fb5480639b0 2022-07-12T08:46:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb5477fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5480ac5a0 0x7fb54813f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb547fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 0x7fb548150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb547fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 0x7fb548150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:34734/0 (socket says 172.21.15.61:34734) 2022-07-12T08:46:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb547fff700 1 -- 172.21.15.61:0/1924976608 learned_addr learned my addr 172.21.15.61:0/1924976608 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.292+0000 7fb54cc92700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5480639b0 0x7fb5480ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5477fe700 1 -- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5480639b0 msgr2=0x7fb5480ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5480639b0 0x7fb5480ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5477fe700 1 -- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 msgr2=0x7fb548150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 0x7fb548150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5477fe700 1 -- 172.21.15.61:0/1924976608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5480af060 con 0x7fb5480ac5a0 2022-07-12T08:46:11.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb547fff700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 0x7fb548150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:11.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5477fe700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5480ac5a0 0x7fb54813f960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fb53c00c000 tx=0x7fb53c019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:11.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53c01cc70 con 0x7fb5480ac5a0 2022-07-12T08:46:11.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c000e90 con 0x7fb5480ac5a0 2022-07-12T08:46:11.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5480ad330 con 0x7fb5480ac5a0 2022-07-12T08:46:11.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53c00be60 con 0x7fb5480ac5a0 2022-07-12T08:46:11.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5480ad800 con 0x7fb5480ac5a0 2022-07-12T08:46:11.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb548057460 con 0x7fb5480ac5a0 2022-07-12T08:46:11.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb53c02a020 con 0x7fb5480ac5a0 2022-07-12T08:46:11.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5457fa700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb528053a60 0x7fb528055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.296+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb53c06d1d0 con 0x7fb5480ac5a0 2022-07-12T08:46:11.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.300+0000 7fb547fff700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb528053a60 0x7fb528055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.300+0000 7fb547fff700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb528053a60 0x7fb528055f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb538001110 tx=0x7fb538009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:11.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.304+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb53c0440e0 con 0x7fb5480ac5a0 2022-07-12T08:46:11.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.472+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb54806b2b0 con 0x7fb5480ac5a0 2022-07-12T08:46:11.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.472+0000 7fb5457fa700 1 -- 172.21.15.61:0/1924976608 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+1781 (secure 0 0 0) 0x7fb53c047700 con 0x7fb5480ac5a0 2022-07-12T08:46:11.476 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:11.476 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":25,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21}],"filesystems":[{"mdsmap":{"epoch":25,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:44:48.629894+0000","modified":"2022-07-12T08:46:11.145461+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-12T08:46:11.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.472+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb528053a60 msgr2=0x7fb528055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.472+0000 7fb54d493700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb528053a60 0x7fb528055f20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fb538001110 tx=0x7fb538009040).stop 2022-07-12T08:46:11.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.472+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5480ac5a0 msgr2=0x7fb54813f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.472+0000 7fb54d493700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5480ac5a0 0x7fb54813f960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7fb53c00c000 tx=0x7fb53c019000).stop 2022-07-12T08:46:11.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 shutdown_connections 2022-07-12T08:46:11.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5480639b0 0x7fb5480ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5480ac5a0 0x7fb54813f960 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb528053a60 0x7fb528055f20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 --2- 172.21.15.61:0/1924976608 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb534005610 0x7fb548150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 >> 172.21.15.61:0/1924976608 conn(0x7fb5480b1950 msgr2=0x7fb5480b3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:11.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 shutdown_connections 2022-07-12T08:46:11.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.476+0000 7fb54d493700 1 -- 172.21.15.61:0/1924976608 wait complete. 2022-07-12T08:46:11.480 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 25 2022-07-12T08:46:11.488 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:46:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 Processor -- start 2022-07-12T08:46:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 -- start start 2022-07-12T08:46:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9a80b52e0 0x7fa9a80b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b4050 0x7fa9a80b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a80b56c0 con 0x7fa9a80b2b90 2022-07-12T08:46:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a80b3000 con 0x7fa9a80b52e0 2022-07-12T08:46:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9b043d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8142f00 con 0x7fa9a80b4050 2022-07-12T08:46:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9ae9da700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa9a80b2b90 0x7fa9a805bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44390/0 (socket says 172.21.15.61:44390) 2022-07-12T08:46:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9ae9da700 1 -- 172.21.15.61:0/2809339810 learned_addr learned my addr 172.21.15.61:0/2809339810 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9ad9d8700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b4050 0x7fa9a80b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9ae1d9700 1 --2- 172.21.15.61:0/2809339810 >> v2:172.21.15.111:3300/0 conn(0x7fa9a80b52e0 0x7fa9a80b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9ad1d7700 1 -- 172.21.15.61:0/2809339810 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327954576 0 0) 0x7fa9a80b56c0 con 0x7fa9a80b2b90 2022-07-12T08:46:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.596+0000 7fa9ad1d7700 1 -- 172.21.15.61:0/2809339810 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa990003260 con 0x7fa9a80b2b90 2022-07-12T08:46:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ae1d9700 1 -- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b4050 msgr2=0x7fa9a80b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ae1d9700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b4050 0x7fa9a80b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ae1d9700 1 -- 172.21.15.61:0/2809339810 >> v1:172.21.15.61:6789/0 conn(0x7fa9a80b2b90 legacy=0x7fa9a805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ae1d9700 1 -- 172.21.15.61:0/2809339810 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a80b4bd0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ad9d8700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b4050 0x7fa9a80b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ae1d9700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b52e0 0x7fa9a80b21b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fa9a4002f70 tx=0x7fa9a4006df0).ready entity=mon.1 client_cookie=2f478f6d491531f3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ad1d7700 1 -- 172.21.15.61:0/2809339810 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a400d4a0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ad1d7700 1 -- 172.21.15.61:0/2809339810 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a4000e90 con 0x7fa9a80b52e0 2022-07-12T08:46:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9ad1d7700 1 -- 172.21.15.61:0/2809339810 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a400d7f0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 -- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b52e0 msgr2=0x7fa9a80b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b52e0 0x7fa9a80b21b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fa9a4002f70 tx=0x7fa9a4006df0).stop 2022-07-12T08:46:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 -- 172.21.15.61:0/2809339810 shutdown_connections 2022-07-12T08:46:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b4050 0x7fa9a80b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2809339810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b52e0 0x7fa9a80b21b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 -- 172.21.15.61:0/2809339810 >> 172.21.15.61:0/2809339810 conn(0x7fa9a8067020 msgr2=0x7fa9a8067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:11.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 -- 172.21.15.61:0/2809339810 shutdown_connections 2022-07-12T08:46:11.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 -- 172.21.15.61:0/2809339810 wait complete. 2022-07-12T08:46:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 Processor -- start 2022-07-12T08:46:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.600+0000 7fa9b043d700 1 -- start start 2022-07-12T08:46:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b2b90 0x7fa9a80adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b4050 0x7fa9a80af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 0x7fa9a8146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a80ae1d0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8156170 con 0x7fa9a80b4050 2022-07-12T08:46:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a8156410 con 0x7fa9a80b2b90 2022-07-12T08:46:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 0x7fa9a8146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 0x7fa9a8146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56178/0 (socket says 172.21.15.61:56178) 2022-07-12T08:46:11.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 -- 172.21.15.61:0/2053646341 learned_addr learned my addr 172.21.15.61:0/2053646341 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ad9d8700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b4050 0x7fa9a80af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae1d9700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b2b90 0x7fa9a80adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 -- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b2b90 msgr2=0x7fa9a80adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b2b90 0x7fa9a80adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 -- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b4050 msgr2=0x7fa9a80af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b4050 0x7fa9a80af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 -- 172.21.15.61:0/2053646341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c00b180 con 0x7fa9a80b52e0 2022-07-12T08:46:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9ae9da700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 0x7fa9a8146f70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fa9a0001490 tx=0x7fa9a0007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:11.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a0005ba0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0005d00 con 0x7fa9a80b52e0 2022-07-12T08:46:11.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a400b180 con 0x7fa9a80b52e0 2022-07-12T08:46:11.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a00072c0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a8147460 con 0x7fa9a80b52e0 2022-07-12T08:46:11.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.604+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a8057460 con 0x7fa9a80b52e0 2022-07-12T08:46:11.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.608+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa9a001a060 con 0x7fa9a80b52e0 2022-07-12T08:46:11.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.608+0000 7fa99b7fe700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa988053a40 0x7fa988055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:11.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.608+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa9a006bc40 con 0x7fa9a80b52e0 2022-07-12T08:46:11.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.608+0000 7fa9ae1d9700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa988053a40 0x7fa988055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:11.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.608+0000 7fa9ae1d9700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa988053a40 0x7fa988055f00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fa9a4018000 tx=0x7fa9a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:11.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.616+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9a00429c0 con 0x7fa9a80b52e0 2022-07-12T08:46:11.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:11.780+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa9a80b7fa0 con 0x7fa9a80b52e0 2022-07-12T08:46:12.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa99b7fe700 1 -- 172.21.15.61:0/2053646341 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa9a0018070 con 0x7fa9a80b52e0 2022-07-12T08:46:12.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa988053a40 msgr2=0x7fa988055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa988053a40 0x7fa988055f00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fa9a4018000 tx=0x7fa9a4014000).stop 2022-07-12T08:46:12.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 msgr2=0x7fa9a8146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 0x7fa9a8146f70 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7fa9a0001490 tx=0x7fa9a0007e00).stop 2022-07-12T08:46:12.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 shutdown_connections 2022-07-12T08:46:12.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a80b52e0 0x7fa9a8146f70 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa988053a40 0x7fa988055f00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a80b4050 0x7fa9a80af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.164+0000 7fa9b043d700 1 --2- 172.21.15.61:0/2053646341 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a80b2b90 0x7fa9a80adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.168+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 >> 172.21.15.61:0/2053646341 conn(0x7fa9a8067020 msgr2=0x7fa9a80b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:12.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.168+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 shutdown_connections 2022-07-12T08:46:12.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.168+0000 7fa9b043d700 1 -- 172.21.15.61:0/2053646341 wait complete. 2022-07-12T08:46:12.186 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:46:12.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 Processor -- start 2022-07-12T08:46:12.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 -- start start 2022-07-12T08:46:12.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f714c064490 0x7f714c0b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:12.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c142d60 con 0x7f714c0b2a50 2022-07-12T08:46:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c142f90 con 0x7f714c1394b0 2022-07-12T08:46:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714c13fa60 con 0x7f714c064490 2022-07-12T08:46:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f714c064490 0x7f714c0b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:12.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f714c0b2a50 0x7f714c05f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44400/0 (socket says 172.21.15.61:44400) 2022-07-12T08:46:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714b7fe700 1 -- 172.21.15.61:0/880850049 learned_addr learned my addr 172.21.15.61:0/880850049 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714affd700 1 -- 172.21.15.61:0/880850049 >> v1:172.21.15.111:6789/0 conn(0x7f714c1394b0 legacy=0x7f714c13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714affd700 1 -- 172.21.15.61:0/880850049 >> v1:172.21.15.61:6789/0 conn(0x7f714c0b2a50 legacy=0x7f714c05f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714affd700 1 -- 172.21.15.61:0/880850049 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c13f760 con 0x7f714c064490 2022-07-12T08:46:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714affd700 1 --2- 172.21.15.61:0/880850049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c064490 0x7f714c0b2e30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f713c002f70 tx=0x7f713c006df0).ready entity=mon.0 client_cookie=5f4ba897c325ec49 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714a7fc700 1 -- 172.21.15.61:0/880850049 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f713c00d610 con 0x7f714c064490 2022-07-12T08:46:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714a7fc700 1 -- 172.21.15.61:0/880850049 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c000e90 con 0x7f714c064490 2022-07-12T08:46:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f714a7fc700 1 -- 172.21.15.61:0/880850049 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f713c00d960 con 0x7f714c064490 2022-07-12T08:46:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 -- 172.21.15.61:0/880850049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c064490 msgr2=0x7f714c0b2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.300+0000 7f7150e11700 1 --2- 172.21.15.61:0/880850049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c064490 0x7f714c0b2e30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f713c002f70 tx=0x7f713c006df0).stop 2022-07-12T08:46:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- 172.21.15.61:0/880850049 shutdown_connections 2022-07-12T08:46:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 --2- 172.21.15.61:0/880850049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c064490 0x7f714c0b2e30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- 172.21.15.61:0/880850049 >> 172.21.15.61:0/880850049 conn(0x7f714c061e70 msgr2=0x7f714c063a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:12.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- 172.21.15.61:0/880850049 shutdown_connections 2022-07-12T08:46:12.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- 172.21.15.61:0/880850049 wait complete. 2022-07-12T08:46:12.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 Processor -- start 2022-07-12T08:46:12.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- start start 2022-07-12T08:46:12.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f714c064490 0x7f714c066ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 0x7f714c0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f714c1394b0 0x7f714c05c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c1383a0 con 0x7f714c0b2a50 2022-07-12T08:46:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c154050 con 0x7f714c1394b0 2022-07-12T08:46:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f7150e11700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c153b10 con 0x7f714c064490 2022-07-12T08:46:12.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f714affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 0x7f714c0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:12.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f714affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 0x7f714c0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56190/0 (socket says 172.21.15.61:56190) 2022-07-12T08:46:12.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f714b7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f714c064490 0x7f714c066ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f714affd700 1 -- 172.21.15.61:0/3369284089 learned_addr learned my addr 172.21.15.61:0/3369284089 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.304+0000 7f714bfff700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f714c1394b0 0x7f714c05c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f714affd700 1 -- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f714c064490 msgr2=0x7f714c066ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f714affd700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f714c064490 0x7f714c066ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f714affd700 1 -- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f714c1394b0 msgr2=0x7f714c05c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:12.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f714affd700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f714c1394b0 0x7f714c05c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:12.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f714affd700 1 -- 172.21.15.61:0/3369284089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c00b180 con 0x7f714c0b2a50 2022-07-12T08:46:12.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f714affd700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 0x7f714c0afc60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f713c002f70 tx=0x7f713c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:12.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f713c00d990 con 0x7f714c0b2a50 2022-07-12T08:46:12.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c011b80 con 0x7f714c0b2a50 2022-07-12T08:46:12.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c153db0 con 0x7f714c0b2a50 2022-07-12T08:46:12.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f713c01e460 con 0x7f714c0b2a50 2022-07-12T08:46:12.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c0aed70 con 0x7f714c0b2a50 2022-07-12T08:46:12.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c057460 con 0x7f714c0b2a50 2022-07-12T08:46:12.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f713c025030 con 0x7f714c0b2a50 2022-07-12T08:46:12.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.308+0000 7f7148ff9700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71300539b0 0x7f7130055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:12.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.312+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f713c06d200 con 0x7f714c0b2a50 2022-07-12T08:46:12.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.312+0000 7f714b7fe700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71300539b0 0x7f7130055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:12.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.312+0000 7f714b7fe700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71300539b0 0x7f7130055e70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f713400d000 tx=0x7f7134009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:12.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.316+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f713c044000 con 0x7f714c0b2a50 2022-07-12T08:46:12.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:12.440+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f714c05ee40 con 0x7f714c0b2a50 2022-07-12T08:46:13.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.164+0000 7f7148ff9700 1 -- 172.21.15.61:0/3369284089 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v55) v1 ==== 185+0+0 (secure 0 0 0) 0x7f713c047620 con 0x7f714c0b2a50 2022-07-12T08:46:13.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71300539b0 msgr2=0x7f7130055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:13.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71300539b0 0x7f7130055e70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f713400d000 tx=0x7f7134009040).stop 2022-07-12T08:46:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 msgr2=0x7f714c0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 0x7f714c0afc60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f713c002f70 tx=0x7f713c015000).stop 2022-07-12T08:46:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 shutdown_connections 2022-07-12T08:46:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f71300539b0 0x7f7130055e70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f714c1394b0 0x7f714c05c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f714c0b2a50 0x7f714c0afc60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 --2- 172.21.15.61:0/3369284089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f714c064490 0x7f714c066ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 >> 172.21.15.61:0/3369284089 conn(0x7f714c061e70 msgr2=0x7f714c138da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 shutdown_connections 2022-07-12T08:46:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.168+0000 7f7150e11700 1 -- 172.21.15.61:0/3369284089 wait complete. 2022-07-12T08:46:13.187 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:46:13.188 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:46:13.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.312+0000 7fd19af2f700 1 Processor -- start 2022-07-12T08:46:13.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.312+0000 7fd19af2f700 1 -- start start 2022-07-12T08:46:13.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.312+0000 7fd19af2f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd19405d060 0x7fd194145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:13.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd1940b3c60 0x7fd194145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:13.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1940afca0 con 0x7fd194067860 2022-07-12T08:46:13.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19405d440 con 0x7fd19405d060 2022-07-12T08:46:13.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1940aff00 con 0x7fd1940b3c60 2022-07-12T08:46:13.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19a72e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd194067860 0x7fd194146760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44414/0 (socket says 172.21.15.61:44414) 2022-07-12T08:46:13.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19a72e700 1 -- 172.21.15.61:0/1056337204 learned_addr learned my addr 172.21.15.61:0/1056337204 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:13.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd199f2d700 1 --2- 172.21.15.61:0/1056337204 >> v2:172.21.15.61:3300/0 conn(0x7fd19405d060 0x7fd194145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:13.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19972c700 1 --2- 172.21.15.61:0/1056337204 >> v2:172.21.15.111:3300/0 conn(0x7fd1940b3c60 0x7fd194145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:13.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd199f2d700 1 -- 172.21.15.61:0/1056337204 >> v2:172.21.15.111:3300/0 conn(0x7fd1940b3c60 msgr2=0x7fd194145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:13.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd199f2d700 1 --2- 172.21.15.61:0/1056337204 >> v2:172.21.15.111:3300/0 conn(0x7fd1940b3c60 0x7fd194145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd199f2d700 1 -- 172.21.15.61:0/1056337204 >> v1:172.21.15.61:6789/0 conn(0x7fd194067860 legacy=0x7fd194146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:13.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd198f2b700 1 -- 172.21.15.61:0/1056337204 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660972200 0 0) 0x7fd1940afca0 con 0x7fd194067860 2022-07-12T08:46:13.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd199f2d700 1 -- 172.21.15.61:0/1056337204 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194146060 con 0x7fd19405d060 2022-07-12T08:46:13.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd199f2d700 1 --2- 172.21.15.61:0/1056337204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd194145680 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fd188002f70 tx=0x7fd188006df0).ready entity=mon.0 client_cookie=3e2330b6781a3b4a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:13.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd198f2b700 1 -- 172.21.15.61:0/1056337204 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18800d570 con 0x7fd19405d060 2022-07-12T08:46:13.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd198f2b700 1 -- 172.21.15.61:0/1056337204 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd188000e90 con 0x7fd19405d060 2022-07-12T08:46:13.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd198f2b700 1 -- 172.21.15.61:0/1056337204 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18800d8c0 con 0x7fd19405d060 2022-07-12T08:46:13.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- 172.21.15.61:0/1056337204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 msgr2=0x7fd194145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:13.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 --2- 172.21.15.61:0/1056337204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd194145680 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fd188002f70 tx=0x7fd188006df0).stop 2022-07-12T08:46:13.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- 172.21.15.61:0/1056337204 shutdown_connections 2022-07-12T08:46:13.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 --2- 172.21.15.61:0/1056337204 >> v2:172.21.15.111:3300/0 conn(0x7fd1940b3c60 0x7fd194145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 --2- 172.21.15.61:0/1056337204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd194145680 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- 172.21.15.61:0/1056337204 >> 172.21.15.61:0/1056337204 conn(0x7fd1940b67c0 msgr2=0x7fd194066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:13.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- 172.21.15.61:0/1056337204 shutdown_connections 2022-07-12T08:46:13.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.316+0000 7fd19af2f700 1 -- 172.21.15.61:0/1056337204 wait complete. 2022-07-12T08:46:13.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 Processor -- start 2022-07-12T08:46:13.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 -- start start 2022-07-12T08:46:13.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd19409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:13.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd194067860 0x7fd194098000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1940b3c60 0x7fd19409df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19409d6c0 con 0x7fd19405d060 2022-07-12T08:46:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd194151df0 con 0x7fd1940b3c60 2022-07-12T08:46:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19409e460 con 0x7fd194067860 2022-07-12T08:46:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd19409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd19409cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56200/0 (socket says 172.21.15.61:56200) 2022-07-12T08:46:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 -- 172.21.15.61:0/2046375301 learned_addr learned my addr 172.21.15.61:0/2046375301 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19972c700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd194067860 0x7fd194098000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:13.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19a72e700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1940b3c60 0x7fd19409df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:13.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 -- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd194067860 msgr2=0x7fd194098000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:13.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd194067860 0x7fd194098000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 -- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1940b3c60 msgr2=0x7fd19409df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:13.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1940b3c60 0x7fd19409df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:13.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 -- 172.21.15.61:0/2046375301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18800b180 con 0x7fd19405d060 2022-07-12T08:46:13.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19972c700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd194067860 0x7fd194098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:13.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19a72e700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1940b3c60 0x7fd19409df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:13.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd199f2d700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd19409cfd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fd188002f70 tx=0x7fd188014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:13.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18800dea0 con 0x7fd19405d060 2022-07-12T08:46:13.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18801e400 con 0x7fd19405d060 2022-07-12T08:46:13.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1940915d0 con 0x7fd19405d060 2022-07-12T08:46:13.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18801e760 con 0x7fd19405d060 2022-07-12T08:46:13.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.320+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1940912c0 con 0x7fd19405d060 2022-07-12T08:46:13.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.324+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1941521e0 con 0x7fd19405d060 2022-07-12T08:46:13.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.324+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd188025030 con 0x7fd19405d060 2022-07-12T08:46:13.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.324+0000 7fd186ffd700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd174053ad0 0x7fd174055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:13.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.324+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fd18806cbf0 con 0x7fd19405d060 2022-07-12T08:46:13.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.324+0000 7fd19972c700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd174053ad0 0x7fd174055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:13.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.324+0000 7fd19972c700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd174053ad0 0x7fd174055f90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fd19000b000 tx=0x7fd190008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:13.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.328+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd1941521e0 con 0x7fd19405d060 2022-07-12T08:46:13.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:13.452+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd1941521e0 con 0x7fd19405d060 2022-07-12T08:46:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.180+0000 7fd186ffd700 1 -- 172.21.15.61:0/2046375301 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v56) v1 ==== 173+0+0 (secure 0 0 0) 0x7fd1941521e0 con 0x7fd19405d060 2022-07-12T08:46:14.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.184+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd174053ad0 msgr2=0x7fd174055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.184+0000 7fd19af2f700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd174053ad0 0x7fd174055f90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fd19000b000 tx=0x7fd190008040).stop 2022-07-12T08:46:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.184+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 msgr2=0x7fd19409cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.184+0000 7fd19af2f700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd19409cfd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7fd188002f70 tx=0x7fd188014000).stop 2022-07-12T08:46:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.188+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 shutdown_connections 2022-07-12T08:46:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.188+0000 7fd19af2f700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd174053ad0 0x7fd174055f90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.188+0000 7fd19af2f700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1940b3c60 0x7fd19409df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.188+0000 7fd19af2f700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd194067860 0x7fd194098000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.188+0000 7fd19af2f700 1 --2- 172.21.15.61:0/2046375301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19405d060 0x7fd19409cfd0 secure :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0x7fd188002f70 tx=0x7fd188014000).stop 2022-07-12T08:46:14.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.188+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 >> 172.21.15.61:0/2046375301 conn(0x7fd1940b67c0 msgr2=0x7fd19405fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:14.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.192+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 shutdown_connections 2022-07-12T08:46:14.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.192+0000 7fd19af2f700 1 -- 172.21.15.61:0/2046375301 wait complete. 2022-07-12T08:46:14.198 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:46:14.207 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:46:14.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.300+0000 7fc5e5278700 1 Processor -- start 2022-07-12T08:46:14.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.300+0000 7fc5e5278700 1 -- start start 2022-07-12T08:46:14.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5e00aaa50 0x7fc5e0140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e0140d20 0x7fc5e013d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e0144cf0 con 0x7fc5e00b24b0 2022-07-12T08:46:14.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e0141a40 con 0x7fc5e00aaa50 2022-07-12T08:46:14.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00b3190 con 0x7fc5e0140d20 2022-07-12T08:46:14.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5e00aaa50 0x7fc5e0140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5e00aaa50 0x7fc5e0140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56210/0 (socket says 172.21.15.61:56210) 2022-07-12T08:46:14.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 -- 172.21.15.61:0/248718303 learned_addr learned my addr 172.21.15.61:0/248718303 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:14.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e4a77700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e0140d20 0x7fc5e013d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 -- 172.21.15.61:0/248718303 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e0140d20 msgr2=0x7fc5e013d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:14.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e0140d20 0x7fc5e013d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 -- 172.21.15.61:0/248718303 >> v1:172.21.15.111:6789/0 conn(0x7fc5e00b24b0 legacy=0x7fc5e01445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 -- 172.21.15.61:0/248718303 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0143ea0 con 0x7fc5e00aaa50 2022-07-12T08:46:14.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e4a77700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e0140d20 0x7fc5e013d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:14.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5d77fe700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00aaa50 0x7fc5e0140830 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fc5c8002f70 tx=0x7fc5c8006eb0).ready entity=mon.0 client_cookie=6587c58053514763 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:14.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5df7fe700 1 -- 172.21.15.61:0/248718303 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c800d5c0 con 0x7fc5e00aaa50 2022-07-12T08:46:14.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5df7fe700 1 -- 172.21.15.61:0/248718303 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c8000e90 con 0x7fc5e00aaa50 2022-07-12T08:46:14.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5df7fe700 1 -- 172.21.15.61:0/248718303 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c800d950 con 0x7fc5e00aaa50 2022-07-12T08:46:14.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- 172.21.15.61:0/248718303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00aaa50 msgr2=0x7fc5e0140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00aaa50 0x7fc5e0140830 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fc5c8002f70 tx=0x7fc5c8006eb0).stop 2022-07-12T08:46:14.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- 172.21.15.61:0/248718303 shutdown_connections 2022-07-12T08:46:14.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e0140d20 0x7fc5e013d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 --2- 172.21.15.61:0/248718303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00aaa50 0x7fc5e0140830 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- 172.21.15.61:0/248718303 >> 172.21.15.61:0/248718303 conn(0x7fc5e00b5860 msgr2=0x7fc5e00b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:14.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- 172.21.15.61:0/248718303 shutdown_connections 2022-07-12T08:46:14.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.304+0000 7fc5e5278700 1 -- 172.21.15.61:0/248718303 wait complete. 2022-07-12T08:46:14.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 Processor -- start 2022-07-12T08:46:14.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 -- start start 2022-07-12T08:46:14.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e00aaa50 0x7fc5e00acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 0x7fc5e014f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5e0140d20 0x7fc5e014fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00ae4e0 con 0x7fc5e00b24b0 2022-07-12T08:46:14.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00afa60 con 0x7fc5e0140d20 2022-07-12T08:46:14.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00ab940 con 0x7fc5e00aaa50 2022-07-12T08:46:14.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 0x7fc5e014f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 0x7fc5e014f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56216/0 (socket says 172.21.15.61:56216) 2022-07-12T08:46:14.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 -- 172.21.15.61:0/4215309098 learned_addr learned my addr 172.21.15.61:0/4215309098 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:14.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e4a77700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5e0140d20 0x7fc5e014fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5dffff700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e00aaa50 0x7fc5e00acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 -- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e00aaa50 msgr2=0x7fc5e00acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e00aaa50 0x7fc5e00acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 -- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5e0140d20 msgr2=0x7fc5e014fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5e0140d20 0x7fc5e014fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 -- 172.21.15.61:0/4215309098 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c800b1e0 con 0x7fc5e00b24b0 2022-07-12T08:46:14.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5d77fe700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 0x7fc5e014f960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fc5e0140c70 tx=0x7fc5c8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:14.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c800de80 con 0x7fc5e00b24b0 2022-07-12T08:46:14.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c8011b00 con 0x7fc5e00b24b0 2022-07-12T08:46:14.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e00afc90 con 0x7fc5e00b24b0 2022-07-12T08:46:14.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c8011e60 con 0x7fc5e00b24b0 2022-07-12T08:46:14.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.308+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e00abbe0 con 0x7fc5e00b24b0 2022-07-12T08:46:14.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.312+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e00ad620 con 0x7fc5e00b24b0 2022-07-12T08:46:14.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.312+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc5c8025030 con 0x7fc5e00b24b0 2022-07-12T08:46:14.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.312+0000 7fc5ddffb700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc5c4053a60 0x7fc5c4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.312+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc5c806cdc0 con 0x7fc5e00b24b0 2022-07-12T08:46:14.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.312+0000 7fc5dffff700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc5c4053a60 0x7fc5c4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.312+0000 7fc5dffff700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc5c4053a60 0x7fc5c4055f20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc5d000d000 tx=0x7fc5d0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:14.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.316+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5c8043f60 con 0x7fc5e00b24b0 2022-07-12T08:46:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.444+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc5e00ab6e0 con 0x7fc5e00b24b0 2022-07-12T08:46:14.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.452+0000 7fc5ddffb700 1 -- 172.21.15.61:0/4215309098 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc5c8023020 con 0x7fc5e00b24b0 2022-07-12T08:46:14.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.452+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc5c4053a60 msgr2=0x7fc5c4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.452+0000 7fc5e5278700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc5c4053a60 0x7fc5c4055f20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc5d000d000 tx=0x7fc5d0007b90).stop 2022-07-12T08:46:14.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.452+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 msgr2=0x7fc5e014f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.452+0000 7fc5e5278700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 0x7fc5e014f960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fc5e0140c70 tx=0x7fc5c8015000).stop 2022-07-12T08:46:14.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 shutdown_connections 2022-07-12T08:46:14.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc5c4053a60 0x7fc5c4055f20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5e0140d20 0x7fc5e014fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5e00b24b0 0x7fc5e014f960 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 --2- 172.21.15.61:0/4215309098 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc5e00aaa50 0x7fc5e00acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 >> 172.21.15.61:0/4215309098 conn(0x7fc5e00b5860 msgr2=0x7fc5e0146710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:14.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 shutdown_connections 2022-07-12T08:46:14.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.456+0000 7fc5e5278700 1 -- 172.21.15.61:0/4215309098 wait complete. 2022-07-12T08:46:14.474 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-12T08:46:14.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.580+0000 7f383ffff700 1 Processor -- start 2022-07-12T08:46:14.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.580+0000 7f383ffff700 1 -- start start 2022-07-12T08:46:14.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f38400677b0 0x7f3840146b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3840141f20 0x7f3840142d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384005d230 con 0x7f384005fb90 2022-07-12T08:46:14.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384005ff70 con 0x7f38400677b0 2022-07-12T08:46:14.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383ffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384005ccf0 con 0x7f3840141f20 2022-07-12T08:46:14.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f384005fb90 0x7f3840141810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44440/0 (socket says 172.21.15.61:44440) 2022-07-12T08:46:14.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383effd700 1 -- 172.21.15.61:0/2563459686 learned_addr learned my addr 172.21.15.61:0/2563459686 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:14.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f38367fc700 1 --2- 172.21.15.61:0/2563459686 >> v2:172.21.15.61:3300/0 conn(0x7f38400677b0 0x7f3840146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383f7fe700 1 --2- 172.21.15.61:0/2563459686 >> v2:172.21.15.111:3300/0 conn(0x7f3840141f20 0x7f3840142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383e7fc700 1 -- 172.21.15.61:0/2563459686 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038235649 0 0) 0x7f384005d230 con 0x7f384005fb90 2022-07-12T08:46:14.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383e7fc700 1 -- 172.21.15.61:0/2563459686 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3820003260 con 0x7f384005fb90 2022-07-12T08:46:14.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f38367fc700 1 -- 172.21.15.61:0/2563459686 >> v2:172.21.15.111:3300/0 conn(0x7f3840141f20 msgr2=0x7f3840142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:14.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f38367fc700 1 --2- 172.21.15.61:0/2563459686 >> v2:172.21.15.111:3300/0 conn(0x7f3840141f20 0x7f3840142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f38367fc700 1 -- 172.21.15.61:0/2563459686 >> v1:172.21.15.61:6789/0 conn(0x7f384005fb90 legacy=0x7f3840141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f38367fc700 1 -- 172.21.15.61:0/2563459686 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384005cf90 con 0x7f38400677b0 2022-07-12T08:46:14.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f38367fc700 1 --2- 172.21.15.61:0/2563459686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f3840146b30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f3828002f70 tx=0x7f3828006df0).ready entity=mon.0 client_cookie=b70a5d64b6cc7724 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:14.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383e7fc700 1 -- 172.21.15.61:0/2563459686 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f382800d4f0 con 0x7f38400677b0 2022-07-12T08:46:14.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383e7fc700 1 -- 172.21.15.61:0/2563459686 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828000e90 con 0x7f38400677b0 2022-07-12T08:46:14.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383e7fc700 1 -- 172.21.15.61:0/2563459686 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f382800d840 con 0x7f38400677b0 2022-07-12T08:46:14.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.584+0000 7f383ffff700 1 -- 172.21.15.61:0/2563459686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 msgr2=0x7f3840146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 --2- 172.21.15.61:0/2563459686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f3840146b30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f3828002f70 tx=0x7f3828006df0).stop 2022-07-12T08:46:14.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 -- 172.21.15.61:0/2563459686 shutdown_connections 2022-07-12T08:46:14.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 --2- 172.21.15.61:0/2563459686 >> v2:172.21.15.111:3300/0 conn(0x7f3840141f20 0x7f3840142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 --2- 172.21.15.61:0/2563459686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f3840146b30 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 -- 172.21.15.61:0/2563459686 >> 172.21.15.61:0/2563459686 conn(0x7f3840063cd0 msgr2=0x7f384005f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:14.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 -- 172.21.15.61:0/2563459686 shutdown_connections 2022-07-12T08:46:14.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 -- 172.21.15.61:0/2563459686 wait complete. 2022-07-12T08:46:14.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 Processor -- start 2022-07-12T08:46:14.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.588+0000 7f383ffff700 1 -- start start 2022-07-12T08:46:14.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f384005fb90 0x7f3840146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f384013d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3840141f20 0x7f3840144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384013db20 con 0x7f38400677b0 2022-07-12T08:46:14.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840144a00 con 0x7f3840141f20 2022-07-12T08:46:14.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384013d880 con 0x7f384005fb90 2022-07-12T08:46:14.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f384013d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f384013d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56230/0 (socket says 172.21.15.61:56230) 2022-07-12T08:46:14.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383effd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f384005fb90 0x7f3840146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 -- 172.21.15.61:0/3521720448 learned_addr learned my addr 172.21.15.61:0/3521720448 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:14.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383f7fe700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3840141f20 0x7f3840144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 -- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f384005fb90 msgr2=0x7f3840146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f384005fb90 0x7f3840146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 -- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3840141f20 msgr2=0x7f3840144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:14.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3840141f20 0x7f3840144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:14.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 -- 172.21.15.61:0/3521720448 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382800b180 con 0x7f38400677b0 2022-07-12T08:46:14.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383effd700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f384005fb90 0x7f3840146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:14.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f38367fc700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f384013d1e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f3828002f70 tx=0x7f3828014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:14.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3828016070 con 0x7f38400677b0 2022-07-12T08:46:14.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828013850 con 0x7f38400677b0 2022-07-12T08:46:14.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3840155030 con 0x7f38400677b0 2022-07-12T08:46:14.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3828013bb0 con 0x7f38400677b0 2022-07-12T08:46:14.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.592+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3840149aa0 con 0x7f38400677b0 2022-07-12T08:46:14.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.596+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384013d6d0 con 0x7f38400677b0 2022-07-12T08:46:14.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.596+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3828025030 con 0x7f38400677b0 2022-07-12T08:46:14.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.596+0000 7f383cff9700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3824053a40 0x7f3824055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:14.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.596+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f382806ca40 con 0x7f38400677b0 2022-07-12T08:46:14.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.596+0000 7f383effd700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3824053a40 0x7f3824055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:14.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.596+0000 7f383effd700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3824053a40 0x7f3824055f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f383000d000 tx=0x7f3830009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:14.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.604+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3828043b70 con 0x7f38400677b0 2022-07-12T08:46:14.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:14.776+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f38401551c0 con 0x7f38400677b0 2022-07-12T08:46:15.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.180+0000 7f383cff9700 1 -- 172.21.15.61:0/3521720448 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v223) v1 ==== 152+0+0 (secure 0 0 0) 0x7f3828023020 con 0x7f38400677b0 2022-07-12T08:46:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3824053a40 msgr2=0x7f3824055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:15.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3824053a40 0x7f3824055f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f383000d000 tx=0x7f3830009040).stop 2022-07-12T08:46:15.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 msgr2=0x7f384013d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:15.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f384013d1e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f3828002f70 tx=0x7f3828014000).stop 2022-07-12T08:46:15.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 shutdown_connections 2022-07-12T08:46:15.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3824053a40 0x7f3824055f00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3840141f20 0x7f3840144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38400677b0 0x7f384013d1e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 --2- 172.21.15.61:0/3521720448 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f384005fb90 0x7f3840146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 >> 172.21.15.61:0/3521720448 conn(0x7f3840063cd0 msgr2=0x7f38400af530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:15.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 shutdown_connections 2022-07-12T08:46:15.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.184+0000 7f383ffff700 1 -- 172.21.15.61:0/3521720448 wait complete. 2022-07-12T08:46:15.205 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-12T08:46:15.206 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-12T08:46:15.207 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-12T08:46:15.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 Processor -- start 2022-07-12T08:46:15.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 -- start start 2022-07-12T08:46:15.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1bd40b6470 0x7f1bd40b6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:15.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1bd409e320 0x7f1bd409e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:15.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd4146470 con 0x7f1bd409d370 2022-07-12T08:46:15.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd4146710 con 0x7f1bd409e320 2022-07-12T08:46:15.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bdaf25700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd41469b0 con 0x7f1bd40b6470 2022-07-12T08:46:15.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bd9722700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1bd409d370 0x7f1bd4142120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44456/0 (socket says 172.21.15.61:44456) 2022-07-12T08:46:15.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bd9722700 1 -- 172.21.15.61:0/2571923216 learned_addr learned my addr 172.21.15.61:0/2571923216 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:15.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.312+0000 7f1bda724700 1 --2- 172.21.15.61:0/2571923216 >> v2:172.21.15.61:3300/0 conn(0x7f1bd409e320 0x7f1bd409e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:15.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd9f23700 1 --2- 172.21.15.61:0/2571923216 >> v2:172.21.15.111:3300/0 conn(0x7f1bd40b6470 0x7f1bd40b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:15.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd8f21700 1 -- 172.21.15.61:0/2571923216 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535899744 0 0) 0x7f1bd4146470 con 0x7f1bd409d370 2022-07-12T08:46:15.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd8f21700 1 -- 172.21.15.61:0/2571923216 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc003260 con 0x7f1bd409d370 2022-07-12T08:46:15.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd9f23700 1 -- 172.21.15.61:0/2571923216 >> v2:172.21.15.61:3300/0 conn(0x7f1bd409e320 msgr2=0x7f1bd409e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:15.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd9f23700 1 --2- 172.21.15.61:0/2571923216 >> v2:172.21.15.61:3300/0 conn(0x7f1bd409e320 0x7f1bd409e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd9f23700 1 -- 172.21.15.61:0/2571923216 >> v1:172.21.15.61:6789/0 conn(0x7f1bd409d370 legacy=0x7f1bd4142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:15.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd9f23700 1 -- 172.21.15.61:0/2571923216 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4146e50 con 0x7f1bd40b6470 2022-07-12T08:46:15.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd9f23700 1 --2- 172.21.15.61:0/2571923216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd40b6470 0x7f1bd40b6850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f1bd0002f70 tx=0x7f1bd0006d90).ready entity=mon.1 client_cookie=347d9cfa73a4490d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:15.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd8f21700 1 -- 172.21.15.61:0/2571923216 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd000e470 con 0x7f1bd40b6470 2022-07-12T08:46:15.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd8f21700 1 -- 172.21.15.61:0/2571923216 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0000e90 con 0x7f1bd40b6470 2022-07-12T08:46:15.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bd8f21700 1 -- 172.21.15.61:0/2571923216 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bd000e7c0 con 0x7f1bd40b6470 2022-07-12T08:46:15.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2571923216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd40b6470 msgr2=0x7f1bd40b6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:15.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2571923216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd40b6470 0x7f1bd40b6850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f1bd0002f70 tx=0x7f1bd0006d90).stop 2022-07-12T08:46:15.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2571923216 shutdown_connections 2022-07-12T08:46:15.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2571923216 >> v2:172.21.15.61:3300/0 conn(0x7f1bd409e320 0x7f1bd409e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2571923216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd40b6470 0x7f1bd40b6850 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2571923216 >> 172.21.15.61:0/2571923216 conn(0x7f1bd40b9d50 msgr2=0x7f1bd40b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:15.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2571923216 shutdown_connections 2022-07-12T08:46:15.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.316+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2571923216 wait complete. 2022-07-12T08:46:15.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 Processor -- start 2022-07-12T08:46:15.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 -- start start 2022-07-12T08:46:15.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1bd409d370 0x7f1bd4145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:15.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd409e320 0x7f1bd41523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:15.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 0x7f1bd4134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:15.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40b3670 con 0x7f1bd40b6470 2022-07-12T08:46:15.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40b2cd0 con 0x7f1bd409e320 2022-07-12T08:46:15.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4152890 con 0x7f1bd409d370 2022-07-12T08:46:15.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 0x7f1bd4134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:15.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 0x7f1bd4134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56246/0 (socket says 172.21.15.61:56246) 2022-07-12T08:46:15.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 -- 172.21.15.61:0/2045448549 learned_addr learned my addr 172.21.15.61:0/2045448549 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:15.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bd9722700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd409e320 0x7f1bd41523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:15.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bd9f23700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1bd409d370 0x7f1bd4145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:15.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 -- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1bd409d370 msgr2=0x7f1bd4145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:15.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1bd409d370 0x7f1bd4145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 -- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd409e320 msgr2=0x7f1bd41523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:15.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd409e320 0x7f1bd41523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:15.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 -- 172.21.15.61:0/2045448549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd000b120 con 0x7f1bd40b6470 2022-07-12T08:46:15.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bda724700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 0x7f1bd4134480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f1bcc006e00 tx=0x7f1bcc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:15.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bcc00c640 con 0x7f1bd40b6470 2022-07-12T08:46:15.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc00c7a0 con 0x7f1bd40b6470 2022-07-12T08:46:15.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bcc00cb00 con 0x7f1bd40b6470 2022-07-12T08:46:15.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd40b2f30 con 0x7f1bd40b6470 2022-07-12T08:46:15.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.320+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd40af5d0 con 0x7f1bd40b6470 2022-07-12T08:46:15.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.324+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd405d2e0 con 0x7f1bd40b6470 2022-07-12T08:46:15.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.324+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1bcc022070 con 0x7f1bd40b6470 2022-07-12T08:46:15.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.324+0000 7f1bc2ffd700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bb4053ad0 0x7f1bb4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:15.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.324+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1bcc06c180 con 0x7f1bd40b6470 2022-07-12T08:46:15.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.324+0000 7f1bd9f23700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bb4053ad0 0x7f1bb4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:15.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.324+0000 7f1bd9f23700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bb4053ad0 0x7f1bb4055f90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f1bd001a000 tx=0x7f1bd0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:15.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.328+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1bcc0433c0 con 0x7f1bd40b6470 2022-07-12T08:46:15.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:15.500+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f1bd4057460 con 0x7f1bd40b6470 2022-07-12T08:46:16.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.180+0000 7f1bc2ffd700 1 -- 172.21.15.61:0/2045448549 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v224) v1 ==== 150+0+0 (secure 0 0 0) 0x7f1bcc0469e0 con 0x7f1bd40b6470 2022-07-12T08:46:16.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.184+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bb4053ad0 msgr2=0x7f1bb4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.184+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bb4053ad0 0x7f1bb4055f90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f1bd001a000 tx=0x7f1bd0014000).stop 2022-07-12T08:46:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.184+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 msgr2=0x7f1bd4134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.184+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 0x7f1bd4134480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f1bcc006e00 tx=0x7f1bcc0091d0).stop 2022-07-12T08:46:16.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.184+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 shutdown_connections 2022-07-12T08:46:16.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bd40b6470 0x7f1bd4134480 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bb4053ad0 0x7f1bb4055f90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1bd409e320 0x7f1bd41523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 --2- 172.21.15.61:0/2045448549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1bd409d370 0x7f1bd4145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 >> 172.21.15.61:0/2045448549 conn(0x7f1bd40b9d50 msgr2=0x7f1bd40b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:16.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 shutdown_connections 2022-07-12T08:46:16.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.188+0000 7f1bdaf25700 1 -- 172.21.15.61:0/2045448549 wait complete. 2022-07-12T08:46:16.211 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:46:16.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 Processor -- start 2022-07-12T08:46:16.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 -- start start 2022-07-12T08:46:16.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040602f0 0x7f0e040ab130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0e04095900 0x7f0e040ab910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0413d460 con 0x7f0e040633f0 2022-07-12T08:46:16.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e04096b60 con 0x7f0e04095900 2022-07-12T08:46:16.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e0a3b2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04096e00 con 0x7f0e040602f0 2022-07-12T08:46:16.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0e04095900 0x7f0e040ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0e04095900 0x7f0e040ab910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56254/0 (socket says 172.21.15.61:56254) 2022-07-12T08:46:16.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 -- 172.21.15.61:0/3162213534 learned_addr learned my addr 172.21.15.61:0/3162213534 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:16.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e093b0700 1 --2- 172.21.15.61:0/3162213534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040602f0 0x7f0e040ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 -- 172.21.15.61:0/3162213534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040602f0 msgr2=0x7f0e040ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:16.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 --2- 172.21.15.61:0/3162213534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040602f0 0x7f0e040ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 -- 172.21.15.61:0/3162213534 >> v1:172.21.15.111:6789/0 conn(0x7f0e040633f0 legacy=0x7f0e0413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 -- 172.21.15.61:0/3162213534 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0413fab0 con 0x7f0e04095900 2022-07-12T08:46:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e08baf700 1 --2- 172.21.15.61:0/3162213534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e040ab910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f0df8002f70 tx=0x7f0df8006eb0).ready entity=mon.0 client_cookie=74ce99157c1000c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e03fff700 1 -- 172.21.15.61:0/3162213534 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0df800d570 con 0x7f0e04095900 2022-07-12T08:46:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e03fff700 1 -- 172.21.15.61:0/3162213534 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df8000e90 con 0x7f0e04095900 2022-07-12T08:46:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.320+0000 7f0e03fff700 1 -- 172.21.15.61:0/3162213534 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0df800d900 con 0x7f0e04095900 2022-07-12T08:46:16.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3162213534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 msgr2=0x7f0e040ab910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3162213534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e040ab910 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f0df8002f70 tx=0x7f0df8006eb0).stop 2022-07-12T08:46:16.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3162213534 shutdown_connections 2022-07-12T08:46:16.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3162213534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e040ab910 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3162213534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040602f0 0x7f0e040ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3162213534 >> 172.21.15.61:0/3162213534 conn(0x7f0e040b3600 msgr2=0x7f0e04060d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:16.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3162213534 shutdown_connections 2022-07-12T08:46:16.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3162213534 wait complete. 2022-07-12T08:46:16.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 Processor -- start 2022-07-12T08:46:16.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.324+0000 7f0e0a3b2700 1 -- start start 2022-07-12T08:46:16.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e040602f0 0x7f0e0413ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040633f0 0x7f0e04138e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e0413ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e040b54d0 con 0x7f0e04095900 2022-07-12T08:46:16.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0413c2a0 con 0x7f0e040602f0 2022-07-12T08:46:16.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e04152750 con 0x7f0e040633f0 2022-07-12T08:46:16.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e0413ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e08baf700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040633f0 0x7f0e04138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e093b0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e040602f0 0x7f0e0413ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e0413ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56262/0 (socket says 172.21.15.61:56262) 2022-07-12T08:46:16.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 -- 172.21.15.61:0/3642892521 learned_addr learned my addr 172.21.15.61:0/3642892521 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 -- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040633f0 msgr2=0x7f0e04138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040633f0 0x7f0e04138e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 -- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e040602f0 msgr2=0x7f0e0413ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e040602f0 0x7f0e0413ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 -- 172.21.15.61:0/3642892521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df800b1e0 con 0x7f0e04095900 2022-07-12T08:46:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e09bb1700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e0413ac00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f0e0413c430 tx=0x7f0dfc007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dfc012400 con 0x7f0e04095900 2022-07-12T08:46:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc012560 con 0x7f0e04095900 2022-07-12T08:46:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dfc0128c0 con 0x7f0e04095900 2022-07-12T08:46:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e041529f0 con 0x7f0e04095900 2022-07-12T08:46:16.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.328+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0413da80 con 0x7f0e04095900 2022-07-12T08:46:16.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.332+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0dfc019030 con 0x7f0e04095900 2022-07-12T08:46:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.332+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e04057460 con 0x7f0e04095900 2022-07-12T08:46:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.332+0000 7f0e027fc700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0de4053ad0 0x7f0de4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.332+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0dfc06ae10 con 0x7f0e04095900 2022-07-12T08:46:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.332+0000 7f0e093b0700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0de4053ad0 0x7f0de4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.332+0000 7f0e093b0700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0de4053ad0 0x7f0de4055f90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f0df400b000 tx=0x7f0df40091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.340+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0dfc042050 con 0x7f0e04095900 2022-07-12T08:46:16.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.468+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0e04062340 con 0x7f0e04095900 2022-07-12T08:46:16.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.476+0000 7f0e027fc700 1 -- 172.21.15.61:0/3642892521 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0dfc045670 con 0x7f0e04095900 2022-07-12T08:46:16.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0de4053ad0 msgr2=0x7f0de4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0de4053ad0 0x7f0de4055f90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f0df400b000 tx=0x7f0df40091d0).stop 2022-07-12T08:46:16.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 msgr2=0x7f0e0413ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e0413ac00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f0e0413c430 tx=0x7f0dfc007b90).stop 2022-07-12T08:46:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 shutdown_connections 2022-07-12T08:46:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e04095900 0x7f0e0413ac00 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e040633f0 0x7f0e04138e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0de4053ad0 0x7f0de4055f90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 --2- 172.21.15.61:0/3642892521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e040602f0 0x7f0e0413ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 >> 172.21.15.61:0/3642892521 conn(0x7f0e040b3600 msgr2=0x7f0e04061350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:16.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 shutdown_connections 2022-07-12T08:46:16.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.480+0000 7f0e0a3b2700 1 -- 172.21.15.61:0/3642892521 wait complete. 2022-07-12T08:46:16.494 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:46:16.495 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:16.518 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:16.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:16.575 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:16.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:16.628 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:46:16.629 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:16.630 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:16.630 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:16.639 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:46:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:16.642 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:46:16.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:16.644 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:16.645 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:46:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:16.656 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:46:16.657 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:46:16.658 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:16.659 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:16.659 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:46:16.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:16.674 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:46:16.674 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:46:16.675 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:46:16.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 Processor -- start 2022-07-12T08:46:16.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- start start 2022-07-12T08:46:16.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fddc40b5aa0 0x7fddc40b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc4147000 0x7fddc41474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddc405d3d0 con 0x7fddc405bfc0 2022-07-12T08:46:16.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddc414b590 con 0x7fddc40b5aa0 2022-07-12T08:46:16.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40b5e80 con 0x7fddc4147000 2022-07-12T08:46:16.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fddc40b5aa0 0x7fddc40b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fddc40b5aa0 0x7fddc40b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56266/0 (socket says 172.21.15.61:56266) 2022-07-12T08:46:16.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 -- 172.21.15.61:0/3608763799 learned_addr learned my addr 172.21.15.61:0/3608763799 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc3fff700 1 --2- 172.21.15.61:0/3608763799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc4147000 0x7fddc41474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 -- 172.21.15.61:0/3608763799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc4147000 msgr2=0x7fddc41474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 --2- 172.21.15.61:0/3608763799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc4147000 0x7fddc41474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 -- 172.21.15.61:0/3608763799 >> v1:172.21.15.111:6789/0 conn(0x7fddc405bfc0 legacy=0x7fddc414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 -- 172.21.15.61:0/3608763799 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc414a740 con 0x7fddc40b5aa0 2022-07-12T08:46:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc37fe700 1 --2- 172.21.15.61:0/3608763799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc40b5aa0 0x7fddc40b0380 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fddb4002f70 tx=0x7fddb4006eb0).ready entity=mon.0 client_cookie=c42bb71b138a56 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc2ffd700 1 -- 172.21.15.61:0/3608763799 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddb400d5c0 con 0x7fddc40b5aa0 2022-07-12T08:46:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc2ffd700 1 -- 172.21.15.61:0/3608763799 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb4000e90 con 0x7fddc40b5aa0 2022-07-12T08:46:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddc2ffd700 1 -- 172.21.15.61:0/3608763799 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddb400d950 con 0x7fddc40b5aa0 2022-07-12T08:46:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- 172.21.15.61:0/3608763799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc40b5aa0 msgr2=0x7fddc40b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 --2- 172.21.15.61:0/3608763799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc40b5aa0 0x7fddc40b0380 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fddb4002f70 tx=0x7fddb4006eb0).stop 2022-07-12T08:46:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- 172.21.15.61:0/3608763799 shutdown_connections 2022-07-12T08:46:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 --2- 172.21.15.61:0/3608763799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc4147000 0x7fddc41474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 --2- 172.21.15.61:0/3608763799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc40b5aa0 0x7fddc40b0380 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- 172.21.15.61:0/3608763799 >> 172.21.15.61:0/3608763799 conn(0x7fddc4063520 msgr2=0x7fddc40b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- 172.21.15.61:0/3608763799 shutdown_connections 2022-07-12T08:46:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.784+0000 7fddca0d0700 1 -- 172.21.15.61:0/3608763799 wait complete. 2022-07-12T08:46:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.788+0000 7fddca0d0700 1 Processor -- start 2022-07-12T08:46:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.788+0000 7fddca0d0700 1 -- start start 2022-07-12T08:46:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.788+0000 7fddca0d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddc405bfc0 0x7fddc409e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc40b5aa0 0x7fddc40a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 0x7fddc40a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40a0d60 con 0x7fddc4147000 2022-07-12T08:46:16.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc40a1000 con 0x7fddc405bfc0 2022-07-12T08:46:16.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc4066140 con 0x7fddc40b5aa0 2022-07-12T08:46:16.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 0x7fddc40a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 0x7fddc40a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56274/0 (socket says 172.21.15.61:56274) 2022-07-12T08:46:16.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 -- 172.21.15.61:0/52236587 learned_addr learned my addr 172.21.15.61:0/52236587 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:16.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddbaffd700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc40b5aa0 0x7fddc40a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc37fe700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddc405bfc0 0x7fddc409e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 -- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc40b5aa0 msgr2=0x7fddc40a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc40b5aa0 0x7fddc40a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 -- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddc405bfc0 msgr2=0x7fddc409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:16.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddc405bfc0 0x7fddc409e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:16.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 -- 172.21.15.61:0/52236587 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb400b1e0 con 0x7fddc4147000 2022-07-12T08:46:16.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc3fff700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 0x7fddc40a17b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fddbc00a000 tx=0x7fddbc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:16.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddbc0127e0 con 0x7fddc4147000 2022-07-12T08:46:16.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddbc012940 con 0x7fddc4147000 2022-07-12T08:46:16.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddc40656d0 con 0x7fddc4147000 2022-07-12T08:46:16.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddbc012ca0 con 0x7fddc4147000 2022-07-12T08:46:16.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc40a1ec0 con 0x7fddc4147000 2022-07-12T08:46:16.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc4066300 con 0x7fddc4147000 2022-07-12T08:46:16.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.792+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fddbc013480 con 0x7fddc4147000 2022-07-12T08:46:16.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.796+0000 7fddc17fa700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdda8053920 0x7fdda8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:16.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.796+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fddbc06bfa0 con 0x7fddc4147000 2022-07-12T08:46:16.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.796+0000 7fddc37fe700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdda8053920 0x7fdda8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:16.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.796+0000 7fddc37fe700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdda8053920 0x7fdda8055de0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fddb401c000 tx=0x7fddb4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:16.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.800+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fddbc043070 con 0x7fddc4147000 2022-07-12T08:46:16.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.928+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fddc4057460 con 0x7fddc4147000 2022-07-12T08:46:16.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.928+0000 7fddc17fa700 1 -- 172.21.15.61:0/52236587 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v56) v1 ==== 81+0+28876 (secure 0 0 0) 0x7fddbc012e00 con 0x7fddc4147000 2022-07-12T08:46:16.932 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:16.934 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:46:16.935 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 56, 2022-07-12T08:46:16.935 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:46:16.936 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:46:16.936 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:46:14.167984+0000", 2022-07-12T08:46:16.937 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:16.937 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:16.938 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:46:16.938 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:46:16.939 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:46:16.939 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:46:16.940 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:46:16.941 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:46:16.941 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:46:16.942 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:16.943 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:46:16.944 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:46:16.944 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:46:16.945 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:46:16.945 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:46:16.946 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 10, 2022-07-12T08:46:16.946 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:46:16.947 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:46:16.947 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:46:16.948 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:46:16.948 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:46:16.950 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:16.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:46:16.951 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:46:16.951 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:46:16.952 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:16.952 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:16.953 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:16.953 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:16.953 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:16.954 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:16.954 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:16.954 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:16.955 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:46:16.955 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:46:16.956 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:46:16.956 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:46:16.956 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:46:16.957 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:16.957 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:16.957 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:16.958 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:16.958 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:16.958 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:16.959 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:16.959 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:16.960 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:46:16.960 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:16.960 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:16.961 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:16.961 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:16.962 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:16.962 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:16.962 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:16.963 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:16.964 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:16.964 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:16.965 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:16.965 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:16.965 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:16.971 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:16.972 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:16.973 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:16.973 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:16.973 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:16.974 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:16.975 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:16.975 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:16.976 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:16.976 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:16.976 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:16.978 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:16.978 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:16.980 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:16.980 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:16.983 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:16.983 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:46:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:46:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:46:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:16.986 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:16.986 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:46:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:46:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:46:16.988 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:16.998 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:16.999 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:16.999 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:16.999 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:17.000 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:17.000 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:17.000 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:17.001 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:17.001 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:17.002 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:17.002 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:17.003 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:17.003 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:17.004 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:17.004 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:17.004 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:17.005 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:17.005 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:17.007 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:17.008 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:46:17.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:17.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:17.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:17.010 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:17.011 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:17.011 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:17.011 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:17.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:17.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:17.016 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:17.016 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:17.018 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:17.018 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:17.018 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.019 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:17.019 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:17.020 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:17.020 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:17.020 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:17.020 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:17.022 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:17.022 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:46:17.022 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:17.022 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:46:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:46:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.024 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:17.024 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:46:17.024 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.024 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:46:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:46:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:46:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:46:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.033 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.033 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:46:17.035 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:46:17.037 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.037 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.038 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.038 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.039 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.041 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.041 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:46:17.042 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.042 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:46:17.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:17.044 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.045 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.046 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:46:17.046 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:46:17.046 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:46:17.047 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:46:17.047 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.050 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:17.050 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:46:17.051 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.051 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.052 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.052 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.053 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.053 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.053 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.054 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:17.054 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.055 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.055 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.056 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:46:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.058 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:46:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.061 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.061 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.062 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:46:17.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.071 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:46:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.075 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:46:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.079 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.080 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:46:17.080 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.081 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.081 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.082 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.082 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.083 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.083 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.084 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:46:17.084 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.085 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:46:17.085 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:17.087 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.087 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.088 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.088 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:46:17.089 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:46:17.089 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:46:17.089 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:46:17.090 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.090 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.091 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.091 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.092 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:17.093 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:46:17.093 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.094 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.094 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.095 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.095 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.096 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.096 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.097 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 51, 2022-07-12T08:46:17.097 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.098 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.098 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.099 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:46:17.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.102 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:46:17.103 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.104 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.104 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.105 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.105 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.106 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:46:17.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.107 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:46:17.109 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.112 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:46:17.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.114 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.114 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.115 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:46:17.115 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.116 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.116 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.117 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.118 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.119 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:46:17.120 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.120 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.121 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.121 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:46:17.122 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:17.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.123 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.124 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:46:17.124 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:46:17.125 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:46:17.125 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:46:17.125 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.126 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.126 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.126 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.127 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.127 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.127 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:17.128 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:46:17.128 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.129 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.130 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.130 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.131 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.131 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.131 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.132 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-12T08:46:17.132 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.133 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.133 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.134 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.134 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:46:17.135 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:46:17.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.137 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.137 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.138 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:46:17.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:46:17.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.141 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.141 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.142 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:46:17.142 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.143 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.143 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:46:17.144 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.145 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.145 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.146 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.147 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.148 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:46:17.148 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.149 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.150 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.150 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:46:17.151 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:17.151 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.152 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.152 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.152 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:46:17.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:46:17.153 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:46:17.154 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:46:17.154 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.155 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.155 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.155 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.156 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.157 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:17.158 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:46:17.158 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.159 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.159 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.159 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.160 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.160 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.161 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.161 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:17.162 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.162 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.163 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.163 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.164 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.164 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.164 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:46:17.165 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.165 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.166 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.167 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:46:17.167 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.168 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.168 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.169 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.169 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:46:17.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.173 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:46:17.174 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.174 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.176 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.176 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.178 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:46:17.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.179 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.180 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.180 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:46:17.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.181 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.182 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.182 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.182 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.183 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.183 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.183 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:46:17.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.185 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.185 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.186 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:46:17.186 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:17.186 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.187 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.187 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.188 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:46:17.188 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:46:17.188 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:46:17.189 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:46:17.189 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.189 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.189 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.190 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.190 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.191 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.191 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:17.192 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:46:17.192 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.192 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.193 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.193 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.194 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.194 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.195 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.195 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-12T08:46:17.195 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.196 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.196 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.197 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:46:17.199 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.199 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.200 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.200 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:46:17.201 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.201 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.201 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.202 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.202 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.203 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.203 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:46:17.204 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.205 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:46:17.206 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.207 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.207 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.208 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.209 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.210 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:46:17.210 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.211 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:46:17.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.212 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.212 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.213 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.213 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:46:17.214 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.214 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.214 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:46:17.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:17.215 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.216 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.216 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:46:17.216 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:46:17.216 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:46:17.217 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:46:17.217 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.217 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.218 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.218 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.218 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.219 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:17.219 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:46:17.219 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.219 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.219 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.220 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.220 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.221 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.221 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 51, 2022-07-12T08:46:17.222 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.222 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.223 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.224 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:46:17.225 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.226 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.226 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.226 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:46:17.227 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.227 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.228 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.229 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.238 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:46:17.239 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.240 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:46:17.241 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.241 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.241 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.242 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:46:17.244 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:46:17.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.246 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.247 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.248 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:46:17.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:46:17.252 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:17.252 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.253 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.254 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:46:17.254 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:46:17.255 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:46:17.255 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:46:17.255 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.256 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.256 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.257 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.258 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.259 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:17.260 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:46:17.260 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:17.261 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:17.261 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:17.261 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:17.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:17.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:17.262 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:17.262 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-12T08:46:17.262 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:17.263 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:17.263 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:17.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:46:17.265 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.266 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.267 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:46:17.267 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.269 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.270 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:17.270 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.271 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.271 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.271 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:46:17.271 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.272 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.272 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.272 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:46:17.273 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.273 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.274 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:17.275 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.275 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.276 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.276 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:46:17.277 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:46:17.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.280 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.280 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:17.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:17.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.281 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:17.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:46:17.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:17.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:46:17.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:17.283 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.283 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.284 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:46:17.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:46:17.284 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:46:17.284 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:46:17.285 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:17.285 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:17.285 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:17.285 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:17.286 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.286 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:17.286 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:46:17.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.287 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:17.287 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.287 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.287 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.288 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.288 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.288 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:46:17.289 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.289 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.289 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:17.290 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.290 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.290 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.290 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.291 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:46:17.292 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.293 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:17.293 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.293 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.293 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.294 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.294 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:46:17.294 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.295 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.295 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:17.295 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.295 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.295 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.296 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.296 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.296 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:46:17.296 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.297 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.297 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:17.297 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.297 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.298 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.298 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.298 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.298 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:46:17.299 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.299 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.299 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:17.300 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.300 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.300 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.300 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.301 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.301 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:46:17.301 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.301 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.302 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.302 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:17.302 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.302 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.303 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.303 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.303 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.303 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:46:17.303 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.304 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:17.304 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:17.304 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:17.304 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:17.305 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:17.305 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:17.305 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:17.305 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:46:17.306 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:17.306 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.306 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:17.306 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:46:17.306 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:46:17.307 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:46:17.307 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:46:17.307 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:46:17.307 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/4155742423": "2022-07-13T08:46:10.747830+0000", 2022-07-12T08:46:17.308 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/4155742423": "2022-07-13T08:46:10.747830+0000" 2022-07-12T08:46:17.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.308 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:46:17.308 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:46:17.309 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:46:17.309 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:46:17.309 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:46:17.309 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:46:17.309 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:46:17.310 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:46:17.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:17.310 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:17.311 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:46:17.311 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:46:17.311 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:46:17.311 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:46:17.312 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:46:17.312 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:46:17.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdda8053920 msgr2=0x7fdda8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdda8053920 0x7fdda8055de0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7fddb401c000 tx=0x7fddb4015000).stop 2022-07-12T08:46:17.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 msgr2=0x7fddc40a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 0x7fddc40a17b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fddbc00a000 tx=0x7fddbc008040).stop 2022-07-12T08:46:17.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 shutdown_connections 2022-07-12T08:46:17.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddc4147000 0x7fddc40a17b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fddc40b5aa0 0x7fddc40a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdda8053920 0x7fdda8055de0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 --2- 172.21.15.61:0/52236587 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddc405bfc0 0x7fddc409e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 >> 172.21.15.61:0/52236587 conn(0x7fddc4063520 msgr2=0x7fddc4069e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 shutdown_connections 2022-07-12T08:46:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:16.932+0000 7fddca0d0700 1 -- 172.21.15.61:0/52236587 wait complete. 2022-07-12T08:46:17.318 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:17.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 Processor -- start 2022-07-12T08:46:17.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 -- start start 2022-07-12T08:46:17.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4d440b1080 0x7f4d440b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4d44142f30 0x7f4d440afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440b5740 con 0x7f4d440b4090 2022-07-12T08:46:17.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4405cc00 con 0x7f4d44142f30 2022-07-12T08:46:17.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.428+0000 7f4d4af45700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440b1460 con 0x7f4d440b1080 2022-07-12T08:46:17.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d494e2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4d44142f30 0x7f4d440afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d43fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4d440b4090 0x7f4d4405c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44496/0 (socket says 172.21.15.61:44496) 2022-07-12T08:46:17.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d43fff700 1 -- 172.21.15.61:0/2788499541 learned_addr learned my addr 172.21.15.61:0/2788499541 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:17.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d48ce1700 1 --2- 172.21.15.61:0/2788499541 >> v2:172.21.15.111:3300/0 conn(0x7f4d440b1080 0x7f4d440b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d494e2700 1 -- 172.21.15.61:0/2788499541 >> v2:172.21.15.111:3300/0 conn(0x7f4d440b1080 msgr2=0x7f4d440b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:17.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d494e2700 1 --2- 172.21.15.61:0/2788499541 >> v2:172.21.15.111:3300/0 conn(0x7f4d440b1080 0x7f4d440b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d494e2700 1 -- 172.21.15.61:0/2788499541 >> v1:172.21.15.61:6789/0 conn(0x7f4d440b4090 legacy=0x7f4d4405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d437fe700 1 -- 172.21.15.61:0/2788499541 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646848225 0 0) 0x7f4d440b5740 con 0x7f4d440b4090 2022-07-12T08:46:17.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d494e2700 1 -- 172.21.15.61:0/2788499541 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d440b6880 con 0x7f4d44142f30 2022-07-12T08:46:17.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d494e2700 1 --2- 172.21.15.61:0/2788499541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d440afb90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f4d3c0032f0 tx=0x7f4d3c0091d0).ready entity=mon.0 client_cookie=8d6360c5b9416629 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:17.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d437fe700 1 -- 172.21.15.61:0/2788499541 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c00b680 con 0x7f4d44142f30 2022-07-12T08:46:17.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d437fe700 1 -- 172.21.15.61:0/2788499541 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c00b7e0 con 0x7f4d44142f30 2022-07-12T08:46:17.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d437fe700 1 -- 172.21.15.61:0/2788499541 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c00b950 con 0x7f4d44142f30 2022-07-12T08:46:17.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 -- 172.21.15.61:0/2788499541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 msgr2=0x7f4d440afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 --2- 172.21.15.61:0/2788499541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d440afb90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f4d3c0032f0 tx=0x7f4d3c0091d0).stop 2022-07-12T08:46:17.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 -- 172.21.15.61:0/2788499541 shutdown_connections 2022-07-12T08:46:17.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 --2- 172.21.15.61:0/2788499541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d440afb90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 --2- 172.21.15.61:0/2788499541 >> v2:172.21.15.111:3300/0 conn(0x7f4d440b1080 0x7f4d440b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 -- 172.21.15.61:0/2788499541 >> 172.21.15.61:0/2788499541 conn(0x7f4d44067020 msgr2=0x7f4d44067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:17.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 -- 172.21.15.61:0/2788499541 shutdown_connections 2022-07-12T08:46:17.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.432+0000 7f4d4af45700 1 -- 172.21.15.61:0/2788499541 wait complete. 2022-07-12T08:46:17.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 Processor -- start 2022-07-12T08:46:17.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 -- start start 2022-07-12T08:46:17.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b1080 0x7f4d44112c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440b4090 0x7f4d44141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d44142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440ae730 con 0x7f4d44142f30 2022-07-12T08:46:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44113110 con 0x7f4d440b1080 2022-07-12T08:46:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d4af45700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44113350 con 0x7f4d440b4090 2022-07-12T08:46:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d494e2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d44142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d494e2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d44142410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56288/0 (socket says 172.21.15.61:56288) 2022-07-12T08:46:17.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.436+0000 7f4d494e2700 1 -- 172.21.15.61:0/1827919407 learned_addr learned my addr 172.21.15.61:0/1827919407 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d43fff700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440b4090 0x7f4d44141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d48ce1700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b1080 0x7f4d44112c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d494e2700 1 -- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440b4090 msgr2=0x7f4d44141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d494e2700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440b4090 0x7f4d44141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d494e2700 1 -- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b1080 msgr2=0x7f4d44112c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d494e2700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b1080 0x7f4d44112c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d494e2700 1 -- 172.21.15.61:0/1827919407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3c009470 con 0x7f4d44142f30 2022-07-12T08:46:17.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d43fff700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440b4090 0x7f4d44141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:17.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d494e2700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d44142410 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f4d3c0032f0 tx=0x7f4d3c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:17.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c00b950 con 0x7f4d44142f30 2022-07-12T08:46:17.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c021630 con 0x7f4d44142f30 2022-07-12T08:46:17.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c00b680 con 0x7f4d44142f30 2022-07-12T08:46:17.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d441135f0 con 0x7f4d44142f30 2022-07-12T08:46:17.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440af620 con 0x7f4d44142f30 2022-07-12T08:46:17.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.440+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d440af1c0 con 0x7f4d44142f30 2022-07-12T08:46:17.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.444+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4d3c00f070 con 0x7f4d44142f30 2022-07-12T08:46:17.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.444+0000 7f4d41ffb700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053ad0 0x7f4d24055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.444+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4d3c06c850 con 0x7f4d44142f30 2022-07-12T08:46:17.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.444+0000 7f4d48ce1700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053ad0 0x7f4d24055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.444+0000 7f4d48ce1700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053ad0 0x7f4d24055f90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4d3400b000 tx=0x7f4d34006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:17.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.452+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4d440af1c0 con 0x7f4d44142f30 2022-07-12T08:46:17.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.612+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d440af1c0 con 0x7f4d44142f30 2022-07-12T08:46:17.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.612+0000 7f4d41ffb700 1 -- 172.21.15.61:0/1827919407 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f4d440af1c0 con 0x7f4d44142f30 2022-07-12T08:46:17.619 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:17.620 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3933421999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3933421999},{"type":"v1","addr":"172.21.15.61:6833","nonce":3933421999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27}],"filesystems":[]} 2022-07-12T08:46:17.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053ad0 msgr2=0x7f4d24055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053ad0 0x7f4d24055f90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f4d3400b000 tx=0x7f4d34006df0).stop 2022-07-12T08:46:17.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 msgr2=0x7f4d44142410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d44142410 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f4d3c0032f0 tx=0x7f4d3c016000).stop 2022-07-12T08:46:17.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 shutdown_connections 2022-07-12T08:46:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44142f30 0x7f4d44142410 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440b4090 0x7f4d44141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053ad0 0x7f4d24055f90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 --2- 172.21.15.61:0/1827919407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b1080 0x7f4d44112c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 >> 172.21.15.61:0/1827919407 conn(0x7f4d44067020 msgr2=0x7f4d441433a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:17.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 shutdown_connections 2022-07-12T08:46:17.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.616+0000 7f4d4af45700 1 -- 172.21.15.61:0/1827919407 wait complete. 2022-07-12T08:46:17.627 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 27 2022-07-12T08:46:17.632 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:17.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.732+0000 7f8cbd098700 1 Processor -- start 2022-07-12T08:46:17.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.732+0000 7f8cbd098700 1 -- start start 2022-07-12T08:46:17.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8cb80b1320 0x7f8cb80ae500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8cb813ec70 0x7f8cb813f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80acee0 con 0x7f8cb80b0120 2022-07-12T08:46:17.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80ad180 con 0x7f8cb813ec70 2022-07-12T08:46:17.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80acb60 con 0x7f8cb80b1320 2022-07-12T08:46:17.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8cb813ec70 0x7f8cb813f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8cb813ec70 0x7f8cb813f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56296/0 (socket says 172.21.15.61:56296) 2022-07-12T08:46:17.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 -- 172.21.15.61:0/2018266299 learned_addr learned my addr 172.21.15.61:0/2018266299 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:17.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cb7fff700 1 --2- 172.21.15.61:0/2018266299 >> v2:172.21.15.111:3300/0 conn(0x7f8cb80b1320 0x7f8cb80ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 -- 172.21.15.61:0/2018266299 >> v2:172.21.15.111:3300/0 conn(0x7f8cb80b1320 msgr2=0x7f8cb80ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:17.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 --2- 172.21.15.61:0/2018266299 >> v2:172.21.15.111:3300/0 conn(0x7f8cb80b1320 0x7f8cb80ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 -- 172.21.15.61:0/2018266299 >> v1:172.21.15.111:6789/0 conn(0x7f8cb80b0120 legacy=0x7f8cb80ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 -- 172.21.15.61:0/2018266299 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80b0630 con 0x7f8cb813ec70 2022-07-12T08:46:17.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbc897700 1 --2- 172.21.15.61:0/2018266299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb813f100 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f8cb00032f0 tx=0x7f8cb0009290).ready entity=mon.0 client_cookie=45b222c94d4c3730 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:17.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cb77fe700 1 -- 172.21.15.61:0/2018266299 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb000b870 con 0x7f8cb813ec70 2022-07-12T08:46:17.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cb77fe700 1 -- 172.21.15.61:0/2018266299 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb000b9d0 con 0x7f8cb813ec70 2022-07-12T08:46:17.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cb77fe700 1 -- 172.21.15.61:0/2018266299 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb000bb40 con 0x7f8cb813ec70 2022-07-12T08:46:17.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- 172.21.15.61:0/2018266299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 msgr2=0x7f8cb813f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 --2- 172.21.15.61:0/2018266299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb813f100 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f8cb00032f0 tx=0x7f8cb0009290).stop 2022-07-12T08:46:17.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- 172.21.15.61:0/2018266299 shutdown_connections 2022-07-12T08:46:17.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 --2- 172.21.15.61:0/2018266299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb813f100 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 --2- 172.21.15.61:0/2018266299 >> v2:172.21.15.111:3300/0 conn(0x7f8cb80b1320 0x7f8cb80ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- 172.21.15.61:0/2018266299 >> 172.21.15.61:0/2018266299 conn(0x7f8cb80b5780 msgr2=0x7f8cb80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- 172.21.15.61:0/2018266299 shutdown_connections 2022-07-12T08:46:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.736+0000 7f8cbd098700 1 -- 172.21.15.61:0/2018266299 wait complete. 2022-07-12T08:46:17.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 Processor -- start 2022-07-12T08:46:17.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 -- start start 2022-07-12T08:46:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cb80b0120 0x7f8cb81385c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cb80b1320 0x7f8cb813c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb8065160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb8133a00 con 0x7f8cb813ec70 2022-07-12T08:46:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb813ac20 con 0x7f8cb80b0120 2022-07-12T08:46:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbd098700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb813a240 con 0x7f8cb80b1320 2022-07-12T08:46:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb8065160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb8065160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56302/0 (socket says 172.21.15.61:56302) 2022-07-12T08:46:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 -- 172.21.15.61:0/3760120278 learned_addr learned my addr 172.21.15.61:0/3760120278 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cb7fff700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cb80b0120 0x7f8cb81385c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8caffff700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cb80b1320 0x7f8cb813c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 -- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cb80b1320 msgr2=0x7f8cb813c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cb80b1320 0x7f8cb813c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 -- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cb80b0120 msgr2=0x7f8cb81385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cb80b0120 0x7f8cb81385c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.740+0000 7f8cbc897700 1 -- 172.21.15.61:0/3760120278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb0009530 con 0x7f8cb813ec70 2022-07-12T08:46:17.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cbc897700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb8065160 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f8cb00032f0 tx=0x7f8cb0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:17.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb000b870 con 0x7f8cb813ec70 2022-07-12T08:46:17.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0021480 con 0x7f8cb813ec70 2022-07-12T08:46:17.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb813a4e0 con 0x7f8cb813ec70 2022-07-12T08:46:17.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cb00217e0 con 0x7f8cb813ec70 2022-07-12T08:46:17.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb8139ec0 con 0x7f8cb813ec70 2022-07-12T08:46:17.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb8057460 con 0x7f8cb813ec70 2022-07-12T08:46:17.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8cb000f070 con 0x7f8cb813ec70 2022-07-12T08:46:17.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb5ffb700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c9c0539b0 0x7f8c9c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:17.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8cb006cfe0 con 0x7f8cb813ec70 2022-07-12T08:46:17.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.744+0000 7f8cb7fff700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c9c0539b0 0x7f8c9c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:17.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.748+0000 7f8cb7fff700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c9c0539b0 0x7f8c9c055e70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8ca800b000 tx=0x7f8ca8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:17.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.752+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8cb0047e60 con 0x7f8cb813ec70 2022-07-12T08:46:17.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.920+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cb8068400 con 0x7f8cb813ec70 2022-07-12T08:46:17.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.920+0000 7f8cb5ffb700 1 -- 172.21.15.61:0/3760120278 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f8cb0015070 con 0x7f8cb813ec70 2022-07-12T08:46:17.924 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:17.924 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1234119172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1234119172},{"type":"v1","addr":"172.21.15.111:6835","nonce":1234119172}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":21},{"gid":5889,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3933421999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3933421999},{"type":"v1","addr":"172.21.15.61:6833","nonce":3933421999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":27}],"filesystems":[]} 2022-07-12T08:46:17.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c9c0539b0 msgr2=0x7f8c9c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c9c0539b0 0x7f8c9c055e70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f8ca800b000 tx=0x7f8ca8006df0).stop 2022-07-12T08:46:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 msgr2=0x7f8cb8065160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb8065160 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f8cb00032f0 tx=0x7f8cb0016000).stop 2022-07-12T08:46:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 shutdown_connections 2022-07-12T08:46:17.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cb813ec70 0x7f8cb8065160 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cb80b1320 0x7f8cb813c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c9c0539b0 0x7f8c9c055e70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 --2- 172.21.15.61:0/3760120278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cb80b0120 0x7f8cb81385c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:17.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 >> 172.21.15.61:0/3760120278 conn(0x7f8cb80b5780 msgr2=0x7f8cb80b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:17.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 shutdown_connections 2022-07-12T08:46:17.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:17.924+0000 7f8cbd098700 1 -- 172.21.15.61:0/3760120278 wait complete. 2022-07-12T08:46:17.929 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 27 2022-07-12T08:46:17.938 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:46:17.938 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:46:17.939 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:46:17.939 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:46:17.940 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:46:17.940 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:46:17.941 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:46:17.941 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:46:17.955 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:46:17.956 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:46:17.958 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:46:17.959 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:46:17.962 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:46:17.963 DEBUG:teuthology.parallel:result is None 2022-07-12T08:46:18.005 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:46:18.006 DEBUG:teuthology.parallel:result is None 2022-07-12T08:46:18.007 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:46:18.039 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:46:18.080 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:46:18.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 Processor -- start 2022-07-12T08:46:18.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 -- start start 2022-07-12T08:46:18.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f0146790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:18.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f69f005cdd0 0x7f69f0141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:18.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f00b6b50 con 0x7f69f00b6770 2022-07-12T08:46:18.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f00b4f10 con 0x7f69f005cdd0 2022-07-12T08:46:18.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f715e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f00b51b0 con 0x7f69f00b4720 2022-07-12T08:46:18.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f0146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:18.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f0146790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38516/0 (socket says 172.21.15.61:38516) 2022-07-12T08:46:18.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 -- 172.21.15.61:0/3386512938 learned_addr learned my addr 172.21.15.61:0/3386512938 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:18.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f695d700 1 --2- 172.21.15.61:0/3386512938 >> v2:172.21.15.111:3300/0 conn(0x7f69f005cdd0 0x7f69f0141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:18.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 -- 172.21.15.61:0/3386512938 >> v2:172.21.15.111:3300/0 conn(0x7f69f005cdd0 msgr2=0x7f69f0141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:18.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 --2- 172.21.15.61:0/3386512938 >> v2:172.21.15.111:3300/0 conn(0x7f69f005cdd0 0x7f69f0141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:18.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 -- 172.21.15.61:0/3386512938 >> v1:172.21.15.111:6789/0 conn(0x7f69f00b6770 legacy=0x7f69f0146080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:18.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 -- 172.21.15.61:0/3386512938 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f00b5450 con 0x7f69f00b4720 2022-07-12T08:46:18.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.112+0000 7f69f595b700 1 --2- 172.21.15.61:0/3386512938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f0146790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f69ec002f70 tx=0x7f69ec006e50).ready entity=mon.2 client_cookie=e6ee01618b8509b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:18.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f515a700 1 -- 172.21.15.61:0/3386512938 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69ec00d5c0 con 0x7f69f00b4720 2022-07-12T08:46:18.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f515a700 1 -- 172.21.15.61:0/3386512938 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec000e90 con 0x7f69f00b4720 2022-07-12T08:46:18.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f515a700 1 -- 172.21.15.61:0/3386512938 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69ec00d950 con 0x7f69f00b4720 2022-07-12T08:46:18.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 -- 172.21.15.61:0/3386512938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 msgr2=0x7f69f0146790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:18.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 --2- 172.21.15.61:0/3386512938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f0146790 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f69ec002f70 tx=0x7f69ec006e50).stop 2022-07-12T08:46:18.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 -- 172.21.15.61:0/3386512938 shutdown_connections 2022-07-12T08:46:18.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 --2- 172.21.15.61:0/3386512938 >> v2:172.21.15.111:3300/0 conn(0x7f69f005cdd0 0x7f69f0141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:18.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 --2- 172.21.15.61:0/3386512938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f0146790 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:18.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 -- 172.21.15.61:0/3386512938 >> 172.21.15.61:0/3386512938 conn(0x7f69f00b9d50 msgr2=0x7f69f00b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:18.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 -- 172.21.15.61:0/3386512938 shutdown_connections 2022-07-12T08:46:18.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.116+0000 7f69f715e700 1 -- 172.21.15.61:0/3386512938 wait complete. 2022-07-12T08:46:18.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 Processor -- start 2022-07-12T08:46:18.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 -- start start 2022-07-12T08:46:18.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f005cdd0 0x7f69f0155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:18.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f00b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:18.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 0x7f69f0151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:18.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0155160 con 0x7f69f00b6770 2022-07-12T08:46:18.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f0155400 con 0x7f69f005cdd0 2022-07-12T08:46:18.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f01556a0 con 0x7f69f00b4720 2022-07-12T08:46:18.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 0x7f69f0151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:18.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 0x7f69f0151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56328/0 (socket says 172.21.15.61:56328) 2022-07-12T08:46:18.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 -- 172.21.15.61:0/1219501674 learned_addr learned my addr 172.21.15.61:0/1219501674 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:18.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f595b700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f00b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:18.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f615c700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f005cdd0 0x7f69f0155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:18.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 -- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 msgr2=0x7f69f00b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:18.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f00b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:18.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 -- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f005cdd0 msgr2=0x7f69f0155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:18.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f005cdd0 0x7f69f0155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:18.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 -- 172.21.15.61:0/1219501674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec00b1e0 con 0x7f69f00b6770 2022-07-12T08:46:18.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f615c700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f005cdd0 0x7f69f0155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:18.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f695d700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 0x7f69f0151890 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f69e800a000 tx=0x7f69e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:18.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e8013620 con 0x7f69f00b6770 2022-07-12T08:46:18.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8013780 con 0x7f69f00b6770 2022-07-12T08:46:18.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69e8013ae0 con 0x7f69f00b6770 2022-07-12T08:46:18.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f013ddc0 con 0x7f69f00b6770 2022-07-12T08:46:18.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.120+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f0151d80 con 0x7f69f00b6770 2022-07-12T08:46:18.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.124+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f0155860 con 0x7f69f00b6770 2022-07-12T08:46:18.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.124+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f69e8013c40 con 0x7f69f00b6770 2022-07-12T08:46:18.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.124+0000 7f69df7fe700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f69d0053920 0x7f69d0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:18.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.124+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f69e806bd10 con 0x7f69f00b6770 2022-07-12T08:46:18.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.124+0000 7f69f615c700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f69d0053920 0x7f69d0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:18.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.124+0000 7f69f615c700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f69d0053920 0x7f69d0055de0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f69e000d000 tx=0x7f69e0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:18.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.128+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f69e8042ef0 con 0x7f69f00b6770 2022-07-12T08:46:18.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:18.264+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f69f0057460 con 0x7f69f00b6770 2022-07-12T08:46:19.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.200+0000 7f69df7fe700 1 -- 172.21.15.61:0/1219501674 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v58) v1 ==== 97+0+0 (secure 0 0 0) 0x7f69e8046510 con 0x7f69f00b6770 2022-07-12T08:46:19.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f69d0053920 msgr2=0x7f69d0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f69d0053920 0x7f69d0055de0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f69e000d000 tx=0x7f69e0009040).stop 2022-07-12T08:46:19.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 msgr2=0x7f69f0151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 0x7f69f0151890 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f69e800a000 tx=0x7f69e8008040).stop 2022-07-12T08:46:19.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 shutdown_connections 2022-07-12T08:46:19.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69f00b6770 0x7f69f0151890 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f69f00b4720 0x7f69f00b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f69d0053920 0x7f69d0055de0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 --2- 172.21.15.61:0/1219501674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f69f005cdd0 0x7f69f0155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 >> 172.21.15.61:0/1219501674 conn(0x7f69f00b9d50 msgr2=0x7f69f00604e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:19.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 shutdown_connections 2022-07-12T08:46:19.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.204+0000 7f69f715e700 1 -- 172.21.15.61:0/1219501674 wait complete. 2022-07-12T08:46:19.216 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:46:19.231 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:46:19.231 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:46:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 Processor -- start 2022-07-12T08:46:19.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 -- start start 2022-07-12T08:46:19.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f3c0b7b10 0x7f8f3c067560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c145500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c0b27b0 con 0x7f8f3c0b35f0 2022-07-12T08:46:19.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3c0b7ef0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f4096a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c05cb50 con 0x7f8f3c146980 2022-07-12T08:46:19.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f3c0b7b10 0x7f8f3c067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f3c0b7b10 0x7f8f3c067560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56334/0 (socket says 172.21.15.61:56334) 2022-07-12T08:46:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 -- 172.21.15.61:0/3017301494 learned_addr learned my addr 172.21.15.61:0/3017301494 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3bfff700 1 --2- 172.21.15.61:0/3017301494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 -- 172.21.15.61:0/3017301494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 msgr2=0x7f8f3c145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 --2- 172.21.15.61:0/3017301494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 -- 172.21.15.61:0/3017301494 >> v1:172.21.15.111:6789/0 conn(0x7f8f3c0b35f0 legacy=0x7f8f3c146270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 -- 172.21.15.61:0/3017301494 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c146dd0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.372+0000 7f8f3b7fe700 1 --2- 172.21.15.61:0/3017301494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c067560 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f8f30002f70 tx=0x7f8f30006eb0).ready entity=mon.0 client_cookie=3584bd9d65672239 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f3a7fc700 1 -- 172.21.15.61:0/3017301494 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f3000d5c0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f3a7fc700 1 -- 172.21.15.61:0/3017301494 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30000e90 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f3a7fc700 1 -- 172.21.15.61:0/3017301494 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f3000d950 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 -- 172.21.15.61:0/3017301494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 msgr2=0x7f8f3c067560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3017301494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c067560 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f8f30002f70 tx=0x7f8f30006eb0).stop 2022-07-12T08:46:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 -- 172.21.15.61:0/3017301494 shutdown_connections 2022-07-12T08:46:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3017301494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3017301494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c067560 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 -- 172.21.15.61:0/3017301494 >> 172.21.15.61:0/3017301494 conn(0x7f8f3c0655c0 msgr2=0x7f8f3c0b70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:19.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 -- 172.21.15.61:0/3017301494 shutdown_connections 2022-07-12T08:46:19.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.376+0000 7f8f4096a700 1 -- 172.21.15.61:0/3017301494 wait complete. 2022-07-12T08:46:19.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 Processor -- start 2022-07-12T08:46:19.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 -- start start 2022-07-12T08:46:19.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c0b35f0 0x7f8f3c097430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c097920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c0992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c152b80 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c097f00 con 0x7f8f3c0b35f0 2022-07-12T08:46:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c152e00 con 0x7f8f3c146980 2022-07-12T08:46:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c097920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c0992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c097920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56342/0 (socket says 172.21.15.61:56342) 2022-07-12T08:46:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 -- 172.21.15.61:0/3148557310 learned_addr learned my addr 172.21.15.61:0/3148557310 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c0992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38536/0 (socket says 172.21.15.61:38536) 2022-07-12T08:46:19.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3b7fe700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c0b35f0 0x7f8f3c097430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 -- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 msgr2=0x7f8f3c0992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c0992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 -- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c0b35f0 msgr2=0x7f8f3c097430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c0b35f0 0x7f8f3c097430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 -- 172.21.15.61:0/3148557310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3000b1e0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f3affd700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c097920 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f8f3c0b7010 tx=0x7f8f2c00c840).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:19.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f2c006aa0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c006c00 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3c067130 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f2c007030 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.380+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3c05c920 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.384+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3c152fc0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.384+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f2c010070 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.384+0000 7f8f38ff9700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f1c053af0 0x7f8f1c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.384+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8f2c06b650 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.384+0000 7f8f3b7fe700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f1c053af0 0x7f8f1c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.384+0000 7f8f3b7fe700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f1c053af0 0x7f8f1c055fb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f8f3001c000 tx=0x7f8f30015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:19.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.388+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f2c042840 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8f3c0980c0 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f38ff9700 1 -- 172.21.15.61:0/3148557310 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f8f2c019030 con 0x7f8f3c0b7b10 2022-07-12T08:46:19.570 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:19.570 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:46:19.573 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:46:19.575 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.576 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:46:19.576 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:46:19.576 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.577 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:46:19.577 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.578 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.579 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.579 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.579 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.580 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:46:19.580 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:46:19.581 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:46:19.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.582 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.582 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.582 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.583 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.583 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.584 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:46:19.585 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:46:19.585 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.585 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.586 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.587 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.587 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.587 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.587 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.588 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:46:19.588 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:46:19.588 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.588 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.589 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.589 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.589 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.590 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.590 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.591 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:46:19.591 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:46:19.591 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.593 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.594 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.594 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.595 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.595 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:46:19.596 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:46:19.597 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.597 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.598 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.598 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.599 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.599 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.600 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.600 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:46:19.601 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:46:19.601 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.602 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.602 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.602 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.603 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.603 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.604 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.604 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:46:19.605 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:46:19.605 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.605 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.605 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.605 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.606 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.606 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.607 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.607 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:46:19.607 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:46:19.608 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.610 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.610 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.611 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.611 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.611 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.612 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.612 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:46:19.613 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:46:19.613 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.613 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.614 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.614 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.615 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.615 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.615 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.616 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:46:19.616 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:46:19.617 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.617 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:46:19.617 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:46:19.618 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:46:19.619 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.627 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.627 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.629 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:46:19.629 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:46:19.629 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.630 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:46:19.630 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:46:19.630 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:46:19.631 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.631 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.631 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.631 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:46:19.632 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:46:19.632 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.632 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:46:19.632 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:46:19.632 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:46:19.633 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.633 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.634 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.634 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:46:19.634 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:46:19.635 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.635 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:46:19.635 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:46:19.636 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:46:19.636 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.636 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.636 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.637 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.637 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:46:19.637 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:46:19.638 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.638 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:46:19.639 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:46:19.639 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.639 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.640 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:19.640 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:19.640 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:46:19.641 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:46:19.641 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:46:19.641 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:46:19.642 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:46:19.642 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:46:19.642 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.643 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:19.643 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:19.643 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:46:19.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f1c053af0 msgr2=0x7f8f1c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f1c053af0 0x7f8f1c055fb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f8f3001c000 tx=0x7f8f30015000).stop 2022-07-12T08:46:19.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 msgr2=0x7f8f3c097920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c097920 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f8f3c0b7010 tx=0x7f8f2c00c840).stop 2022-07-12T08:46:19.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 shutdown_connections 2022-07-12T08:46:19.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f3c146980 0x7f8f3c0992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3c0b7b10 0x7f8f3c097920 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f1c053af0 0x7f8f1c055fb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 --2- 172.21.15.61:0/3148557310 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3c0b35f0 0x7f8f3c097430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 >> 172.21.15.61:0/3148557310 conn(0x7f8f3c0655c0 msgr2=0x7f8f3c145ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:19.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 shutdown_connections 2022-07-12T08:46:19.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.560+0000 7f8f4096a700 1 -- 172.21.15.61:0/3148557310 wait complete. 2022-07-12T08:46:19.651 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:46:19.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 Processor -- start 2022-07-12T08:46:19.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 -- start start 2022-07-12T08:46:19.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0b02a0 0x7fa40c0b0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa40c0614a0 0x7fa40c146420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa40c0b1a60 0x7fa40c0af510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40c05c150 con 0x7fa40c0614a0 2022-07-12T08:46:19.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40c0b0ff0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa411c15700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c0acab0 con 0x7fa40c0b02a0 2022-07-12T08:46:19.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa40affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa40c0614a0 0x7fa40c146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa40affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa40c0614a0 0x7fa40c146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56348/0 (socket says 172.21.15.61:56348) 2022-07-12T08:46:19.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa40affd700 1 -- 172.21.15.61:0/67596638 learned_addr learned my addr 172.21.15.61:0/67596638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:19.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa40bfff700 1 --2- 172.21.15.61:0/67596638 >> v2:172.21.15.111:3300/0 conn(0x7fa40c0b1a60 0x7fa40c0af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa40b7fe700 1 --2- 172.21.15.61:0/67596638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0b02a0 0x7fa40c0b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.756+0000 7fa40affd700 1 -- 172.21.15.61:0/67596638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0b02a0 msgr2=0x7fa40c0b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40affd700 1 --2- 172.21.15.61:0/67596638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0b02a0 0x7fa40c0b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40affd700 1 -- 172.21.15.61:0/67596638 >> v2:172.21.15.111:3300/0 conn(0x7fa40c0b1a60 msgr2=0x7fa40c0af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:19.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40affd700 1 --2- 172.21.15.61:0/67596638 >> v2:172.21.15.111:3300/0 conn(0x7fa40c0b1a60 0x7fa40c0af510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40affd700 1 -- 172.21.15.61:0/67596638 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40c0abfb0 con 0x7fa40c0614a0 2022-07-12T08:46:19.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40affd700 1 --2- 172.21.15.61:0/67596638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0614a0 0x7fa40c146420 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fa400002f70 tx=0x7fa400006eb0).ready entity=mon.0 client_cookie=82b629c9000e0c5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:19.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40a7fc700 1 -- 172.21.15.61:0/67596638 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa40000d570 con 0x7fa40c0614a0 2022-07-12T08:46:19.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40a7fc700 1 -- 172.21.15.61:0/67596638 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa400000e90 con 0x7fa40c0614a0 2022-07-12T08:46:19.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa40a7fc700 1 -- 172.21.15.61:0/67596638 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa40000d900 con 0x7fa40c0614a0 2022-07-12T08:46:19.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 -- 172.21.15.61:0/67596638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0614a0 msgr2=0x7fa40c146420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 --2- 172.21.15.61:0/67596638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0614a0 0x7fa40c146420 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fa400002f70 tx=0x7fa400006eb0).stop 2022-07-12T08:46:19.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 -- 172.21.15.61:0/67596638 shutdown_connections 2022-07-12T08:46:19.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 --2- 172.21.15.61:0/67596638 >> v2:172.21.15.111:3300/0 conn(0x7fa40c0b1a60 0x7fa40c0af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 --2- 172.21.15.61:0/67596638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0614a0 0x7fa40c146420 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 --2- 172.21.15.61:0/67596638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0b02a0 0x7fa40c0b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 -- 172.21.15.61:0/67596638 >> 172.21.15.61:0/67596638 conn(0x7fa40c05f140 msgr2=0x7fa40c05f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:19.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 -- 172.21.15.61:0/67596638 shutdown_connections 2022-07-12T08:46:19.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.760+0000 7fa411c15700 1 -- 172.21.15.61:0/67596638 wait complete. 2022-07-12T08:46:19.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 Processor -- start 2022-07-12T08:46:19.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 -- start start 2022-07-12T08:46:19.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0614a0 0x7fa40c09b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa40c0b02a0 0x7fa40c094780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 0x7fa40c095f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c093da0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c094040 con 0x7fa40c0b02a0 2022-07-12T08:46:19.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c0942e0 con 0x7fa40c0614a0 2022-07-12T08:46:19.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 0x7fa40c095f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 0x7fa40c095f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56356/0 (socket says 172.21.15.61:56356) 2022-07-12T08:46:19.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 -- 172.21.15.61:0/3987483107 learned_addr learned my addr 172.21.15.61:0/3987483107 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:19.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40b7fe700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0614a0 0x7fa40c09b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40affd700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa40c0b02a0 0x7fa40c094780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 -- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0614a0 msgr2=0x7fa40c09b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0614a0 0x7fa40c09b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 -- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa40c0b02a0 msgr2=0x7fa40c094780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa40c0b02a0 0x7fa40c094780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 -- 172.21.15.61:0/3987483107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40000b1e0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40b7fe700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0614a0 0x7fa40c09b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:19.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40affd700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa40c0b02a0 0x7fa40c094780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:46:19.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa40bfff700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 0x7fa40c095f60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fa3fc00c000 tx=0x7fa3fc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:19.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3fc010780 con 0x7fa40c0b1a60 2022-07-12T08:46:19.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3fc0108e0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40c092eb0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3fc010c40 con 0x7fa40c0b1a60 2022-07-12T08:46:19.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.764+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40c0b26f0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.768+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40c0944a0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.768+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa3fc01c460 con 0x7fa40c0b1a60 2022-07-12T08:46:19.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.768+0000 7fa408ff9700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa3ec053a40 0x7fa3ec055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:19.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.768+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa3fc06c090 con 0x7fa40c0b1a60 2022-07-12T08:46:19.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.768+0000 7fa40b7fe700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa3ec053a40 0x7fa3ec055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:19.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.768+0000 7fa40b7fe700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa3ec053a40 0x7fa3ec055f00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fa40400d000 tx=0x7fa404008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:19.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.772+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3fc0431b0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.900+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa40c057460 con 0x7fa40c0b1a60 2022-07-12T08:46:19.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.900+0000 7fa408ff9700 1 -- 172.21.15.61:0/3987483107 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13230 (secure 0 0 0) 0x7fa3fc0467d0 con 0x7fa40c0b1a60 2022-07-12T08:46:19.906 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:19.906 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":58,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:46:19.190476+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:46:19.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa3ec053a40 msgr2=0x7fa3ec055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa3ec053a40 0x7fa3ec055f00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fa40400d000 tx=0x7fa404008040).stop 2022-07-12T08:46:19.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 msgr2=0x7fa40c095f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:19.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 0x7fa40c095f60 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fa3fc00c000 tx=0x7fa3fc006df0).stop 2022-07-12T08:46:19.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 shutdown_connections 2022-07-12T08:46:19.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa40c0b1a60 0x7fa40c095f60 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa3ec053a40 0x7fa3ec055f00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa40c0b02a0 0x7fa40c094780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 --2- 172.21.15.61:0/3987483107 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa40c0614a0 0x7fa40c09b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:19.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 >> 172.21.15.61:0/3987483107 conn(0x7fa40c05f140 msgr2=0x7fa40c0b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:19.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 shutdown_connections 2022-07-12T08:46:19.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:19.904+0000 7fa411c15700 1 -- 172.21.15.61:0/3987483107 wait complete. 2022-07-12T08:46:19.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:46:19.919 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:46:20.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 Processor -- start 2022-07-12T08:46:20.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 -- start start 2022-07-12T08:46:20.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8a2805d060 0x7f8a280b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8a28142ab0 0x7f8a280afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2805d440 con 0x7f8a28142f10 2022-07-12T08:46:20.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a28143d80 con 0x7f8a2805d060 2022-07-12T08:46:20.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a2c83e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280674b0 con 0x7f8a28142ab0 2022-07-12T08:46:20.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.028+0000 7f8a277fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8a2805d060 0x7f8a280b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a27fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8a28142f10 0x7f8a281433c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44582/0 (socket says 172.21.15.61:44582) 2022-07-12T08:46:20.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a27fff700 1 -- 172.21.15.61:0/1182170894 learned_addr learned my addr 172.21.15.61:0/1182170894 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:20.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a26ffd700 1 --2- 172.21.15.61:0/1182170894 >> v2:172.21.15.111:3300/0 conn(0x7f8a28142ab0 0x7f8a280afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a277fe700 1 -- 172.21.15.61:0/1182170894 >> v2:172.21.15.111:3300/0 conn(0x7f8a28142ab0 msgr2=0x7f8a280afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:20.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a277fe700 1 --2- 172.21.15.61:0/1182170894 >> v2:172.21.15.111:3300/0 conn(0x7f8a28142ab0 0x7f8a280afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a277fe700 1 -- 172.21.15.61:0/1182170894 >> v1:172.21.15.61:6789/0 conn(0x7f8a28142f10 legacy=0x7f8a281433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a267fc700 1 -- 172.21.15.61:0/1182170894 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468388525 0 0) 0x7f8a2805d440 con 0x7f8a28142f10 2022-07-12T08:46:20.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a277fe700 1 -- 172.21.15.61:0/1182170894 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28067750 con 0x7f8a2805d060 2022-07-12T08:46:20.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a277fe700 1 --2- 172.21.15.61:0/1182170894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a280b0130 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f8a1c002f70 tx=0x7f8a1c006df0).ready entity=mon.0 client_cookie=abda60a2ec961596 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a267fc700 1 -- 172.21.15.61:0/1182170894 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c00d570 con 0x7f8a2805d060 2022-07-12T08:46:20.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a267fc700 1 -- 172.21.15.61:0/1182170894 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c000e90 con 0x7f8a2805d060 2022-07-12T08:46:20.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a267fc700 1 -- 172.21.15.61:0/1182170894 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c00d8c0 con 0x7f8a2805d060 2022-07-12T08:46:20.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 -- 172.21.15.61:0/1182170894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 msgr2=0x7f8a280b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/1182170894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a280b0130 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f8a1c002f70 tx=0x7f8a1c006df0).stop 2022-07-12T08:46:20.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 -- 172.21.15.61:0/1182170894 shutdown_connections 2022-07-12T08:46:20.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/1182170894 >> v2:172.21.15.111:3300/0 conn(0x7f8a28142ab0 0x7f8a280afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/1182170894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a280b0130 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 -- 172.21.15.61:0/1182170894 >> 172.21.15.61:0/1182170894 conn(0x7f8a280b6900 msgr2=0x7f8a28066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 -- 172.21.15.61:0/1182170894 shutdown_connections 2022-07-12T08:46:20.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 -- 172.21.15.61:0/1182170894 wait complete. 2022-07-12T08:46:20.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 Processor -- start 2022-07-12T08:46:20.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.032+0000 7f8a2c83e700 1 -- start start 2022-07-12T08:46:20.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a281498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a28142f10 0x7f8a28155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a28149dc0 0x7f8a28155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28142860 con 0x7f8a2805d060 2022-07-12T08:46:20.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28156420 con 0x7f8a28149dc0 2022-07-12T08:46:20.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280b1c20 con 0x7f8a28142f10 2022-07-12T08:46:20.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a281498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a281498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56366/0 (socket says 172.21.15.61:56366) 2022-07-12T08:46:20.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 -- 172.21.15.61:0/3807365853 learned_addr learned my addr 172.21.15.61:0/3807365853 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:20.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a26ffd700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a28142f10 0x7f8a28155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a27fff700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a28149dc0 0x7f8a28155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 -- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a28142f10 msgr2=0x7f8a28155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a28142f10 0x7f8a28155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 -- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a28149dc0 msgr2=0x7f8a28155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a28149dc0 0x7f8a28155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 -- 172.21.15.61:0/3807365853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c00b180 con 0x7f8a2805d060 2022-07-12T08:46:20.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a26ffd700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a28142f10 0x7f8a28155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:46:20.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a27fff700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a28149dc0 0x7f8a28155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:46:20.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a277fe700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a281498d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f8a28156560 tx=0x7f8a1c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c00dea0 con 0x7f8a2805d060 2022-07-12T08:46:20.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c01e400 con 0x7f8a2805d060 2022-07-12T08:46:20.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a280b1980 con 0x7f8a2805d060 2022-07-12T08:46:20.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c01e760 con 0x7f8a2805d060 2022-07-12T08:46:20.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2809c420 con 0x7f8a2805d060 2022-07-12T08:46:20.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.036+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a281557a0 con 0x7f8a2805d060 2022-07-12T08:46:20.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.040+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8a1c025030 con 0x7f8a2805d060 2022-07-12T08:46:20.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.040+0000 7f8a24ff9700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8a08053920 0x7f8a08055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.040+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8a1c06c8d0 con 0x7f8a2805d060 2022-07-12T08:46:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.040+0000 7f8a26ffd700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8a08053920 0x7f8a08055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.040+0000 7f8a26ffd700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8a08053920 0x7f8a08055de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8a1800b000 tx=0x7f8a18008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.044+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8a281557a0 con 0x7f8a2805d060 2022-07-12T08:46:20.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.168+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8a28057460 con 0x7f8a2805d060 2022-07-12T08:46:20.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a24ff9700 1 -- 172.21.15.61:0/3807365853 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v58) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8a1c023070 con 0x7f8a2805d060 2022-07-12T08:46:20.174 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:46:20.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8a08053920 msgr2=0x7f8a08055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8a08053920 0x7f8a08055de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8a1800b000 tx=0x7f8a18008040).stop 2022-07-12T08:46:20.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 msgr2=0x7f8a281498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a281498d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f8a28156560 tx=0x7f8a1c014000).stop 2022-07-12T08:46:20.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 shutdown_connections 2022-07-12T08:46:20.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8a08053920 0x7f8a08055de0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8a28149dc0 0x7f8a28155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8a28142f10 0x7f8a28155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 --2- 172.21.15.61:0/3807365853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8a2805d060 0x7f8a281498d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 >> 172.21.15.61:0/3807365853 conn(0x7f8a280b6900 msgr2=0x7f8a280b3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 shutdown_connections 2022-07-12T08:46:20.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.172+0000 7f8a2c83e700 1 -- 172.21.15.61:0/3807365853 wait complete. 2022-07-12T08:46:20.196 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:46:20.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 Processor -- start 2022-07-12T08:46:20.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 -- start start 2022-07-12T08:46:20.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc00c05d0f0 0x7fc00c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c0b25e0 0x7fc00c067200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc00c0afcf0 con 0x7fc00c0b1390 2022-07-12T08:46:20.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc00c143f10 con 0x7fc00c05d0f0 2022-07-12T08:46:20.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc0117b4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c1430c0 con 0x7fc00c0b25e0 2022-07-12T08:46:20.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc00b7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc00c05d0f0 0x7fc00c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc010fb3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c0b25e0 0x7fc00c067200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc00bfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fc00c0b1390 0x7fc00c143800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:58194/0 (socket says 172.21.15.61:58194) 2022-07-12T08:46:20.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.316+0000 7fc00bfff700 1 -- 172.21.15.61:0/2433244248 learned_addr learned my addr 172.21.15.61:0/2433244248 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:20.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00b7fe700 1 -- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c0b25e0 msgr2=0x7fc00c067200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:20.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00b7fe700 1 --2- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c0b25e0 0x7fc00c067200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00b7fe700 1 -- 172.21.15.61:0/2433244248 >> v1:172.21.15.111:6789/0 conn(0x7fc00c0b1390 legacy=0x7fc00c143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00b7fe700 1 -- 172.21.15.61:0/2433244248 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00c143360 con 0x7fc00c05d0f0 2022-07-12T08:46:20.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc010fb3700 1 --2- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c0b25e0 0x7fc00c067200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:20.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00b7fe700 1 --2- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c05d0f0 0x7fc00c0b20f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fc000002f70 tx=0x7fc000006df0).ready entity=mon.1 client_cookie=ad53ca13faa364c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00affd700 1 -- 172.21.15.61:0/2433244248 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc000014070 con 0x7fc00c05d0f0 2022-07-12T08:46:20.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00affd700 1 -- 172.21.15.61:0/2433244248 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc000000e90 con 0x7fc00c05d0f0 2022-07-12T08:46:20.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc00affd700 1 -- 172.21.15.61:0/2433244248 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00000fb60 con 0x7fc00c05d0f0 2022-07-12T08:46:20.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 -- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c05d0f0 msgr2=0x7fc00c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 --2- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c05d0f0 0x7fc00c0b20f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fc000002f70 tx=0x7fc000006df0).stop 2022-07-12T08:46:20.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 -- 172.21.15.61:0/2433244248 shutdown_connections 2022-07-12T08:46:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 --2- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c0b25e0 0x7fc00c067200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 --2- 172.21.15.61:0/2433244248 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c05d0f0 0x7fc00c0b20f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 -- 172.21.15.61:0/2433244248 >> 172.21.15.61:0/2433244248 conn(0x7fc00c063c50 msgr2=0x7fc00c05f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 -- 172.21.15.61:0/2433244248 shutdown_connections 2022-07-12T08:46:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.320+0000 7fc0117b4700 1 -- 172.21.15.61:0/2433244248 wait complete. 2022-07-12T08:46:20.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 Processor -- start 2022-07-12T08:46:20.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 -- start start 2022-07-12T08:46:20.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c05d0f0 0x7fc00c14d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c0b1390 0x7fc00c0a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 0x7fc00c0b3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c14e990 con 0x7fc00c0b25e0 2022-07-12T08:46:20.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0abdc0 con 0x7fc00c0b1390 2022-07-12T08:46:20.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc0117b4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00c0ac060 con 0x7fc00c05d0f0 2022-07-12T08:46:20.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 0x7fc00c0b3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 0x7fc00c0b3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56384/0 (socket says 172.21.15.61:56384) 2022-07-12T08:46:20.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 -- 172.21.15.61:0/1551418152 learned_addr learned my addr 172.21.15.61:0/1551418152 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:20.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc00b7fe700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c0b1390 0x7fc00c0a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc00bfff700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c05d0f0 0x7fc00c14d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 -- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c05d0f0 msgr2=0x7fc00c14d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c05d0f0 0x7fc00c14d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 -- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c0b1390 msgr2=0x7fc00c0a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c0b1390 0x7fc00c0a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.324+0000 7fc010fb3700 1 -- 172.21.15.61:0/1551418152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004009470 con 0x7fc00c0b25e0 2022-07-12T08:46:20.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc010fb3700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 0x7fc00c0b3890 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fc00400c000 tx=0x7fc0040091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00400c6a0 con 0x7fc00c0b25e0 2022-07-12T08:46:20.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00400c800 con 0x7fc00c0b25e0 2022-07-12T08:46:20.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc00000b180 con 0x7fc00c0b25e0 2022-07-12T08:46:20.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc00400cb60 con 0x7fc00c0b25e0 2022-07-12T08:46:20.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc00c151310 con 0x7fc00c0b25e0 2022-07-12T08:46:20.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00c0ac220 con 0x7fc00c0b25e0 2022-07-12T08:46:20.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc004022070 con 0x7fc00c0b25e0 2022-07-12T08:46:20.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0097fa700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfec0539d0 0x7fbfec055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc00406c280 con 0x7fc00c0b25e0 2022-07-12T08:46:20.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.328+0000 7fc00bfff700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfec0539d0 0x7fbfec055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.332+0000 7fc00bfff700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfec0539d0 0x7fbfec055e90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fbffc00d000 tx=0x7fbffc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.336+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc004043400 con 0x7fc00c0b25e0 2022-07-12T08:46:20.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.460+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc00c057460 con 0x7fc00c0b25e0 2022-07-12T08:46:20.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0097fa700 1 -- 172.21.15.61:0/1551418152 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v58) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc0040160a0 con 0x7fc00c0b25e0 2022-07-12T08:46:20.467 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:46:20.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfec0539d0 msgr2=0x7fbfec055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfec0539d0 0x7fbfec055e90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fbffc00d000 tx=0x7fbffc009040).stop 2022-07-12T08:46:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 msgr2=0x7fc00c0b3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 0x7fc00c0b3890 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fc00400c000 tx=0x7fc0040091d0).stop 2022-07-12T08:46:20.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 shutdown_connections 2022-07-12T08:46:20.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc00c0b25e0 0x7fc00c0b3890 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfec0539d0 0x7fbfec055e90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc00c0b1390 0x7fc00c0a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 --2- 172.21.15.61:0/1551418152 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc00c05d0f0 0x7fc00c14d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 >> 172.21.15.61:0/1551418152 conn(0x7fc00c063c50 msgr2=0x7fc00c05f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 shutdown_connections 2022-07-12T08:46:20.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.464+0000 7fc0117b4700 1 -- 172.21.15.61:0/1551418152 wait complete. 2022-07-12T08:46:20.482 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:46:20.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.588+0000 7f658b4c3700 1 Processor -- start 2022-07-12T08:46:20.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.588+0000 7f658b4c3700 1 -- start start 2022-07-12T08:46:20.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f658405c8e0 0x7f65840b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 0x7f65840b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65840601f0 con 0x7f65840b2fa0 2022-07-12T08:46:20.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658405f9e0 con 0x7f658405c8e0 2022-07-12T08:46:20.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658405ccc0 con 0x7f658405f4d0 2022-07-12T08:46:20.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 0x7f65840b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 0x7f65840b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38584/0 (socket says 172.21.15.61:38584) 2022-07-12T08:46:20.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 -- 172.21.15.61:0/3560745381 learned_addr learned my addr 172.21.15.61:0/3560745381 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:20.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658a4c1700 1 --2- 172.21.15.61:0/3560745381 >> v2:172.21.15.111:3300/0 conn(0x7f658405c8e0 0x7f65840b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 -- 172.21.15.61:0/3560745381 >> v2:172.21.15.111:3300/0 conn(0x7f658405c8e0 msgr2=0x7f65840b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:20.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 --2- 172.21.15.61:0/3560745381 >> v2:172.21.15.111:3300/0 conn(0x7f658405c8e0 0x7f65840b13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 -- 172.21.15.61:0/3560745381 >> v1:172.21.15.111:6789/0 conn(0x7f65840b2fa0 legacy=0x7f6584145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 -- 172.21.15.61:0/3560745381 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584146e30 con 0x7f658405f4d0 2022-07-12T08:46:20.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f6589cc0700 1 --2- 172.21.15.61:0/3560745381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 0x7f65840b2ab0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6574002f70 tx=0x7f6574006eb0).ready entity=mon.2 client_cookie=d367af662abd4ed2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f65894bf700 1 -- 172.21.15.61:0/3560745381 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f657400d570 con 0x7f658405f4d0 2022-07-12T08:46:20.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f65894bf700 1 -- 172.21.15.61:0/3560745381 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574000e90 con 0x7f658405f4d0 2022-07-12T08:46:20.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f65894bf700 1 -- 172.21.15.61:0/3560745381 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f657400d900 con 0x7f658405f4d0 2022-07-12T08:46:20.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- 172.21.15.61:0/3560745381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 msgr2=0x7f65840b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3560745381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 0x7f65840b2ab0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f6574002f70 tx=0x7f6574006eb0).stop 2022-07-12T08:46:20.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- 172.21.15.61:0/3560745381 shutdown_connections 2022-07-12T08:46:20.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3560745381 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f658405f4d0 0x7f65840b2ab0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3560745381 >> v2:172.21.15.111:3300/0 conn(0x7f658405c8e0 0x7f65840b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- 172.21.15.61:0/3560745381 >> 172.21.15.61:0/3560745381 conn(0x7f6584063a60 msgr2=0x7f6584063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- 172.21.15.61:0/3560745381 shutdown_connections 2022-07-12T08:46:20.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.592+0000 7f658b4c3700 1 -- 172.21.15.61:0/3560745381 wait complete. 2022-07-12T08:46:20.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 Processor -- start 2022-07-12T08:46:20.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 -- start start 2022-07-12T08:46:20.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658405c8e0 0x7f6584065970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 0x7f65841463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65840b2fa0 0x7f6584156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6584066af0 con 0x7f658405f4d0 2022-07-12T08:46:20.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658413f4f0 con 0x7f658405c8e0 2022-07-12T08:46:20.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65841562d0 con 0x7f65840b2fa0 2022-07-12T08:46:20.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 0x7f65841463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 0x7f65841463e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56396/0 (socket says 172.21.15.61:56396) 2022-07-12T08:46:20.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 -- 172.21.15.61:0/3287585965 learned_addr learned my addr 172.21.15.61:0/3287585965 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:20.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658a4c1700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658405c8e0 0x7f6584065970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658acc2700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65840b2fa0 0x7f6584156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 -- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65840b2fa0 msgr2=0x7f6584156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65840b2fa0 0x7f6584156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 -- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658405c8e0 msgr2=0x7f6584065970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658405c8e0 0x7f6584065970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 -- 172.21.15.61:0/3287585965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657400b1e0 con 0x7f658405f4d0 2022-07-12T08:46:20.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f6589cc0700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 0x7f65841463e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f6574002f70 tx=0x7f6574015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65740117e0 con 0x7f658405f4d0 2022-07-12T08:46:20.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574011940 con 0x7f658405f4d0 2022-07-12T08:46:20.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6584159790 con 0x7f658405f4d0 2022-07-12T08:46:20.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.596+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6574011ca0 con 0x7f658405f4d0 2022-07-12T08:46:20.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.600+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6584064f00 con 0x7f658405f4d0 2022-07-12T08:46:20.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.600+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6584057460 con 0x7f658405f4d0 2022-07-12T08:46:20.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.600+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f657401a040 con 0x7f658405f4d0 2022-07-12T08:46:20.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.600+0000 7f65737fe700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6564053a40 0x7f6564055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.600+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f657406c9f0 con 0x7f658405f4d0 2022-07-12T08:46:20.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.600+0000 7f658a4c1700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6564053a40 0x7f6564055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.604+0000 7f658a4c1700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6564053a40 0x7f6564055f00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f658000b000 tx=0x7f6580006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.608+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f657406ee60 con 0x7f658405f4d0 2022-07-12T08:46:20.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.736+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6584067ab0 con 0x7f658405f4d0 2022-07-12T08:46:20.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.736+0000 7f65737fe700 1 -- 172.21.15.61:0/3287585965 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v58) v1 ==== 84+0+143 (secure 0 0 0) 0x7f6574023020 con 0x7f658405f4d0 2022-07-12T08:46:20.754 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:20.754 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:46:20.755 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:20.755 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:46:20.756 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:46:20.756 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:20.756 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:20.757 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:46:20.758 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:46:20.758 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:20.759 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:46:20.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6564053a40 msgr2=0x7f6564055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6564053a40 0x7f6564055f00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f658000b000 tx=0x7f6580006df0).stop 2022-07-12T08:46:20.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 msgr2=0x7f65841463e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 0x7f65841463e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f6574002f70 tx=0x7f6574015000).stop 2022-07-12T08:46:20.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 shutdown_connections 2022-07-12T08:46:20.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65840b2fa0 0x7f6584156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f658405f4d0 0x7f65841463e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6564053a40 0x7f6564055f00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 --2- 172.21.15.61:0/3287585965 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f658405c8e0 0x7f6584065970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 >> 172.21.15.61:0/3287585965 conn(0x7f6584063a60 msgr2=0x7f6584149950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 shutdown_connections 2022-07-12T08:46:20.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.740+0000 7f658b4c3700 1 -- 172.21.15.61:0/3287585965 wait complete. 2022-07-12T08:46:20.771 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:46:20.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:46:20.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb36232b700 1 Processor -- start 2022-07-12T08:46:20.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb36232b700 1 -- start start 2022-07-12T08:46:20.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb36232b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb35c0aaf60 0x7fb35c060520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb36232b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b0070 con 0x7fb35c060a10 2022-07-12T08:46:20.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb36232b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35c0b1560 con 0x7fb35c0ac6c0 2022-07-12T08:46:20.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb36232b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35c0acaa0 con 0x7fb35c0aaf60 2022-07-12T08:46:20.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3608c8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb35c060a10 0x7fb35c14a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44624/0 (socket says 172.21.15.61:44624) 2022-07-12T08:46:20.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3537fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb35c0aaf60 0x7fb35c060520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3608c8700 1 -- 172.21.15.61:0/2971970856 learned_addr learned my addr 172.21.15.61:0/2971970856 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:20.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb35b7fe700 1 -- 172.21.15.61:0/2971970856 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674794793 0 0) 0x7fb35c0b0070 con 0x7fb35c060a10 2022-07-12T08:46:20.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb35b7fe700 1 -- 172.21.15.61:0/2971970856 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33c003260 con 0x7fb35c060a10 2022-07-12T08:46:20.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3537fe700 1 -- 172.21.15.61:0/2971970856 >> v1:172.21.15.111:6789/0 conn(0x7fb35c0ac6c0 legacy=0x7fb35c0615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3537fe700 1 -- 172.21.15.61:0/2971970856 >> v1:172.21.15.61:6789/0 conn(0x7fb35c060a10 legacy=0x7fb35c14a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3537fe700 1 -- 172.21.15.61:0/2971970856 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c0acdf0 con 0x7fb35c0aaf60 2022-07-12T08:46:20.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb3537fe700 1 --2- 172.21.15.61:0/2971970856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c060520 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fb344002f70 tx=0x7fb344006df0).ready entity=mon.0 client_cookie=2412cb6a8b86f496 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb35b7fe700 1 -- 172.21.15.61:0/2971970856 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb34400d610 con 0x7fb35c0aaf60 2022-07-12T08:46:20.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb35b7fe700 1 -- 172.21.15.61:0/2971970856 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb344000e90 con 0x7fb35c0aaf60 2022-07-12T08:46:20.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.900+0000 7fb35b7fe700 1 -- 172.21.15.61:0/2971970856 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb34400d960 con 0x7fb35c0aaf60 2022-07-12T08:46:20.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- 172.21.15.61:0/2971970856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 msgr2=0x7fb35c060520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 --2- 172.21.15.61:0/2971970856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c060520 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fb344002f70 tx=0x7fb344006df0).stop 2022-07-12T08:46:20.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- 172.21.15.61:0/2971970856 shutdown_connections 2022-07-12T08:46:20.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 --2- 172.21.15.61:0/2971970856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c060520 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- 172.21.15.61:0/2971970856 >> 172.21.15.61:0/2971970856 conn(0x7fb35c0b23d0 msgr2=0x7fb35c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:20.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- 172.21.15.61:0/2971970856 shutdown_connections 2022-07-12T08:46:20.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- 172.21.15.61:0/2971970856 wait complete. 2022-07-12T08:46:20.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 Processor -- start 2022-07-12T08:46:20.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- start start 2022-07-12T08:46:20.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb35c060a10 0x7fb35c11e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c065470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb35c0ac6c0 0x7fb35c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c066220 con 0x7fb35c0aaf60 2022-07-12T08:46:20.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c063b40 con 0x7fb35c0ac6c0 2022-07-12T08:46:20.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb36232b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb35c0af800 con 0x7fb35c060a10 2022-07-12T08:46:20.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb3537fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c065470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb3537fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c065470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56410/0 (socket says 172.21.15.61:56410) 2022-07-12T08:46:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb3537fe700 1 -- 172.21.15.61:0/1912270111 learned_addr learned my addr 172.21.15.61:0/1912270111 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.904+0000 7fb35bfff700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb35c060a10 0x7fb35c11e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3608c8700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb35c0ac6c0 0x7fb35c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3537fe700 1 -- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb35c060a10 msgr2=0x7fb35c11e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3537fe700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb35c060a10 0x7fb35c11e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3537fe700 1 -- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb35c0ac6c0 msgr2=0x7fb35c1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3537fe700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb35c0ac6c0 0x7fb35c1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3537fe700 1 -- 172.21.15.61:0/1912270111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34400b180 con 0x7fb35c0aaf60 2022-07-12T08:46:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb3537fe700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c065470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fb344002f70 tx=0x7fb344014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb344016070 con 0x7fb35c0aaf60 2022-07-12T08:46:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34400de90 con 0x7fb35c0aaf60 2022-07-12T08:46:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35c063e00 con 0x7fb35c0aaf60 2022-07-12T08:46:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb344013da0 con 0x7fb35c0aaf60 2022-07-12T08:46:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c063740 con 0x7fb35c0aaf60 2022-07-12T08:46:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35c064d90 con 0x7fb35c0aaf60 2022-07-12T08:46:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.908+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb344024030 con 0x7fb35c0aaf60 2022-07-12T08:46:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.912+0000 7fb359ffb700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb340053a40 0x7fb340055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.912+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb34406c720 con 0x7fb35c0aaf60 2022-07-12T08:46:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.912+0000 7fb35bfff700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb340053a40 0x7fb340055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.912+0000 7fb35bfff700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb340053a40 0x7fb340055f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb34c00d000 tx=0x7fb34c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:20.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:20.916+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb35c064d90 con 0x7fb35c0aaf60 2022-07-12T08:46:21.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.044+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb35c064d60 con 0x7fb35c0aaf60 2022-07-12T08:46:21.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.216+0000 7fb359ffb700 1 -- 172.21.15.61:0/1912270111 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v59) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb344022070 con 0x7fb35c0aaf60 2022-07-12T08:46:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb340053a40 msgr2=0x7fb340055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:21.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb340053a40 0x7fb340055f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb34c00d000 tx=0x7fb34c007b90).stop 2022-07-12T08:46:21.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 msgr2=0x7fb35c065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:21.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c065470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fb344002f70 tx=0x7fb344014000).stop 2022-07-12T08:46:21.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 shutdown_connections 2022-07-12T08:46:21.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb340053a40 0x7fb340055f00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb35c0ac6c0 0x7fb35c1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb35c0aaf60 0x7fb35c065470 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 --2- 172.21.15.61:0/1912270111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb35c060a10 0x7fb35c11e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 >> 172.21.15.61:0/1912270111 conn(0x7fb35c0b23d0 msgr2=0x7fb35c0acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:21.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 shutdown_connections 2022-07-12T08:46:21.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.220+0000 7fb36232b700 1 -- 172.21.15.61:0/1912270111 wait complete. 2022-07-12T08:46:21.233 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:46:21.241 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:46:21.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.356+0000 7f1081a3c700 1 Processor -- start 2022-07-12T08:46:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.356+0000 7f1081a3c700 1 -- start start 2022-07-12T08:46:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f107c0b0c20 0x7f107c144310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f107c144800 0x7f107c140110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c05d240 con 0x7f107c0aea80 2022-07-12T08:46:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c064400 con 0x7f107c144800 2022-07-12T08:46:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c0646a0 con 0x7f107c0b0c20 2022-07-12T08:46:21.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f107c144800 0x7f107c140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:21.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f107c144800 0x7f107c140110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56418/0 (socket says 172.21.15.61:56418) 2022-07-12T08:46:21.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 -- 172.21.15.61:0/307578224 learned_addr learned my addr 172.21.15.61:0/307578224 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:21.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107a7fc700 1 --2- 172.21.15.61:0/307578224 >> v2:172.21.15.111:3300/0 conn(0x7f107c0b0c20 0x7f107c144310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:21.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 -- 172.21.15.61:0/307578224 >> v2:172.21.15.111:3300/0 conn(0x7f107c0b0c20 msgr2=0x7f107c144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:21.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 --2- 172.21.15.61:0/307578224 >> v2:172.21.15.111:3300/0 conn(0x7f107c0b0c20 0x7f107c144310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 -- 172.21.15.61:0/307578224 >> v1:172.21.15.111:6789/0 conn(0x7f107c0aea80 legacy=0x7f107c064b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:21.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 -- 172.21.15.61:0/307578224 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0affd0 con 0x7f107c144800 2022-07-12T08:46:21.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107a7fc700 1 --2- 172.21.15.61:0/307578224 >> v2:172.21.15.111:3300/0 conn(0x7f107c0b0c20 0x7f107c144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:21.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f107b7fe700 1 --2- 172.21.15.61:0/307578224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c144800 0x7f107c140110 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f106c002f70 tx=0x7f106c006eb0).ready entity=mon.0 client_cookie=984ccc79d2d7f108 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:21.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1079ffb700 1 -- 172.21.15.61:0/307578224 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f106c00d610 con 0x7f107c144800 2022-07-12T08:46:21.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1079ffb700 1 -- 172.21.15.61:0/307578224 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c000e90 con 0x7f107c144800 2022-07-12T08:46:21.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1079ffb700 1 -- 172.21.15.61:0/307578224 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f106c00d9a0 con 0x7f107c144800 2022-07-12T08:46:21.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- 172.21.15.61:0/307578224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c144800 msgr2=0x7f107c140110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:21.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 --2- 172.21.15.61:0/307578224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c144800 0x7f107c140110 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f106c002f70 tx=0x7f106c006eb0).stop 2022-07-12T08:46:21.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- 172.21.15.61:0/307578224 shutdown_connections 2022-07-12T08:46:21.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 --2- 172.21.15.61:0/307578224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c144800 0x7f107c140110 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 --2- 172.21.15.61:0/307578224 >> v2:172.21.15.111:3300/0 conn(0x7f107c0b0c20 0x7f107c144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- 172.21.15.61:0/307578224 >> 172.21.15.61:0/307578224 conn(0x7f107c0b5290 msgr2=0x7f107c0b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:21.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- 172.21.15.61:0/307578224 shutdown_connections 2022-07-12T08:46:21.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.360+0000 7f1081a3c700 1 -- 172.21.15.61:0/307578224 wait complete. 2022-07-12T08:46:21.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 Processor -- start 2022-07-12T08:46:21.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 -- start start 2022-07-12T08:46:21.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f107c0aea80 0x7f107c14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:21.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 0x7f107c09e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:21.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f107c144800 0x7f107c13f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:21.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1502b0 con 0x7f107c0b0c20 2022-07-12T08:46:21.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c150510 con 0x7f107c0aea80 2022-07-12T08:46:21.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1507b0 con 0x7f107c144800 2022-07-12T08:46:21.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 0x7f107c09e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:21.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 0x7f107c09e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56424/0 (socket says 172.21.15.61:56424) 2022-07-12T08:46:21.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 -- 172.21.15.61:0/775132060 learned_addr learned my addr 172.21.15.61:0/775132060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:21.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107b7fe700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f107c144800 0x7f107c13f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:21.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107affd700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f107c0aea80 0x7f107c14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:21.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 -- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f107c144800 msgr2=0x7f107c13f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:21.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f107c144800 0x7f107c13f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 -- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f107c0aea80 msgr2=0x7f107c14fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f107c0aea80 0x7f107c14fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 -- 172.21.15.61:0/775132060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c00b1e0 con 0x7f107c0b0c20 2022-07-12T08:46:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f107a7fc700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 0x7f107c09e370 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f106400a000 tx=0x7f1064008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:21.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1064011780 con 0x7f107c0b0c20 2022-07-12T08:46:21.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10640118e0 con 0x7f107c0b0c20 2022-07-12T08:46:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c150a50 con 0x7f107c0b0c20 2022-07-12T08:46:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1064011c40 con 0x7f107c0b0c20 2022-07-12T08:46:21.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.364+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c09ea80 con 0x7f107c0b0c20 2022-07-12T08:46:21.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.368+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c13f930 con 0x7f107c0b0c20 2022-07-12T08:46:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.368+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f106401c460 con 0x7f107c0b0c20 2022-07-12T08:46:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.368+0000 7f1073fff700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1060053ad0 0x7f1060055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.368+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f106406c550 con 0x7f107c0b0c20 2022-07-12T08:46:21.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.368+0000 7f107affd700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1060053ad0 0x7f1060055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:21.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.368+0000 7f107affd700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1060053ad0 0x7f1060055f90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f107400d000 tx=0x7f1074007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.376+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1064043510 con 0x7f107c0b0c20 2022-07-12T08:46:21.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:21.504+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f107c13f930 con 0x7f107c0b0c20 2022-07-12T08:46:22.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.092+0000 7f1073fff700 1 -- 172.21.15.61:0/775132060 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v60) v1 ==== 125+0+0 (secure 0 0 0) 0x7f107c13f930 con 0x7f107c0b0c20 2022-07-12T08:46:22.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1060053ad0 msgr2=0x7f1060055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1060053ad0 0x7f1060055f90 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f107400d000 tx=0x7f1074007f10).stop 2022-07-12T08:46:22.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 msgr2=0x7f107c09e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 0x7f107c09e370 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f106400a000 tx=0x7f1064008040).stop 2022-07-12T08:46:22.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 shutdown_connections 2022-07-12T08:46:22.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f107c144800 0x7f107c13f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f107c0b0c20 0x7f107c09e370 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1060053ad0 0x7f1060055f90 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 --2- 172.21.15.61:0/775132060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f107c0aea80 0x7f107c14fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 >> 172.21.15.61:0/775132060 conn(0x7f107c0b5290 msgr2=0x7f107c0b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:22.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 shutdown_connections 2022-07-12T08:46:22.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.100+0000 7f1081a3c700 1 -- 172.21.15.61:0/775132060 wait complete. 2022-07-12T08:46:22.127 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:46:22.130 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:46:22.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.236+0000 7fb642d3a700 1 Processor -- start 2022-07-12T08:46:22.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.236+0000 7fb642d3a700 1 -- start start 2022-07-12T08:46:22.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb642d3a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb63c0b27c0 0x7fb63c0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb642d3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 0x7fb63c0af590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb642d3a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63c0b30b0 con 0x7fb63c0afa80 2022-07-12T08:46:22.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb642d3a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63c0b2ba0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb642d3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c0b32e0 con 0x7fb63c060250 2022-07-12T08:46:22.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 0x7fb63c0af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb640ad6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb63c0b27c0 0x7fb63c0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 0x7fb63c0af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38630/0 (socket says 172.21.15.61:38630) 2022-07-12T08:46:22.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb640ad6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb63c0b27c0 0x7fb63c0ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:35006/0 (socket says 172.21.15.61:35006) 2022-07-12T08:46:22.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 -- 172.21.15.61:0/3660105921 learned_addr learned my addr 172.21.15.61:0/3660105921 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:22.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb6412d7700 1 --1- 172.21.15.61:0/3660105921 >> v1:172.21.15.111:6789/0 conn(0x7fb63c0afa80 0x7fb63c13ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:58256/0 (socket says 172.21.15.61:58256) 2022-07-12T08:46:22.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 -- 172.21.15.61:0/3660105921 >> v2:172.21.15.111:3300/0 conn(0x7fb63c0b27c0 msgr2=0x7fb63c0ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:22.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 --2- 172.21.15.61:0/3660105921 >> v2:172.21.15.111:3300/0 conn(0x7fb63c0b27c0 0x7fb63c0ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 -- 172.21.15.61:0/3660105921 >> v1:172.21.15.111:6789/0 conn(0x7fb63c0afa80 legacy=0x7fb63c13ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 -- 172.21.15.61:0/3660105921 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63c0a9310 con 0x7fb63c060250 2022-07-12T08:46:22.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63bfff700 1 --2- 172.21.15.61:0/3660105921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 0x7fb63c0af590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb628002f70 tx=0x7fb628006eb0).ready entity=mon.2 client_cookie=8d7fbf66c5e123d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63b7fe700 1 -- 172.21.15.61:0/3660105921 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb62800d570 con 0x7fb63c060250 2022-07-12T08:46:22.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63b7fe700 1 -- 172.21.15.61:0/3660105921 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628000e90 con 0x7fb63c060250 2022-07-12T08:46:22.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.240+0000 7fb63b7fe700 1 -- 172.21.15.61:0/3660105921 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb62800d900 con 0x7fb63c060250 2022-07-12T08:46:22.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- 172.21.15.61:0/3660105921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 msgr2=0x7fb63c0af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 --2- 172.21.15.61:0/3660105921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 0x7fb63c0af590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fb628002f70 tx=0x7fb628006eb0).stop 2022-07-12T08:46:22.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- 172.21.15.61:0/3660105921 shutdown_connections 2022-07-12T08:46:22.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 --2- 172.21.15.61:0/3660105921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c060250 0x7fb63c0af590 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 --2- 172.21.15.61:0/3660105921 >> v2:172.21.15.111:3300/0 conn(0x7fb63c0b27c0 0x7fb63c0ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- 172.21.15.61:0/3660105921 >> 172.21.15.61:0/3660105921 conn(0x7fb63c0633a0 msgr2=0x7fb63c0637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:22.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- 172.21.15.61:0/3660105921 shutdown_connections 2022-07-12T08:46:22.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- 172.21.15.61:0/3660105921 wait complete. 2022-07-12T08:46:22.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 Processor -- start 2022-07-12T08:46:22.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- start start 2022-07-12T08:46:22.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb63c060250 0x7fb63c141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c0afa80 0x7fb63c13a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 0x7fb63c13ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c14f6f0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c14f990 con 0x7fb63c060250 2022-07-12T08:46:22.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.244+0000 7fb642d3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63c14f380 con 0x7fb63c0afa80 2022-07-12T08:46:22.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 0x7fb63c13ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 0x7fb63c13ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56442/0 (socket says 172.21.15.61:56442) 2022-07-12T08:46:22.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 -- 172.21.15.61:0/2992508344 learned_addr learned my addr 172.21.15.61:0/2992508344 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:22.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb640ad6700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb63c060250 0x7fb63c141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb63bfff700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c0afa80 0x7fb63c13a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 -- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c0afa80 msgr2=0x7fb63c13a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c0afa80 0x7fb63c13a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 -- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb63c060250 msgr2=0x7fb63c141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb63c060250 0x7fb63c141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 -- 172.21.15.61:0/2992508344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62800b1e0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb6412d7700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 0x7fb63c13ad80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fb63c0b2e70 tx=0x7fb634007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb634005ae0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634005c40 con 0x7fb63c0b27c0 2022-07-12T08:46:22.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb63c066ca0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6340070d0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63c141710 con 0x7fb63c0b27c0 2022-07-12T08:46:22.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.248+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63c14f540 con 0x7fb63c0b27c0 2022-07-12T08:46:22.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.252+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb63401a060 con 0x7fb63c0b27c0 2022-07-12T08:46:22.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.252+0000 7fb639ffb700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb61c053a40 0x7fb61c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.252+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fb63406b7a0 con 0x7fb63c0b27c0 2022-07-12T08:46:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.252+0000 7fb640ad6700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb61c053a40 0x7fb61c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.252+0000 7fb640ad6700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb61c053a40 0x7fb61c055f00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fb63000b000 tx=0x7fb630006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.260+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb634042770 con 0x7fb63c0b27c0 2022-07-12T08:46:22.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.424+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fb63c057460 con 0x7fb63c0b27c0 2022-07-12T08:46:22.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.436+0000 7fb639ffb700 1 -- 172.21.15.61:0/2992508344 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 11 and data pool 12 v30) v1 ==== 189+0+0 (secure 0 0 0) 0x7fb634018070 con 0x7fb63c0b27c0 2022-07-12T08:46:22.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb61c053a40 msgr2=0x7fb61c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb61c053a40 0x7fb61c055f00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fb63000b000 tx=0x7fb630006df0).stop 2022-07-12T08:46:22.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 msgr2=0x7fb63c13ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 0x7fb63c13ad80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fb63c0b2e70 tx=0x7fb634007f40).stop 2022-07-12T08:46:22.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 shutdown_connections 2022-07-12T08:46:22.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb63c0b27c0 0x7fb63c13ad80 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb63c0afa80 0x7fb63c13a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb61c053a40 0x7fb61c055f00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 --2- 172.21.15.61:0/2992508344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb63c060250 0x7fb63c141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.444+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 >> 172.21.15.61:0/2992508344 conn(0x7fb63c0633a0 msgr2=0x7fb63c13ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:22.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.448+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 shutdown_connections 2022-07-12T08:46:22.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.448+0000 7fb642d3a700 1 -- 172.21.15.61:0/2992508344 wait complete. 2022-07-12T08:46:22.463 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 11 and data pool 12 2022-07-12T08:46:22.478 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:46:22.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.580+0000 7f00a7fff700 1 Processor -- start 2022-07-12T08:46:22.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.580+0000 7f00a7fff700 1 -- start start 2022-07-12T08:46:22.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00a805f480 0x7f00a805f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f00a8140160 0x7f00a813c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a805fd50 con 0x7f00a813cc00 2022-07-12T08:46:22.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a8142420 con 0x7f00a805f480 2022-07-12T08:46:22.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a8140680 con 0x7f00a8140160 2022-07-12T08:46:22.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00a805f480 0x7f00a805f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00a805f480 0x7f00a805f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56458/0 (socket says 172.21.15.61:56458) 2022-07-12T08:46:22.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 -- 172.21.15.61:0/1206311976 learned_addr learned my addr 172.21.15.61:0/1206311976 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:22.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f009ffff700 1 --2- 172.21.15.61:0/1206311976 >> v2:172.21.15.111:3300/0 conn(0x7f00a8140160 0x7f00a813c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 -- 172.21.15.61:0/1206311976 >> v2:172.21.15.111:3300/0 conn(0x7f00a8140160 msgr2=0x7f00a813c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:22.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 --2- 172.21.15.61:0/1206311976 >> v2:172.21.15.111:3300/0 conn(0x7f00a8140160 0x7f00a813c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 -- 172.21.15.61:0/1206311976 >> v1:172.21.15.111:6789/0 conn(0x7f00a813cc00 legacy=0x7f00a8142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 -- 172.21.15.61:0/1206311976 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a8140cd0 con 0x7f00a805f480 2022-07-12T08:46:22.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a6ffd700 1 --2- 172.21.15.61:0/1206311976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a805f480 0x7f00a805f860 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f0098002f70 tx=0x7f0098006f40).ready entity=mon.0 client_cookie=f682e0dabae8df4d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a67fc700 1 -- 172.21.15.61:0/1206311976 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009800dc70 con 0x7f00a805f480 2022-07-12T08:46:22.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a67fc700 1 -- 172.21.15.61:0/1206311976 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0098000e90 con 0x7f00a805f480 2022-07-12T08:46:22.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a67fc700 1 -- 172.21.15.61:0/1206311976 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f009800be90 con 0x7f00a805f480 2022-07-12T08:46:22.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- 172.21.15.61:0/1206311976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a805f480 msgr2=0x7f00a805f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1206311976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a805f480 0x7f00a805f860 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f0098002f70 tx=0x7f0098006f40).stop 2022-07-12T08:46:22.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- 172.21.15.61:0/1206311976 shutdown_connections 2022-07-12T08:46:22.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1206311976 >> v2:172.21.15.111:3300/0 conn(0x7f00a8140160 0x7f00a813c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1206311976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a805f480 0x7f00a805f860 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- 172.21.15.61:0/1206311976 >> 172.21.15.61:0/1206311976 conn(0x7f00a8063a60 msgr2=0x7f00a8063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:22.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- 172.21.15.61:0/1206311976 shutdown_connections 2022-07-12T08:46:22.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.584+0000 7f00a7fff700 1 -- 172.21.15.61:0/1206311976 wait complete. 2022-07-12T08:46:22.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 Processor -- start 2022-07-12T08:46:22.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 -- start start 2022-07-12T08:46:22.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00a805f480 0x7f00a80af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 0x7f00a80b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a8140160 0x7f00a80b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a80b0540 con 0x7f00a813cc00 2022-07-12T08:46:22.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a813ed00 con 0x7f00a8140160 2022-07-12T08:46:22.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f00a7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a81386e0 con 0x7f00a805f480 2022-07-12T08:46:22.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.588+0000 7f009ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 0x7f00a80b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 0x7f00a80b0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56466/0 (socket says 172.21.15.61:56466) 2022-07-12T08:46:22.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 -- 172.21.15.61:0/1106341878 learned_addr learned my addr 172.21.15.61:0/1106341878 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:22.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a6ffd700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00a805f480 0x7f00a80af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a77fe700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a8140160 0x7f00a80b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 -- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00a805f480 msgr2=0x7f00a80af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00a805f480 0x7f00a80af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 -- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a8140160 msgr2=0x7f00a80b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a8140160 0x7f00a80b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 -- 172.21.15.61:0/1106341878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009800b460 con 0x7f00a813cc00 2022-07-12T08:46:22.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f009ffff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 0x7f00a80b0fd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f00a813e3e0 tx=0x7f0090008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0090011780 con 0x7f00a813cc00 2022-07-12T08:46:22.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00900118e0 con 0x7f00a813cc00 2022-07-12T08:46:22.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a813ea60 con 0x7f00a813cc00 2022-07-12T08:46:22.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0090011c40 con 0x7f00a813cc00 2022-07-12T08:46:22.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a813f460 con 0x7f00a813cc00 2022-07-12T08:46:22.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0090012420 con 0x7f00a813cc00 2022-07-12T08:46:22.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a8057460 con 0x7f00a813cc00 2022-07-12T08:46:22.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a4ff9700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f008c053ad0 0x7f008c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f009006be90 con 0x7f00a813cc00 2022-07-12T08:46:22.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.592+0000 7f00a6ffd700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f008c053ad0 0x7f008c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.596+0000 7f00a6ffd700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f008c053ad0 0x7f008c055f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f009801c000 tx=0x7f00a80ae4e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.600+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0090042d90 con 0x7f00a813cc00 2022-07-12T08:46:22.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.724+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00a8138b00 con 0x7f00a813cc00 2022-07-12T08:46:22.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.728+0000 7f00a4ff9700 1 -- 172.21.15.61:0/1106341878 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f00900463b0 con 0x7f00a813cc00 2022-07-12T08:46:22.736 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:22.736 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":61,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:46:22.431627+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:46:21.097184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-12T08:46:21.563425+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:46:22.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.728+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f008c053ad0 msgr2=0x7f008c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.728+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f008c053ad0 0x7f008c055f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f009801c000 tx=0x7f00a80ae4e0).stop 2022-07-12T08:46:22.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.728+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 msgr2=0x7f00a80b0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.728+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 0x7f00a80b0fd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f00a813e3e0 tx=0x7f0090008040).stop 2022-07-12T08:46:22.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 shutdown_connections 2022-07-12T08:46:22.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f008c053ad0 0x7f008c055f90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00a8140160 0x7f00a80b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00a813cc00 0x7f00a80b0fd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 --2- 172.21.15.61:0/1106341878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00a805f480 0x7f00a80af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 >> 172.21.15.61:0/1106341878 conn(0x7f00a8063a60 msgr2=0x7f00a8139790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:22.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 shutdown_connections 2022-07-12T08:46:22.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.732+0000 7f00a7fff700 1 -- 172.21.15.61:0/1106341878 wait complete. 2022-07-12T08:46:22.764 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:46:22.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.872+0000 7f54b5a7b700 1 Processor -- start 2022-07-12T08:46:22.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.872+0000 7f54b5a7b700 1 -- start start 2022-07-12T08:46:22.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b00b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f54b00b47f0 0x7f54b00b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b013d240 con 0x7f54b0142d50 2022-07-12T08:46:22.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54b00b24c0 con 0x7f54b00b47f0 2022-07-12T08:46:22.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b01431f0 con 0x7f54b013ce60 2022-07-12T08:46:22.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54aeffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f54b0142d50 0x7f54b00b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44690/0 (socket says 172.21.15.61:44690) 2022-07-12T08:46:22.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54aeffd700 1 -- 172.21.15.61:0/3741591874 learned_addr learned my addr 172.21.15.61:0/3741591874 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:22.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54affff700 1 --2- 172.21.15.61:0/3741591874 >> v2:172.21.15.61:3300/0 conn(0x7f54b00b47f0 0x7f54b00b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54af7fe700 1 --2- 172.21.15.61:0/3741591874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b00b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54ae7fc700 1 -- 172.21.15.61:0/3741591874 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572579482 0 0) 0x7f54b013d240 con 0x7f54b0142d50 2022-07-12T08:46:22.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54ae7fc700 1 -- 172.21.15.61:0/3741591874 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5494003260 con 0x7f54b0142d50 2022-07-12T08:46:22.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54af7fe700 1 -- 172.21.15.61:0/3741591874 >> v2:172.21.15.61:3300/0 conn(0x7f54b00b47f0 msgr2=0x7f54b00b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:22.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54af7fe700 1 --2- 172.21.15.61:0/3741591874 >> v2:172.21.15.61:3300/0 conn(0x7f54b00b47f0 0x7f54b00b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54af7fe700 1 -- 172.21.15.61:0/3741591874 >> v1:172.21.15.61:6789/0 conn(0x7f54b0142d50 legacy=0x7f54b00b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54af7fe700 1 -- 172.21.15.61:0/3741591874 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0143490 con 0x7f54b013ce60 2022-07-12T08:46:22.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54af7fe700 1 --2- 172.21.15.61:0/3741591874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b00b15a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f54a0002f70 tx=0x7f54a0006df0).ready entity=mon.2 client_cookie=93570fada432b354 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54ae7fc700 1 -- 172.21.15.61:0/3741591874 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54a000d4a0 con 0x7f54b013ce60 2022-07-12T08:46:22.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54ae7fc700 1 -- 172.21.15.61:0/3741591874 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a0000e90 con 0x7f54b013ce60 2022-07-12T08:46:22.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54ae7fc700 1 -- 172.21.15.61:0/3741591874 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54a000d7f0 con 0x7f54b013ce60 2022-07-12T08:46:22.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- 172.21.15.61:0/3741591874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 msgr2=0x7f54b00b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/3741591874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b00b15a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f54a0002f70 tx=0x7f54a0006df0).stop 2022-07-12T08:46:22.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- 172.21.15.61:0/3741591874 shutdown_connections 2022-07-12T08:46:22.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/3741591874 >> v2:172.21.15.61:3300/0 conn(0x7f54b00b47f0 0x7f54b00b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/3741591874 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b00b15a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- 172.21.15.61:0/3741591874 >> 172.21.15.61:0/3741591874 conn(0x7f54b005f9f0 msgr2=0x7f54b0066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:22.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- 172.21.15.61:0/3741591874 shutdown_connections 2022-07-12T08:46:22.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.876+0000 7f54b5a7b700 1 -- 172.21.15.61:0/3741591874 wait complete. 2022-07-12T08:46:22.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 Processor -- start 2022-07-12T08:46:22.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 -- start start 2022-07-12T08:46:22.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b00b47f0 0x7f54b01477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b0146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 0x7f54b0151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b0146c10 con 0x7f54b0142d50 2022-07-12T08:46:22.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b013ed10 con 0x7f54b00b47f0 2022-07-12T08:46:22.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54b5a7b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b013efb0 con 0x7f54b013ce60 2022-07-12T08:46:22.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 0x7f54b0151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 0x7f54b0151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56482/0 (socket says 172.21.15.61:56482) 2022-07-12T08:46:22.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b0146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54af7fe700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b00b47f0 0x7f54b01477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.880+0000 7f54affff700 1 -- 172.21.15.61:0/2500593575 learned_addr learned my addr 172.21.15.61:0/2500593575 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:22.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54affff700 1 -- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 msgr2=0x7f54b0146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54affff700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b0146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54affff700 1 -- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b00b47f0 msgr2=0x7f54b01477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:22.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54affff700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b00b47f0 0x7f54b01477a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:22.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54affff700 1 -- 172.21.15.61:0/2500593575 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a000b180 con 0x7f54b0142d50 2022-07-12T08:46:22.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54affff700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 0x7f54b0151c80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f54a8006e00 tx=0x7f54a80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54a8012600 con 0x7f54b0142d50 2022-07-12T08:46:22.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54a8012760 con 0x7f54b0142d50 2022-07-12T08:46:22.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b013f250 con 0x7f54b0142d50 2022-07-12T08:46:22.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54a8012ac0 con 0x7f54b0142d50 2022-07-12T08:46:22.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b013f910 con 0x7f54b0142d50 2022-07-12T08:46:22.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b013dbe0 con 0x7f54b0142d50 2022-07-12T08:46:22.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f54a8022070 con 0x7f54b0142d50 2022-07-12T08:46:22.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54acff9700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5488053a40 0x7f5488055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:22.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f54a806c510 con 0x7f54b0142d50 2022-07-12T08:46:22.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54af7fe700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5488053a40 0x7f5488055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:22.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.884+0000 7f54af7fe700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5488053a40 0x7f5488055f00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f54a001a000 tx=0x7f54a0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:22.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:22.892+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54b013dbe0 con 0x7f54b0142d50 2022-07-12T08:46:23.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.020+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54b013dbe0 con 0x7f54b0142d50 2022-07-12T08:46:23.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.020+0000 7f54acff9700 1 -- 172.21.15.61:0/2500593575 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f54b013dbe0 con 0x7f54b0142d50 2022-07-12T08:46:23.033 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:23.033 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":61,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:46:22.431627+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:46:21.097184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-12T08:46:21.563425+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:46:23.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5488053a40 msgr2=0x7f5488055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5488053a40 0x7f5488055f00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f54a001a000 tx=0x7f54a0014000).stop 2022-07-12T08:46:23.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 msgr2=0x7f54b0151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 0x7f54b0151c80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f54a8006e00 tx=0x7f54a80091d0).stop 2022-07-12T08:46:23.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 shutdown_connections 2022-07-12T08:46:23.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54b0142d50 0x7f54b0151c80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f54b013ce60 0x7f54b0146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5488053a40 0x7f5488055f00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 --2- 172.21.15.61:0/2500593575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54b00b47f0 0x7f54b01477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 >> 172.21.15.61:0/2500593575 conn(0x7f54b005f9f0 msgr2=0x7f54b00b4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:23.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 shutdown_connections 2022-07-12T08:46:23.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.024+0000 7f54b5a7b700 1 -- 172.21.15.61:0/2500593575 wait complete. 2022-07-12T08:46:23.041 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:46:23.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.148+0000 7f326359e700 1 Processor -- start 2022-07-12T08:46:23.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.148+0000 7f326359e700 1 -- start start 2022-07-12T08:46:23.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 0x7f32640af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f32640afdc0 0x7f326409a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3264099010 con 0x7f3264060c10 2022-07-12T08:46:23.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326414b780 con 0x7f32640afdc0 2022-07-12T08:46:23.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3264060ff0 con 0x7f32640b1000 2022-07-12T08:46:23.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3261d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 0x7f32640af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3262d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f32640afdc0 0x7f326409a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3261d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 0x7f32640af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38684/0 (socket says 172.21.15.61:38684) 2022-07-12T08:46:23.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3261d9b700 1 -- 172.21.15.61:0/2430055834 learned_addr learned my addr 172.21.15.61:0/2430055834 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:23.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326259c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3264060c10 0x7f326414b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:58306/0 (socket says 172.21.15.61:58306) 2022-07-12T08:46:23.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3262d9d700 1 -- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 msgr2=0x7f32640af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:23.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3262d9d700 1 --2- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 0x7f32640af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3262d9d700 1 -- 172.21.15.61:0/2430055834 >> v1:172.21.15.111:6789/0 conn(0x7f3264060c10 legacy=0x7f326414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326159a700 1 -- 172.21.15.61:0/2430055834 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843591072 0 0) 0x7f3264099010 con 0x7f3264060c10 2022-07-12T08:46:23.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3262d9d700 1 -- 172.21.15.61:0/2430055834 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326414a930 con 0x7f32640afdc0 2022-07-12T08:46:23.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3261d9b700 1 --2- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 0x7f32640af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:23.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f3262d9d700 1 --2- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640afdc0 0x7f326409a1f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f325c002f70 tx=0x7f325c006df0).ready entity=mon.1 client_cookie=fd629f0235e41445 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326159a700 1 -- 172.21.15.61:0/2430055834 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325c00d570 con 0x7f32640afdc0 2022-07-12T08:46:23.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326159a700 1 -- 172.21.15.61:0/2430055834 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c000e90 con 0x7f32640afdc0 2022-07-12T08:46:23.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326159a700 1 -- 172.21.15.61:0/2430055834 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325c00d8c0 con 0x7f32640afdc0 2022-07-12T08:46:23.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640afdc0 msgr2=0x7f326409a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 --2- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640afdc0 0x7f326409a1f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f325c002f70 tx=0x7f325c006df0).stop 2022-07-12T08:46:23.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- 172.21.15.61:0/2430055834 shutdown_connections 2022-07-12T08:46:23.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 --2- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640afdc0 0x7f326409a1f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 --2- 172.21.15.61:0/2430055834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f32640b1000 0x7f32640af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- 172.21.15.61:0/2430055834 >> 172.21.15.61:0/2430055834 conn(0x7f326405bd10 msgr2=0x7f326405c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:23.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- 172.21.15.61:0/2430055834 shutdown_connections 2022-07-12T08:46:23.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.152+0000 7f326359e700 1 -- 172.21.15.61:0/2430055834 wait complete. 2022-07-12T08:46:23.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 Processor -- start 2022-07-12T08:46:23.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 -- start start 2022-07-12T08:46:23.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3264060c10 0x7f3264099fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 0x7f3264148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b1000 0x7f3264148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32640aeb00 con 0x7f32640afdc0 2022-07-12T08:46:23.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32640ae860 con 0x7f32640b1000 2022-07-12T08:46:23.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3264062860 con 0x7f3264060c10 2022-07-12T08:46:23.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 0x7f3264148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 0x7f3264148110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56494/0 (socket says 172.21.15.61:56494) 2022-07-12T08:46:23.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 -- 172.21.15.61:0/3963045696 learned_addr learned my addr 172.21.15.61:0/3963045696 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:23.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326259c700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3264060c10 0x7f3264099fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3262d9d700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b1000 0x7f3264148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 -- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3264060c10 msgr2=0x7f3264099fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3264060c10 0x7f3264099fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 -- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b1000 msgr2=0x7f3264148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b1000 0x7f3264148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 -- 172.21.15.61:0/3963045696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3254009530 con 0x7f32640afdc0 2022-07-12T08:46:23.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3262d9d700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b1000 0x7f3264148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:23.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f3261d9b700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 0x7f3264148110 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f3254006b60 tx=0x7f3254009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325400c920 con 0x7f32640afdc0 2022-07-12T08:46:23.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325400ca80 con 0x7f32640afdc0 2022-07-12T08:46:23.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c00b180 con 0x7f32640afdc0 2022-07-12T08:46:23.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325400cde0 con 0x7f32640afdc0 2022-07-12T08:46:23.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.156+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3264148af0 con 0x7f32640afdc0 2022-07-12T08:46:23.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.160+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3264057460 con 0x7f32640afdc0 2022-07-12T08:46:23.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.160+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3254022070 con 0x7f32640afdc0 2022-07-12T08:46:23.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.160+0000 7f32537fe700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3240053a40 0x7f3240055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.160+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f325406c750 con 0x7f32640afdc0 2022-07-12T08:46:23.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.160+0000 7f326259c700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3240053a40 0x7f3240055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.164+0000 7f326259c700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3240053a40 0x7f3240055f00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f325800d000 tx=0x7f3258007a90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.172+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f32540435a0 con 0x7f32640afdc0 2022-07-12T08:46:23.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.336+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f326406b2b0 con 0x7f32640afdc0 2022-07-12T08:46:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.444+0000 7f32537fe700 1 -- 172.21.15.61:0/3963045696 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3254046bc0 con 0x7f32640afdc0 2022-07-12T08:46:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3240053a40 msgr2=0x7f3240055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3240053a40 0x7f3240055f00 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f325800d000 tx=0x7f3258007a90).stop 2022-07-12T08:46:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 msgr2=0x7f3264148110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 0x7f3264148110 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f3254006b60 tx=0x7f3254009290).stop 2022-07-12T08:46:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 shutdown_connections 2022-07-12T08:46:23.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3240053a40 0x7f3240055f00 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32640b1000 0x7f3264148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32640afdc0 0x7f3264148110 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 --2- 172.21.15.61:0/3963045696 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3264060c10 0x7f3264099fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 >> 172.21.15.61:0/3963045696 conn(0x7f326405bd10 msgr2=0x7f326405c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 shutdown_connections 2022-07-12T08:46:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.448+0000 7f326359e700 1 -- 172.21.15.61:0/3963045696 wait complete. 2022-07-12T08:46:23.478 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:23.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 Processor -- start 2022-07-12T08:46:23.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 -- start start 2022-07-12T08:46:23.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc500af280 0x7efc50060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc500af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc500aecb0 con 0x7efc500b3f60 2022-07-12T08:46:23.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc50143b00 con 0x7efc500af280 2022-07-12T08:46:23.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc56802700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc501435c0 con 0x7efc5009bce0 2022-07-12T08:46:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc500af280 0x7efc50060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc500af280 0x7efc50060680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56500/0 (socket says 172.21.15.61:56500) 2022-07-12T08:46:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 -- 172.21.15.61:0/804851194 learned_addr learned my addr 172.21.15.61:0/804851194 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc54fff700 1 --2- 172.21.15.61:0/804851194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc500af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 -- 172.21.15.61:0/804851194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 msgr2=0x7efc500af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 --2- 172.21.15.61:0/804851194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc500af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 -- 172.21.15.61:0/804851194 >> v1:172.21.15.111:6789/0 conn(0x7efc500b3f60 legacy=0x7efc500b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 -- 172.21.15.61:0/804851194 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50143860 con 0x7efc500af280 2022-07-12T08:46:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc55800700 1 --2- 172.21.15.61:0/804851194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50060680 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7efc4c0032f0 tx=0x7efc4c009250).ready entity=mon.0 client_cookie=3cbaeeb6b631677e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc47fff700 1 -- 172.21.15.61:0/804851194 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc4c009ea0 con 0x7efc500af280 2022-07-12T08:46:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc47fff700 1 -- 172.21.15.61:0/804851194 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c00b8a0 con 0x7efc500af280 2022-07-12T08:46:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.588+0000 7efc47fff700 1 -- 172.21.15.61:0/804851194 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc4c00ba10 con 0x7efc500af280 2022-07-12T08:46:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 -- 172.21.15.61:0/804851194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 msgr2=0x7efc50060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 --2- 172.21.15.61:0/804851194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50060680 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7efc4c0032f0 tx=0x7efc4c009250).stop 2022-07-12T08:46:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 -- 172.21.15.61:0/804851194 shutdown_connections 2022-07-12T08:46:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 --2- 172.21.15.61:0/804851194 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc500af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 --2- 172.21.15.61:0/804851194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50060680 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 -- 172.21.15.61:0/804851194 >> 172.21.15.61:0/804851194 conn(0x7efc500b92c0 msgr2=0x7efc500b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 -- 172.21.15.61:0/804851194 shutdown_connections 2022-07-12T08:46:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 -- 172.21.15.61:0/804851194 wait complete. 2022-07-12T08:46:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 Processor -- start 2022-07-12T08:46:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.592+0000 7efc56802700 1 -- start start 2022-07-12T08:46:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc50155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc500b3f60 0x7efc5009c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b0880 con 0x7efc500af280 2022-07-12T08:46:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b24b0 con 0x7efc500b3f60 2022-07-12T08:46:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc50142690 con 0x7efc5009bce0 2022-07-12T08:46:23.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc55800700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc50155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56508/0 (socket says 172.21.15.61:56508) 2022-07-12T08:46:23.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 -- 172.21.15.61:0/441037251 learned_addr learned my addr 172.21.15.61:0/441037251 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:23.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56001700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc500b3f60 0x7efc5009c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 -- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 msgr2=0x7efc50155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc50155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 -- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc500b3f60 msgr2=0x7efc5009c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc500b3f60 0x7efc5009c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 -- 172.21.15.61:0/441037251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c0094d0 con 0x7efc500af280 2022-07-12T08:46:23.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc54fff700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50147b10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7efc4000c000 tx=0x7efc40006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc40010780 con 0x7efc500af280 2022-07-12T08:46:23.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc400108e0 con 0x7efc500af280 2022-07-12T08:46:23.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc500b1340 con 0x7efc500af280 2022-07-12T08:46:23.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc40010c40 con 0x7efc500af280 2022-07-12T08:46:23.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.596+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5009d000 con 0x7efc500af280 2022-07-12T08:46:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.600+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc50057460 con 0x7efc500af280 2022-07-12T08:46:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.600+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efc4001c460 con 0x7efc500af280 2022-07-12T08:46:23.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.600+0000 7efc467fc700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc38053a40 0x7efc38055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.600+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efc4006c240 con 0x7efc500af280 2022-07-12T08:46:23.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.600+0000 7efc55800700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc38053a40 0x7efc38055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.604+0000 7efc55800700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc38053a40 0x7efc38055f00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7efc4c01b000 tx=0x7efc4c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.608+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc40043140 con 0x7efc500af280 2022-07-12T08:46:23.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.772+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc500afeb0 con 0x7efc500af280 2022-07-12T08:46:23.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.776+0000 7efc467fc700 1 -- 172.21.15.61:0/441037251 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7efc40046760 con 0x7efc500af280 2022-07-12T08:46:23.781 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:23.781 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:23.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc38053a40 msgr2=0x7efc38055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc38053a40 0x7efc38055f00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7efc4c01b000 tx=0x7efc4c016000).stop 2022-07-12T08:46:23.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 msgr2=0x7efc50147b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50147b10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7efc4000c000 tx=0x7efc40006df0).stop 2022-07-12T08:46:23.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 shutdown_connections 2022-07-12T08:46:23.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc38053a40 0x7efc38055f00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc500b3f60 0x7efc5009c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc500af280 0x7efc50147b10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 --2- 172.21.15.61:0/441037251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc5009bce0 0x7efc50155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 >> 172.21.15.61:0/441037251 conn(0x7efc500b92c0 msgr2=0x7efc500b8190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:23.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 shutdown_connections 2022-07-12T08:46:23.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.780+0000 7efc56802700 1 -- 172.21.15.61:0/441037251 wait complete. 2022-07-12T08:46:23.798 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:23.807 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:46:23.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 Processor -- start 2022-07-12T08:46:23.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 -- start start 2022-07-12T08:46:23.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380ab290 0x7f2c380ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2c380afa60 0x7f2c380aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c380abc70 con 0x7f2c3805f140 2022-07-12T08:46:23.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c3805f580 con 0x7f2c380afa60 2022-07-12T08:46:23.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.932+0000 7f2c3759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3805c720 con 0x7f2c380ab290 2022-07-12T08:46:23.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2c380afa60 0x7f2c380aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2c380afa60 0x7f2c380aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56516/0 (socket says 172.21.15.61:56516) 2022-07-12T08:46:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 -- 172.21.15.61:0/2948191115 learned_addr learned my addr 172.21.15.61:0/2948191115 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:23.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3659c700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380ab290 0x7f2c380ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 -- 172.21.15.61:0/2948191115 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380ab290 msgr2=0x7f2c380ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380ab290 0x7f2c380ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 -- 172.21.15.61:0/2948191115 >> v1:172.21.15.61:6789/0 conn(0x7f2c3805f140 legacy=0x7f2c3805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 -- 172.21.15.61:0/2948191115 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3805c9c0 con 0x7f2c380afa60 2022-07-12T08:46:23.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3659c700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380ab290 0x7f2c380ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:23.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c36d9d700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c380afa60 0x7f2c380aff10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f2c28002f70 tx=0x7f2c28006eb0).ready entity=mon.0 client_cookie=2555ecaac2f25261 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:23.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3559a700 1 -- 172.21.15.61:0/2948191115 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c2800d5c0 con 0x7f2c380afa60 2022-07-12T08:46:23.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3559a700 1 -- 172.21.15.61:0/2948191115 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c28000e90 con 0x7f2c380afa60 2022-07-12T08:46:23.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3559a700 1 -- 172.21.15.61:0/2948191115 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c2800d950 con 0x7f2c380afa60 2022-07-12T08:46:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 -- 172.21.15.61:0/2948191115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c380afa60 msgr2=0x7f2c380aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c380afa60 0x7f2c380aff10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f2c28002f70 tx=0x7f2c28006eb0).stop 2022-07-12T08:46:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 -- 172.21.15.61:0/2948191115 shutdown_connections 2022-07-12T08:46:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c380afa60 0x7f2c380aff10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 --2- 172.21.15.61:0/2948191115 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380ab290 0x7f2c380ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 -- 172.21.15.61:0/2948191115 >> 172.21.15.61:0/2948191115 conn(0x7f2c380b1950 msgr2=0x7f2c380b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 -- 172.21.15.61:0/2948191115 shutdown_connections 2022-07-12T08:46:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 -- 172.21.15.61:0/2948191115 wait complete. 2022-07-12T08:46:23.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 Processor -- start 2022-07-12T08:46:23.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.936+0000 7f2c3759e700 1 -- start start 2022-07-12T08:46:23.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 0x7f2c380acba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c380ab290 0x7f2c38064970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380afa60 0x7f2c381597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:23.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38063d00 con 0x7f2c3805f140 2022-07-12T08:46:23.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38064230 con 0x7f2c380ab290 2022-07-12T08:46:23.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c380644d0 con 0x7f2c380afa60 2022-07-12T08:46:23.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 0x7f2c380acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 0x7f2c380acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56520/0 (socket says 172.21.15.61:56520) 2022-07-12T08:46:23.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 -- 172.21.15.61:0/3372803338 learned_addr learned my addr 172.21.15.61:0/3372803338 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:23.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c36d9d700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380afa60 0x7f2c381597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c35d9b700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c380ab290 0x7f2c38064970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:23.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 -- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380afa60 msgr2=0x7f2c381597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380afa60 0x7f2c381597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 -- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c380ab290 msgr2=0x7f2c38064970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:23.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c380ab290 0x7f2c38064970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:23.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 -- 172.21.15.61:0/3372803338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2800b1e0 con 0x7f2c3805f140 2022-07-12T08:46:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3659c700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 0x7f2c380acba0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f2c30006cc0 tx=0x7f2c30008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c30013620 con 0x7f2c3805f140 2022-07-12T08:46:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c30013780 con 0x7f2c3805f140 2022-07-12T08:46:24.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c30013ae0 con 0x7f2c3805f140 2022-07-12T08:46:24.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c380ad740 con 0x7f2c3805f140 2022-07-12T08:46:24.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c38063860 con 0x7f2c3805f140 2022-07-12T08:46:24.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c38063b40 con 0x7f2c3805f140 2022-07-12T08:46:24.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.940+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2c30013c40 con 0x7f2c3805f140 2022-07-12T08:46:24.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.944+0000 7f2c2f7fe700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c1c053af0 0x7f2c1c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.944+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2c3006c030 con 0x7f2c3805f140 2022-07-12T08:46:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.944+0000 7f2c35d9b700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c1c053af0 0x7f2c1c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.944+0000 7f2c35d9b700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c1c053af0 0x7f2c1c055fb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2c2000d000 tx=0x7f2c20000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:23.948+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c30042f30 con 0x7f2c3805f140 2022-07-12T08:46:24.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.076+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c38057460 con 0x7f2c3805f140 2022-07-12T08:46:24.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.076+0000 7f2c2f7fe700 1 -- 172.21.15.61:0/3372803338 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f2c30046550 con 0x7f2c3805f140 2022-07-12T08:46:24.084 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:24.085 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:46:23.436615+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:46:21.097184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-12T08:46:21.563425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:46:24.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.076+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c1c053af0 msgr2=0x7f2c1c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c1c053af0 0x7f2c1c055fb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f2c2000d000 tx=0x7f2c20000bc0).stop 2022-07-12T08:46:24.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 msgr2=0x7f2c380acba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 0x7f2c380acba0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f2c30006cc0 tx=0x7f2c30008040).stop 2022-07-12T08:46:24.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 shutdown_connections 2022-07-12T08:46:24.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2c380afa60 0x7f2c381597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c1c053af0 0x7f2c1c055fb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c380ab290 0x7f2c38064970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 --2- 172.21.15.61:0/3372803338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c3805f140 0x7f2c380acba0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 >> 172.21.15.61:0/3372803338 conn(0x7f2c380b1950 msgr2=0x7f2c380afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 shutdown_connections 2022-07-12T08:46:24.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.080+0000 7f2c3759e700 1 -- 172.21.15.61:0/3372803338 wait complete. 2022-07-12T08:46:24.099 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:46:24.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c7fff700 1 Processor -- start 2022-07-12T08:46:24.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c7fff700 1 -- start start 2022-07-12T08:46:24.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc6c80b02f0 0x7fc6c80618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c8142b90 con 0x7fc6c8138f40 2022-07-12T08:46:24.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c8060ef0 con 0x7fc6c81427b0 2022-07-12T08:46:24.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c8061160 con 0x7fc6c80b02f0 2022-07-12T08:46:24.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc6c80b02f0 0x7fc6c80618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc6c8138f40 0x7fc6c8140370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44750/0 (socket says 172.21.15.61:44750) 2022-07-12T08:46:24.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c77fe700 1 -- 172.21.15.61:0/2808111716 learned_addr learned my addr 172.21.15.61:0/2808111716 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c5ffb700 1 -- 172.21.15.61:0/2808111716 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71501223 0 0) 0x7fc6c8142b90 con 0x7fc6c8138f40 2022-07-12T08:46:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c5ffb700 1 -- 172.21.15.61:0/2808111716 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ac004370 con 0x7fc6c8138f40 2022-07-12T08:46:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c67fc700 1 -- 172.21.15.61:0/2808111716 >> v1:172.21.15.111:6789/0 conn(0x7fc6c81427b0 legacy=0x7fc6c8138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c67fc700 1 -- 172.21.15.61:0/2808111716 >> v1:172.21.15.61:6789/0 conn(0x7fc6c8138f40 legacy=0x7fc6c8140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c5ffb700 1 -- 172.21.15.61:0/2808111716 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567449669 0 0) 0x7fc6c8060ef0 con 0x7fc6c81427b0 2022-07-12T08:46:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.220+0000 7fc6c67fc700 1 -- 172.21.15.61:0/2808111716 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c8061400 con 0x7fc6c80b02f0 2022-07-12T08:46:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c67fc700 1 --2- 172.21.15.61:0/2808111716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c80b02f0 0x7fc6c80618d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fc6bc002f70 tx=0x7fc6bc006df0).ready entity=mon.0 client_cookie=c1c9c9d3c69ee8e7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c5ffb700 1 -- 172.21.15.61:0/2808111716 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6bc00d610 con 0x7fc6c80b02f0 2022-07-12T08:46:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c5ffb700 1 -- 172.21.15.61:0/2808111716 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc000e90 con 0x7fc6c80b02f0 2022-07-12T08:46:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c5ffb700 1 -- 172.21.15.61:0/2808111716 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6bc00d960 con 0x7fc6c80b02f0 2022-07-12T08:46:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 -- 172.21.15.61:0/2808111716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c80b02f0 msgr2=0x7fc6c80618d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/2808111716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c80b02f0 0x7fc6c80618d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fc6bc002f70 tx=0x7fc6bc006df0).stop 2022-07-12T08:46:24.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 -- 172.21.15.61:0/2808111716 shutdown_connections 2022-07-12T08:46:24.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/2808111716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c80b02f0 0x7fc6c80618d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 -- 172.21.15.61:0/2808111716 >> 172.21.15.61:0/2808111716 conn(0x7fc6c80b5860 msgr2=0x7fc6c80b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:24.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 -- 172.21.15.61:0/2808111716 shutdown_connections 2022-07-12T08:46:24.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 -- 172.21.15.61:0/2808111716 wait complete. 2022-07-12T08:46:24.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 Processor -- start 2022-07-12T08:46:24.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.224+0000 7fc6c7fff700 1 -- start start 2022-07-12T08:46:24.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6c80b02f0 0x7fc6c813f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c8138f40 0x7fc6c8066b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 0x7fc6c8151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c80ae530 con 0x7fc6c81427b0 2022-07-12T08:46:24.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c80ae7d0 con 0x7fc6c8138f40 2022-07-12T08:46:24.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c80aea70 con 0x7fc6c80b02f0 2022-07-12T08:46:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 0x7fc6c8151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c6ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6c80b02f0 0x7fc6c813f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 0x7fc6c8151250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56538/0 (socket says 172.21.15.61:56538) 2022-07-12T08:46:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 -- 172.21.15.61:0/1371362422 learned_addr learned my addr 172.21.15.61:0/1371362422 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c67fc700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c8138f40 0x7fc6c8066b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 -- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6c80b02f0 msgr2=0x7fc6c813f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6c80b02f0 0x7fc6c813f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 -- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c8138f40 msgr2=0x7fc6c8066b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c8138f40 0x7fc6c8066b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 -- 172.21.15.61:0/1371362422 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc00b180 con 0x7fc6c81427b0 2022-07-12T08:46:24.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.228+0000 7fc6c77fe700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 0x7fc6c8151250 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fc6c0001490 tx=0x7fc6c0007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c0005ba0 con 0x7fc6c81427b0 2022-07-12T08:46:24.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c80ae220 con 0x7fc6c81427b0 2022-07-12T08:46:24.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c0005d00 con 0x7fc6c81427b0 2022-07-12T08:46:24.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c00072c0 con 0x7fc6c81427b0 2022-07-12T08:46:24.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c80adfd0 con 0x7fc6c81427b0 2022-07-12T08:46:24.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c8142e00 con 0x7fc6c81427b0 2022-07-12T08:46:24.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc6c0010070 con 0x7fc6c81427b0 2022-07-12T08:46:24.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.232+0000 7fc6b7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6a4053920 0x7fc6a4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.236+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc6c006b7e0 con 0x7fc6c81427b0 2022-07-12T08:46:24.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.236+0000 7fc6c6ffd700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6a4053920 0x7fc6a4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.236+0000 7fc6c6ffd700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6a4053920 0x7fc6a4055de0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc6b800d000 tx=0x7fc6b8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.236+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6c00426a0 con 0x7fc6c81427b0 2022-07-12T08:46:24.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.404+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc6c80aec30 con 0x7fc6c81427b0 2022-07-12T08:46:24.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.412+0000 7fc6b7fff700 1 -- 172.21.15.61:0/1371362422 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc6c0018070 con 0x7fc6c81427b0 2022-07-12T08:46:24.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6a4053920 msgr2=0x7fc6a4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6a4053920 0x7fc6a4055de0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc6b800d000 tx=0x7fc6b8007b90).stop 2022-07-12T08:46:24.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 msgr2=0x7fc6c8151250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 0x7fc6c8151250 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fc6c0001490 tx=0x7fc6c0007e00).stop 2022-07-12T08:46:24.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 shutdown_connections 2022-07-12T08:46:24.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6c81427b0 0x7fc6c8151250 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6a4053920 0x7fc6a4055de0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6c8138f40 0x7fc6c8066b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 --2- 172.21.15.61:0/1371362422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6c80b02f0 0x7fc6c813f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 >> 172.21.15.61:0/1371362422 conn(0x7fc6c80b5860 msgr2=0x7fc6c8138810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 shutdown_connections 2022-07-12T08:46:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.416+0000 7fc6c7fff700 1 -- 172.21.15.61:0/1371362422 wait complete. 2022-07-12T08:46:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:46:24.430 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:46:24.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934f59e700 1 Processor -- start 2022-07-12T08:46:24.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934f59e700 1 -- start start 2022-07-12T08:46:24.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 0x7f9350146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f935005ff60 con 0x7f935005d100 2022-07-12T08:46:24.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350146200 con 0x7f93500af490 2022-07-12T08:46:24.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93501464a0 con 0x7f935005cd20 2022-07-12T08:46:24.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f935005d100 0x7f935005f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44764/0 (socket says 172.21.15.61:44764) 2022-07-12T08:46:24.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.544+0000 7f934ed9d700 1 -- 172.21.15.61:0/3217637644 learned_addr learned my addr 172.21.15.61:0/3217637644 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:24.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 0x7f9350146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171747997 0 0) 0x7f935005ff60 con 0x7f935005d100 2022-07-12T08:46:24.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330003260 con 0x7f935005d100 2022-07-12T08:46:24.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779057927 0 0) 0x7f9350146200 con 0x7f93500af490 2022-07-12T08:46:24.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935005ff60 con 0x7f93500af490 2022-07-12T08:46:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 972333593 0 0) 0x7f9330003260 con 0x7f935005d100 2022-07-12T08:46:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 msgr2=0x7f9350146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 0x7f9350146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 >> v1:172.21.15.111:6789/0 conn(0x7f93500af490 legacy=0x7f935005f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350146740 con 0x7f935005d100 2022-07-12T08:46:24.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f93480019e0 con 0x7f935005d100 2022-07-12T08:46:24.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 >> v1:172.21.15.61:6789/0 conn(0x7f935005d100 legacy=0x7f935005f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9330003910 0x7f9330003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9330004500 0x7f9330004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 0x7f9330004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93501464a0 con 0x7f9330004500 2022-07-12T08:46:24.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933000b800 con 0x7f9330003910 2022-07-12T08:46:24.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933000b990 con 0x7f935005cd20 2022-07-12T08:46:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9330003910 0x7f9330003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934ed9d700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 0x7f9330004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934dd9b700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9330004500 0x7f9330004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 -- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 msgr2=0x7f9330004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935005cd20 0x7f9330004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 -- 172.21.15.61:0/3217637644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9330004500 msgr2=0x7f9330004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:24.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9330004500 0x7f9330004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 -- 172.21.15.61:0/3217637644 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9330003260 con 0x7f9330003910 2022-07-12T08:46:24.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934e59c700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9330003910 0x7f9330003d10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f933800c000 tx=0x7f9338006df0).ready entity=mon.1 client_cookie=ac624ce9a1c87602 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f933800e830 con 0x7f9330003910 2022-07-12T08:46:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f933800e990 con 0x7f9330003910 2022-07-12T08:46:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934d59a700 1 -- 172.21.15.61:0/3217637644 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f933800ecf0 con 0x7f9330003910 2022-07-12T08:46:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934f59e700 1 -- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9330003910 msgr2=0x7f9330003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934f59e700 1 --2- 172.21.15.61:0/3217637644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9330003910 0x7f9330003d10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f933800c000 tx=0x7f9338006df0).stop 2022-07-12T08:46:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934ed9d700 1 -- 172.21.15.61:0/3217637644 reap_dead start 2022-07-12T08:46:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934f59e700 1 -- 172.21.15.61:0/3217637644 shutdown_connections 2022-07-12T08:46:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934f59e700 1 -- 172.21.15.61:0/3217637644 >> 172.21.15.61:0/3217637644 conn(0x7f93500b9e20 msgr2=0x7f93500b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934f59e700 1 -- 172.21.15.61:0/3217637644 shutdown_connections 2022-07-12T08:46:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.548+0000 7f934f59e700 1 -- 172.21.15.61:0/3217637644 wait complete. 2022-07-12T08:46:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 Processor -- start 2022-07-12T08:46:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 -- start start 2022-07-12T08:46:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93500b74a0 0x7f935009e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93501553e0 0x7f935009eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 0x7f935009eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500b7880 con 0x7f93501553e0 2022-07-12T08:46:24.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9350156260 con 0x7f93500b74a0 2022-07-12T08:46:24.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9350155880 con 0x7f935009daf0 2022-07-12T08:46:24.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93501553e0 0x7f935009eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 0x7f935009eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93500b74a0 0x7f935009e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 0x7f935009eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:38754/0 (socket says 172.21.15.61:38754) 2022-07-12T08:46:24.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93501553e0 0x7f935009eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56556/0 (socket says 172.21.15.61:56556) 2022-07-12T08:46:24.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 -- 172.21.15.61:0/3859018753 learned_addr learned my addr 172.21.15.61:0/3859018753 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:24.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934e59c700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93500b74a0 0x7f935009e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:35128/0 (socket says 172.21.15.61:35128) 2022-07-12T08:46:24.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 -- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93500b74a0 msgr2=0x7f935009e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93500b74a0 0x7f935009e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 -- 172.21.15.61:0/3859018753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93501553e0 msgr2=0x7f935009eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93501553e0 0x7f935009eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 -- 172.21.15.61:0/3859018753 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9350155b20 con 0x7f935009daf0 2022-07-12T08:46:24.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934ed9d700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 0x7f935009eff0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9348000b60 tx=0x7f9348002080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93480116b0 con 0x7f935009daf0 2022-07-12T08:46:24.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9350155dc0 con 0x7f935009daf0 2022-07-12T08:46:24.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9348011810 con 0x7f935009daf0 2022-07-12T08:46:24.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9348011b70 con 0x7f935009daf0 2022-07-12T08:46:24.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.552+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935009c4a0 con 0x7f935009daf0 2022-07-12T08:46:24.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.556+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9350155f50 con 0x7f935009daf0 2022-07-12T08:46:24.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.556+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9348011cd0 con 0x7f935009daf0 2022-07-12T08:46:24.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.556+0000 7f93477fe700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93340539b0 0x7f9334055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.556+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f934806b9f0 con 0x7f935009daf0 2022-07-12T08:46:24.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.556+0000 7f934e59c700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93340539b0 0x7f9334055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.556+0000 7f934e59c700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93340539b0 0x7f9334055e70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f933801a000 tx=0x7f9338019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.568+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9348042760 con 0x7f935009daf0 2022-07-12T08:46:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.736+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9350057460 con 0x7f935009daf0 2022-07-12T08:46:24.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.748+0000 7f93477fe700 1 -- 172.21.15.61:0/3859018753 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9348019030 con 0x7f935009daf0 2022-07-12T08:46:24.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93340539b0 msgr2=0x7f9334055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93340539b0 0x7f9334055e70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f933801a000 tx=0x7f9338019000).stop 2022-07-12T08:46:24.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 msgr2=0x7f935009eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 0x7f935009eff0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9348000b60 tx=0x7f9348002080).stop 2022-07-12T08:46:24.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 shutdown_connections 2022-07-12T08:46:24.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f935009daf0 0x7f935009eff0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93501553e0 0x7f935009eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93340539b0 0x7f9334055e70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 --2- 172.21.15.61:0/3859018753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93500b74a0 0x7f935009e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 >> 172.21.15.61:0/3859018753 conn(0x7f93500b9e20 msgr2=0x7f93501413b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:24.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 shutdown_connections 2022-07-12T08:46:24.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.752+0000 7f934f59e700 1 -- 172.21.15.61:0/3859018753 wait complete. 2022-07-12T08:46:24.759 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:46:24.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:46:24.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 Processor -- start 2022-07-12T08:46:24.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 -- start start 2022-07-12T08:46:24.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f97bc05cd20 0x7f97bc146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc0af490 0x7f97bc05d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc146200 con 0x7f97bc05f140 2022-07-12T08:46:24.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc1464a0 con 0x7f97bc05cd20 2022-07-12T08:46:24.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c328f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc146740 con 0x7f97bc0af490 2022-07-12T08:46:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c228d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f97bc05cd20 0x7f97bc146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c1a8c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc0af490 0x7f97bc05d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c228d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f97bc05cd20 0x7f97bc146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56566/0 (socket says 172.21.15.61:56566) 2022-07-12T08:46:24.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.888+0000 7f97c228d700 1 -- 172.21.15.61:0/4061586930 learned_addr learned my addr 172.21.15.61:0/4061586930 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c128b700 1 -- 172.21.15.61:0/4061586930 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435661715 0 0) 0x7f97bc146200 con 0x7f97bc05f140 2022-07-12T08:46:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c128b700 1 -- 172.21.15.61:0/4061586930 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a4003260 con 0x7f97bc05f140 2022-07-12T08:46:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c228d700 1 -- 172.21.15.61:0/4061586930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc0af490 msgr2=0x7f97bc05d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c228d700 1 --2- 172.21.15.61:0/4061586930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc0af490 0x7f97bc05d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c228d700 1 -- 172.21.15.61:0/4061586930 >> v1:172.21.15.111:6789/0 conn(0x7f97bc05f140 legacy=0x7f97bc05f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c228d700 1 -- 172.21.15.61:0/4061586930 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc060870 con 0x7f97bc05cd20 2022-07-12T08:46:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c1a8c700 1 --2- 172.21.15.61:0/4061586930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc0af490 0x7f97bc05d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c228d700 1 --2- 172.21.15.61:0/4061586930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05cd20 0x7f97bc146be0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f97ac002f70 tx=0x7f97ac006eb0).ready entity=mon.0 client_cookie=70e07444fc62a6f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c128b700 1 -- 172.21.15.61:0/4061586930 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97ac00d5c0 con 0x7f97bc05cd20 2022-07-12T08:46:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c128b700 1 -- 172.21.15.61:0/4061586930 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac000e90 con 0x7f97bc05cd20 2022-07-12T08:46:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c128b700 1 -- 172.21.15.61:0/4061586930 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97ac00d950 con 0x7f97bc05cd20 2022-07-12T08:46:24.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 -- 172.21.15.61:0/4061586930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05cd20 msgr2=0x7f97bc146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 --2- 172.21.15.61:0/4061586930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05cd20 0x7f97bc146be0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f97ac002f70 tx=0x7f97ac006eb0).stop 2022-07-12T08:46:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 -- 172.21.15.61:0/4061586930 shutdown_connections 2022-07-12T08:46:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 --2- 172.21.15.61:0/4061586930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc0af490 0x7f97bc05d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 --2- 172.21.15.61:0/4061586930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05cd20 0x7f97bc146be0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 -- 172.21.15.61:0/4061586930 >> 172.21.15.61:0/4061586930 conn(0x7f97bc0b9e20 msgr2=0x7f97bc0b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 -- 172.21.15.61:0/4061586930 shutdown_connections 2022-07-12T08:46:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 -- 172.21.15.61:0/4061586930 wait complete. 2022-07-12T08:46:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 Processor -- start 2022-07-12T08:46:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.892+0000 7f97c328f700 1 -- start start 2022-07-12T08:46:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc05cd20 0x7f97bc0b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 0x7f97bc0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc0af490 0x7f97bc0b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc09b520 con 0x7f97bc05f140 2022-07-12T08:46:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0b1250 con 0x7f97bc0af490 2022-07-12T08:46:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc09b750 con 0x7f97bc05cd20 2022-07-12T08:46:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 0x7f97bc0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 0x7f97bc0b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56570/0 (socket says 172.21.15.61:56570) 2022-07-12T08:46:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 -- 172.21.15.61:0/1666663597 learned_addr learned my addr 172.21.15.61:0/1666663597 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c228d700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc05cd20 0x7f97bc0b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c2a8e700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc0af490 0x7f97bc0b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 -- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc05cd20 msgr2=0x7f97bc0b7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc05cd20 0x7f97bc0b7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 -- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc0af490 msgr2=0x7f97bc0b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc0af490 0x7f97bc0b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:24.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 -- 172.21.15.61:0/1666663597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b800b180 con 0x7f97bc05f140 2022-07-12T08:46:24.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c1a8c700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 0x7f97bc0b0870 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f97b800c000 tx=0x7f97b8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97b8010780 con 0x7f97bc05f140 2022-07-12T08:46:24.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b80108e0 con 0x7f97bc05f140 2022-07-12T08:46:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ac00b1e0 con 0x7f97bc05f140 2022-07-12T08:46:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f97b8010c40 con 0x7f97bc05f140 2022-07-12T08:46:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc09f380 con 0x7f97bc05f140 2022-07-12T08:46:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.896+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc0b3950 con 0x7f97bc05f140 2022-07-12T08:46:24.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.900+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f97b801c460 con 0x7f97bc05f140 2022-07-12T08:46:24.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.900+0000 7f97ab7fe700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f979c0539b0 0x7f979c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:24.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.900+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f97b806c3f0 con 0x7f97bc05f140 2022-07-12T08:46:24.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.900+0000 7f97c228d700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f979c0539b0 0x7f979c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:24.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.900+0000 7f97c228d700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f979c0539b0 0x7f979c055e70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f97ac01c000 tx=0x7f97ac015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:24.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:24.904+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f97b80431e0 con 0x7f97bc05f140 2022-07-12T08:46:25.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.072+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f97bc057460 con 0x7f97bc05f140 2022-07-12T08:46:25.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.080+0000 7f97ab7fe700 1 -- 172.21.15.61:0/1666663597 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7f97b8046800 con 0x7f97bc05f140 2022-07-12T08:46:25.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.080+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f979c0539b0 msgr2=0x7f979c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.080+0000 7f97c328f700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f979c0539b0 0x7f979c055e70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f97ac01c000 tx=0x7f97ac015000).stop 2022-07-12T08:46:25.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.080+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 msgr2=0x7f97bc0b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.080+0000 7f97c328f700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 0x7f97bc0b0870 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f97b800c000 tx=0x7f97b8006df0).stop 2022-07-12T08:46:25.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 shutdown_connections 2022-07-12T08:46:25.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f979c0539b0 0x7f979c055e70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f97bc0af490 0x7f97bc0b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f97bc05f140 0x7f97bc0b0870 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 --2- 172.21.15.61:0/1666663597 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f97bc05cd20 0x7f97bc0b7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 >> 172.21.15.61:0/1666663597 conn(0x7f97bc0b9e20 msgr2=0x7f97bc0b8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:25.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 shutdown_connections 2022-07-12T08:46:25.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.084+0000 7f97c328f700 1 -- 172.21.15.61:0/1666663597 wait complete. 2022-07-12T08:46:25.093 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:46:25.100 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d5ad51700 1 Processor -- start 2022-07-12T08:46:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d5ad51700 1 -- start start 2022-07-12T08:46:25.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d5ad51700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5d54140ae0 0x7f5d540b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d5ad51700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540b32c0 con 0x7f5d540b0b10 2022-07-12T08:46:25.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d5ad51700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540a93f0 con 0x7f5d540b2ee0 2022-07-12T08:46:25.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d5ad51700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d54060b50 con 0x7f5d54140ae0 2022-07-12T08:46:25.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d53fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5d540b0b10 0x7f5d540aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44796/0 (socket says 172.21.15.61:44796) 2022-07-12T08:46:25.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d53fff700 1 -- 172.21.15.61:0/1579874199 learned_addr learned my addr 172.21.15.61:0/1579874199 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:25.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d592ee700 1 --2- 172.21.15.61:0/1579874199 >> v2:172.21.15.111:3300/0 conn(0x7f5d54140ae0 0x7f5d540b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407002693 0 0) 0x7f5d540b32c0 con 0x7f5d540b0b10 2022-07-12T08:46:25.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d38003260 con 0x7f5d540b0b10 2022-07-12T08:46:25.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350082741 0 0) 0x7f5d540a93f0 con 0x7f5d540b2ee0 2022-07-12T08:46:25.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d540b32c0 con 0x7f5d540b2ee0 2022-07-12T08:46:25.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136616908 0 0) 0x7f5d38003260 con 0x7f5d540b0b10 2022-07-12T08:46:25.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 >> v2:172.21.15.111:3300/0 conn(0x7f5d54140ae0 msgr2=0x7f5d540b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:25.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 --2- 172.21.15.61:0/1579874199 >> v2:172.21.15.111:3300/0 conn(0x7f5d54140ae0 0x7f5d540b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 >> v1:172.21.15.111:6789/0 conn(0x7f5d540b2ee0 legacy=0x7f5d541403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540a9690 con 0x7f5d540b0b10 2022-07-12T08:46:25.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.204+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f5d3c001660 con 0x7f5d540b0b10 2022-07-12T08:46:25.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 >> v1:172.21.15.61:6789/0 conn(0x7f5d540b0b10 legacy=0x7f5d540aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d38004940 0x7f5d38004d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d380034a0 0x7f5d38003920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d38003e10 0x7f5d3800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54060b50 con 0x7f5d38003e10 2022-07-12T08:46:25.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3800bcf0 con 0x7f5d380034a0 2022-07-12T08:46:25.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3800beb0 con 0x7f5d38004940 2022-07-12T08:46:25.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d38003e10 0x7f5d3800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d53fff700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d380034a0 0x7f5d38003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d58aed700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d38004940 0x7f5d38004d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 -- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d38004940 msgr2=0x7f5d38004d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:25.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d38004940 0x7f5d38004d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 -- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d380034a0 msgr2=0x7f5d38003920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:25.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d380034a0 0x7f5d38003920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 -- 172.21.15.61:0/1579874199 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d38003260 con 0x7f5d38003e10 2022-07-12T08:46:25.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 -- 172.21.15.61:0/1579874199 reap_dead start 2022-07-12T08:46:25.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d592ee700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d38003e10 0x7f5d3800b750 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f5d4c00c000 tx=0x7f5d4c006df0).ready entity=mon.0 client_cookie=7c2dfcca6da879a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:25.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d4c00e800 con 0x7f5d38003e10 2022-07-12T08:46:25.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00e960 con 0x7f5d38003e10 2022-07-12T08:46:25.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d537fe700 1 -- 172.21.15.61:0/1579874199 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d4c00ecc0 con 0x7f5d38003e10 2022-07-12T08:46:25.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 -- 172.21.15.61:0/1579874199 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d38003e10 msgr2=0x7f5d3800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d38003e10 0x7f5d3800b750 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7f5d4c00c000 tx=0x7f5d4c006df0).stop 2022-07-12T08:46:25.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 -- 172.21.15.61:0/1579874199 shutdown_connections 2022-07-12T08:46:25.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/1579874199 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d38003e10 0x7f5d3800b750 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 -- 172.21.15.61:0/1579874199 >> 172.21.15.61:0/1579874199 conn(0x7f5d54063ab0 msgr2=0x7f5d54063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:25.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 -- 172.21.15.61:0/1579874199 shutdown_connections 2022-07-12T08:46:25.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 -- 172.21.15.61:0/1579874199 wait complete. 2022-07-12T08:46:25.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 Processor -- start 2022-07-12T08:46:25.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.208+0000 7f5d5ad51700 1 -- start start 2022-07-12T08:46:25.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d38003e10 0x7f5d540b5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 0x7f5d54132470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d54132960 0x7f5d54132de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d541332d0 con 0x7f5d540acba0 2022-07-12T08:46:25.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54141160 con 0x7f5d38003e10 2022-07-12T08:46:25.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54142b40 con 0x7f5d54132960 2022-07-12T08:46:25.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 0x7f5d54132470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 0x7f5d54132470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56590/0 (socket says 172.21.15.61:56590) 2022-07-12T08:46:25.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 -- 172.21.15.61:0/3927608468 learned_addr learned my addr 172.21.15.61:0/3927608468 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:25.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d592ee700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d54132960 0x7f5d54132de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d58aed700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d38003e10 0x7f5d540b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 -- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d54132960 msgr2=0x7f5d54132de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d54132960 0x7f5d54132de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 -- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d38003e10 msgr2=0x7f5d540b5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d38003e10 0x7f5d540b5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 -- 172.21.15.61:0/3927608468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54142de0 con 0x7f5d540acba0 2022-07-12T08:46:25.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d53fff700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 0x7f5d54132470 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f5d3c00d000 tx=0x7f5d540b39f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:25.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d3c0116b0 con 0x7f5d540acba0 2022-07-12T08:46:25.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3c011810 con 0x7f5d540acba0 2022-07-12T08:46:25.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d540a8a10 con 0x7f5d540acba0 2022-07-12T08:46:25.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d3c011b70 con 0x7f5d540acba0 2022-07-12T08:46:25.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54141730 con 0x7f5d540acba0 2022-07-12T08:46:25.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.212+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5d3c01b030 con 0x7f5d540acba0 2022-07-12T08:46:25.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.216+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d54057460 con 0x7f5d540acba0 2022-07-12T08:46:25.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.216+0000 7f5d51ffb700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d2c053a40 0x7f5d2c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.216+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5d3c06b8f0 con 0x7f5d540acba0 2022-07-12T08:46:25.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.216+0000 7f5d58aed700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d2c053a40 0x7f5d2c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.216+0000 7f5d58aed700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d2c053a40 0x7f5d2c055f00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f5d44003000 tx=0x7f5d44007d80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:25.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.220+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d3c0427b0 con 0x7f5d540acba0 2022-07-12T08:46:25.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.388+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d540679a0 con 0x7f5d540acba0 2022-07-12T08:46:25.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.388+0000 7f5d51ffb700 1 -- 172.21.15.61:0/3927608468 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f5d3c019030 con 0x7f5d540acba0 2022-07-12T08:46:25.396 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:25.396 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:25.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d2c053a40 msgr2=0x7f5d2c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d2c053a40 0x7f5d2c055f00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f5d44003000 tx=0x7f5d44007d80).stop 2022-07-12T08:46:25.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 msgr2=0x7f5d54132470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 0x7f5d54132470 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f5d3c00d000 tx=0x7f5d540b39f0).stop 2022-07-12T08:46:25.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 shutdown_connections 2022-07-12T08:46:25.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d54132960 0x7f5d54132de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d540acba0 0x7f5d54132470 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d2c053a40 0x7f5d2c055f00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 --2- 172.21.15.61:0/3927608468 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d38003e10 0x7f5d540b5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 >> 172.21.15.61:0/3927608468 conn(0x7f5d54063ab0 msgr2=0x7f5d540afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:25.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 shutdown_connections 2022-07-12T08:46:25.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.392+0000 7f5d5ad51700 1 -- 172.21.15.61:0/3927608468 wait complete. 2022-07-12T08:46:25.406 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:25.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:46:22.431962+0000', 'modified': '2022-07-12T08:46:23.437118+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5979}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5979': {'gid': 5979, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3698454378', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3698454378}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3698454378}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:46:25.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:46:25.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:46:25.735 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5979,"want_state":"up:active","state":"up:active","rank_uptime":3.2929505489999999,"mdsmap_epoch":31,"osdmap_epoch":62,"osdmap_epoch_barrier":61,"uptime":7.6860782040000002} 2022-07-12T08:46:25.736 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5979, "mdsmap_epoch": 31, "osdmap_epoch": 62, "osdmap_epoch_barrier": 61, "rank_uptime": 3.292950549, "state": "up:active", "uptime": 7.686078204, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:46:25.737 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:46:25.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.832+0000 7ff546263700 1 Processor -- start 2022-07-12T08:46:25.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.832+0000 7ff546263700 1 -- start start 2022-07-12T08:46:25.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff54009bce0 0x7ff5400af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff5400b3f60 0x7ff5400b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5400aecb0 con 0x7ff5400af280 2022-07-12T08:46:25.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff540143b00 con 0x7ff5400b3f60 2022-07-12T08:46:25.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5401435c0 con 0x7ff54009bce0 2022-07-12T08:46:25.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff5400b3f60 0x7ff5400b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff5400b3f60 0x7ff5400b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56600/0 (socket says 172.21.15.61:56600) 2022-07-12T08:46:25.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545261700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff5400af280 0x7ff540060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44814/0 (socket says 172.21.15.61:44814) 2022-07-12T08:46:25.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 -- 172.21.15.61:0/2265276204 learned_addr learned my addr 172.21.15.61:0/2265276204 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:25.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff544a60700 1 --2- 172.21.15.61:0/2265276204 >> v2:172.21.15.111:3300/0 conn(0x7ff54009bce0 0x7ff5400af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 -- 172.21.15.61:0/2265276204 >> v2:172.21.15.111:3300/0 conn(0x7ff54009bce0 msgr2=0x7ff5400af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:25.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 --2- 172.21.15.61:0/2265276204 >> v2:172.21.15.111:3300/0 conn(0x7ff54009bce0 0x7ff5400af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 -- 172.21.15.61:0/2265276204 >> v1:172.21.15.61:6789/0 conn(0x7ff5400af280 legacy=0x7ff540060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 -- 172.21.15.61:0/2265276204 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff540143860 con 0x7ff5400b3f60 2022-07-12T08:46:25.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff545a62700 1 --2- 172.21.15.61:0/2265276204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5400b3f60 0x7ff5400b4810 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7ff5380032f0 tx=0x7ff538009290).ready entity=mon.0 client_cookie=3690aae44d450ed2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:25.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff52ffff700 1 -- 172.21.15.61:0/2265276204 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff53800b870 con 0x7ff5400b3f60 2022-07-12T08:46:25.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff52ffff700 1 -- 172.21.15.61:0/2265276204 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53800b9d0 con 0x7ff5400b3f60 2022-07-12T08:46:25.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff52ffff700 1 -- 172.21.15.61:0/2265276204 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff53800bb40 con 0x7ff5400b3f60 2022-07-12T08:46:25.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 -- 172.21.15.61:0/2265276204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5400b3f60 msgr2=0x7ff5400b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 --2- 172.21.15.61:0/2265276204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5400b3f60 0x7ff5400b4810 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7ff5380032f0 tx=0x7ff538009290).stop 2022-07-12T08:46:25.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.836+0000 7ff546263700 1 -- 172.21.15.61:0/2265276204 shutdown_connections 2022-07-12T08:46:25.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 --2- 172.21.15.61:0/2265276204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff5400b3f60 0x7ff5400b4810 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 --2- 172.21.15.61:0/2265276204 >> v2:172.21.15.111:3300/0 conn(0x7ff54009bce0 0x7ff5400af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 -- 172.21.15.61:0/2265276204 >> 172.21.15.61:0/2265276204 conn(0x7ff5400b92c0 msgr2=0x7ff5400b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:25.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 -- 172.21.15.61:0/2265276204 shutdown_connections 2022-07-12T08:46:25.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 -- 172.21.15.61:0/2265276204 wait complete. 2022-07-12T08:46:25.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 Processor -- start 2022-07-12T08:46:25.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.840+0000 7ff546263700 1 -- start start 2022-07-12T08:46:25.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 0x7ff54013e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5400af280 0x7ff540142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5400b3f60 0x7ff5401559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54009c460 con 0x7ff54009bce0 2022-07-12T08:46:25.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54009c700 con 0x7ff5400b3f60 2022-07-12T08:46:25.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5400b06d0 con 0x7ff5400af280 2022-07-12T08:46:25.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 0x7ff54013e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 0x7ff54013e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56602/0 (socket says 172.21.15.61:56602) 2022-07-12T08:46:25.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 -- 172.21.15.61:0/3591673437 learned_addr learned my addr 172.21.15.61:0/3591673437 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:25.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff544a60700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5400af280 0x7ff540142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545a62700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5400b3f60 0x7ff5401559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 -- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5400af280 msgr2=0x7ff540142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5400af280 0x7ff540142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 -- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5400b3f60 msgr2=0x7ff5401559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5400b3f60 0x7ff5401559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 -- 172.21.15.61:0/3591673437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538009530 con 0x7ff54009bce0 2022-07-12T08:46:25.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff545261700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 0x7ff54013e560 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff5400b70b0 tx=0x7ff53c007b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:25.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff53c0070d0 con 0x7ff54009bce0 2022-07-12T08:46:25.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c007230 con 0x7ff54009bce0 2022-07-12T08:46:25.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff540155560 con 0x7ff54009bce0 2022-07-12T08:46:25.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff53c0073a0 con 0x7ff54009bce0 2022-07-12T08:46:25.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff540142e90 con 0x7ff54009bce0 2022-07-12T08:46:25.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.844+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5400b0890 con 0x7ff54009bce0 2022-07-12T08:46:25.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.848+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff53c01a060 con 0x7ff54009bce0 2022-07-12T08:46:25.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.848+0000 7ff52e7fc700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff520053ad0 0x7ff520055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:25.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.848+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff53c06b4e0 con 0x7ff54009bce0 2022-07-12T08:46:25.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.848+0000 7ff544a60700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff520053ad0 0x7ff520055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:25.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.848+0000 7ff544a60700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff520053ad0 0x7ff520055f90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7ff53000c000 tx=0x7ff530008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:25.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.852+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff53c042440 con 0x7ff54009bce0 2022-07-12T08:46:25.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.976+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5400afeb0 con 0x7ff54009bce0 2022-07-12T08:46:25.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.976+0000 7ff52e7fc700 1 -- 172.21.15.61:0/3591673437 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16089 (secure 0 0 0) 0x7ff53c018070 con 0x7ff54009bce0 2022-07-12T08:46:25.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:25.982 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:46:23.436615+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:46:21.097184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-12T08:46:21.563425+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:46:25.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff520053ad0 msgr2=0x7ff520055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff520053ad0 0x7ff520055f90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7ff53000c000 tx=0x7ff530008040).stop 2022-07-12T08:46:25.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 msgr2=0x7ff54013e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:25.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 0x7ff54013e560 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff5400b70b0 tx=0x7ff53c007b30).stop 2022-07-12T08:46:25.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 shutdown_connections 2022-07-12T08:46:25.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff520053ad0 0x7ff520055f90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5400b3f60 0x7ff5401559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff5400af280 0x7ff540142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 --2- 172.21.15.61:0/3591673437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff54009bce0 0x7ff54013e560 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:25.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 >> 172.21.15.61:0/3591673437 conn(0x7ff5400b92c0 msgr2=0x7ff5400b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:25.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 shutdown_connections 2022-07-12T08:46:25.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:25.980+0000 7ff546263700 1 -- 172.21.15.61:0/3591673437 wait complete. 2022-07-12T08:46:25.999 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:46:21.097184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:46:21.563425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:46:26.000 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:46:26.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05d002700 1 Processor -- start 2022-07-12T08:46:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05d002700 1 -- start start 2022-07-12T08:46:26.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05d002700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa0580afc00 0x7fa058145c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05d002700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0580b5e80 con 0x7fa0580b5aa0 2022-07-12T08:46:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05d002700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0580b0910 con 0x7fa0580b01b0 2022-07-12T08:46:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05d002700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0580b0720 con 0x7fa0580afc00 2022-07-12T08:46:26.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa056d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa0580b5aa0 0x7fa058145540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44828/0 (socket says 172.21.15.61:44828) 2022-07-12T08:46:26.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa056d9d700 1 -- 172.21.15.61:0/379572183 learned_addr learned my addr 172.21.15.61:0/379572183 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:26.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa05759e700 1 --2- 172.21.15.61:0/379572183 >> v2:172.21.15.61:3300/0 conn(0x7fa0580afc00 0x7fa058145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.112+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778899522 0 0) 0x7fa0580b5e80 con 0x7fa0580b5aa0 2022-07-12T08:46:26.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa038003260 con 0x7fa0580b5aa0 2022-07-12T08:46:26.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098091935 0 0) 0x7fa038003260 con 0x7fa0580b5aa0 2022-07-12T08:46:26.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 >> v2:172.21.15.61:3300/0 conn(0x7fa0580afc00 msgr2=0x7fa058145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:26.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 --2- 172.21.15.61:0/379572183 >> v2:172.21.15.61:3300/0 conn(0x7fa0580afc00 0x7fa058145c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 >> v1:172.21.15.111:6789/0 conn(0x7fa0580b01b0 legacy=0x7fa0580addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0580b0bb0 con 0x7fa0580b5aa0 2022-07-12T08:46:26.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fa050004dc0 con 0x7fa0580b5aa0 2022-07-12T08:46:26.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 >> v1:172.21.15.61:6789/0 conn(0x7fa0580b5aa0 legacy=0x7fa058145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa038003780 0x7fa038003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa038004500 0x7fa038004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038004e70 0x7fa03800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03800bcf0 con 0x7fa038003780 2022-07-12T08:46:26.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03800be60 con 0x7fa038004500 2022-07-12T08:46:26.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03800c020 con 0x7fa038004e70 2022-07-12T08:46:26.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05759e700 1 --2- 172.21.15.61:0/379572183 >> v2:172.21.15.61:3300/0 conn(0x7fa0580afc00 0x7fa058145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:26.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa038003780 0x7fa038003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05659c700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa038004500 0x7fa038004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05759e700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038004e70 0x7fa03800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 -- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038004e70 msgr2=0x7fa03800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:26.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038004e70 0x7fa03800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 -- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa038004500 msgr2=0x7fa038004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:26.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa038004500 0x7fa038004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05759e700 1 -- 172.21.15.61:0/379572183 reap_dead start 2022-07-12T08:46:26.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 -- 172.21.15.61:0/379572183 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa038003260 con 0x7fa038003780 2022-07-12T08:46:26.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05659c700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa038004500 0x7fa038004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:26.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa056d9d700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa038003780 0x7fa038003ba0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fa050004ad0 tx=0x7fa0580b0aa0).ready entity=mon.0 client_cookie=3b2a182e248b88ee server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0500070d0 con 0x7fa038003780 2022-07-12T08:46:26.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa050000e90 con 0x7fa038003780 2022-07-12T08:46:26.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa055d9b700 1 -- 172.21.15.61:0/379572183 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa050007420 con 0x7fa038003780 2022-07-12T08:46:26.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 -- 172.21.15.61:0/379572183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa038003780 msgr2=0x7fa038003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa038003780 0x7fa038003ba0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7fa050004ad0 tx=0x7fa0580b0aa0).stop 2022-07-12T08:46:26.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 -- 172.21.15.61:0/379572183 shutdown_connections 2022-07-12T08:46:26.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 --2- 172.21.15.61:0/379572183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa038003780 0x7fa038003ba0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 -- 172.21.15.61:0/379572183 >> 172.21.15.61:0/379572183 conn(0x7fa0580634d0 msgr2=0x7fa0580b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 -- 172.21.15.61:0/379572183 shutdown_connections 2022-07-12T08:46:26.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.116+0000 7fa05d002700 1 -- 172.21.15.61:0/379572183 wait complete. 2022-07-12T08:46:26.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 Processor -- start 2022-07-12T08:46:26.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 -- start start 2022-07-12T08:46:26.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038003780 0x7fa0581558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0580a3140 0x7fa058155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 0x7fa058143f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0580a4f40 con 0x7fa0580a2830 2022-07-12T08:46:26.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa058153240 con 0x7fa0580a3140 2022-07-12T08:46:26.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0580abbd0 con 0x7fa038003780 2022-07-12T08:46:26.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 0x7fa058143f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 0x7fa058143f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56624/0 (socket says 172.21.15.61:56624) 2022-07-12T08:46:26.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 -- 172.21.15.61:0/1882015573 learned_addr learned my addr 172.21.15.61:0/1882015573 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:26.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa056d9d700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038003780 0x7fa0581558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05659c700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0580a3140 0x7fa058155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 -- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038003780 msgr2=0x7fa0581558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038003780 0x7fa0581558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 -- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0580a3140 msgr2=0x7fa058155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0580a3140 0x7fa058155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 -- 172.21.15.61:0/1882015573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0580a4130 con 0x7fa0580a2830 2022-07-12T08:46:26.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05659c700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0580a3140 0x7fa058155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:26.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa056d9d700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038003780 0x7fa0581558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:26.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05759e700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 0x7fa058143f10 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fa04800b000 tx=0x7fa0480091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa04801a070 con 0x7fa0580a2830 2022-07-12T08:46:26.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04800fa50 con 0x7fa0580a2830 2022-07-12T08:46:26.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0580a43d0 con 0x7fa0580a2830 2022-07-12T08:46:26.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa04800fdb0 con 0x7fa0580a2830 2022-07-12T08:46:26.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.120+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0580ab830 con 0x7fa0580a2830 2022-07-12T08:46:26.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.124+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0580a4560 con 0x7fa0580a2830 2022-07-12T08:46:26.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.124+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa048014440 con 0x7fa0580a2830 2022-07-12T08:46:26.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.124+0000 7fa04ffff700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa03c053ad0 0x7fa03c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.124+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa04806c6f0 con 0x7fa0580a2830 2022-07-12T08:46:26.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.124+0000 7fa056d9d700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa03c053ad0 0x7fa03c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.124+0000 7fa056d9d700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa03c053ad0 0x7fa03c055f90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fa050017000 tx=0x7fa050013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.128+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0480435f0 con 0x7fa0580a2830 2022-07-12T08:46:26.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.256+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa058057460 con 0x7fa0580a2830 2022-07-12T08:46:26.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.256+0000 7fa04ffff700 1 -- 172.21.15.61:0/1882015573 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v62) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa048046c10 con 0x7fa0580a2830 2022-07-12T08:46:26.263 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:46:26.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa03c053ad0 msgr2=0x7fa03c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa03c053ad0 0x7fa03c055f90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fa050017000 tx=0x7fa050013000).stop 2022-07-12T08:46:26.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 msgr2=0x7fa058143f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 0x7fa058143f10 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fa04800b000 tx=0x7fa0480091d0).stop 2022-07-12T08:46:26.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 shutdown_connections 2022-07-12T08:46:26.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa0580a2830 0x7fa058143f10 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa03c053ad0 0x7fa03c055f90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0580a3140 0x7fa058155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 --2- 172.21.15.61:0/1882015573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa038003780 0x7fa0581558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 >> 172.21.15.61:0/1882015573 conn(0x7fa0580634d0 msgr2=0x7fa0580b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 shutdown_connections 2022-07-12T08:46:26.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.260+0000 7fa05d002700 1 -- 172.21.15.61:0/1882015573 wait complete. 2022-07-12T08:46:26.277 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:46:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.388+0000 7fb5c677d700 1 Processor -- start 2022-07-12T08:46:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- start start 2022-07-12T08:46:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5c00b36c0 0x7fb5c0143120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb5c00613f0 0x7fb5c0143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c005f650 con 0x7fb5c00a9b40 2022-07-12T08:46:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c00b1dc0 con 0x7fb5c00b36c0 2022-07-12T08:46:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c0143610 con 0x7fb5c00613f0 2022-07-12T08:46:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c4d1a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb5c00a9b40 0x7fb5c00b2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44848/0 (socket says 172.21.15.61:44848) 2022-07-12T08:46:26.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c4d1a700 1 -- 172.21.15.61:0/2230629382 learned_addr learned my addr 172.21.15.61:0/2230629382 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:26.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bffff700 1 --2- 172.21.15.61:0/2230629382 >> v2:172.21.15.61:3300/0 conn(0x7fb5c00b36c0 0x7fb5c0143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bf7fe700 1 --2- 172.21.15.61:0/2230629382 >> v2:172.21.15.111:3300/0 conn(0x7fb5c00613f0 0x7fb5c0143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5beffd700 1 -- 172.21.15.61:0/2230629382 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886266186 0 0) 0x7fb5c005f650 con 0x7fb5c00a9b40 2022-07-12T08:46:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5beffd700 1 -- 172.21.15.61:0/2230629382 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a0003260 con 0x7fb5c00a9b40 2022-07-12T08:46:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bffff700 1 -- 172.21.15.61:0/2230629382 >> v2:172.21.15.111:3300/0 conn(0x7fb5c00613f0 msgr2=0x7fb5c0143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bffff700 1 --2- 172.21.15.61:0/2230629382 >> v2:172.21.15.111:3300/0 conn(0x7fb5c00613f0 0x7fb5c0143ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bffff700 1 -- 172.21.15.61:0/2230629382 >> v1:172.21.15.61:6789/0 conn(0x7fb5c00a9b40 legacy=0x7fb5c00b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bffff700 1 -- 172.21.15.61:0/2230629382 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c005f2d0 con 0x7fb5c00b36c0 2022-07-12T08:46:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5bffff700 1 --2- 172.21.15.61:0/2230629382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00b36c0 0x7fb5c0143120 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fb5b00032f0 tx=0x7fb5b00091d0).ready entity=mon.0 client_cookie=ff399f1243124ac8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5beffd700 1 -- 172.21.15.61:0/2230629382 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5b000b680 con 0x7fb5c00b36c0 2022-07-12T08:46:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5beffd700 1 -- 172.21.15.61:0/2230629382 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b000b7e0 con 0x7fb5c00b36c0 2022-07-12T08:46:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5beffd700 1 -- 172.21.15.61:0/2230629382 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5b000b950 con 0x7fb5c00b36c0 2022-07-12T08:46:26.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- 172.21.15.61:0/2230629382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00b36c0 msgr2=0x7fb5c0143120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 --2- 172.21.15.61:0/2230629382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00b36c0 0x7fb5c0143120 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fb5b00032f0 tx=0x7fb5b00091d0).stop 2022-07-12T08:46:26.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- 172.21.15.61:0/2230629382 shutdown_connections 2022-07-12T08:46:26.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 --2- 172.21.15.61:0/2230629382 >> v2:172.21.15.111:3300/0 conn(0x7fb5c00613f0 0x7fb5c0143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 --2- 172.21.15.61:0/2230629382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00b36c0 0x7fb5c0143120 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- 172.21.15.61:0/2230629382 >> 172.21.15.61:0/2230629382 conn(0x7fb5c00b6e40 msgr2=0x7fb5c00b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.392+0000 7fb5c677d700 1 -- 172.21.15.61:0/2230629382 shutdown_connections 2022-07-12T08:46:26.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.396+0000 7fb5c677d700 1 -- 172.21.15.61:0/2230629382 wait complete. 2022-07-12T08:46:26.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.396+0000 7fb5c677d700 1 Processor -- start 2022-07-12T08:46:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.396+0000 7fb5c677d700 1 -- start start 2022-07-12T08:46:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 0x7fb5c009a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c00a9b40 0x7fb5c0155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5c00b36c0 0x7fb5c01403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0140ed0 con 0x7fb5c00613f0 2022-07-12T08:46:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0141100 con 0x7fb5c00a9b40 2022-07-12T08:46:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0155cb0 con 0x7fb5c00b36c0 2022-07-12T08:46:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 0x7fb5c009a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 0x7fb5c009a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56636/0 (socket says 172.21.15.61:56636) 2022-07-12T08:46:26.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 -- 172.21.15.61:0/3376186406 learned_addr learned my addr 172.21.15.61:0/3376186406 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:26.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bf7fe700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c00a9b40 0x7fb5c0155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c4d1a700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5c00b36c0 0x7fb5c01403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 -- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5c00b36c0 msgr2=0x7fb5c01403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5c00b36c0 0x7fb5c01403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 -- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c00a9b40 msgr2=0x7fb5c0155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c00a9b40 0x7fb5c0155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 -- 172.21.15.61:0/3376186406 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b0009470 con 0x7fb5c00613f0 2022-07-12T08:46:26.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bffff700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 0x7fb5c009a310 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fb5b00032f0 tx=0x7fb5b0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5b0011da0 con 0x7fb5c00613f0 2022-07-12T08:46:26.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b0021440 con 0x7fb5c00613f0 2022-07-12T08:46:26.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c0155f50 con 0x7fb5c00613f0 2022-07-12T08:46:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5b00217a0 con 0x7fb5c00613f0 2022-07-12T08:46:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c0148600 con 0x7fb5c00613f0 2022-07-12T08:46:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.400+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c0057460 con 0x7fb5c00613f0 2022-07-12T08:46:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.404+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb5b0027020 con 0x7fb5c00613f0 2022-07-12T08:46:26.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.404+0000 7fb5bd7fa700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5a4053920 0x7fb5a4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.404+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb5b006d320 con 0x7fb5c00613f0 2022-07-12T08:46:26.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.404+0000 7fb5bf7fe700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5a4053920 0x7fb5a4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.404+0000 7fb5bf7fe700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5a4053920 0x7fb5a4055de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb5a800c000 tx=0x7fb5a8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.412+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5b0044170 con 0x7fb5c00613f0 2022-07-12T08:46:26.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb5c01560e0 con 0x7fb5c00613f0 2022-07-12T08:46:26.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5bd7fa700 1 -- 172.21.15.61:0/3376186406 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v62) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb5b00150a0 con 0x7fb5c00613f0 2022-07-12T08:46:26.551 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:46:26.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5a4053920 msgr2=0x7fb5a4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5a4053920 0x7fb5a4055de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb5a800c000 tx=0x7fb5a8008040).stop 2022-07-12T08:46:26.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 msgr2=0x7fb5c009a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 0x7fb5c009a310 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fb5b00032f0 tx=0x7fb5b0016000).stop 2022-07-12T08:46:26.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 shutdown_connections 2022-07-12T08:46:26.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5c00b36c0 0x7fb5c01403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5a4053920 0x7fb5a4055de0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5c00a9b40 0x7fb5c0155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 --2- 172.21.15.61:0/3376186406 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5c00613f0 0x7fb5c009a310 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 >> 172.21.15.61:0/3376186406 conn(0x7fb5c00b6e40 msgr2=0x7fb5c0094010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 shutdown_connections 2022-07-12T08:46:26.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.540+0000 7fb5c677d700 1 -- 172.21.15.61:0/3376186406 wait complete. 2022-07-12T08:46:26.567 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:46:26.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f873f8700 1 Processor -- start 2022-07-12T08:46:26.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f873f8700 1 -- start start 2022-07-12T08:46:26.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f873f8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9f8005d190 0x7f9f80142330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f873f8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f80146c20 con 0x7f9f800acbd0 2022-07-12T08:46:26.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f873f8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f80066820 con 0x7f9f800b7eb0 2022-07-12T08:46:26.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f873f8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f800b2760 con 0x7f9f8005d190 2022-07-12T08:46:26.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f84993700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9f800acbd0 0x7f9f80066110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44864/0 (socket says 172.21.15.61:44864) 2022-07-12T08:46:26.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.680+0000 7f9f84993700 1 -- 172.21.15.61:0/1921206052 learned_addr learned my addr 172.21.15.61:0/1921206052 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:26.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f85995700 1 --2- 172.21.15.61:0/1921206052 >> v2:172.21.15.111:3300/0 conn(0x7f9f8005d190 0x7f9f80142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704809230 0 0) 0x7f9f80146c20 con 0x7f9f800acbd0 2022-07-12T08:46:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6c003260 con 0x7f9f800acbd0 2022-07-12T08:46:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091269803 0 0) 0x7f9f80066820 con 0x7f9f800b7eb0 2022-07-12T08:46:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f80146c20 con 0x7f9f800b7eb0 2022-07-12T08:46:26.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344138943 0 0) 0x7f9f6c003260 con 0x7f9f800acbd0 2022-07-12T08:46:26.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f85995700 1 -- 172.21.15.61:0/1921206052 >> v1:172.21.15.111:6789/0 conn(0x7f9f800b7eb0 legacy=0x7f9f80146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f85995700 1 -- 172.21.15.61:0/1921206052 >> v1:172.21.15.61:6789/0 conn(0x7f9f800acbd0 legacy=0x7f9f80066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f85995700 1 -- 172.21.15.61:0/1921206052 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800b8290 con 0x7f9f8005d190 2022-07-12T08:46:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f85995700 1 --2- 172.21.15.61:0/1921206052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80142330 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f9f74001110 tx=0x7f9f780091d0).ready entity=mon.1 client_cookie=20df951b69ee8fd2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f74001660 con 0x7f9f8005d190 2022-07-12T08:46:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7800f510 con 0x7f9f8005d190 2022-07-12T08:46:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f7ffff700 1 -- 172.21.15.61:0/1921206052 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7800f680 con 0x7f9f8005d190 2022-07-12T08:46:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 -- 172.21.15.61:0/1921206052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 msgr2=0x7f9f80142330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 --2- 172.21.15.61:0/1921206052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80142330 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f9f74001110 tx=0x7f9f780091d0).stop 2022-07-12T08:46:26.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 -- 172.21.15.61:0/1921206052 shutdown_connections 2022-07-12T08:46:26.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 --2- 172.21.15.61:0/1921206052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80142330 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 -- 172.21.15.61:0/1921206052 >> 172.21.15.61:0/1921206052 conn(0x7f9f80065850 msgr2=0x7f9f800b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 -- 172.21.15.61:0/1921206052 shutdown_connections 2022-07-12T08:46:26.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.684+0000 7f9f873f8700 1 -- 172.21.15.61:0/1921206052 wait complete. 2022-07-12T08:46:26.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 Processor -- start 2022-07-12T08:46:26.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 -- start start 2022-07-12T08:46:26.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 0x7f9f800aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f800b7eb0 0x7f9f800af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80143810 con 0x7f9f800acbd0 2022-07-12T08:46:26.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f801432d0 con 0x7f9f8005d190 2022-07-12T08:46:26.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f873f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80143570 con 0x7f9f800b7eb0 2022-07-12T08:46:26.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 0x7f9f800aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 0x7f9f800aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56652/0 (socket says 172.21.15.61:56652) 2022-07-12T08:46:26.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 -- 172.21.15.61:0/2631265786 learned_addr learned my addr 172.21.15.61:0/2631265786 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:26.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f85194700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f85995700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f800b7eb0 0x7f9f800af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 -- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f800b7eb0 msgr2=0x7f9f800af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f800b7eb0 0x7f9f800af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 -- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 msgr2=0x7f9f80147f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80147f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.688+0000 7f9f84993700 1 -- 172.21.15.61:0/2631265786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f78009470 con 0x7f9f800acbd0 2022-07-12T08:46:26.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f84993700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 0x7f9f800aed00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f9f74000fe0 tx=0x7f9f74001b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f74007170 con 0x7f9f800acbd0 2022-07-12T08:46:26.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f740072d0 con 0x7f9f800acbd0 2022-07-12T08:46:26.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f80060f70 con 0x7f9f800acbd0 2022-07-12T08:46:26.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f74004920 con 0x7f9f800acbd0 2022-07-12T08:46:26.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f800af6e0 con 0x7f9f800acbd0 2022-07-12T08:46:26.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f800aeaa0 con 0x7f9f800acbd0 2022-07-12T08:46:26.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f7401b070 con 0x7f9f800acbd0 2022-07-12T08:46:26.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f7e7fc700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f600539b0 0x7f9f60055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.692+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f7406b8c0 con 0x7f9f800acbd0 2022-07-12T08:46:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.696+0000 7f9f85194700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f600539b0 0x7f9f60055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.696+0000 7f9f85194700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f600539b0 0x7f9f60055e70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f9f7000b000 tx=0x7f9f70009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.700+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f74042710 con 0x7f9f800acbd0 2022-07-12T08:46:26.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.824+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f80057460 con 0x7f9f800acbd0 2022-07-12T08:46:26.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f7e7fc700 1 -- 172.21.15.61:0/2631265786 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9f74001c60 con 0x7f9f800acbd0 2022-07-12T08:46:26.830 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:46:26.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f600539b0 msgr2=0x7f9f60055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f600539b0 0x7f9f60055e70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f9f7000b000 tx=0x7f9f70009040).stop 2022-07-12T08:46:26.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 msgr2=0x7f9f800aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 0x7f9f800aed00 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f9f74000fe0 tx=0x7f9f74001b60).stop 2022-07-12T08:46:26.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 shutdown_connections 2022-07-12T08:46:26.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9f800b7eb0 0x7f9f800af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f800acbd0 0x7f9f800aed00 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9f600539b0 0x7f9f60055e70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 --2- 172.21.15.61:0/2631265786 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9f8005d190 0x7f9f80147f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 >> 172.21.15.61:0/2631265786 conn(0x7f9f80065850 msgr2=0x7f9f80148df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 shutdown_connections 2022-07-12T08:46:26.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.828+0000 7f9f873f8700 1 -- 172.21.15.61:0/2631265786 wait complete. 2022-07-12T08:46:26.844 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:46:26.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf76372700 1 Processor -- start 2022-07-12T08:46:26.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf76372700 1 -- start start 2022-07-12T08:46:26.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf76372700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcf700b5920 0x7fcf700b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf76372700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf70097c80 con 0x7fcf70098c80 2022-07-12T08:46:26.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf76372700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700b5d00 con 0x7fcf700987f0 2022-07-12T08:46:26.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf76372700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf700990f0 con 0x7fcf700b5920 2022-07-12T08:46:26.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75b71700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcf70098c80 0x7fcf7014b3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44880/0 (socket says 172.21.15.61:44880) 2022-07-12T08:46:26.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75b71700 1 -- 172.21.15.61:0/4226201970 learned_addr learned my addr 172.21.15.61:0/4226201970 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75370700 1 --2- 172.21.15.61:0/4226201970 >> v2:172.21.15.61:3300/0 conn(0x7fcf700b5920 0x7fcf700b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf6ffff700 1 -- 172.21.15.61:0/4226201970 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494372651 0 0) 0x7fcf70097c80 con 0x7fcf70098c80 2022-07-12T08:46:26.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf6ffff700 1 -- 172.21.15.61:0/4226201970 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5c003260 con 0x7fcf70098c80 2022-07-12T08:46:26.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf6ffff700 1 -- 172.21.15.61:0/4226201970 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911455759 0 0) 0x7fcf700b5d00 con 0x7fcf700987f0 2022-07-12T08:46:26.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75370700 1 -- 172.21.15.61:0/4226201970 >> v1:172.21.15.111:6789/0 conn(0x7fcf700987f0 legacy=0x7fcf70143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75370700 1 -- 172.21.15.61:0/4226201970 >> v1:172.21.15.61:6789/0 conn(0x7fcf70098c80 legacy=0x7fcf7014b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75370700 1 -- 172.21.15.61:0/4226201970 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7005c7f0 con 0x7fcf700b5920 2022-07-12T08:46:26.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.956+0000 7fcf75370700 1 --2- 172.21.15.61:0/4226201970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700b5920 0x7fcf700b01f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7fcf64002f70 tx=0x7fcf64006df0).ready entity=mon.0 client_cookie=a1d05fe7e751e296 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf6ffff700 1 -- 172.21.15.61:0/4226201970 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf6400d610 con 0x7fcf700b5920 2022-07-12T08:46:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf6ffff700 1 -- 172.21.15.61:0/4226201970 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf64000e90 con 0x7fcf700b5920 2022-07-12T08:46:26.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf6ffff700 1 -- 172.21.15.61:0/4226201970 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf6400d960 con 0x7fcf700b5920 2022-07-12T08:46:26.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 -- 172.21.15.61:0/4226201970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700b5920 msgr2=0x7fcf700b01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 --2- 172.21.15.61:0/4226201970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700b5920 0x7fcf700b01f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7fcf64002f70 tx=0x7fcf64006df0).stop 2022-07-12T08:46:26.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 -- 172.21.15.61:0/4226201970 shutdown_connections 2022-07-12T08:46:26.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 --2- 172.21.15.61:0/4226201970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700b5920 0x7fcf700b01f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 -- 172.21.15.61:0/4226201970 >> 172.21.15.61:0/4226201970 conn(0x7fcf70063340 msgr2=0x7fcf700b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:26.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 -- 172.21.15.61:0/4226201970 shutdown_connections 2022-07-12T08:46:26.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 -- 172.21.15.61:0/4226201970 wait complete. 2022-07-12T08:46:26.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 Processor -- start 2022-07-12T08:46:26.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.960+0000 7fcf76372700 1 -- start start 2022-07-12T08:46:26.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 0x7fcf70144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf70098c80 0x7fcf701501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf700b5920 0x7fcf701506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:26.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7013def0 con 0x7fcf700987f0 2022-07-12T08:46:26.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 0x7fcf70144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70140c70 con 0x7fcf70098c80 2022-07-12T08:46:26.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf70067ce0 con 0x7fcf700b5920 2022-07-12T08:46:26.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 0x7fcf70144620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56664/0 (socket says 172.21.15.61:56664) 2022-07-12T08:46:26.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 -- 172.21.15.61:0/4158211644 learned_addr learned my addr 172.21.15.61:0/4158211644 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:26.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf74b6f700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf70098c80 0x7fcf701501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75b71700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf700b5920 0x7fcf701506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:26.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 -- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf700b5920 msgr2=0x7fcf701506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf700b5920 0x7fcf701506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 -- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf70098c80 msgr2=0x7fcf701501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:26.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf70098c80 0x7fcf701501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:26.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 -- 172.21.15.61:0/4158211644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6400b180 con 0x7fcf700987f0 2022-07-12T08:46:26.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75b71700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf700b5920 0x7fcf701506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:26.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf74b6f700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf70098c80 0x7fcf701501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:26.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf75370700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 0x7fcf70144620 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fcf7013e030 tx=0x7fcf64014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:26.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf6400d290 con 0x7fcf700987f0 2022-07-12T08:46:26.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6401e400 con 0x7fcf700987f0 2022-07-12T08:46:26.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf70140ea0 con 0x7fcf700987f0 2022-07-12T08:46:26.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf6401e760 con 0x7fcf700987f0 2022-07-12T08:46:26.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.964+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf70064290 con 0x7fcf700987f0 2022-07-12T08:46:26.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.968+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf70141340 con 0x7fcf700987f0 2022-07-12T08:46:26.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.968+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcf64025030 con 0x7fcf700987f0 2022-07-12T08:46:27.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.968+0000 7fcf6e7fc700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf50053a40 0x7fcf50055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.968+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf6406cd70 con 0x7fcf700987f0 2022-07-12T08:46:27.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.968+0000 7fcf74b6f700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf50053a40 0x7fcf50055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.968+0000 7fcf74b6f700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf50053a40 0x7fcf50055f00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fcf6000b000 tx=0x7fcf60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:27.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:26.972+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf64043cd0 con 0x7fcf700987f0 2022-07-12T08:46:27.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.100+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf70057460 con 0x7fcf700987f0 2022-07-12T08:46:27.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.100+0000 7fcf6e7fc700 1 -- 172.21.15.61:0/4158211644 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcf64023020 con 0x7fcf700987f0 2022-07-12T08:46:27.106 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:46:27.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf50053a40 msgr2=0x7fcf50055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf50053a40 0x7fcf50055f00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fcf6000b000 tx=0x7fcf60009040).stop 2022-07-12T08:46:27.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 msgr2=0x7fcf70144620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 0x7fcf70144620 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7fcf7013e030 tx=0x7fcf64014000).stop 2022-07-12T08:46:27.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 shutdown_connections 2022-07-12T08:46:27.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf700b5920 0x7fcf701506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf50053a40 0x7fcf50055f00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf70098c80 0x7fcf701501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 --2- 172.21.15.61:0/4158211644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf700987f0 0x7fcf70144620 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 >> 172.21.15.61:0/4158211644 conn(0x7fcf70063340 msgr2=0x7fcf70060740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:27.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 shutdown_connections 2022-07-12T08:46:27.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.104+0000 7fcf76372700 1 -- 172.21.15.61:0/4158211644 wait complete. 2022-07-12T08:46:27.117 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:46:27.118 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:27.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 Processor -- start 2022-07-12T08:46:27.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 -- start start 2022-07-12T08:46:27.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8600b0150 0x7fd86009a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd860142ef0 0x7fd860143340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd860067eb0 con 0x7fd86005f140 2022-07-12T08:46:27.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86005f650 con 0x7fd8600b0150 2022-07-12T08:46:27.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd865323700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600b0530 con 0x7fd860142ef0 2022-07-12T08:46:27.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd857fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd86005f140 0x7fd86009a850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44890/0 (socket says 172.21.15.61:44890) 2022-07-12T08:46:27.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd857fff700 1 -- 172.21.15.61:0/4254154631 learned_addr learned my addr 172.21.15.61:0/4254154631 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:27.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd85ffff700 1 --2- 172.21.15.61:0/4254154631 >> v2:172.21.15.61:3300/0 conn(0x7fd8600b0150 0x7fd86009a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd864b22700 1 --2- 172.21.15.61:0/4254154631 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd860142ef0 0x7fd860143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.220+0000 7fd85f7fe700 1 -- 172.21.15.61:0/4254154631 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884199887 0 0) 0x7fd860067eb0 con 0x7fd86005f140 2022-07-12T08:46:27.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85f7fe700 1 -- 172.21.15.61:0/4254154631 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd840003260 con 0x7fd86005f140 2022-07-12T08:46:27.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85ffff700 1 -- 172.21.15.61:0/4254154631 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd860142ef0 msgr2=0x7fd860143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:27.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85ffff700 1 --2- 172.21.15.61:0/4254154631 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd860142ef0 0x7fd860143340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85ffff700 1 -- 172.21.15.61:0/4254154631 >> v1:172.21.15.61:6789/0 conn(0x7fd86005f140 legacy=0x7fd86009a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85f7fe700 1 -- 172.21.15.61:0/4254154631 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3561843715 0 0) 0x7fd840003260 con 0x7fd86005f140 2022-07-12T08:46:27.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85ffff700 1 -- 172.21.15.61:0/4254154631 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860067c10 con 0x7fd8600b0150 2022-07-12T08:46:27.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85ffff700 1 --2- 172.21.15.61:0/4254154631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8600b0150 0x7fd86009a360 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fd848001820 tx=0x7fd850006df0).ready entity=mon.0 client_cookie=769fa34d7ab79ed3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:27.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85f7fe700 1 -- 172.21.15.61:0/4254154631 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd848001680 con 0x7fd8600b0150 2022-07-12T08:46:27.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85f7fe700 1 -- 172.21.15.61:0/4254154631 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd850000e90 con 0x7fd8600b0150 2022-07-12T08:46:27.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd85f7fe700 1 -- 172.21.15.61:0/4254154631 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd850012460 con 0x7fd8600b0150 2022-07-12T08:46:27.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 -- 172.21.15.61:0/4254154631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8600b0150 msgr2=0x7fd86009a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 --2- 172.21.15.61:0/4254154631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8600b0150 0x7fd86009a360 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fd848001820 tx=0x7fd850006df0).stop 2022-07-12T08:46:27.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 -- 172.21.15.61:0/4254154631 shutdown_connections 2022-07-12T08:46:27.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 --2- 172.21.15.61:0/4254154631 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd860142ef0 0x7fd860143340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 --2- 172.21.15.61:0/4254154631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8600b0150 0x7fd86009a360 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 -- 172.21.15.61:0/4254154631 >> 172.21.15.61:0/4254154631 conn(0x7fd8600b1950 msgr2=0x7fd8600b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:27.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 -- 172.21.15.61:0/4254154631 shutdown_connections 2022-07-12T08:46:27.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 -- 172.21.15.61:0/4254154631 wait complete. 2022-07-12T08:46:27.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 Processor -- start 2022-07-12T08:46:27.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 -- start start 2022-07-12T08:46:27.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd86005f140 0x7fd860153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.224+0000 7fd865323700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8600b0150 0x7fd8600ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 0x7fd860065a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600640d0 con 0x7fd860142ef0 2022-07-12T08:46:27.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860153830 con 0x7fd8600b0150 2022-07-12T08:46:27.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd860153540 con 0x7fd86005f140 2022-07-12T08:46:27.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 0x7fd860065a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 0x7fd860065a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56682/0 (socket says 172.21.15.61:56682) 2022-07-12T08:46:27.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 -- 172.21.15.61:0/2818763730 learned_addr learned my addr 172.21.15.61:0/2818763730 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:27.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd85ffff700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd86005f140 0x7fd860153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd857fff700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8600b0150 0x7fd8600ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 -- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd86005f140 msgr2=0x7fd860153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd86005f140 0x7fd860153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 -- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8600b0150 msgr2=0x7fd8600ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8600b0150 0x7fd8600ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 -- 172.21.15.61:0/2818763730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85000b180 con 0x7fd860142ef0 2022-07-12T08:46:27.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd864b22700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 0x7fd860065a00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fd858006e00 tx=0x7fd8580091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:27.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd85800c7f0 con 0x7fd860142ef0 2022-07-12T08:46:27.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85800c950 con 0x7fd860142ef0 2022-07-12T08:46:27.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd85800ccb0 con 0x7fd860142ef0 2022-07-12T08:46:27.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8600675c0 con 0x7fd860142ef0 2022-07-12T08:46:27.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8600ad770 con 0x7fd860142ef0 2022-07-12T08:46:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860057460 con 0x7fd860142ef0 2022-07-12T08:46:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.228+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd858022070 con 0x7fd860142ef0 2022-07-12T08:46:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.232+0000 7fd85dffb700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8440539b0 0x7fd844055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.232+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd85806c6b0 con 0x7fd860142ef0 2022-07-12T08:46:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.232+0000 7fd85ffff700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8440539b0 0x7fd844055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.232+0000 7fd85ffff700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8440539b0 0x7fd844055e70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fd850018000 tx=0x7fd850014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.236+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd858043480 con 0x7fd860142ef0 2022-07-12T08:46:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.400+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd860062b80 con 0x7fd860142ef0 2022-07-12T08:46:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd85dffb700 1 -- 172.21.15.61:0/2818763730 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fd858016090 con 0x7fd860142ef0 2022-07-12T08:46:27.409 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:27.409 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8440539b0 msgr2=0x7fd844055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8440539b0 0x7fd844055e70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fd850018000 tx=0x7fd850014000).stop 2022-07-12T08:46:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 msgr2=0x7fd860065a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 0x7fd860065a00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7fd858006e00 tx=0x7fd8580091d0).stop 2022-07-12T08:46:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 shutdown_connections 2022-07-12T08:46:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd860142ef0 0x7fd860065a00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8440539b0 0x7fd844055e70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8600b0150 0x7fd8600ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 --2- 172.21.15.61:0/2818763730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd86005f140 0x7fd860153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:27.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 >> 172.21.15.61:0/2818763730 conn(0x7fd8600b1950 msgr2=0x7fd8600ab070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 shutdown_connections 2022-07-12T08:46:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:27.404+0000 7fd865323700 1 -- 172.21.15.61:0/2818763730 wait complete. 2022-07-12T08:46:27.418 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:27.631 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 31, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:46:22.431962+0000', 'modified': '2022-07-12T08:46:23.437118+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5979}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5979': {'gid': 5979, 'name': 'a', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3698454378', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3698454378}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3698454378}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:46:27.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:46:27.632 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:46:27.751 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5979,"want_state":"up:active","state":"up:active","rank_uptime":5.3087120700000003,"mdsmap_epoch":31,"osdmap_epoch":62,"osdmap_epoch_barrier":61,"uptime":9.7018394620000006} 2022-07-12T08:46:27.752 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5979, "mdsmap_epoch": 31, "osdmap_epoch": 62, "osdmap_epoch_barrier": 61, "rank_uptime": 5.30871207, "state": "up:active", "uptime": 9.701839462, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:46:27.753 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:46:27.754 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:46:27.755 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:46:27.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:27.756 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:27.767 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:46:27.794 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:46:27.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:46:27.808 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:46:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:46:27.810 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:27.811 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:46:27.861 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:46:27.863 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:27.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:46:27.886 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:46:27.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:46:27.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21512]: starting ceph client 2022-07-12T08:46:27.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:46:27.942+0000 7f92b0d05080 -1 init, newargv = 0x562521a80430 newargc=9 2022-07-12T08:46:27.951 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21512]: starting fuse 2022-07-12T08:46:27.962 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:46:27.963 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:46:27.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:27.965 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:46:27.982 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:46:27.983 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:46:27.984 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:27.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:46:27.985 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:46:27.986 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:46:27.987 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:46:27.988 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:46:27.989 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:27.990 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:46:27.991 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:46:27.992 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:46:27.993 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:27.994 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:46:27.995 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:46:27.995 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:46:27.996 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:46:27.997 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:27.998 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:46:27.999 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:46:28.000 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:46:28.001 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:46:28.002 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:46:28.003 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:28.004 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:46:28.005 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:46:28.055 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21512.asok 2022-07-12T08:46:28.056 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21512.asok 2022-07-12T08:46:28.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:28.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21512.asok status 2022-07-12T08:46:28.151 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:46:28.152 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:46:28.152 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:46:28.152 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:46:28.153 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:46:28.153 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:46:28.153 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:46:28.153 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21512", 2022-07-12T08:46:28.154 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:46:28.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:46:28.154 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:46:28.155 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:46:28.155 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6156, 2022-07-12T08:46:28.155 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:46:28.155 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:46:28.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:46:28.156 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6156 2022-07-12T08:46:28.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:46:28.157 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:46:28.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:46:28.157 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:46:28.158 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2803727365 2022-07-12T08:46:28.158 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:46:28.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:46:28.158 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:46:28.159 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:46:28.159 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:46:28.159 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2803727365 2022-07-12T08:46:28.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:46:28.160 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6156 172.21.15.121:0/2803727365", 2022-07-12T08:46:28.160 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2803727365", 2022-07-12T08:46:28.160 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:46:28.160 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 31, 2022-07-12T08:46:28.161 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 62, 2022-07-12T08:46:28.161 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:46:28.161 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:46:28.162 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:46:28.162 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:46:28.164 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:28.164 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:28.165 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:28.174 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:46:28.174 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:46:28.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:28.176 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:28.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:28.196 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:28.196 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:28.251 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:46:28.251 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:46:28.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:46:28.253 DEBUG:teuthology.orchestra.run.smithi121:> import sys 2022-07-12T08:46:28.255 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-07-12T08:46:28.255 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:46:28.256 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:28.257 DEBUG:teuthology.orchestra.run.smithi121:> n = 1000 2022-07-12T08:46:28.257 DEBUG:teuthology.orchestra.run.smithi121:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-12T08:46:28.259 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:28.260 DEBUG:teuthology.orchestra.run.smithi121:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-12T08:46:28.261 DEBUG:teuthology.orchestra.run.smithi121:> os.makedirs(os.path.dirname(abs_path)) 2022-07-12T08:46:28.263 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:46:28.264 DEBUG:teuthology.orchestra.run.smithi121:> for i in range(0, n): 2022-07-12T08:46:28.264 DEBUG:teuthology.orchestra.run.smithi121:> fname = "{0}_{1}".format(abs_path, i) 2022-07-12T08:46:28.265 DEBUG:teuthology.orchestra.run.smithi121:> with open(fname, '"'"'w'"'"') as f: 2022-07-12T08:46:28.265 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-07-12T08:46:28.265 DEBUG:teuthology.orchestra.run.smithi121:> if True: 2022-07-12T08:46:28.266 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-07-12T08:46:28.266 DEBUG:teuthology.orchestra.run.smithi121:> os.fsync(f.fileno()) 2022-07-12T08:46:28.267 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:46:32.898 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:32.902 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:32.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:32.913 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:32.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:46:37.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:38.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.064+0000 7f2918c0a700 1 Processor -- start 2022-07-12T08:46:38.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.064+0000 7f2918c0a700 1 -- start start 2022-07-12T08:46:38.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f29140af6e0 0x7f2914143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140afac0 con 0x7f2914060c30 2022-07-12T08:46:38.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29140611b0 con 0x7f291413cb80 2022-07-12T08:46:38.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2914143970 con 0x7f29140af6e0 2022-07-12T08:46:38.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2912ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2914060c30 0x7f291413c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44906/0 (socket says 172.21.15.61:44906) 2022-07-12T08:46:38.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2912ffd700 1 -- 172.21.15.61:0/3197009190 learned_addr learned my addr 172.21.15.61:0/3197009190 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:38.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29137fe700 1 --2- 172.21.15.61:0/3197009190 >> v2:172.21.15.111:3300/0 conn(0x7f29140af6e0 0x7f2914143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928975551 0 0) 0x7f29140afac0 con 0x7f2914060c30 2022-07-12T08:46:38.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2900003260 con 0x7f2914060c30 2022-07-12T08:46:38.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2168893144 0 0) 0x7f2900003260 con 0x7f2914060c30 2022-07-12T08:46:38.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 >> v2:172.21.15.111:3300/0 conn(0x7f29140af6e0 msgr2=0x7f2914143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 --2- 172.21.15.61:0/3197009190 >> v2:172.21.15.111:3300/0 conn(0x7f29140af6e0 0x7f2914143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 >> v1:172.21.15.111:6789/0 conn(0x7f291413cb80 legacy=0x7f29140b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2914061450 con 0x7f2914060c30 2022-07-12T08:46:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f2908001660 con 0x7f2914060c30 2022-07-12T08:46:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 >> v1:172.21.15.61:6789/0 conn(0x7f2914060c30 legacy=0x7f291413c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900003750 0x7f2900003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2900004500 0x7f2900004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2900004e70 0x7f290000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2914143970 con 0x7f2900004e70 2022-07-12T08:46:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290000bd20 con 0x7f2900004500 2022-07-12T08:46:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290000bee0 con 0x7f2900003750 2022-07-12T08:46:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2912ffd700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2900004500 0x7f2900004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2900004e70 0x7f290000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29137fe700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900003750 0x7f2900003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 -- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900003750 msgr2=0x7f2900003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900003750 0x7f2900003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 -- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2900004500 msgr2=0x7f2900004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2900004500 0x7f2900004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 -- 172.21.15.61:0/3197009190 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900003260 con 0x7f2900004e70 2022-07-12T08:46:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 -- 172.21.15.61:0/3197009190 reap_dead start 2022-07-12T08:46:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2913fff700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2900004e70 0x7f290000b750 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f2914062ea0 tx=0x7f290c007d80).ready entity=mon.0 client_cookie=cb5a34d3f11f225c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290c001620 con 0x7f2900004e70 2022-07-12T08:46:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c006890 con 0x7f2900004e70 2022-07-12T08:46:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f29127fc700 1 -- 172.21.15.61:0/3197009190 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290c006bf0 con 0x7f2900004e70 2022-07-12T08:46:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- 172.21.15.61:0/3197009190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2900004e70 msgr2=0x7f290000b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2900004e70 0x7f290000b750 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f2914062ea0 tx=0x7f290c007d80).stop 2022-07-12T08:46:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- 172.21.15.61:0/3197009190 shutdown_connections 2022-07-12T08:46:38.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3197009190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2900004e70 0x7f290000b750 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- 172.21.15.61:0/3197009190 >> 172.21.15.61:0/3197009190 conn(0x7f29140b5780 msgr2=0x7f29140b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:38.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- 172.21.15.61:0/3197009190 shutdown_connections 2022-07-12T08:46:38.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.068+0000 7f2918c0a700 1 -- 172.21.15.61:0/3197009190 wait complete. 2022-07-12T08:46:38.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 Processor -- start 2022-07-12T08:46:38.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- start start 2022-07-12T08:46:38.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900004e70 0x7f2914145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2914064d30 0x7f29140ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 0x7f291413ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2914065680 con 0x7f29140adf70 2022-07-12T08:46:38.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140aeae0 con 0x7f2914064d30 2022-07-12T08:46:38.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2914064350 con 0x7f2900004e70 2022-07-12T08:46:38.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 0x7f291413ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f29137fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900004e70 0x7f2914145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 0x7f291413ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56700/0 (socket says 172.21.15.61:56700) 2022-07-12T08:46:38.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 -- 172.21.15.61:0/3776937667 learned_addr learned my addr 172.21.15.61:0/3776937667 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:38.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2912ffd700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2914064d30 0x7f29140ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 -- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900004e70 msgr2=0x7f2914145a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900004e70 0x7f2914145a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 -- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2914064d30 msgr2=0x7f29140ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2914064d30 0x7f29140ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 -- 172.21.15.61:0/3776937667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29140645f0 con 0x7f29140adf70 2022-07-12T08:46:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2913fff700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 0x7f291413ad80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f2914062ea0 tx=0x7f290c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290c00c860 con 0x7f29140adf70 2022-07-12T08:46:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c00c9c0 con 0x7f29140adf70 2022-07-12T08:46:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2914064890 con 0x7f29140adf70 2022-07-12T08:46:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290c00cd20 con 0x7f29140adf70 2022-07-12T08:46:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291413b490 con 0x7f29140adf70 2022-07-12T08:46:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.072+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2914064a20 con 0x7f29140adf70 2022-07-12T08:46:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.076+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f290c010030 con 0x7f29140adf70 2022-07-12T08:46:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.076+0000 7f2910ff9700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28f4053a40 0x7f28f4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.076+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f290c06d020 con 0x7f29140adf70 2022-07-12T08:46:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.076+0000 7f29137fe700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28f4053a40 0x7f28f4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.076+0000 7f29137fe700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28f4053a40 0x7f28f4055f00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f290400b000 tx=0x7f2904008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.080+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f290c043f80 con 0x7f29140adf70 2022-07-12T08:46:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.240+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2914057460 con 0x7f29140adf70 2022-07-12T08:46:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.240+0000 7f2910ff9700 1 -- 172.21.15.61:0/3776937667 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f290c0475a0 con 0x7f29140adf70 2022-07-12T08:46:38.245 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:38.245 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:38.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28f4053a40 msgr2=0x7f28f4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28f4053a40 0x7f28f4055f00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f290400b000 tx=0x7f2904008040).stop 2022-07-12T08:46:38.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 msgr2=0x7f291413ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 0x7f291413ad80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f2914062ea0 tx=0x7f290c016000).stop 2022-07-12T08:46:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 shutdown_connections 2022-07-12T08:46:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29140adf70 0x7f291413ad80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f28f4053a40 0x7f28f4055f00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2914064d30 0x7f29140ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 --2- 172.21.15.61:0/3776937667 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2900004e70 0x7f2914145a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 >> 172.21.15.61:0/3776937667 conn(0x7f29140b5780 msgr2=0x7f29140b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:38.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 shutdown_connections 2022-07-12T08:46:38.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.244+0000 7f2918c0a700 1 -- 172.21.15.61:0/3776937667 wait complete. 2022-07-12T08:46:38.252 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:38.264 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-12T08:46:38.370 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-12T08:46:38.371 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-12T08:46:38.372 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:38.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632812700 1 Processor -- start 2022-07-12T08:46:38.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632812700 1 -- start start 2022-07-12T08:46:38.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632812700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 0x7f962c14b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632812700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c067eb0 con 0x7f962c05f140 2022-07-12T08:46:38.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632812700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962c05f650 con 0x7f962c0b0150 2022-07-12T08:46:38.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632812700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0b0530 con 0x7f962c1495a0 2022-07-12T08:46:38.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963100f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f962c05f140 0x7f962c09aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44924/0 (socket says 172.21.15.61:44924) 2022-07-12T08:46:38.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963100f700 1 -- 172.21.15.61:0/2732879609 learned_addr learned my addr 172.21.15.61:0/2732879609 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:38.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f9632011700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 0x7f962c14b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197929806 0 0) 0x7f962c067eb0 con 0x7f962c05f140 2022-07-12T08:46:38.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9614003260 con 0x7f962c05f140 2022-07-12T08:46:38.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924390035 0 0) 0x7f9614003260 con 0x7f962c05f140 2022-07-12T08:46:38.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 msgr2=0x7f962c14b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 0x7f962c14b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 >> v1:172.21.15.111:6789/0 conn(0x7f962c0b0150 legacy=0x7f962c09a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c067c10 con 0x7f962c05f140 2022-07-12T08:46:38.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f9628001660 con 0x7f962c05f140 2022-07-12T08:46:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.472+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 >> v1:172.21.15.61:6789/0 conn(0x7f962c05f140 legacy=0x7f962c09aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614003780 0x7f9614003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9614004500 0x7f9614004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 0x7f9614004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0b0530 con 0x7f9614004500 2022-07-12T08:46:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961400b750 con 0x7f9614003780 2022-07-12T08:46:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961400b8c0 con 0x7f962c1495a0 2022-07-12T08:46:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9614004500 0x7f9614004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9631810700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614003780 0x7f9614003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632011700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 0x7f9614004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 -- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 msgr2=0x7f9614004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c1495a0 0x7f9614004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 -- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614003780 msgr2=0x7f9614003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614003780 0x7f9614003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 -- 172.21.15.61:0/2732879609 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614003260 con 0x7f9614004500 2022-07-12T08:46:38.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963100f700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9614004500 0x7f9614004980 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f9628001370 tx=0x7f962c05cf00).ready entity=mon.0 client_cookie=95940985fdb00d3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96280070d0 con 0x7f9614004500 2022-07-12T08:46:38.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9628007230 con 0x7f9614004500 2022-07-12T08:46:38.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f963080e700 1 -- 172.21.15.61:0/2732879609 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962800c960 con 0x7f9614004500 2022-07-12T08:46:38.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 -- 172.21.15.61:0/2732879609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9614004500 msgr2=0x7f9614004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 --2- 172.21.15.61:0/2732879609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9614004500 0x7f9614004980 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f9628001370 tx=0x7f962c05cf00).stop 2022-07-12T08:46:38.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632011700 1 -- 172.21.15.61:0/2732879609 reap_dead start 2022-07-12T08:46:38.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 -- 172.21.15.61:0/2732879609 shutdown_connections 2022-07-12T08:46:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 -- 172.21.15.61:0/2732879609 >> 172.21.15.61:0/2732879609 conn(0x7f962c0b1950 msgr2=0x7f962c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 -- 172.21.15.61:0/2732879609 shutdown_connections 2022-07-12T08:46:38.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 -- 172.21.15.61:0/2732879609 wait complete. 2022-07-12T08:46:38.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 Processor -- start 2022-07-12T08:46:38.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.476+0000 7f9632812700 1 -- start start 2022-07-12T08:46:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614004500 0x7f962c1416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 0x7f962c0ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c0addb0 0x7f962c0ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0aef70 con 0x7f962c064270 2022-07-12T08:46:38.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0ae860 con 0x7f9614004500 2022-07-12T08:46:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962c0ad9b0 con 0x7f962c0addb0 2022-07-12T08:46:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 0x7f962c0ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 0x7f962c0ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56718/0 (socket says 172.21.15.61:56718) 2022-07-12T08:46:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 -- 172.21.15.61:0/2506162564 learned_addr learned my addr 172.21.15.61:0/2506162564 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:38.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9631810700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614004500 0x7f962c1416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632011700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c0addb0 0x7f962c0ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 -- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c0addb0 msgr2=0x7f962c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c0addb0 0x7f962c0ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 -- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614004500 msgr2=0x7f962c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614004500 0x7f962c1416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 -- 172.21.15.61:0/2506162564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c0adb70 con 0x7f962c064270 2022-07-12T08:46:38.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f963100f700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 0x7f962c0ac2f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f9628001370 tx=0x7f9628013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9628017070 con 0x7f962c064270 2022-07-12T08:46:38.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9628004920 con 0x7f962c064270 2022-07-12T08:46:38.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c0accd0 con 0x7f962c064270 2022-07-12T08:46:38.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96280070d0 con 0x7f962c064270 2022-07-12T08:46:38.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c13bd80 con 0x7f962c064270 2022-07-12T08:46:38.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962c0af170 con 0x7f962c064270 2022-07-12T08:46:38.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.480+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9628010070 con 0x7f962c064270 2022-07-12T08:46:38.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.484+0000 7f961affd700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f960c053a60 0x7f960c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.484+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f962801a040 con 0x7f962c064270 2022-07-12T08:46:38.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.484+0000 7f9631810700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f960c053a60 0x7f960c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.484+0000 7f9631810700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f960c053a60 0x7f960c055f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f961c003000 tx=0x7f961c0079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.488+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9628043780 con 0x7f962c064270 2022-07-12T08:46:38.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.656+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f962c0aea20 con 0x7f962c064270 2022-07-12T08:46:38.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f961affd700 1 -- 172.21.15.61:0/2506162564 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f9628046da0 con 0x7f962c064270 2022-07-12T08:46:38.661 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:38.662 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:38.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f960c053a60 msgr2=0x7f960c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f960c053a60 0x7f960c055f20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f961c003000 tx=0x7f961c0079e0).stop 2022-07-12T08:46:38.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 msgr2=0x7f962c0ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 0x7f962c0ac2f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f9628001370 tx=0x7f9628013000).stop 2022-07-12T08:46:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 shutdown_connections 2022-07-12T08:46:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f962c0addb0 0x7f962c0ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962c064270 0x7f962c0ac2f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f960c053a60 0x7f960c055f20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 --2- 172.21.15.61:0/2506162564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9614004500 0x7f962c1416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 >> 172.21.15.61:0/2506162564 conn(0x7f962c0b1950 msgr2=0x7f962c1430c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 shutdown_connections 2022-07-12T08:46:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.660+0000 7f9632812700 1 -- 172.21.15.61:0/2506162564 wait complete. 2022-07-12T08:46:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:38.675 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-12T08:46:38.780 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-12T08:46:38.781 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-12T08:46:38.781 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 Processor -- start 2022-07-12T08:46:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 -- start start 2022-07-12T08:46:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f47340ab290 0x7f47340ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f473405f140 0x7f473405ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340abc70 con 0x7f47340afa60 2022-07-12T08:46:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473405f580 con 0x7f473405f140 2022-07-12T08:46:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4738a31700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473405c720 con 0x7f47340ab290 2022-07-12T08:46:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f473405f140 0x7f473405ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f473405f140 0x7f473405ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56728/0 (socket says 172.21.15.61:56728) 2022-07-12T08:46:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 -- 172.21.15.61:0/2229183635 learned_addr learned my addr 172.21.15.61:0/2229183635 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f47337fe700 1 --2- 172.21.15.61:0/2229183635 >> v2:172.21.15.111:3300/0 conn(0x7f47340ab290 0x7f47340ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 -- 172.21.15.61:0/2229183635 >> v2:172.21.15.111:3300/0 conn(0x7f47340ab290 msgr2=0x7f47340ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 --2- 172.21.15.61:0/2229183635 >> v2:172.21.15.111:3300/0 conn(0x7f47340ab290 0x7f47340ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 -- 172.21.15.61:0/2229183635 >> v1:172.21.15.111:6789/0 conn(0x7f47340afa60 legacy=0x7f473414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.880+0000 7f4732ffd700 1 -- 172.21.15.61:0/2229183635 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473405c9c0 con 0x7f473405f140 2022-07-12T08:46:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4732ffd700 1 --2- 172.21.15.61:0/2229183635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f473405ce60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f471c002f70 tx=0x7f471c006eb0).ready entity=mon.0 client_cookie=a2d75a004550b839 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f47327fc700 1 -- 172.21.15.61:0/2229183635 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c00d610 con 0x7f473405f140 2022-07-12T08:46:38.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f47327fc700 1 -- 172.21.15.61:0/2229183635 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c000e90 con 0x7f473405f140 2022-07-12T08:46:38.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f47327fc700 1 -- 172.21.15.61:0/2229183635 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c00d9a0 con 0x7f473405f140 2022-07-12T08:46:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 -- 172.21.15.61:0/2229183635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 msgr2=0x7f473405ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 --2- 172.21.15.61:0/2229183635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f473405ce60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f471c002f70 tx=0x7f471c006eb0).stop 2022-07-12T08:46:38.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 -- 172.21.15.61:0/2229183635 shutdown_connections 2022-07-12T08:46:38.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 --2- 172.21.15.61:0/2229183635 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f473405ce60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 --2- 172.21.15.61:0/2229183635 >> v2:172.21.15.111:3300/0 conn(0x7f47340ab290 0x7f47340ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 -- 172.21.15.61:0/2229183635 >> 172.21.15.61:0/2229183635 conn(0x7f47340b1950 msgr2=0x7f47340b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 -- 172.21.15.61:0/2229183635 shutdown_connections 2022-07-12T08:46:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 -- 172.21.15.61:0/2229183635 wait complete. 2022-07-12T08:46:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.884+0000 7f4738a31700 1 Processor -- start 2022-07-12T08:46:38.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 -- start start 2022-07-12T08:46:38.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f4734099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47340ab290 0x7f47340ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47340afa60 0x7f4734156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4734063d80 con 0x7f473405f140 2022-07-12T08:46:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4734064020 con 0x7f47340ab290 2022-07-12T08:46:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47340adfd0 con 0x7f47340afa60 2022-07-12T08:46:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f4734099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f4734099970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56732/0 (socket says 172.21.15.61:56732) 2022-07-12T08:46:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 -- 172.21.15.61:0/3102310839 learned_addr learned my addr 172.21.15.61:0/3102310839 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4732ffd700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47340ab290 0x7f47340ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4733fff700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47340afa60 0x7f4734156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 -- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47340afa60 msgr2=0x7f4734156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47340afa60 0x7f4734156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 -- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47340ab290 msgr2=0x7f47340ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:38.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47340ab290 0x7f47340ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:38.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 -- 172.21.15.61:0/3102310839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c00b1e0 con 0x7f473405f140 2022-07-12T08:46:38.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4732ffd700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47340ab290 0x7f47340ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:38.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f47337fe700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f4734099970 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f472c003430 tx=0x7f472c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f472c013620 con 0x7f473405f140 2022-07-12T08:46:38.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c013780 con 0x7f473405f140 2022-07-12T08:46:38.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47340ae470 con 0x7f473405f140 2022-07-12T08:46:38.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f472c013ae0 con 0x7f473405f140 2022-07-12T08:46:38.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.888+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4734157030 con 0x7f473405f140 2022-07-12T08:46:38.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.892+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47340641e0 con 0x7f473405f140 2022-07-12T08:46:38.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.892+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f472c013c40 con 0x7f473405f140 2022-07-12T08:46:38.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.892+0000 7f4730ff9700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4718053a40 0x7f4718055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:38.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.892+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f472c06c0c0 con 0x7f473405f140 2022-07-12T08:46:38.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.892+0000 7f4732ffd700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4718053a40 0x7f4718055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:38.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.892+0000 7f4732ffd700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4718053a40 0x7f4718055f00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f471c01a000 tx=0x7f471c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:38.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:38.896+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f472c042fc0 con 0x7f473405f140 2022-07-12T08:46:39.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.064+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4734057460 con 0x7f473405f140 2022-07-12T08:46:39.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.064+0000 7f4730ff9700 1 -- 172.21.15.61:0/3102310839 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f472c0465e0 con 0x7f473405f140 2022-07-12T08:46:39.069 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:39.070 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:39.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4718053a40 msgr2=0x7f4718055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4718053a40 0x7f4718055f00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f471c01a000 tx=0x7f471c015000).stop 2022-07-12T08:46:39.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 msgr2=0x7f4734099970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f4734099970 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f472c003430 tx=0x7f472c008040).stop 2022-07-12T08:46:39.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 shutdown_connections 2022-07-12T08:46:39.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47340afa60 0x7f4734156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4718053a40 0x7f4718055f00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f47340ab290 0x7f47340ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 --2- 172.21.15.61:0/3102310839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f473405f140 0x7f4734099970 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 >> 172.21.15.61:0/3102310839 conn(0x7f47340b1950 msgr2=0x7f473414d870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:39.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 shutdown_connections 2022-07-12T08:46:39.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.068+0000 7f4738a31700 1 -- 172.21.15.61:0/3102310839 wait complete. 2022-07-12T08:46:39.083 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:39.084 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 10 2022-07-12T08:46:39.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.188+0000 7f8e2318f700 1 Processor -- start 2022-07-12T08:46:39.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.188+0000 7f8e2318f700 1 -- start start 2022-07-12T08:46:39.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8e1c0aa830 0x7f8e1c0aac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c05c3e0 con 0x7f8e1c0b0280 2022-07-12T08:46:39.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c0b5d00 con 0x7f8e1c0aa830 2022-07-12T08:46:39.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c05c610 con 0x7f8e1c0b5920 2022-07-12T08:46:39.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2298e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8e1c0aa830 0x7f8e1c0aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2198c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8e1c0b0280 0x7f8e1c14aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44958/0 (socket says 172.21.15.61:44958) 2022-07-12T08:46:39.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2198c700 1 -- 172.21.15.61:0/1097655174 learned_addr learned my addr 172.21.15.61:0/1097655174 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:39.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1097655174 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2118b700 1 -- 172.21.15.61:0/1097655174 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301340938 0 0) 0x7f8e1c05c3e0 con 0x7f8e1c0b0280 2022-07-12T08:46:39.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2118b700 1 -- 172.21.15.61:0/1097655174 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e04003260 con 0x7f8e1c0b0280 2022-07-12T08:46:39.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2298e700 1 -- 172.21.15.61:0/1097655174 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 msgr2=0x7f8e1c14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2298e700 1 --2- 172.21.15.61:0/1097655174 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c14a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2298e700 1 -- 172.21.15.61:0/1097655174 >> v1:172.21.15.61:6789/0 conn(0x7f8e1c0b0280 legacy=0x7f8e1c14aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2298e700 1 -- 172.21.15.61:0/1097655174 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c05c8b0 con 0x7f8e1c0aa830 2022-07-12T08:46:39.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2298e700 1 --2- 172.21.15.61:0/1097655174 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c0aac80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f8e14002f70 tx=0x7f8e14006df0).ready entity=mon.0 client_cookie=91d33f2ec067b1b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2118b700 1 -- 172.21.15.61:0/1097655174 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e1400d4a0 con 0x7f8e1c0aa830 2022-07-12T08:46:39.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2118b700 1 -- 172.21.15.61:0/1097655174 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14000e90 con 0x7f8e1c0aa830 2022-07-12T08:46:39.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2118b700 1 -- 172.21.15.61:0/1097655174 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e1400d7f0 con 0x7f8e1c0aa830 2022-07-12T08:46:39.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- 172.21.15.61:0/1097655174 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 msgr2=0x7f8e1c0aac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 --2- 172.21.15.61:0/1097655174 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c0aac80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f8e14002f70 tx=0x7f8e14006df0).stop 2022-07-12T08:46:39.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- 172.21.15.61:0/1097655174 shutdown_connections 2022-07-12T08:46:39.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 --2- 172.21.15.61:0/1097655174 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c0aac80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 --2- 172.21.15.61:0/1097655174 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c14a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- 172.21.15.61:0/1097655174 >> 172.21.15.61:0/1097655174 conn(0x7f8e1c063340 msgr2=0x7f8e1c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:39.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- 172.21.15.61:0/1097655174 shutdown_connections 2022-07-12T08:46:39.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.192+0000 7f8e2318f700 1 -- 172.21.15.61:0/1097655174 wait complete. 2022-07-12T08:46:39.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 Processor -- start 2022-07-12T08:46:39.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 -- start start 2022-07-12T08:46:39.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c14d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e1c0b0280 0x7f8e1c1483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c064dc0 con 0x7f8e1c0aa830 2022-07-12T08:46:39.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c14e410 con 0x7f8e1c0b0280 2022-07-12T08:46:39.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c063e00 con 0x7f8e1c0b5920 2022-07-12T08:46:39.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c14d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c14d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56746/0 (socket says 172.21.15.61:56746) 2022-07-12T08:46:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 -- 172.21.15.61:0/3551315053 learned_addr learned my addr 172.21.15.61:0/3551315053 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e1c0b0280 0x7f8e1c1483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2298e700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 -- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 msgr2=0x7f8e1c148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 -- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e1c0b0280 msgr2=0x7f8e1c1483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e1c0b0280 0x7f8e1c1483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 -- 172.21.15.61:0/3551315053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1400b180 con 0x7f8e1c0aa830 2022-07-12T08:46:39.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2298e700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:39.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c14d040 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f8e1c065710 tx=0x7f8e0c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e0b7fe700 1 -- 172.21.15.61:0/3551315053 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c010780 con 0x7f8e1c0aa830 2022-07-12T08:46:39.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e0b7fe700 1 -- 172.21.15.61:0/3551315053 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c0108e0 con 0x7f8e1c0aa830 2022-07-12T08:46:39.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c0640a0 con 0x7f8e1c0aa830 2022-07-12T08:46:39.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e0b7fe700 1 -- 172.21.15.61:0/3551315053 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c010c40 con 0x7f8e1c0aa830 2022-07-12T08:46:39.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.196+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c155d70 con 0x7f8e1c0aa830 2022-07-12T08:46:39.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.200+0000 7f8e0b7fe700 1 -- 172.21.15.61:0/3551315053 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8e0c01c460 con 0x7f8e1c0aa830 2022-07-12T08:46:39.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.200+0000 7f8e0b7fe700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dfc053a40 0x7f8dfc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.200+0000 7f8e0b7fe700 1 -- 172.21.15.61:0/3551315053 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e0c06c290 con 0x7f8e1c0aa830 2022-07-12T08:46:39.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.200+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dfc053a40 0x7f8dfc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.200+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dfc053a40 0x7f8dfc055f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8e1800d000 tx=0x7f8e18009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 Processor -- start 2022-07-12T08:46:39.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- start start 2022-07-12T08:46:39.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 0x29d5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x29d6230 0x29d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29d3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2947130 con 0x2a100c0 2022-07-12T08:46:39.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2927430 con 0x29d39b0 2022-07-12T08:46:39.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x29d1980 con 0x29d6230 2022-07-12T08:46:39.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 0x29d5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 0x29d5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56754/0 (socket says 172.21.15.61:56754) 2022-07-12T08:46:39.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 -- 172.21.15.61:0/1702589224 learned_addr learned my addr 172.21.15.61:0/1702589224 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:39.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2198c700 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x29d6230 0x29d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2298e700 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29d3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 -- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x29d6230 msgr2=0x29d66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x29d6230 0x29d66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 -- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 msgr2=0x29d3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29d3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 -- 172.21.15.61:0/1702589224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x297ebf0 con 0x2a100c0 2022-07-12T08:46:39.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2298e700 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29d3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:39.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 0x29d5d40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f8e0c06c4a0 tx=0x7f8e0c014000).ready entity=mon.0 client_cookie=90cb5d136b6c4234 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2318f700 1 -- 172.21.15.61:0/1702589224 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c06c0b0 con 0x2a100c0 2022-07-12T08:46:39.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2318f700 1 -- 172.21.15.61:0/1702589224 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c0451f0 con 0x2a100c0 2022-07-12T08:46:39.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2318f700 1 -- 172.21.15.61:0/1702589224 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c043560 con 0x2a100c0 2022-07-12T08:46:39.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- 172.21.15.61:0/1702589224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 msgr2=0x29d5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 0x29d5d40 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f8e0c06c4a0 tx=0x7f8e0c014000).stop 2022-07-12T08:46:39.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- 172.21.15.61:0/1702589224 shutdown_connections 2022-07-12T08:46:39.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29d3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x29d6230 0x29d66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1702589224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x2a100c0 0x29d5d40 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- 172.21.15.61:0/1702589224 >> 172.21.15.61:0/1702589224 conn(0x29d29b0 msgr2=0x2a0f800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:39.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- 172.21.15.61:0/1702589224 shutdown_connections 2022-07-12T08:46:39.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.204+0000 7f8e2e648740 1 -- 172.21.15.61:0/1702589224 wait complete. 2022-07-12T08:46:39.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- start start 2022-07-12T08:46:39.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 Processor -- start 2022-07-12T08:46:39.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29da8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a1a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a1ad30 con 0x29d6230 2022-07-12T08:46:39.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a098b0 con 0x29d39b0 2022-07-12T08:46:39.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x29daf60 con 0x2a100c0 2022-07-12T08:46:39.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 10 client.0 ll_register_callbacks cb 0x298e630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:46:39.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a1a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a1a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56762/0 (socket says 172.21.15.61:56762) 2022-07-12T08:46:39.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 -- 172.21.15.61:0/1931652175 learned_addr learned my addr 172.21.15.61:0/1931652175 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:39.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29da8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2298e700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 -- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 msgr2=0x29d99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 -- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 msgr2=0x29da8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29da8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 -- 172.21.15.61:0/1931652175 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c155d70 con 0x29d6230 2022-07-12T08:46:39.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2198c700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a1a840 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f8e18018000 tx=0x7f8e18016040).ready entity=mon.0 client_cookie=90841565dd23c144 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8df97fa700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-12T08:46:39.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8df97fa700 1 -- 172.21.15.61:0/1931652175 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e18012560 con 0x29d6230 2022-07-12T08:46:39.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8df97fa700 1 -- 172.21.15.61:0/1931652175 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e180126c0 con 0x29d6230 2022-07-12T08:46:39.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x293d1a0 con 0x29d6230 2022-07-12T08:46:39.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8df97fa700 1 -- 172.21.15.61:0/1931652175 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e18012880 con 0x29d6230 2022-07-12T08:46:39.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 10 client.6189 fetch_fsmap learned FSMap version 31 2022-07-12T08:46:39.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x292c110 con 0x29d6230 2022-07-12T08:46:39.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8df97fa700 1 -- 172.21.15.61:0/1931652175 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x293d1a0 con 0x29d6230 2022-07-12T08:46:39.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8df97fa700 1 -- 172.21.15.61:0/1931652175 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 31) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f8e18021430 con 0x29d6230 2022-07-12T08:46:39.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 10 client.6189 fetch_fsmap finished waiting for FSMap version 31 2022-07-12T08:46:39.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 10 client.6189 resolve_mds: resolved ID 'a' to GID 5979 2022-07-12T08:46:39.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 20 client.6189 populate_metadata read hostname 'smithi061' 2022-07-12T08:46:39.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x2a1dfa0 0x2a1e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 4 client.6189 mds_command: new command op to 5979 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-12T08:46:39.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 --> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2935220 con 0x2a1dfa0 2022-07-12T08:46:39.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.208+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x2a1dfa0 0x2a1e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2218d700 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x2a1dfa0 0x2a1e380 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=b1eae57b21570a3e server_cookie=e97efb77796822a in_seq=0 out_seq=0 2022-07-12T08:46:39.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8df97fa700 10 client.6189 ms_handle_connect on v2:172.21.15.61:6832/3698454378 2022-07-12T08:46:39.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8df97fa700 1 -- 172.21.15.61:0/1931652175 <== mds.0 v2:172.21.15.61:6832/3698454378 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x2935220 con 0x2a1dfa0 2022-07-12T08:46:39.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8df97fa700 10 client.6189 handle_command_reply: tid=0 2022-07-12T08:46:39.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 client.6189 shutdown 2022-07-12T08:46:39.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 msgr2=0x2a1a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a1a840 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f8e18018000 tx=0x7f8e18016040).stop 2022-07-12T08:46:39.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 shutdown_connections 2022-07-12T08:46:39.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x2a1dfa0 0x2a1e380 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8df97fa700 0 client.6189 ms_handle_reset on v2:172.21.15.61:6832/3698454378 2022-07-12T08:46:39.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a1a840 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 --2- 172.21.15.61:0/1931652175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x29da8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 >> 172.21.15.61:0/1931652175 conn(0x29d29b0 msgr2=0x2a0f710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:39.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 shutdown_connections 2022-07-12T08:46:39.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 1 -- 172.21.15.61:0/1931652175 wait complete. 2022-07-12T08:46:39.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.212+0000 7f8e2e648740 20 client.6189 trim_cache size 0 max 16384 2022-07-12T08:46:39.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.224+0000 7f8e2e648740 1 Processor -- start 2022-07-12T08:46:39.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.224+0000 7f8e2e648740 1 -- start start 2022-07-12T08:46:39.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 0x29d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d6230 0x29d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x292cd00 con 0x29d39b0 2022-07-12T08:46:39.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2973730 con 0x29d6230 2022-07-12T08:46:39.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x2959aa0 con 0x2a100c0 2022-07-12T08:46:39.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 0x29d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 0x29d0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56768/0 (socket says 172.21.15.61:56768) 2022-07-12T08:46:39.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 -- 172.21.15.61:0/3180717924 learned_addr learned my addr 172.21.15.61:0/3180717924 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2298e700 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d6230 0x29d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 -- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 msgr2=0x29d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 -- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d6230 msgr2=0x29d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d6230 0x29d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 -- 172.21.15.61:0/3180717924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2936790 con 0x29d39b0 2022-07-12T08:46:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2298e700 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 0x29d0410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f8e0c01d000 tx=0x7f8e0c027000).ready entity=mon.0 client_cookie=32e05a87832e3927 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8dfb7fe700 1 -- 172.21.15.61:0/3180717924 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c034040 con 0x29d39b0 2022-07-12T08:46:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8dfb7fe700 1 -- 172.21.15.61:0/3180717924 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c06c0b0 con 0x29d39b0 2022-07-12T08:46:39.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8dfb7fe700 1 -- 172.21.15.61:0/3180717924 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c03a0d0 con 0x29d39b0 2022-07-12T08:46:39.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- 172.21.15.61:0/3180717924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 msgr2=0x29d0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 0x29d0410 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f8e0c01d000 tx=0x7f8e0c027000).stop 2022-07-12T08:46:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- 172.21.15.61:0/3180717924 shutdown_connections 2022-07-12T08:46:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x29d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d6230 0x29d0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3180717924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d39b0 0x29d0410 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- 172.21.15.61:0/3180717924 >> 172.21.15.61:0/3180717924 conn(0x29d29b0 msgr2=0x29d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- 172.21.15.61:0/3180717924 shutdown_connections 2022-07-12T08:46:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- 172.21.15.61:0/3180717924 wait complete. 2022-07-12T08:46:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.228+0000 7f8e2e648740 1 -- start start 2022-07-12T08:46:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 Processor -- start 2022-07-12T08:46:39.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x2a0bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a6e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x2a6cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a6dc20 con 0x29d6230 2022-07-12T08:46:39.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a6dec0 con 0x29d39b0 2022-07-12T08:46:39.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a18160 con 0x2a100c0 2022-07-12T08:46:39.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 10 client.0 ll_register_callbacks cb 0x29de130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-12T08:46:39.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2198c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a6e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x2a0bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2198c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a6e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56776/0 (socket says 172.21.15.61:56776) 2022-07-12T08:46:39.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2198c700 1 -- 172.21.15.61:0/3095506160 learned_addr learned my addr 172.21.15.61:0/3095506160 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:39.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2298e700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x2a6cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 -- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 msgr2=0x2a6cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x2a6cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 -- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 msgr2=0x2a6e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a6e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 -- 172.21.15.61:0/3095506160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c05c8b0 con 0x29d39b0 2022-07-12T08:46:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a6e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2218d700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x2a0bb90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8e0c01d000 tx=0x7f8e0c036000).ready entity=mon.1 client_cookie=576ce61aff3bb514 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8dfa7fc700 10 client.0 ms_handle_connect on v2:172.21.15.111:3300/0 2022-07-12T08:46:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8dfa7fc700 1 -- 172.21.15.61:0/3095506160 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e0c034040 con 0x29d39b0 2022-07-12T08:46:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8dfa7fc700 1 -- 172.21.15.61:0/3095506160 <== mon.1 v2:172.21.15.111:3300/0 2 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8e0c043560 con 0x29d39b0 2022-07-12T08:46:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e0c0451f0 con 0x29d39b0 2022-07-12T08:46:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8dfa7fc700 1 -- 172.21.15.61:0/3095506160 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e0c043720 con 0x29d39b0 2022-07-12T08:46:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 10 client.6082 fetch_fsmap learned FSMap version 31 2022-07-12T08:46:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.232+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x297ebf0 con 0x29d39b0 2022-07-12T08:46:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8dfa7fc700 1 -- 172.21.15.61:0/3095506160 <== mon.1 v2:172.21.15.111:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c0451f0 con 0x29d39b0 2022-07-12T08:46:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8dfa7fc700 1 -- 172.21.15.61:0/3095506160 <== mon.1 v2:172.21.15.111:3300/0 5 ==== fsmap(e 31) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f8e0c046340 con 0x29d39b0 2022-07-12T08:46:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2e648740 10 client.6082 fetch_fsmap finished waiting for FSMap version 31 2022-07-12T08:46:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2e648740 10 client.6082 resolve_mds: resolved ID 'a' to GID 5979 2022-07-12T08:46:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2e648740 20 client.6082 populate_metadata read hostname 'smithi061' 2022-07-12T08:46:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x29d97b0 0x2a6d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2e648740 4 client.6082 mds_command: new command op to 5979 tid=0[{"prefix": "cache drop", "timeout": 10}] 2022-07-12T08:46:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 --> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] -- command(tid 0: {"prefix": "cache drop", "timeout": 10}) v1 -- 0x282a5e0 con 0x29d97b0 2022-07-12T08:46:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x29d97b0 0x2a6d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8dfa7fc700 10 client.6082 ms_handle_connect on v2:172.21.15.61:6832/3698454378 2022-07-12T08:46:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:39.236+0000 7f8e2198c700 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x29d97b0 0x2a6d170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=7b206dcf2f92b306 server_cookie=5b68b7ad0244035e in_seq=0 out_seq=0 2022-07-12T08:46:49.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8dfa7fc700 1 -- 172.21.15.61:0/3095506160 <== mds.0 v2:172.21.15.61:6832/3698454378 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+400 (crc 0 0 0) 0x282a5e0 con 0x29d97b0 2022-07-12T08:46:49.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8dfa7fc700 10 client.6082 handle_command_reply: tid=0 2022-07-12T08:46:49.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 client.6082 shutdown 2022-07-12T08:46:49.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 msgr2=0x2a0bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:49.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x2a0bb90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f8e0c01d000 tx=0x7f8e0c036000).stop 2022-07-12T08:46:49.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 shutdown_connections 2022-07-12T08:46:49.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:6832/3698454378,v1:172.21.15.61:6833/3698454378] conn(0x29d97b0 0x2a6d170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x2a100c0 0x2a6cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x29d6230 0x2a6e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 --2- 172.21.15.61:0/3095506160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x29d39b0 0x2a0bb90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8dfa7fc700 0 client.6082 ms_handle_reset on v2:172.21.15.61:6832/3698454378 2022-07-12T08:46:49.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 >> 172.21.15.61:0/3095506160 conn(0x29d29b0 msgr2=0x29d0410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:49.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 shutdown_connections 2022-07-12T08:46:49.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 1 -- 172.21.15.61:0/3095506160 wait complete. 2022-07-12T08:46:49.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2e648740 20 client.6082 trim_cache size 0 max 16384 2022-07-12T08:46:49.481 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:46:49.481 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-12T08:46:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": -110, 2022-07-12T08:46:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(110) Connection timed out", 2022-07-12T08:46:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 240 2022-07-12T08:46:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-12T08:46:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-12T08:46:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-12T08:46:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:49.484 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-12T08:46:49.484 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 220 2022-07-12T08:46:49.484 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:49.484 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-12T08:46:49.484 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-12T08:46:49.485 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 15438, 2022-07-12T08:46:49.485 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 2428384 2022-07-12T08:46:49.485 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:49.485 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:49.486 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 10.236162144 2022-07-12T08:46:49.486 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:46:49.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dfc053a40 msgr2=0x7f8dfc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:49.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dfc053a40 0x7f8dfc055f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8e1800d000 tx=0x7f8e18009040).stop 2022-07-12T08:46:49.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 msgr2=0x7f8e1c14d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:49.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c14d040 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f8e1c065710 tx=0x7f8e0c006df0).stop 2022-07-12T08:46:49.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 shutdown_connections 2022-07-12T08:46:49.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e1c0b5920 0x7f8e1c148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dfc053a40 0x7f8dfc055f00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e1c0b0280 0x7f8e1c1483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 --2- 172.21.15.61:0/3551315053 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e1c0aa830 0x7f8e1c14d040 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:49.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 >> 172.21.15.61:0/3551315053 conn(0x7f8e1c063340 msgr2=0x7f8e1c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:49.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 shutdown_connections 2022-07-12T08:46:49.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:49.472+0000 7f8e2318f700 1 -- 172.21.15.61:0/3551315053 wait complete. 2022-07-12T08:46:49.491 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:46:49.492 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:49.511 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:49.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:49.561 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:49.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:49.616 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:46:49.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:49.617 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:49.617 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:49.626 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:46:49.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:46:49.629 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:46:49.630 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:46:49.630 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:49.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21512]: fuse finished with error 0 and tester_r 0 2022-07-12T08:46:49.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:49.649 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:49.650 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:49.700 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:46:49.702 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:46:49.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:46:49.762 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:49.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:49.786 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:49.787 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:46:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:49.810 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:46:49.810 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:46:49.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:46:49.812 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:46:49.868 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:46:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:49.873 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:49.874 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:49.875 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:46:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:49.895 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:46:49.895 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:46:49.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:46:49.896 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:46:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:49.905 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:46:49.905 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:46:50.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 Processor -- start 2022-07-12T08:46:50.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 -- start start 2022-07-12T08:46:50.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f75440af4b0 0x7f75440af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f754405bdf0 0x7f754405c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7544143610 con 0x7f7544060b50 2022-07-12T08:46:50.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7544143840 con 0x7f75440af4b0 2022-07-12T08:46:50.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7548e00700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75440afef0 con 0x7f754405bdf0 2022-07-12T08:46:50.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7542ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7544060b50 0x7f7544142120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45002/0 (socket says 172.21.15.61:45002) 2022-07-12T08:46:50.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7542ffd700 1 -- 172.21.15.61:0/367238665 learned_addr learned my addr 172.21.15.61:0/367238665 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:50.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75437fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f75440af4b0 0x7f75440af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f7543fff700 1 --2- 172.21.15.61:0/367238665 >> v2:172.21.15.111:3300/0 conn(0x7f754405bdf0 0x7f754405c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75427fc700 1 -- 172.21.15.61:0/367238665 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629495949 0 0) 0x7f7544143610 con 0x7f7544060b50 2022-07-12T08:46:50.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75427fc700 1 -- 172.21.15.61:0/367238665 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7524003260 con 0x7f7544060b50 2022-07-12T08:46:50.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75437fe700 1 -- 172.21.15.61:0/367238665 >> v2:172.21.15.111:3300/0 conn(0x7f754405bdf0 msgr2=0x7f754405c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:50.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75437fe700 1 --2- 172.21.15.61:0/367238665 >> v2:172.21.15.111:3300/0 conn(0x7f754405bdf0 0x7f754405c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75437fe700 1 -- 172.21.15.61:0/367238665 >> v1:172.21.15.61:6789/0 conn(0x7f7544060b50 legacy=0x7f7544142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75437fe700 1 -- 172.21.15.61:0/367238665 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7544142830 con 0x7f75440af4b0 2022-07-12T08:46:50.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75437fe700 1 --2- 172.21.15.61:0/367238665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75440af4b0 0x7f75440af890 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f752c002f70 tx=0x7f752c006df0).ready entity=mon.0 client_cookie=5ef031b85ae4030 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75427fc700 1 -- 172.21.15.61:0/367238665 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f752c00d4f0 con 0x7f75440af4b0 2022-07-12T08:46:50.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75427fc700 1 -- 172.21.15.61:0/367238665 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752c000e90 con 0x7f75440af4b0 2022-07-12T08:46:50.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.020+0000 7f75427fc700 1 -- 172.21.15.61:0/367238665 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f752c00d840 con 0x7f75440af4b0 2022-07-12T08:46:50.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- 172.21.15.61:0/367238665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75440af4b0 msgr2=0x7f75440af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 --2- 172.21.15.61:0/367238665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75440af4b0 0x7f75440af890 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f752c002f70 tx=0x7f752c006df0).stop 2022-07-12T08:46:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- 172.21.15.61:0/367238665 shutdown_connections 2022-07-12T08:46:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 --2- 172.21.15.61:0/367238665 >> v2:172.21.15.111:3300/0 conn(0x7f754405bdf0 0x7f754405c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 --2- 172.21.15.61:0/367238665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75440af4b0 0x7f75440af890 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- 172.21.15.61:0/367238665 >> 172.21.15.61:0/367238665 conn(0x7f75440b92c0 msgr2=0x7f75440b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:50.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- 172.21.15.61:0/367238665 shutdown_connections 2022-07-12T08:46:50.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- 172.21.15.61:0/367238665 wait complete. 2022-07-12T08:46:50.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 Processor -- start 2022-07-12T08:46:50.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- start start 2022-07-12T08:46:50.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f754405bdf0 0x7f75440b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 0x7f7544098d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75440af4b0 0x7f7544143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544144470 con 0x7f7544060b50 2022-07-12T08:46:50.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544144690 con 0x7f754405bdf0 2022-07-12T08:46:50.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.024+0000 7f7548e00700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7544144930 con 0x7f75440af4b0 2022-07-12T08:46:50.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 0x7f7544098d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f75437fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f754405bdf0 0x7f75440b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 0x7f7544098d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56792/0 (socket says 172.21.15.61:56792) 2022-07-12T08:46:50.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 -- 172.21.15.61:0/598263665 learned_addr learned my addr 172.21.15.61:0/598263665 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:50.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7543fff700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75440af4b0 0x7f7544143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 -- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75440af4b0 msgr2=0x7f7544143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75440af4b0 0x7f7544143f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 -- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f754405bdf0 msgr2=0x7f75440b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f754405bdf0 0x7f75440b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 -- 172.21.15.61:0/598263665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c00b180 con 0x7f7544060b50 2022-07-12T08:46:50.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7542ffd700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 0x7f7544098d20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f7534001110 tx=0x7f7534007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75340070d0 con 0x7f7544060b50 2022-07-12T08:46:50.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7534007230 con 0x7f7544060b50 2022-07-12T08:46:50.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7534005610 con 0x7f7544060b50 2022-07-12T08:46:50.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7544151e10 con 0x7f7544060b50 2022-07-12T08:46:50.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75441524d0 con 0x7f7544060b50 2022-07-12T08:46:50.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7534010070 con 0x7f7544060b50 2022-07-12T08:46:50.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7540ff9700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7528053a40 0x7f7528055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f753406b4d0 con 0x7f7544060b50 2022-07-12T08:46:50.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7544151fd0 con 0x7f7544060b50 2022-07-12T08:46:50.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.028+0000 7f75437fe700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7528053a40 0x7f7528055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.032+0000 7f75437fe700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7528053a40 0x7f7528055f00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f752c01a000 tx=0x7f752c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.036+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7534018020 con 0x7f7544060b50 2022-07-12T08:46:50.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.156+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7544057460 con 0x7f7544060b50 2022-07-12T08:46:50.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.156+0000 7f7540ff9700 1 -- 172.21.15.61:0/598263665 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v62) v1 ==== 81+0+33696 (secure 0 0 0) 0x7f7534042390 con 0x7f7544060b50 2022-07-12T08:46:50.160 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:50.161 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:46:50.161 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 62, 2022-07-12T08:46:50.165 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:46:50.165 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:46:50.165 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:46:23.436615+0000", 2022-07-12T08:46:50.166 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:50.166 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:50.166 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:46:50.167 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:46:50.167 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:46:50.167 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:46:50.167 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:46:50.168 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:46:50.168 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:46:50.168 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:50.169 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:46:50.169 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:46:50.169 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:46:50.169 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:46:50.170 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:46:50.170 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 12, 2022-07-12T08:46:50.170 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:46:50.170 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:46:50.171 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:46:50.171 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:46:50.171 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:46:50.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.172 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:46:50.172 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:46:50.172 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:46:50.172 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:50.173 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:50.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:50.173 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:50.173 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:50.176 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:50.176 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:50.179 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:50.179 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:46:50.181 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:46:50.181 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:46:50.181 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:46:50.182 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:46:50.182 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:50.182 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:50.182 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:50.183 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:50.183 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:50.183 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:50.183 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:50.184 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.184 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:46:50.184 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:50.186 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:50.187 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:50.187 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:50.188 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:50.188 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:50.189 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:50.189 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:50.190 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:50.191 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:50.192 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:50.192 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:50.193 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:50.193 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:50.193 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:50.194 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:50.194 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:50.195 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:50.195 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:50.198 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:50.198 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:50.198 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:50.199 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:50.199 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:50.199 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:50.200 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:50.201 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.201 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:50.202 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:50.203 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:50.203 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:50.204 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:50.204 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:50.205 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:50.205 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:50.206 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:50.206 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:50.207 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:50.208 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:46:50.209 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:46:50.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.209 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:50.210 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:46:50.210 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.211 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:46:50.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:46:50.213 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:46:50.214 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:50.214 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:50.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:50.215 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:50.216 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:50.216 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:50.217 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:50.217 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:50.219 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:50.219 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:50.219 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:50.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:50.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:50.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:50.221 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:50.221 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:50.222 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:50.223 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:50.224 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:50.224 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:46:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:50.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:50.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:50.227 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:50.227 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:46:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:46:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 11, 2022-07-12T08:46:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:46:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:46:21.097184+0000", 2022-07-12T08:46:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "61", 2022-07-12T08:46:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:50.247 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:50.247 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:50.247 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:50.247 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:50.249 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:50.249 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:50.249 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:50.249 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:50.255 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:50.255 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:46:50.255 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:46:50.255 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:46:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:46:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:46:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:46:50.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.257 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 12, 2022-07-12T08:46:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:46:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:46:21.563425+0000", 2022-07-12T08:46:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:50.260 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:50.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:50.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:50.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:50.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:50.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:50.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:50.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:50.261 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "62", 2022-07-12T08:46:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:50.272 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:46:50.272 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:50.272 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:46:50.272 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:46:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:46:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:46:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:46:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:46:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:46:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:46:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:46:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:46:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:46:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:46:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.289 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.289 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:46:50.289 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:46:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:46:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:46:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.291 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.291 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.291 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:50.292 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:46:50.292 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.292 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.292 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.293 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.293 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.293 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:46:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:46:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:46:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:46:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:46:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:46:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:46:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:46:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:46:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:46:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:46:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:46:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:46:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-12T08:46:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:46:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:46:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:46:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.318 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.318 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:46:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.320 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.320 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.322 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.322 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:46:50.323 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.323 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.324 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.324 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:46:50.325 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:46:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.330 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:46:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:50.332 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.332 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.333 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.334 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:46:50.334 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:46:50.335 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:46:50.335 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:46:50.336 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.336 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.337 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.337 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.338 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.338 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.339 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:50.339 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:46:50.340 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.340 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.342 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.342 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.343 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-12T08:46:50.343 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.344 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.344 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.345 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.346 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.346 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:46:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.348 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.348 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:46:50.349 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.349 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.350 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.350 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.351 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.351 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.352 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.352 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:46:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.354 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.354 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.355 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:46:50.355 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.355 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.356 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.356 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.358 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:46:50.358 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.359 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.359 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.360 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.360 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:46:50.360 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:46:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.365 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.365 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:46:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:46:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:46:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:46:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:46:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:46:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.373 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:46:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.376 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.376 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:46:50.377 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.378 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.378 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.379 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.379 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.380 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.380 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:46:50.381 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.381 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.381 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.382 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:46:50.382 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.383 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.383 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.384 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.384 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.384 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.385 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.385 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.386 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:46:50.386 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.387 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.387 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.387 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:46:50.388 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.389 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.389 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.390 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.390 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.390 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:46:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:46:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:46:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:46:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:46:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:46:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:46:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-12T08:46:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:46:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:46:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:46:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:46:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:46:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:46:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:46:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:46:50.413 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:46:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:46:50.416 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:46:50.416 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:46:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.418 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.419 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.419 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:46:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.421 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.421 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.422 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.422 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:50.424 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.424 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:46:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:46:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:46:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:46:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:46:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:46:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:46:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:46:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:46:50.440 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:46:50.440 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:46:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:46:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.442 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.442 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.443 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.443 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.444 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.444 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:46:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:50.447 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:50.447 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:50.448 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:50.448 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-12T08:46:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:50.450 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:50.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:46:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.454 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.455 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:46:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.457 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.457 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.459 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.459 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:46:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.471 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:46:50.471 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.473 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.473 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:50.474 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:46:50.476 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.476 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.477 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.477 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:46:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.480 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:50.480 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:50.482 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:46:50.482 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.483 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.483 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:46:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:50.485 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.486 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.486 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.487 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:46:50.487 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:46:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:46:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:46:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:46:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.494 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.494 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.496 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:46:50.496 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.497 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.497 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.499 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.499 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.500 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.500 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:46:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.502 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.502 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.503 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:50.503 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.505 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.505 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.506 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:46:50.506 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.507 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.508 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:50.508 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.508 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.509 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.509 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.510 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.510 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:46:50.511 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.512 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.512 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.513 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:50.513 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.514 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.514 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.515 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.515 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.516 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:46:50.516 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.517 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.517 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.518 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:50.518 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.519 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.519 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.519 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.520 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.520 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:46:50.521 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.522 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.522 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.523 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:50.524 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.525 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.525 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.526 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.527 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.527 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:46:50.527 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.528 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.528 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:50.529 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:50.529 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:50.530 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:50.530 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:50.530 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:50.531 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:50.531 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:46:50.532 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:50.532 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.532 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:50.533 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:46:50.533 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:46:50.534 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:46:50.535 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:46:50.535 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:46:50.536 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:46:50.536 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:46:50.537 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:46:50.537 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:46:50.538 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:46:50.538 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:46:50.539 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:46:50.539 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:46:50.540 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:50.540 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:50.541 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:46:50.541 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:46:50.542 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:46:50.542 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:46:50.543 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:46:50.543 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:46:50.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7528053a40 msgr2=0x7f7528055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7528053a40 0x7f7528055f00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f752c01a000 tx=0x7f752c014000).stop 2022-07-12T08:46:50.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 msgr2=0x7f7544098d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 0x7f7544098d20 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f7534001110 tx=0x7f7534007b20).stop 2022-07-12T08:46:50.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 shutdown_connections 2022-07-12T08:46:50.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f75440af4b0 0x7f7544143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7544060b50 0x7f7544098d20 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7528053a40 0x7f7528055f00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 --2- 172.21.15.61:0/598263665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f754405bdf0 0x7f75440b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 >> 172.21.15.61:0/598263665 conn(0x7f75440b92c0 msgr2=0x7f75440b7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:50.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 shutdown_connections 2022-07-12T08:46:50.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.160+0000 7f7548e00700 1 -- 172.21.15.61:0/598263665 wait complete. 2022-07-12T08:46:50.554 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:50.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.664+0000 7f897093e700 1 Processor -- start 2022-07-12T08:46:50.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.664+0000 7f897093e700 1 -- start start 2022-07-12T08:46:50.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c0b14e0 0x7f896c146480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896c0b3280 con 0x7f896c0b2ea0 2022-07-12T08:46:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896c0ad810 con 0x7f896c146970 2022-07-12T08:46:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0adab0 con 0x7f896c0b14e0 2022-07-12T08:46:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f896c0b2ea0 0x7f896c0adf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45016/0 (socket says 172.21.15.61:45016) 2022-07-12T08:46:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896a59c700 1 -- 172.21.15.61:0/3141947878 learned_addr learned my addr 172.21.15.61:0/3141947878 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f8969d9b700 1 --2- 172.21.15.61:0/3141947878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c0b14e0 0x7f896c146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889633655 0 0) 0x7f896c0b3280 con 0x7f896c0b2ea0 2022-07-12T08:46:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c003260 con 0x7f896c0b2ea0 2022-07-12T08:46:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917991299 0 0) 0x7f896c0ad810 con 0x7f896c146970 2022-07-12T08:46:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f896c0b3280 con 0x7f896c146970 2022-07-12T08:46:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f8969d9b700 1 -- 172.21.15.61:0/3141947878 >> v1:172.21.15.111:6789/0 conn(0x7f896c146970 legacy=0x7f896c0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f8969d9b700 1 -- 172.21.15.61:0/3141947878 >> v1:172.21.15.61:6789/0 conn(0x7f896c0b2ea0 legacy=0x7f896c0adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f8969d9b700 1 -- 172.21.15.61:0/3141947878 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c05f8e0 con 0x7f896c0b14e0 2022-07-12T08:46:50.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f8969d9b700 1 --2- 172.21.15.61:0/3141947878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c0b14e0 0x7f896c146480 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8954002f70 tx=0x7f8954006df0).ready entity=mon.2 client_cookie=df451a1be659559a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f895400d5c0 con 0x7f896c0b14e0 2022-07-12T08:46:50.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8954000e90 con 0x7f896c0b14e0 2022-07-12T08:46:50.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f896959a700 1 -- 172.21.15.61:0/3141947878 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f895400d910 con 0x7f896c0b14e0 2022-07-12T08:46:50.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- 172.21.15.61:0/3141947878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c0b14e0 msgr2=0x7f896c146480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 --2- 172.21.15.61:0/3141947878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c0b14e0 0x7f896c146480 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8954002f70 tx=0x7f8954006df0).stop 2022-07-12T08:46:50.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- 172.21.15.61:0/3141947878 shutdown_connections 2022-07-12T08:46:50.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 --2- 172.21.15.61:0/3141947878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c0b14e0 0x7f896c146480 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- 172.21.15.61:0/3141947878 >> 172.21.15.61:0/3141947878 conn(0x7f896c063ab0 msgr2=0x7f896c063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:50.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- 172.21.15.61:0/3141947878 shutdown_connections 2022-07-12T08:46:50.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.668+0000 7f897093e700 1 -- 172.21.15.61:0/3141947878 wait complete. 2022-07-12T08:46:50.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 Processor -- start 2022-07-12T08:46:50.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 -- start start 2022-07-12T08:46:50.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f896c0b14e0 0x7f896c0afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 0x7f896c152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c146970 0x7f896c152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0acd20 con 0x7f896c0b2ea0 2022-07-12T08:46:50.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0a9ae0 con 0x7f896c0b14e0 2022-07-12T08:46:50.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f897093e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0b0170 con 0x7f896c146970 2022-07-12T08:46:50.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 0x7f896c152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 0x7f896c152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56806/0 (socket says 172.21.15.61:56806) 2022-07-12T08:46:50.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 -- 172.21.15.61:0/3568861089 learned_addr learned my addr 172.21.15.61:0/3568861089 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:50.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f896a59c700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f896c0b14e0 0x7f896c0afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f896ad9d700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c146970 0x7f896c152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 -- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c146970 msgr2=0x7f896c152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c146970 0x7f896c152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 -- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f896c0b14e0 msgr2=0x7f896c0afc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f896c0b14e0 0x7f896c0afc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f8969d9b700 1 -- 172.21.15.61:0/3568861089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895400b180 con 0x7f896c0b2ea0 2022-07-12T08:46:50.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.672+0000 7f896a59c700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f896c0b14e0 0x7f896c0afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:50.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f8969d9b700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 0x7f896c152320 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f8954002f70 tx=0x7f8954014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8954016070 con 0x7f896c0b2ea0 2022-07-12T08:46:50.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540137d0 con 0x7f896c0b2ea0 2022-07-12T08:46:50.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896c064600 con 0x7f896c0b2ea0 2022-07-12T08:46:50.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8954013b30 con 0x7f896c0b2ea0 2022-07-12T08:46:50.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896c064b10 con 0x7f896c0b2ea0 2022-07-12T08:46:50.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896c057460 con 0x7f896c0b2ea0 2022-07-12T08:46:50.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8954025030 con 0x7f896c0b2ea0 2022-07-12T08:46:50.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f89637fe700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8950053ad0 0x7f8950055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f895406cdb0 con 0x7f896c0b2ea0 2022-07-12T08:46:50.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.676+0000 7f896a59c700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8950053ad0 0x7f8950055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.680+0000 7f896a59c700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8950053ad0 0x7f8950055f90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f895c00d000 tx=0x7f895c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.684+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8954043d10 con 0x7f896c0b2ea0 2022-07-12T08:46:50.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.848+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f896c1551a0 con 0x7f896c0b2ea0 2022-07-12T08:46:50.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.848+0000 7f89637fe700 1 -- 172.21.15.61:0/3568861089 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f8954023020 con 0x7f896c0b2ea0 2022-07-12T08:46:50.851 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:50.852 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":31,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:23.437118+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5979},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5979":{"gid":5979,"name":"a","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3698454378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3698454378},{"type":"v1","addr":"172.21.15.61:6833","nonce":3698454378}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:50.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8950053ad0 msgr2=0x7f8950055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8950053ad0 0x7f8950055f90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f895c00d000 tx=0x7f895c009040).stop 2022-07-12T08:46:50.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 msgr2=0x7f896c152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 0x7f896c152320 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 rx=0x7f8954002f70 tx=0x7f8954014000).stop 2022-07-12T08:46:50.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 shutdown_connections 2022-07-12T08:46:50.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f896c146970 0x7f896c152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f896c0b2ea0 0x7f896c152320 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8950053ad0 0x7f8950055f90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 --2- 172.21.15.61:0/3568861089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f896c0b14e0 0x7f896c0afc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 >> 172.21.15.61:0/3568861089 conn(0x7f896c063ab0 msgr2=0x7f896c142210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:50.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 shutdown_connections 2022-07-12T08:46:50.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.852+0000 7f897093e700 1 -- 172.21.15.61:0/3568861089 wait complete. 2022-07-12T08:46:50.868 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 31 2022-07-12T08:46:50.870 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:46:50.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.972+0000 7f66cffff700 1 Processor -- start 2022-07-12T08:46:50.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- start start 2022-07-12T08:46:50.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f66d00ab290 0x7f66d00ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d00abc70 con 0x7f66d00afa60 2022-07-12T08:46:50.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d005c720 con 0x7f66d005f140 2022-07-12T08:46:50.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d005f580 con 0x7f66d00ab290 2022-07-12T08:46:50.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f66d00afa60 0x7f66d0147530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45034/0 (socket says 172.21.15.61:45034) 2022-07-12T08:46:50.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cf7fe700 1 -- 172.21.15.61:0/1095646936 learned_addr learned my addr 172.21.15.61:0/1095646936 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66ceffd700 1 --2- 172.21.15.61:0/1095646936 >> v2:172.21.15.111:3300/0 conn(0x7f66d00ab290 0x7f66d00ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953622025 0 0) 0x7f66d00abc70 con 0x7f66d00afa60 2022-07-12T08:46:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b0003260 con 0x7f66d00afa60 2022-07-12T08:46:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011368346 0 0) 0x7f66d005c720 con 0x7f66d005f140 2022-07-12T08:46:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d00abc70 con 0x7f66d005f140 2022-07-12T08:46:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2546365653 0 0) 0x7f66b0003260 con 0x7f66d00afa60 2022-07-12T08:46:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66ceffd700 1 -- 172.21.15.61:0/1095646936 >> v1:172.21.15.111:6789/0 conn(0x7f66d005f140 legacy=0x7f66d005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66ceffd700 1 -- 172.21.15.61:0/1095646936 >> v1:172.21.15.61:6789/0 conn(0x7f66d00afa60 legacy=0x7f66d0147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66ceffd700 1 -- 172.21.15.61:0/1095646936 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d005c9c0 con 0x7f66d00ab290 2022-07-12T08:46:50.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66ceffd700 1 --2- 172.21.15.61:0/1095646936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d00ab290 0x7f66d00ab670 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f66c8001110 tx=0x7f66c00091d0).ready entity=mon.1 client_cookie=c242299002680b60 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c8001660 con 0x7f66d00ab290 2022-07-12T08:46:50.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c000f510 con 0x7f66d00ab290 2022-07-12T08:46:50.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cdffb700 1 -- 172.21.15.61:0/1095646936 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66c000f680 con 0x7f66d00ab290 2022-07-12T08:46:50.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- 172.21.15.61:0/1095646936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d00ab290 msgr2=0x7f66d00ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 --2- 172.21.15.61:0/1095646936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d00ab290 0x7f66d00ab670 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f66c8001110 tx=0x7f66c00091d0).stop 2022-07-12T08:46:50.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- 172.21.15.61:0/1095646936 shutdown_connections 2022-07-12T08:46:50.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 --2- 172.21.15.61:0/1095646936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d00ab290 0x7f66d00ab670 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- 172.21.15.61:0/1095646936 >> 172.21.15.61:0/1095646936 conn(0x7f66d00b1950 msgr2=0x7f66d00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:50.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- 172.21.15.61:0/1095646936 shutdown_connections 2022-07-12T08:46:50.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.976+0000 7f66cffff700 1 -- 172.21.15.61:0/1095646936 wait complete. 2022-07-12T08:46:50.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 Processor -- start 2022-07-12T08:46:50.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 -- start start 2022-07-12T08:46:50.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d005f140 0x7f66d0064b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66d00ab290 0x7f66d014b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 0x7f66d01563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d014a4d0 con 0x7f66d00afa60 2022-07-12T08:46:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d01476f0 con 0x7f66d005f140 2022-07-12T08:46:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d0066e30 con 0x7f66d00ab290 2022-07-12T08:46:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 0x7f66d01563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 0x7f66d01563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56822/0 (socket says 172.21.15.61:56822) 2022-07-12T08:46:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cf7fe700 1 -- 172.21.15.61:0/2717451079 learned_addr learned my addr 172.21.15.61:0/2717451079 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66d00ab290 0x7f66d014b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ceffd700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d005f140 0x7f66d0064b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 -- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d005f140 msgr2=0x7f66d0064b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d005f140 0x7f66d0064b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 -- 172.21.15.61:0/2717451079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 msgr2=0x7f66d01563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:50.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 0x7f66d01563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:50.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 -- 172.21.15.61:0/2717451079 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0009470 con 0x7f66d00ab290 2022-07-12T08:46:50.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cf7fe700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 0x7f66d01563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:50.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66ce7fc700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66d00ab290 0x7f66d014b970 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f66d014a2d0 tx=0x7f66b8007ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66b8012400 con 0x7f66d00ab290 2022-07-12T08:46:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b8012560 con 0x7f66d00ab290 2022-07-12T08:46:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66b80128c0 con 0x7f66d00ab290 2022-07-12T08:46:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d00670d0 con 0x7f66d00ab290 2022-07-12T08:46:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.980+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d0155db0 con 0x7f66d00ab290 2022-07-12T08:46:50.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.984+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d0057460 con 0x7f66d00ab290 2022-07-12T08:46:50.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.984+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f66b8019030 con 0x7f66d00ab290 2022-07-12T08:46:50.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.984+0000 7f66c7fff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66b4053a60 0x7f66b4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:50.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.984+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f66b806b1f0 con 0x7f66d00ab290 2022-07-12T08:46:50.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.984+0000 7f66ceffd700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66b4053a60 0x7f66b4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:50.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.984+0000 7f66ceffd700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66b4053a60 0x7f66b4055f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f66c001b000 tx=0x7f66c0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:51.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:50.992+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66b80420b0 con 0x7f66d00ab290 2022-07-12T08:46:51.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:51.156+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f66d0067720 con 0x7f66d00ab290 2022-07-12T08:46:52.158 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:46:52.156+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:46:52.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.168+0000 7f66c7fff700 1 -- 172.21.15.61:0/2717451079 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f66b80456d0 con 0x7f66d00ab290 2022-07-12T08:46:52.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66b4053a60 msgr2=0x7f66b4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66b4053a60 0x7f66b4055f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f66c001b000 tx=0x7f66c0016000).stop 2022-07-12T08:46:52.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66d00ab290 msgr2=0x7f66d014b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66d00ab290 0x7f66d014b970 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f66d014a2d0 tx=0x7f66b8007ad0).stop 2022-07-12T08:46:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 shutdown_connections 2022-07-12T08:46:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66d00afa60 0x7f66d01563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66d00ab290 0x7f66d014b970 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66b4053a60 0x7f66b4055f20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 --2- 172.21.15.61:0/2717451079 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66d005f140 0x7f66d0064b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 >> 172.21.15.61:0/2717451079 conn(0x7f66d00b1950 msgr2=0x7f66d00b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 shutdown_connections 2022-07-12T08:46:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.172+0000 7f66cffff700 1 -- 172.21.15.61:0/2717451079 wait complete. 2022-07-12T08:46:52.179 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:46:52.190 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:52.210 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:46:52.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b1e14700 1 Processor -- start 2022-07-12T08:46:52.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b1e14700 1 -- start start 2022-07-12T08:46:52.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b1e14700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f09ac05c8e0 0x7f09ac0b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b1e14700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac146110 con 0x7f09ac0605c0 2022-07-12T08:46:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b1e14700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac05ccc0 con 0x7f09ac1467e0 2022-07-12T08:46:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b1e14700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac05f350 con 0x7f09ac05c8e0 2022-07-12T08:46:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09abfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f09ac0605c0 0x7f09ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45058/0 (socket says 172.21.15.61:45058) 2022-07-12T08:46:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09abfff700 1 -- 172.21.15.61:0/4064968604 learned_addr learned my addr 172.21.15.61:0/4064968604 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:52.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09b0e12700 1 --2- 172.21.15.61:0/4064968604 >> v2:172.21.15.111:3300/0 conn(0x7f09ac05c8e0 0x7f09ac0b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.284+0000 7f09ab7fe700 1 -- 172.21.15.61:0/4064968604 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398156571 0 0) 0x7f09ac146110 con 0x7f09ac0605c0 2022-07-12T08:46:52.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09ab7fe700 1 -- 172.21.15.61:0/4064968604 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f098c003260 con 0x7f09ac0605c0 2022-07-12T08:46:52.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b0e12700 1 -- 172.21.15.61:0/4064968604 >> v1:172.21.15.111:6789/0 conn(0x7f09ac1467e0 legacy=0x7f09ac141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b0e12700 1 -- 172.21.15.61:0/4064968604 >> v1:172.21.15.61:6789/0 conn(0x7f09ac0605c0 legacy=0x7f09ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b0e12700 1 -- 172.21.15.61:0/4064968604 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac146340 con 0x7f09ac05c8e0 2022-07-12T08:46:52.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b0e12700 1 --2- 172.21.15.61:0/4064968604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac0b1410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f09ac05d3e0 tx=0x7f099c0091d0).ready entity=mon.1 client_cookie=e18232744f77aafb server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:52.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09ab7fe700 1 -- 172.21.15.61:0/4064968604 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f099c00b680 con 0x7f09ac05c8e0 2022-07-12T08:46:52.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09ab7fe700 1 -- 172.21.15.61:0/4064968604 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c00b7e0 con 0x7f09ac05c8e0 2022-07-12T08:46:52.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09ab7fe700 1 -- 172.21.15.61:0/4064968604 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f099c00b950 con 0x7f09ac05c8e0 2022-07-12T08:46:52.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- 172.21.15.61:0/4064968604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 msgr2=0x7f09ac0b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 --2- 172.21.15.61:0/4064968604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac0b1410 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f09ac05d3e0 tx=0x7f099c0091d0).stop 2022-07-12T08:46:52.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- 172.21.15.61:0/4064968604 shutdown_connections 2022-07-12T08:46:52.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 --2- 172.21.15.61:0/4064968604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac0b1410 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- 172.21.15.61:0/4064968604 >> 172.21.15.61:0/4064968604 conn(0x7f09ac063a60 msgr2=0x7f09ac063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:52.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- 172.21.15.61:0/4064968604 shutdown_connections 2022-07-12T08:46:52.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- 172.21.15.61:0/4064968604 wait complete. 2022-07-12T08:46:52.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 Processor -- start 2022-07-12T08:46:52.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- start start 2022-07-12T08:46:52.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac1441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ac0605c0 0x7f09ac152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 0x7f09ac114f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac13f870 con 0x7f09ac1467e0 2022-07-12T08:46:52.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0b0010 con 0x7f09ac05c8e0 2022-07-12T08:46:52.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1e14700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0ae930 con 0x7f09ac0605c0 2022-07-12T08:46:52.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1613700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 0x7f09ac114f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b0e12700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac1441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09abfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ac0605c0 0x7f09ac152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1613700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 0x7f09ac114f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56848/0 (socket says 172.21.15.61:56848) 2022-07-12T08:46:52.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.288+0000 7f09b1613700 1 -- 172.21.15.61:0/1536939329 learned_addr learned my addr 172.21.15.61:0/1536939329 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:52.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1613700 1 -- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ac0605c0 msgr2=0x7f09ac152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1613700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ac0605c0 0x7f09ac152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1613700 1 -- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 msgr2=0x7f09ac1441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1613700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac1441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1613700 1 -- 172.21.15.61:0/1536939329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c009470 con 0x7f09ac1467e0 2022-07-12T08:46:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1613700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 0x7f09ac114f10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f09ac05d500 tx=0x7f09a4007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a4007030 con 0x7f09ac1467e0 2022-07-12T08:46:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4007190 con 0x7f09ac1467e0 2022-07-12T08:46:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac0aebd0 con 0x7f09ac1467e0 2022-07-12T08:46:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a4006510 con 0x7f09ac1467e0 2022-07-12T08:46:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac152a10 con 0x7f09ac1467e0 2022-07-12T08:46:52.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac0aed60 con 0x7f09ac1467e0 2022-07-12T08:46:52.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f09a4010030 con 0x7f09ac1467e0 2022-07-12T08:46:52.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09a9ffb700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0990053a60 0x7f0990055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f09a406b9a0 con 0x7f09ac1467e0 2022-07-12T08:46:52.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.292+0000 7f09b0e12700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0990053a60 0x7f0990055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.296+0000 7f09b0e12700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0990053a60 0x7f0990055f20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f099c01b000 tx=0x7f099c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:52.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.300+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f09a40428b0 con 0x7f09ac1467e0 2022-07-12T08:46:52.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.460+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09ac057460 con 0x7f09ac1467e0 2022-07-12T08:46:52.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.460+0000 7f09a9ffb700 1 -- 172.21.15.61:0/1536939329 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f09a4045ed0 con 0x7f09ac1467e0 2022-07-12T08:46:52.464 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:52.465 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":32,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28}],"filesystems":[{"mdsmap":{"epoch":32,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:46:22.431962+0000","modified":"2022-07-12T08:46:52.159809+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-12T08:46:52.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0990053a60 msgr2=0x7f0990055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0990053a60 0x7f0990055f20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f099c01b000 tx=0x7f099c016000).stop 2022-07-12T08:46:52.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 msgr2=0x7f09ac114f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 0x7f09ac114f10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f09ac05d500 tx=0x7f09a4007c20).stop 2022-07-12T08:46:52.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 shutdown_connections 2022-07-12T08:46:52.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ac1467e0 0x7f09ac114f10 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ac0605c0 0x7f09ac152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0990053a60 0x7f0990055f20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 --2- 172.21.15.61:0/1536939329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ac05c8e0 0x7f09ac1441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 >> 172.21.15.61:0/1536939329 conn(0x7f09ac063a60 msgr2=0x7f09ac13fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 shutdown_connections 2022-07-12T08:46:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.464+0000 7f09b1e14700 1 -- 172.21.15.61:0/1536939329 wait complete. 2022-07-12T08:46:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 32 2022-07-12T08:46:52.482 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:46:52.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.584+0000 7f7d87ece700 1 Processor -- start 2022-07-12T08:46:52.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.584+0000 7f7d87ece700 1 -- start start 2022-07-12T08:46:52.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d800af050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7d800ae440 0x7f7d800b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d801439b0 con 0x7f7d800ae060 2022-07-12T08:46:52.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d800aeb80 con 0x7f7d800ae440 2022-07-12T08:46:52.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80143c10 con 0x7f7d800afe40 2022-07-12T08:46:52.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7d800ae440 0x7f7d800b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7d800ae440 0x7f7d800b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56856/0 (socket says 172.21.15.61:56856) 2022-07-12T08:46:52.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 -- 172.21.15.61:0/3345470393 learned_addr learned my addr 172.21.15.61:0/3345470393 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:52.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d85469700 1 --2- 172.21.15.61:0/3345470393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d800af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 -- 172.21.15.61:0/3345470393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 msgr2=0x7f7d800af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:52.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 --2- 172.21.15.61:0/3345470393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d800af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 -- 172.21.15.61:0/3345470393 >> v1:172.21.15.111:6789/0 conn(0x7f7d800ae060 legacy=0x7f7d800ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 -- 172.21.15.61:0/3345470393 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800ae8a0 con 0x7f7d800ae440 2022-07-12T08:46:52.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d8646b700 1 --2- 172.21.15.61:0/3345470393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae440 0x7f7d800b0750 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f7d78002f70 tx=0x7f7d78006eb0).ready entity=mon.0 client_cookie=e1acfd88c1c1dc94 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:52.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d84c68700 1 -- 172.21.15.61:0/3345470393 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d7800d570 con 0x7f7d800ae440 2022-07-12T08:46:52.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d84c68700 1 -- 172.21.15.61:0/3345470393 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d78000e90 con 0x7f7d800ae440 2022-07-12T08:46:52.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d84c68700 1 -- 172.21.15.61:0/3345470393 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d7800d900 con 0x7f7d800ae440 2022-07-12T08:46:52.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- 172.21.15.61:0/3345470393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae440 msgr2=0x7f7d800b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 --2- 172.21.15.61:0/3345470393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae440 0x7f7d800b0750 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f7d78002f70 tx=0x7f7d78006eb0).stop 2022-07-12T08:46:52.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- 172.21.15.61:0/3345470393 shutdown_connections 2022-07-12T08:46:52.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 --2- 172.21.15.61:0/3345470393 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae440 0x7f7d800b0750 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 --2- 172.21.15.61:0/3345470393 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d800af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- 172.21.15.61:0/3345470393 >> 172.21.15.61:0/3345470393 conn(0x7f7d800b5290 msgr2=0x7f7d800b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- 172.21.15.61:0/3345470393 shutdown_connections 2022-07-12T08:46:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 -- 172.21.15.61:0/3345470393 wait complete. 2022-07-12T08:46:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.588+0000 7f7d87ece700 1 Processor -- start 2022-07-12T08:46:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 -- start start 2022-07-12T08:46:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae060 0x7f7d80145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d800ae440 0x7f7d80155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d80155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80067b30 con 0x7f7d800ae060 2022-07-12T08:46:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d800667d0 con 0x7f7d800ae440 2022-07-12T08:46:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d80065750 con 0x7f7d800afe40 2022-07-12T08:46:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d8646b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d80155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae060 0x7f7d80145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85469700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d800ae440 0x7f7d80155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d8646b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d80155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39056/0 (socket says 172.21.15.61:39056) 2022-07-12T08:46:52.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d8646b700 1 -- 172.21.15.61:0/4048115969 learned_addr learned my addr 172.21.15.61:0/4048115969 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:52.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 -- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 msgr2=0x7f7d80155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d80155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 -- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d800ae440 msgr2=0x7f7d80155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:52.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d800ae440 0x7f7d80155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:52.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 -- 172.21.15.61:0/4048115969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7800b1e0 con 0x7f7d800ae060 2022-07-12T08:46:52.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85469700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d800ae440 0x7f7d80155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:52.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d8646b700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d80155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:52.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d85c6a700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae060 0x7f7d80145cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f7d80143f90 tx=0x7f7d74007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d7400cdb0 con 0x7f7d800ae060 2022-07-12T08:46:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d74007030 con 0x7f7d800ae060 2022-07-12T08:46:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d74007390 con 0x7f7d800ae060 2022-07-12T08:46:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d800659f0 con 0x7f7d800ae060 2022-07-12T08:46:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.592+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d80140640 con 0x7f7d800ae060 2022-07-12T08:46:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.596+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7d74006510 con 0x7f7d800ae060 2022-07-12T08:46:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.596+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d80057460 con 0x7f7d800ae060 2022-07-12T08:46:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.596+0000 7f7d7effd700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d60053a40 0x7f7d60055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.596+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7d7406b860 con 0x7f7d800ae060 2022-07-12T08:46:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.596+0000 7f7d85469700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d60053a40 0x7f7d60055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.596+0000 7f7d85469700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d60053a40 0x7f7d60055f00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f7d7000b000 tx=0x7f7d700091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.604+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d74042580 con 0x7f7d800ae060 2022-07-12T08:46:52.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:52.768+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7d800447a0 con 0x7f7d800ae060 2022-07-12T08:46:53.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.176+0000 7f7d7effd700 1 -- 172.21.15.61:0/4048115969 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7d74045ba0 con 0x7f7d800ae060 2022-07-12T08:46:53.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.180+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d60053a40 msgr2=0x7f7d60055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.180+0000 7f7d87ece700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d60053a40 0x7f7d60055f00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f7d7000b000 tx=0x7f7d700091d0).stop 2022-07-12T08:46:53.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.180+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae060 msgr2=0x7f7d80145cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.180+0000 7f7d87ece700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae060 0x7f7d80145cb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f7d80143f90 tx=0x7f7d74007b50).stop 2022-07-12T08:46:53.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 shutdown_connections 2022-07-12T08:46:53.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d800afe40 0x7f7d80155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d60053a40 0x7f7d60055f00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d800ae440 0x7f7d80155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 --2- 172.21.15.61:0/4048115969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d800ae060 0x7f7d80145cb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 >> 172.21.15.61:0/4048115969 conn(0x7f7d800b5290 msgr2=0x7f7d80141c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:53.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 shutdown_connections 2022-07-12T08:46:53.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.184+0000 7f7d87ece700 1 -- 172.21.15.61:0/4048115969 wait complete. 2022-07-12T08:46:53.201 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:46:53.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.320+0000 7fb9e78e3700 1 Processor -- start 2022-07-12T08:46:53.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.320+0000 7fb9e78e3700 1 -- start start 2022-07-12T08:46:53.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e78e3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 0x7fb9e014dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e78e3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00ba290 con 0x7fb9e00b3180 2022-07-12T08:46:53.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e78e3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00b4010 con 0x7fb9e00b9eb0 2022-07-12T08:46:53.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e78e3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00b1410 con 0x7fb9e00b4840 2022-07-12T08:46:53.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e4e7e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb9e00b3180 0x7fb9e00b2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45088/0 (socket says 172.21.15.61:45088) 2022-07-12T08:46:53.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e4e7e700 1 -- 172.21.15.61:0/1368259078 learned_addr learned my addr 172.21.15.61:0/1368259078 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:53.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 0x7fb9e014dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045879583 0 0) 0x7fb9e00ba290 con 0x7fb9e00b3180 2022-07-12T08:46:53.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c4003260 con 0x7fb9e00b3180 2022-07-12T08:46:53.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296990267 0 0) 0x7fb9e00b4010 con 0x7fb9e00b9eb0 2022-07-12T08:46:53.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e00ba290 con 0x7fb9e00b9eb0 2022-07-12T08:46:53.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434510383 0 0) 0x7fb9c4003260 con 0x7fb9e00b3180 2022-07-12T08:46:53.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 msgr2=0x7fb9e014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:53.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 0x7fb9e014dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 >> v1:172.21.15.111:6789/0 conn(0x7fb9e00b9eb0 legacy=0x7fb9e00b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e00b42b0 con 0x7fb9e00b3180 2022-07-12T08:46:53.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fb9dc001660 con 0x7fb9e00b3180 2022-07-12T08:46:53.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 >> v1:172.21.15.61:6789/0 conn(0x7fb9e00b3180 legacy=0x7fb9e00b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4003910 0x7fb9c4003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 0x7fb9c4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9c40049f0 0x7fb9c4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00b1410 con 0x7fb9c40049f0 2022-07-12T08:46:53.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c400b780 con 0x7fb9c4003910 2022-07-12T08:46:53.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c400b940 con 0x7fb9e00b4840 2022-07-12T08:46:53.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9c40049f0 0x7fb9c4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e567f700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4003910 0x7fb9c4003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e4e7e700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 0x7fb9c4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 msgr2=0x7fb9c4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:53.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00b4840 0x7fb9c4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4003910 msgr2=0x7fb9c4003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:53.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4003910 0x7fb9c4003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1368259078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c4003260 con 0x7fb9c40049f0 2022-07-12T08:46:53.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e567f700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9c4003910 0x7fb9c4003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:53.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9c40049f0 0x7fb9c4004e70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fb9d800c000 tx=0x7fb9d80091d0).ready entity=mon.0 client_cookie=a6a1077891618a71 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:53.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9d800e6e0 con 0x7fb9c40049f0 2022-07-12T08:46:53.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d800e840 con 0x7fb9c40049f0 2022-07-12T08:46:53.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.324+0000 7fb9cbfff700 1 -- 172.21.15.61:0/1368259078 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9d800eba0 con 0x7fb9c40049f0 2022-07-12T08:46:53.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1368259078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9c40049f0 msgr2=0x7fb9c4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1368259078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9c40049f0 0x7fb9c4004e70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fb9d800c000 tx=0x7fb9d80091d0).stop 2022-07-12T08:46:53.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1368259078 reap_dead start 2022-07-12T08:46:53.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1368259078 shutdown_connections 2022-07-12T08:46:53.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1368259078 >> 172.21.15.61:0/1368259078 conn(0x7fb9e00678c0 msgr2=0x7fb9e00b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:53.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1368259078 shutdown_connections 2022-07-12T08:46:53.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1368259078 wait complete. 2022-07-12T08:46:53.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 Processor -- start 2022-07-12T08:46:53.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- start start 2022-07-12T08:46:53.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00ae840 0x7fb9e01474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e01401c0 0x7fb9e01469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 0x7fb9e0146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e0145970 con 0x7fb9e00af5e0 2022-07-12T08:46:53.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00aec20 con 0x7fb9e01401c0 2022-07-12T08:46:53.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e78e3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e01479b0 con 0x7fb9e00ae840 2022-07-12T08:46:53.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e5e80700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 0x7fb9e0146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e567f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00ae840 0x7fb9e01474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e4e7e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e01401c0 0x7fb9e01469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e5e80700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 0x7fb9e0146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56884/0 (socket says 172.21.15.61:56884) 2022-07-12T08:46:53.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.328+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1980873023 learned_addr learned my addr 172.21.15.61:0/1980873023 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:53.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00ae840 msgr2=0x7fb9e01474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00ae840 0x7fb9e01474c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e01401c0 msgr2=0x7fb9e01469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:53.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e01401c0 0x7fb9e01469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:53.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e5e80700 1 -- 172.21.15.61:0/1980873023 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e01406d0 con 0x7fb9e00af5e0 2022-07-12T08:46:53.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e4e7e700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e01401c0 0x7fb9e01469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:53.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e5e80700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 0x7fb9e0146ee0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fb9e00aedb0 tx=0x7fb9d801a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:53.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9d800e6e0 con 0x7fb9e00af5e0 2022-07-12T08:46:53.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d800eed0 con 0x7fb9e00af5e0 2022-07-12T08:46:53.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9d801b850 con 0x7fb9e00af5e0 2022-07-12T08:46:53.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e0140970 con 0x7fb9e00af5e0 2022-07-12T08:46:53.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e00ae1b0 con 0x7fb9e00af5e0 2022-07-12T08:46:53.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e0057460 con 0x7fb9e00af5e0 2022-07-12T08:46:53.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.332+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb9d802d070 con 0x7fb9e00af5e0 2022-07-12T08:46:53.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.336+0000 7fb9d2ffd700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9bc0539b0 0x7fb9bc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:53.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.336+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb9d806db90 con 0x7fb9e00af5e0 2022-07-12T08:46:53.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.336+0000 7fb9e567f700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9bc0539b0 0x7fb9bc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:53.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.336+0000 7fb9e567f700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9bc0539b0 0x7fb9bc055e70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fb9d4003000 tx=0x7fb9d4007a50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:53.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.340+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb9d8044aa0 con 0x7fb9e00af5e0 2022-07-12T08:46:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:53.468+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb9e0066150 con 0x7fb9e00af5e0 2022-07-12T08:46:54.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.184+0000 7fb9d2ffd700 1 -- 172.21.15.61:0/1980873023 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v64) v1 ==== 185+0+0 (secure 0 0 0) 0x7fb9d8016070 con 0x7fb9e00af5e0 2022-07-12T08:46:54.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9bc0539b0 msgr2=0x7fb9bc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:54.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9bc0539b0 0x7fb9bc055e70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fb9d4003000 tx=0x7fb9d4007a50).stop 2022-07-12T08:46:54.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 msgr2=0x7fb9e0146ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:54.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 0x7fb9e0146ee0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fb9e00aedb0 tx=0x7fb9d801a000).stop 2022-07-12T08:46:54.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 shutdown_connections 2022-07-12T08:46:54.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9e00af5e0 0x7fb9e0146ee0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb9bc0539b0 0x7fb9bc055e70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb9e01401c0 0x7fb9e01469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 --2- 172.21.15.61:0/1980873023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb9e00ae840 0x7fb9e01474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 >> 172.21.15.61:0/1980873023 conn(0x7fb9e00678c0 msgr2=0x7fb9e0142960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:54.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 shutdown_connections 2022-07-12T08:46:54.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.188+0000 7fb9e78e3700 1 -- 172.21.15.61:0/1980873023 wait complete. 2022-07-12T08:46:54.200 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:46:54.208 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:46:54.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.332+0000 7fb24b2f9700 1 Processor -- start 2022-07-12T08:46:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.332+0000 7fb24b2f9700 1 -- start start 2022-07-12T08:46:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440ab670 0x7fb24405f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb2440afa60 0x7fb2440afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2440abc70 con 0x7fb2440ab290 2022-07-12T08:46:54.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2440aba90 con 0x7fb2440afa60 2022-07-12T08:46:54.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24405c720 con 0x7fb2440ab670 2022-07-12T08:46:54.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24a2f7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb2440ab290 0x7fb24405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45106/0 (socket says 172.21.15.61:45106) 2022-07-12T08:46:54.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24a2f7700 1 -- 172.21.15.61:0/741931449 learned_addr learned my addr 172.21.15.61:0/741931449 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:54.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24aaf8700 1 --2- 172.21.15.61:0/741931449 >> v2:172.21.15.111:3300/0 conn(0x7fb2440afa60 0x7fb2440afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:54.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb249af6700 1 --2- 172.21.15.61:0/741931449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440ab670 0x7fb24405f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:54.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb2492f5700 1 -- 172.21.15.61:0/741931449 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369023766 0 0) 0x7fb2440abc70 con 0x7fb2440ab290 2022-07-12T08:46:54.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb2492f5700 1 -- 172.21.15.61:0/741931449 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb230003260 con 0x7fb2440ab290 2022-07-12T08:46:54.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb249af6700 1 -- 172.21.15.61:0/741931449 >> v2:172.21.15.111:3300/0 conn(0x7fb2440afa60 msgr2=0x7fb2440afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:54.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb249af6700 1 --2- 172.21.15.61:0/741931449 >> v2:172.21.15.111:3300/0 conn(0x7fb2440afa60 0x7fb2440afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb2492f5700 1 -- 172.21.15.61:0/741931449 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3908683721 0 0) 0x7fb230003260 con 0x7fb2440ab290 2022-07-12T08:46:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb249af6700 1 -- 172.21.15.61:0/741931449 >> v1:172.21.15.61:6789/0 conn(0x7fb2440ab290 legacy=0x7fb24405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb249af6700 1 -- 172.21.15.61:0/741931449 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24405c9c0 con 0x7fb2440ab670 2022-07-12T08:46:54.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24aaf8700 1 --2- 172.21.15.61:0/741931449 >> v2:172.21.15.111:3300/0 conn(0x7fb2440afa60 0x7fb2440afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:54.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb249af6700 1 --2- 172.21.15.61:0/741931449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440ab670 0x7fb24405f140 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb238001800 tx=0x7fb2340091d0).ready entity=mon.2 client_cookie=a9f64af6299048a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:54.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb2492f5700 1 -- 172.21.15.61:0/741931449 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb238001660 con 0x7fb2440ab670 2022-07-12T08:46:54.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb2492f5700 1 -- 172.21.15.61:0/741931449 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23400f510 con 0x7fb2440ab670 2022-07-12T08:46:54.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb2492f5700 1 -- 172.21.15.61:0/741931449 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb23400f680 con 0x7fb2440ab670 2022-07-12T08:46:54.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- 172.21.15.61:0/741931449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440ab670 msgr2=0x7fb24405f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:54.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/741931449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440ab670 0x7fb24405f140 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fb238001800 tx=0x7fb2340091d0).stop 2022-07-12T08:46:54.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- 172.21.15.61:0/741931449 shutdown_connections 2022-07-12T08:46:54.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/741931449 >> v2:172.21.15.111:3300/0 conn(0x7fb2440afa60 0x7fb2440afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/741931449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440ab670 0x7fb24405f140 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- 172.21.15.61:0/741931449 >> 172.21.15.61:0/741931449 conn(0x7fb2440b1950 msgr2=0x7fb2440b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- 172.21.15.61:0/741931449 shutdown_connections 2022-07-12T08:46:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.336+0000 7fb24b2f9700 1 -- 172.21.15.61:0/741931449 wait complete. 2022-07-12T08:46:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 Processor -- start 2022-07-12T08:46:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 -- start start 2022-07-12T08:46:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 0x7fb244063ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2440ab670 0x7fb244091b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440afa60 0x7fb244092040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb244067780 con 0x7fb2440ab290 2022-07-12T08:46:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440663f0 con 0x7fb2440ab670 2022-07-12T08:46:54.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb244066690 con 0x7fb2440afa60 2022-07-12T08:46:54.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 0x7fb244063ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:54.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 0x7fb244063ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56894/0 (socket says 172.21.15.61:56894) 2022-07-12T08:46:54.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 -- 172.21.15.61:0/1600742203 learned_addr learned my addr 172.21.15.61:0/1600742203 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:54.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24aaf8700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440afa60 0x7fb244092040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:54.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb249af6700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2440ab670 0x7fb244091b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:54.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 -- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440afa60 msgr2=0x7fb244092040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:54.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440afa60 0x7fb244092040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 -- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2440ab670 msgr2=0x7fb244091b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:54.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2440ab670 0x7fb244091b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 -- 172.21.15.61:0/1600742203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c00b180 con 0x7fb2440ab290 2022-07-12T08:46:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24a2f7700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 0x7fb244063ba0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fb2380018c0 tx=0x7fb238001b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb238007170 con 0x7fb2440ab290 2022-07-12T08:46:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2380072d0 con 0x7fb2440ab290 2022-07-12T08:46:54.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234009470 con 0x7fb2440ab290 2022-07-12T08:46:54.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb238004920 con 0x7fb2440ab290 2022-07-12T08:46:54.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.340+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24408f690 con 0x7fb2440ab290 2022-07-12T08:46:54.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.344+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb244066850 con 0x7fb2440ab290 2022-07-12T08:46:54.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.344+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb23801b070 con 0x7fb2440ab290 2022-07-12T08:46:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.344+0000 7fb2437fe700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb224053a40 0x7fb224055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.344+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fb23806b590 con 0x7fb2440ab290 2022-07-12T08:46:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.344+0000 7fb249af6700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb224053a40 0x7fb224055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:54.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.344+0000 7fb249af6700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb224053a40 0x7fb224055f00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb23401b000 tx=0x7fb234016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:54.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.348+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb238042650 con 0x7fb2440ab290 2022-07-12T08:46:54.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:54.476+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb244057460 con 0x7fb2440ab290 2022-07-12T08:46:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.196+0000 7fb2437fe700 1 -- 172.21.15.61:0/1600742203 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v65) v1 ==== 173+0+0 (secure 0 0 0) 0x7fb238045c70 con 0x7fb2440ab290 2022-07-12T08:46:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb224053a40 msgr2=0x7fb224055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb224053a40 0x7fb224055f00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fb23401b000 tx=0x7fb234016000).stop 2022-07-12T08:46:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 msgr2=0x7fb244063ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 0x7fb244063ba0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7fb2380018c0 tx=0x7fb238001b60).stop 2022-07-12T08:46:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 shutdown_connections 2022-07-12T08:46:55.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb2440afa60 0x7fb244092040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb224053a40 0x7fb224055f00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb2440ab670 0x7fb244091b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 --2- 172.21.15.61:0/1600742203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb2440ab290 0x7fb244063ba0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 >> 172.21.15.61:0/1600742203 conn(0x7fb2440b1950 msgr2=0x7fb2440b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:55.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.200+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 shutdown_connections 2022-07-12T08:46:55.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.204+0000 7fb24b2f9700 1 -- 172.21.15.61:0/1600742203 wait complete. 2022-07-12T08:46:55.214 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:46:55.228 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:46:55.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.340+0000 7ff80b4b3700 1 Processor -- start 2022-07-12T08:46:55.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.340+0000 7ff80b4b3700 1 -- start start 2022-07-12T08:46:55.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.340+0000 7ff80b4b3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8040af490 0x7ff804146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.340+0000 7ff80b4b3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff80405d100 0x7ff8041431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804146200 con 0x7ff80405cd20 2022-07-12T08:46:55.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8041464a0 con 0x7ff8040af490 2022-07-12T08:46:55.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff804146740 con 0x7ff80405d100 2022-07-12T08:46:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8040af490 0x7ff804146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8040af490 0x7ff804146be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56902/0 (socket says 172.21.15.61:56902) 2022-07-12T08:46:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 -- 172.21.15.61:0/1437154576 learned_addr learned my addr 172.21.15.61:0/1437154576 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:55.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80acb2700 1 --2- 172.21.15.61:0/1437154576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff80405d100 0x7ff8041431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 -- 172.21.15.61:0/1437154576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff80405d100 msgr2=0x7ff8041431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 --2- 172.21.15.61:0/1437154576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff80405d100 0x7ff8041431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 -- 172.21.15.61:0/1437154576 >> v1:172.21.15.111:6789/0 conn(0x7ff80405cd20 legacy=0x7ff80405f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 -- 172.21.15.61:0/1437154576 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804060870 con 0x7ff8040af490 2022-07-12T08:46:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff809cb0700 1 --2- 172.21.15.61:0/1437154576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8040af490 0x7ff804146be0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7ff800002f70 tx=0x7ff800006eb0).ready entity=mon.0 client_cookie=f9893eb6dcc797ff server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:55.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff8094af700 1 -- 172.21.15.61:0/1437154576 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff80000d570 con 0x7ff8040af490 2022-07-12T08:46:55.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff8094af700 1 -- 172.21.15.61:0/1437154576 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800000e90 con 0x7ff8040af490 2022-07-12T08:46:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff8094af700 1 -- 172.21.15.61:0/1437154576 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff80000d900 con 0x7ff8040af490 2022-07-12T08:46:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- 172.21.15.61:0/1437154576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8040af490 msgr2=0x7ff804146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/1437154576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8040af490 0x7ff804146be0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7ff800002f70 tx=0x7ff800006eb0).stop 2022-07-12T08:46:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- 172.21.15.61:0/1437154576 shutdown_connections 2022-07-12T08:46:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/1437154576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff80405d100 0x7ff8041431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/1437154576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8040af490 0x7ff804146be0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- 172.21.15.61:0/1437154576 >> 172.21.15.61:0/1437154576 conn(0x7ff8040b9e20 msgr2=0x7ff8040b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- 172.21.15.61:0/1437154576 shutdown_connections 2022-07-12T08:46:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- 172.21.15.61:0/1437154576 wait complete. 2022-07-12T08:46:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 Processor -- start 2022-07-12T08:46:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 -- start start 2022-07-12T08:46:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.344+0000 7ff80b4b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80405cd20 0x7ff804151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 0x7ff804144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8040af490 0x7ff804152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80409e4c0 con 0x7ff80405d100 2022-07-12T08:46:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804145b70 con 0x7ff80405cd20 2022-07-12T08:46:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff804145680 con 0x7ff8040af490 2022-07-12T08:46:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 0x7ff804144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 0x7ff804144b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56910/0 (socket says 172.21.15.61:56910) 2022-07-12T08:46:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 -- 172.21.15.61:0/2854731162 learned_addr learned my addr 172.21.15.61:0/2854731162 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80acb2700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8040af490 0x7ff804152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80a4b1700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80405cd20 0x7ff804151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 -- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8040af490 msgr2=0x7ff804152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8040af490 0x7ff804152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 -- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80405cd20 msgr2=0x7ff804151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80405cd20 0x7ff804151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 -- 172.21.15.61:0/2854731162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80000b1e0 con 0x7ff80405d100 2022-07-12T08:46:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff809cb0700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 0x7ff804144b10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7ff800002f70 tx=0x7ff800015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff800011610 con 0x7ff80405d100 2022-07-12T08:46:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff800011770 con 0x7ff80405d100 2022-07-12T08:46:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8041458d0 con 0x7ff80405d100 2022-07-12T08:46:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff800011ad0 con 0x7ff80405d100 2022-07-12T08:46:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80409f230 con 0x7ff80405d100 2022-07-12T08:46:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff800025030 con 0x7ff80405d100 2022-07-12T08:46:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.348+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff804057460 con 0x7ff80405d100 2022-07-12T08:46:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.352+0000 7ff7f37fe700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff7e4053ad0 0x7ff7e4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.352+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff80006ccf0 con 0x7ff80405d100 2022-07-12T08:46:55.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.352+0000 7ff80a4b1700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff7e4053ad0 0x7ff7e4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.352+0000 7ff80a4b1700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff7e4053ad0 0x7ff7e4055f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7ff7f400b000 tx=0x7ff7f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.356+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff800043e90 con 0x7ff80405d100 2022-07-12T08:46:55.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.492+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff8040b04a0 con 0x7ff80405d100 2022-07-12T08:46:55.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.496+0000 7ff7f37fe700 1 -- 172.21.15.61:0/2854731162 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff8000230a0 con 0x7ff80405d100 2022-07-12T08:46:55.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff7e4053ad0 msgr2=0x7ff7e4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff7e4053ad0 0x7ff7e4055f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7ff7f400b000 tx=0x7ff7f4009040).stop 2022-07-12T08:46:55.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 msgr2=0x7ff804144b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 0x7ff804144b10 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7ff800002f70 tx=0x7ff800015000).stop 2022-07-12T08:46:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 shutdown_connections 2022-07-12T08:46:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8040af490 0x7ff804152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff80405d100 0x7ff804144b10 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff7e4053ad0 0x7ff7e4055f90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 --2- 172.21.15.61:0/2854731162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff80405cd20 0x7ff804151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 >> 172.21.15.61:0/2854731162 conn(0x7ff8040b9e20 msgr2=0x7ff8040b8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:55.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 shutdown_connections 2022-07-12T08:46:55.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.500+0000 7ff80b4b3700 1 -- 172.21.15.61:0/2854731162 wait complete. 2022-07-12T08:46:55.517 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-12T08:46:55.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 Processor -- start 2022-07-12T08:46:55.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 -- start start 2022-07-12T08:46:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d740ab290 0x7f1d740ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1d7405f140 0x7f1d7405ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d740abc70 con 0x7f1d740afa60 2022-07-12T08:46:55.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7405f580 con 0x7f1d740ab290 2022-07-12T08:46:55.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d79638700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7405c720 con 0x7f1d7405f140 2022-07-12T08:46:55.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d73fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d740ab290 0x7f1d740ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.620+0000 7f1d73fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d740ab290 0x7f1d740ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56916/0 (socket says 172.21.15.61:56916) 2022-07-12T08:46:55.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d73fff700 1 -- 172.21.15.61:0/1831066037 learned_addr learned my addr 172.21.15.61:0/1831066037 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:55.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d737fe700 1 --2- 172.21.15.61:0/1831066037 >> v2:172.21.15.111:3300/0 conn(0x7f1d7405f140 0x7f1d7405ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d73fff700 1 -- 172.21.15.61:0/1831066037 >> v2:172.21.15.111:3300/0 conn(0x7f1d7405f140 msgr2=0x7f1d7405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:55.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d73fff700 1 --2- 172.21.15.61:0/1831066037 >> v2:172.21.15.111:3300/0 conn(0x7f1d7405f140 0x7f1d7405ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d73fff700 1 -- 172.21.15.61:0/1831066037 >> v1:172.21.15.111:6789/0 conn(0x7f1d740afa60 legacy=0x7f1d7414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d73fff700 1 -- 172.21.15.61:0/1831066037 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7405c9c0 con 0x7f1d740ab290 2022-07-12T08:46:55.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d73fff700 1 --2- 172.21.15.61:0/1831066037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740ab290 0x7f1d740ab670 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f1d6c0032f0 tx=0x7f1d6c009290).ready entity=mon.0 client_cookie=1e467ded52d14fdc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:55.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d72ffd700 1 -- 172.21.15.61:0/1831066037 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6c00b870 con 0x7f1d740ab290 2022-07-12T08:46:55.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d72ffd700 1 -- 172.21.15.61:0/1831066037 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c00b9d0 con 0x7f1d740ab290 2022-07-12T08:46:55.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d72ffd700 1 -- 172.21.15.61:0/1831066037 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6c00bb40 con 0x7f1d740ab290 2022-07-12T08:46:55.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 -- 172.21.15.61:0/1831066037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740ab290 msgr2=0x7f1d740ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 --2- 172.21.15.61:0/1831066037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740ab290 0x7f1d740ab670 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f1d6c0032f0 tx=0x7f1d6c009290).stop 2022-07-12T08:46:55.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 -- 172.21.15.61:0/1831066037 shutdown_connections 2022-07-12T08:46:55.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 --2- 172.21.15.61:0/1831066037 >> v2:172.21.15.111:3300/0 conn(0x7f1d7405f140 0x7f1d7405ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 --2- 172.21.15.61:0/1831066037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740ab290 0x7f1d740ab670 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 -- 172.21.15.61:0/1831066037 >> 172.21.15.61:0/1831066037 conn(0x7f1d740b1950 msgr2=0x7f1d740b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:55.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 -- 172.21.15.61:0/1831066037 shutdown_connections 2022-07-12T08:46:55.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.624+0000 7f1d79638700 1 -- 172.21.15.61:0/1831066037 wait complete. 2022-07-12T08:46:55.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 Processor -- start 2022-07-12T08:46:55.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 -- start start 2022-07-12T08:46:55.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7405f140 0x7f1d74156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d740ab290 0x7f1d74066e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 0x7f1d7414cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74148920 con 0x7f1d740afa60 2022-07-12T08:46:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74065660 con 0x7f1d740ab290 2022-07-12T08:46:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74148610 con 0x7f1d7405f140 2022-07-12T08:46:55.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 0x7f1d7414cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 0x7f1d7414cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56926/0 (socket says 172.21.15.61:56926) 2022-07-12T08:46:55.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 -- 172.21.15.61:0/3250619807 learned_addr learned my addr 172.21.15.61:0/3250619807 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:55.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d73fff700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7405f140 0x7f1d74156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d737fe700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d740ab290 0x7f1d74066e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 -- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7405f140 msgr2=0x7f1d74156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7405f140 0x7f1d74156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 -- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d740ab290 msgr2=0x7f1d74066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:55.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d740ab290 0x7f1d74066e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:55.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 -- 172.21.15.61:0/3250619807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c009530 con 0x7f1d740afa60 2022-07-12T08:46:55.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d737fe700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d740ab290 0x7f1d74066e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:55.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d78e37700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 0x7f1d7414cbb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f1d740615d0 tx=0x7f1d64007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:55.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6400ce30 con 0x7f1d740afa60 2022-07-12T08:46:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64012400 con 0x7f1d740afa60 2022-07-12T08:46:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d74067370 con 0x7f1d740afa60 2022-07-12T08:46:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64012760 con 0x7f1d740afa60 2022-07-12T08:46:55.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.628+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d740af1e0 con 0x7f1d740afa60 2022-07-12T08:46:55.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.632+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d74063ba0 con 0x7f1d740afa60 2022-07-12T08:46:55.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.632+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1d64019030 con 0x7f1d740afa60 2022-07-12T08:46:55.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.632+0000 7f1d717fa700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d54053a40 0x7f1d54055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:55.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.632+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1d6406aeb0 con 0x7f1d740afa60 2022-07-12T08:46:55.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.632+0000 7f1d73fff700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d54053a40 0x7f1d54055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:55.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.632+0000 7f1d73fff700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d54053a40 0x7f1d54055f00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f1d6c019000 tx=0x7f1d6c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:55.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.636+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d74063ba0 con 0x7f1d740afa60 2022-07-12T08:46:55.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:55.808+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f1d74063ba0 con 0x7f1d740afa60 2022-07-12T08:46:56.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.196+0000 7f1d717fa700 1 -- 172.21.15.61:0/3250619807 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v254) v1 ==== 147+0+0 (secure 0 0 0) 0x7f1d74063ba0 con 0x7f1d740afa60 2022-07-12T08:46:56.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d54053a40 msgr2=0x7f1d54055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:56.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d54053a40 0x7f1d54055f00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f1d6c019000 tx=0x7f1d6c016000).stop 2022-07-12T08:46:56.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 msgr2=0x7f1d7414cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:56.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 0x7f1d7414cbb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f1d740615d0 tx=0x7f1d64007bd0).stop 2022-07-12T08:46:56.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 shutdown_connections 2022-07-12T08:46:56.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d740afa60 0x7f1d7414cbb0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d54053a40 0x7f1d54055f00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d740ab290 0x7f1d74066e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 --2- 172.21.15.61:0/3250619807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7405f140 0x7f1d74156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 >> 172.21.15.61:0/3250619807 conn(0x7f1d740b1950 msgr2=0x7f1d7414d870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:56.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 shutdown_connections 2022-07-12T08:46:56.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.200+0000 7f1d79638700 1 -- 172.21.15.61:0/3250619807 wait complete. 2022-07-12T08:46:56.223 INFO:tasks.cephfs_test_runner:test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-12T08:46:56.223 INFO:tasks.cephfs_test_runner:Starting test: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:46:56.224 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-12T08:46:56.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f3330b82700 1 Processor -- start 2022-07-12T08:46:56.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f3330b82700 1 -- start start 2022-07-12T08:46:56.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f3330b82700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f332c0b25c0 0x7f332c0b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:56.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f3330b82700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c140010 con 0x7f332c1436b0 2022-07-12T08:46:56.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f3330b82700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c0668f0 con 0x7f332c0b6780 2022-07-12T08:46:56.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f3330b82700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c142f70 con 0x7f332c0b25c0 2022-07-12T08:46:56.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f332affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f332c0b25c0 0x7f332c0b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:56.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f332affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f332c0b25c0 0x7f332c0b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56932/0 (socket says 172.21.15.61:56932) 2022-07-12T08:46:56.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f332affd700 1 -- 172.21.15.61:0/3206066392 learned_addr learned my addr 172.21.15.61:0/3206066392 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:56.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f332affd700 1 -- 172.21.15.61:0/3206066392 >> v1:172.21.15.111:6789/0 conn(0x7f332c0b6780 legacy=0x7f332c13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:56.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f332affd700 1 -- 172.21.15.61:0/3206066392 >> v1:172.21.15.61:6789/0 conn(0x7f332c1436b0 legacy=0x7f332c065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:56.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.332+0000 7f332affd700 1 -- 172.21.15.61:0/3206066392 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c143210 con 0x7f332c0b25c0 2022-07-12T08:46:56.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f332affd700 1 --2- 172.21.15.61:0/3206066392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c0b6b60 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f332c060230 tx=0x7f3320006eb0).ready entity=mon.0 client_cookie=78e25600c3712388 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:56.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f332a7fc700 1 -- 172.21.15.61:0/3206066392 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f332000fa60 con 0x7f332c0b25c0 2022-07-12T08:46:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f332a7fc700 1 -- 172.21.15.61:0/3206066392 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3320000e90 con 0x7f332c0b25c0 2022-07-12T08:46:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f332a7fc700 1 -- 172.21.15.61:0/3206066392 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f332000fdf0 con 0x7f332c0b25c0 2022-07-12T08:46:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- 172.21.15.61:0/3206066392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 msgr2=0x7f332c0b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 --2- 172.21.15.61:0/3206066392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c0b6b60 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f332c060230 tx=0x7f3320006eb0).stop 2022-07-12T08:46:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- 172.21.15.61:0/3206066392 shutdown_connections 2022-07-12T08:46:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 --2- 172.21.15.61:0/3206066392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c0b6b60 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- 172.21.15.61:0/3206066392 >> 172.21.15.61:0/3206066392 conn(0x7f332c064570 msgr2=0x7f332c0b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- 172.21.15.61:0/3206066392 shutdown_connections 2022-07-12T08:46:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- 172.21.15.61:0/3206066392 wait complete. 2022-07-12T08:46:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 Processor -- start 2022-07-12T08:46:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- start start 2022-07-12T08:46:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c14d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:56.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f332c0b6780 0x7f332c14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:56.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332c1436b0 0x7f332c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:56.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c13c2d0 con 0x7f332c0b25c0 2022-07-12T08:46:56.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c14e870 con 0x7f332c1436b0 2022-07-12T08:46:56.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.336+0000 7f3330b82700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0af0e0 con 0x7f332c0b6780 2022-07-12T08:46:56.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:56.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332c1436b0 0x7f332c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:56.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332affd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f332c0b6780 0x7f332c14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:56.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c14d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56938/0 (socket says 172.21.15.61:56938) 2022-07-12T08:46:56.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 -- 172.21.15.61:0/3804799573 learned_addr learned my addr 172.21.15.61:0/3804799573 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:56.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 -- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f332c0b6780 msgr2=0x7f332c14de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:56.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f332c0b6780 0x7f332c14de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 -- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332c1436b0 msgr2=0x7f332c13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:56.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332c1436b0 0x7f332c13d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:56.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 -- 172.21.15.61:0/3804799573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332000b1e0 con 0x7f332c0b25c0 2022-07-12T08:46:56.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332bfff700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332c1436b0 0x7f332c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:56.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f332b7fe700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c14d920 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f332c1285e0 tx=0x7f33240014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:56.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f332400e4a0 con 0x7f332c0b25c0 2022-07-12T08:46:56.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332400e600 con 0x7f332c0b25c0 2022-07-12T08:46:56.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c14e5d0 con 0x7f332c0b25c0 2022-07-12T08:46:56.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f332400e960 con 0x7f332c0b25c0 2022-07-12T08:46:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c0b0090 con 0x7f332c0b25c0 2022-07-12T08:46:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.340+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c057460 con 0x7f332c0b25c0 2022-07-12T08:46:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.344+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3324019030 con 0x7f332c0b25c0 2022-07-12T08:46:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.344+0000 7f3328ff9700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f330c0539b0 0x7f330c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:56.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.344+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f332406b270 con 0x7f332c0b25c0 2022-07-12T08:46:56.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.344+0000 7f332affd700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f330c0539b0 0x7f330c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:56.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.344+0000 7f332affd700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f330c0539b0 0x7f330c055e70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f332001d000 tx=0x7f3320017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:56.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.348+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f33240424b0 con 0x7f332c0b25c0 2022-07-12T08:46:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:56.524+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7f332c067b80 con 0x7f332c0b25c0 2022-07-12T08:46:57.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.208+0000 7f3328ff9700 1 -- 172.21.15.61:0/3804799573 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v255) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3324045ad0 con 0x7f332c0b25c0 2022-07-12T08:46:57.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f330c0539b0 msgr2=0x7f330c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f330c0539b0 0x7f330c055e70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f332001d000 tx=0x7f3320017000).stop 2022-07-12T08:46:57.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 msgr2=0x7f332c14d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c14d920 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f332c1285e0 tx=0x7f33240014a0).stop 2022-07-12T08:46:57.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 shutdown_connections 2022-07-12T08:46:57.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f330c0539b0 0x7f330c055e70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332c1436b0 0x7f332c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f332c0b6780 0x7f332c14de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 --2- 172.21.15.61:0/3804799573 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332c0b25c0 0x7f332c14d920 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 >> 172.21.15.61:0/3804799573 conn(0x7f332c064570 msgr2=0x7f332c0b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:57.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 shutdown_connections 2022-07-12T08:46:57.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.212+0000 7f3330b82700 1 -- 172.21.15.61:0/3804799573 wait complete. 2022-07-12T08:46:57.248 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:46:57.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.372+0000 7f2a3ffff700 1 Processor -- start 2022-07-12T08:46:57.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.372+0000 7f2a3ffff700 1 -- start start 2022-07-12T08:46:57.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a4009a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a40099010 con 0x7f2a40060c10 2022-07-12T08:46:57.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4014b780 con 0x7f2a400b1000 2022-07-12T08:46:57.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a40060ff0 con 0x7f2a4009a1f0 2022-07-12T08:46:57.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2a40060c10 0x7f2a4014b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45162/0 (socket says 172.21.15.61:45162) 2022-07-12T08:46:57.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3effd700 1 -- 172.21.15.61:0/118222422 learned_addr learned my addr 172.21.15.61:0/118222422 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:57.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3f7fe700 1 --2- 172.21.15.61:0/118222422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a4009a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98562026 0 0) 0x7f2a40099010 con 0x7f2a40060c10 2022-07-12T08:46:57.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a20003260 con 0x7f2a40060c10 2022-07-12T08:46:57.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686448683 0 0) 0x7f2a30001490 con 0x7f2a400b1000 2022-07-12T08:46:57.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a40099010 con 0x7f2a400b1000 2022-07-12T08:46:57.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3f7fe700 1 -- 172.21.15.61:0/118222422 >> v1:172.21.15.111:6789/0 conn(0x7f2a400b1000 legacy=0x7f2a400af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3f7fe700 1 -- 172.21.15.61:0/118222422 >> v1:172.21.15.61:6789/0 conn(0x7f2a40060c10 legacy=0x7f2a4014b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3f7fe700 1 -- 172.21.15.61:0/118222422 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4014a930 con 0x7f2a4009a1f0 2022-07-12T08:46:57.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3f7fe700 1 --2- 172.21.15.61:0/118222422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a4009a630 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2a38002f70 tx=0x7f2a38006df0).ready entity=mon.2 client_cookie=ef9cb89df5c9ff97 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:57.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a3800d5c0 con 0x7f2a4009a1f0 2022-07-12T08:46:57.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a38000e90 con 0x7f2a4009a1f0 2022-07-12T08:46:57.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3dffb700 1 -- 172.21.15.61:0/118222422 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a3800d910 con 0x7f2a4009a1f0 2022-07-12T08:46:57.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- 172.21.15.61:0/118222422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 msgr2=0x7f2a4009a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/118222422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a4009a630 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2a38002f70 tx=0x7f2a38006df0).stop 2022-07-12T08:46:57.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- 172.21.15.61:0/118222422 shutdown_connections 2022-07-12T08:46:57.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/118222422 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a4009a630 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- 172.21.15.61:0/118222422 >> 172.21.15.61:0/118222422 conn(0x7f2a4005bd10 msgr2=0x7f2a4005c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:57.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- 172.21.15.61:0/118222422 shutdown_connections 2022-07-12T08:46:57.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.376+0000 7f2a3ffff700 1 -- 172.21.15.61:0/118222422 wait complete. 2022-07-12T08:46:57.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 Processor -- start 2022-07-12T08:46:57.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 -- start start 2022-07-12T08:46:57.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 0x7f2a400ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a40146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a400b1000 0x7f2a40063410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a400acc00 con 0x7f2a40060c10 2022-07-12T08:46:57.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a40156cd0 con 0x7f2a400b1000 2022-07-12T08:46:57.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a40156f70 con 0x7f2a4009a1f0 2022-07-12T08:46:57.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 0x7f2a400ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 0x7f2a400ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56950/0 (socket says 172.21.15.61:56950) 2022-07-12T08:46:57.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 -- 172.21.15.61:0/3623905879 learned_addr learned my addr 172.21.15.61:0/3623905879 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:57.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3e7fc700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a40146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3f7fe700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a400b1000 0x7f2a40063410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 -- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 msgr2=0x7f2a40146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a40146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 -- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a400b1000 msgr2=0x7f2a40063410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a400b1000 0x7f2a40063410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 -- 172.21.15.61:0/3623905879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3800b180 con 0x7f2a40060c10 2022-07-12T08:46:57.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a3effd700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 0x7f2a400ac710 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f2a280011d0 tx=0x7f2a28007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:57.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a280070d0 con 0x7f2a40060c10 2022-07-12T08:46:57.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.380+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a28007230 con 0x7f2a40060c10 2022-07-12T08:46:57.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a40146a50 con 0x7f2a40060c10 2022-07-12T08:46:57.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a28005610 con 0x7f2a40060c10 2022-07-12T08:46:57.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a40063aa0 con 0x7f2a40060c10 2022-07-12T08:46:57.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a401572f0 con 0x7f2a40060c10 2022-07-12T08:46:57.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2a2801a060 con 0x7f2a40060c10 2022-07-12T08:46:57.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a37fff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a240539b0 0x7f2a24055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f2a2806b2d0 con 0x7f2a40060c10 2022-07-12T08:46:57.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.384+0000 7f2a3e7fc700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a240539b0 0x7f2a24055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.388+0000 7f2a3e7fc700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a240539b0 0x7f2a24055e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2a3000d000 tx=0x7f2a30007cd0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:57.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.388+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2a28042510 con 0x7f2a40060c10 2022-07-12T08:46:57.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.520+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2a40057460 con 0x7f2a40060c10 2022-07-12T08:46:57.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.524+0000 7f2a37fff700 1 -- 172.21.15.61:0/3623905879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2a28018020 con 0x7f2a40060c10 2022-07-12T08:46:57.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a240539b0 msgr2=0x7f2a24055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a240539b0 0x7f2a24055e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f2a3000d000 tx=0x7f2a30007cd0).stop 2022-07-12T08:46:57.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 msgr2=0x7f2a400ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 0x7f2a400ac710 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f2a280011d0 tx=0x7f2a28007b50).stop 2022-07-12T08:46:57.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 shutdown_connections 2022-07-12T08:46:57.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a240539b0 0x7f2a24055e70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2a400b1000 0x7f2a40063410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2a4009a1f0 0x7f2a40146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 --2- 172.21.15.61:0/3623905879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a40060c10 0x7f2a400ac710 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 >> 172.21.15.61:0/3623905879 conn(0x7f2a4005bd10 msgr2=0x7f2a400b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:57.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 shutdown_connections 2022-07-12T08:46:57.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.528+0000 7f2a3ffff700 1 -- 172.21.15.61:0/3623905879 wait complete. 2022-07-12T08:46:57.545 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:46:57.547 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:57.572 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:57.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:57.617 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:57.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:46:57.672 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:46:57.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:57.674 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:57.674 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:46:57.683 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:46:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:57.686 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:46:57.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:57.688 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:57.689 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:46:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:57.702 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:46:57.703 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:46:57.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:46:57.705 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:46:57.705 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:46:57.733 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:46:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:46:57.735 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:46:57.736 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:46:57.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 Processor -- start 2022-07-12T08:46:57.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- start start 2022-07-12T08:46:57.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51780b5aa0 0x7f51780b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f517805bfc0 0x7f517814ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517805d3d0 con 0x7f5178147000 2022-07-12T08:46:57.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51780b5e80 con 0x7f51780b5aa0 2022-07-12T08:46:57.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517814a740 con 0x7f517805bfc0 2022-07-12T08:46:57.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51780b5aa0 0x7f51780b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51780b5aa0 0x7f51780b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56960/0 (socket says 172.21.15.61:56960) 2022-07-12T08:46:57.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 -- 172.21.15.61:0/375161473 learned_addr learned my addr 172.21.15.61:0/375161473 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:57.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f516f7fe700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f517805bfc0 0x7f517814ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 -- 172.21.15.61:0/375161473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f517805bfc0 msgr2=0x7f517814ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:57.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f517805bfc0 0x7f517814ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 -- 172.21.15.61:0/375161473 >> v1:172.21.15.111:6789/0 conn(0x7f5178147000 legacy=0x7f5178144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 -- 172.21.15.61:0/375161473 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517814a9e0 con 0x7f51780b5aa0 2022-07-12T08:46:57.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f516f7fe700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f517805bfc0 0x7f517814ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:46:57.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f5177fff700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780b5aa0 0x7f51780b0380 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f5168002f70 tx=0x7f5168006eb0).ready entity=mon.0 client_cookie=665ffc2cb5cb5b9e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:57.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f51777fe700 1 -- 172.21.15.61:0/375161473 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516800d5c0 con 0x7f51780b5aa0 2022-07-12T08:46:57.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f51777fe700 1 -- 172.21.15.61:0/375161473 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5168000e90 con 0x7f51780b5aa0 2022-07-12T08:46:57.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f51777fe700 1 -- 172.21.15.61:0/375161473 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f516800d950 con 0x7f51780b5aa0 2022-07-12T08:46:57.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- 172.21.15.61:0/375161473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780b5aa0 msgr2=0x7f51780b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780b5aa0 0x7f51780b0380 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f5168002f70 tx=0x7f5168006eb0).stop 2022-07-12T08:46:57.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- 172.21.15.61:0/375161473 shutdown_connections 2022-07-12T08:46:57.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f517805bfc0 0x7f517814ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 --2- 172.21.15.61:0/375161473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780b5aa0 0x7f51780b0380 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- 172.21.15.61:0/375161473 >> 172.21.15.61:0/375161473 conn(0x7f5178063520 msgr2=0x7f51780b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:57.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- 172.21.15.61:0/375161473 shutdown_connections 2022-07-12T08:46:57.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.848+0000 7f517e445700 1 -- 172.21.15.61:0/375161473 wait complete. 2022-07-12T08:46:57.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 Processor -- start 2022-07-12T08:46:57.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 -- start start 2022-07-12T08:46:57.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517805bfc0 0x7f51780b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f51780b5aa0 0x7f5178063a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 0x7f5178063f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51781563f0 con 0x7f5178147000 2022-07-12T08:46:57.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178066220 con 0x7f517805bfc0 2022-07-12T08:46:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5178143fb0 con 0x7f51780b5aa0 2022-07-12T08:46:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 0x7f5178063f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 0x7f5178063f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56968/0 (socket says 172.21.15.61:56968) 2022-07-12T08:46:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 -- 172.21.15.61:0/317175453 learned_addr learned my addr 172.21.15.61:0/317175453 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f5177fff700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517805bfc0 0x7f51780b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f516f7fe700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f51780b5aa0 0x7f5178063a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 -- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f51780b5aa0 msgr2=0x7f5178063a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f51780b5aa0 0x7f5178063a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 -- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517805bfc0 msgr2=0x7f51780b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517805bfc0 0x7f51780b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 -- 172.21.15.61:0/317175453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516800b1e0 con 0x7f5178147000 2022-07-12T08:46:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517c9e2700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 0x7f5178063f10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f5178148150 tx=0x7f5170007f50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f517000f470 con 0x7f5178147000 2022-07-12T08:46:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517000f5d0 con 0x7f5178147000 2022-07-12T08:46:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51780671d0 con 0x7f5178147000 2022-07-12T08:46:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f517000f930 con 0x7f5178147000 2022-07-12T08:46:57.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.852+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5178065870 con 0x7f5178147000 2022-07-12T08:46:57.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.856+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5178066070 con 0x7f5178147000 2022-07-12T08:46:57.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.856+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5170010070 con 0x7f5178147000 2022-07-12T08:46:57.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.856+0000 7f5175ffb700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5150053940 0x7f5150055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:57.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.856+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f517006b0f0 con 0x7f5178147000 2022-07-12T08:46:57.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.856+0000 7f5177fff700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5150053940 0x7f5150055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:57.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.856+0000 7f5177fff700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5150053940 0x7f5150055e00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f516801c000 tx=0x7f5168015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:57.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.864+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5170042290 con 0x7f5178147000 2022-07-12T08:46:57.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.988+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5178067360 con 0x7f5178147000 2022-07-12T08:46:57.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.992+0000 7f5175ffb700 1 -- 172.21.15.61:0/317175453 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v65) v1 ==== 81+0+28876 (secure 0 0 0) 0x7f51700458b0 con 0x7f5178147000 2022-07-12T08:46:57.995 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:57.996 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:46:57.996 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 65, 2022-07-12T08:46:57.996 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:46:57.997 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:46:57.997 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:46:55.192695+0000", 2022-07-12T08:46:57.997 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:57.997 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:46:57.998 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:46:57.998 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:46:57.998 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:46:57.998 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:46:57.999 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:46:57.999 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:46:57.999 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:46:57.999 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:58.000 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:46:58.000 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:46:58.000 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:46:58.000 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:46:58.001 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:46:58.001 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 12, 2022-07-12T08:46:58.001 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:46:58.001 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:46:58.002 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:46:58.002 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:46:58.002 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:46:58.002 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.003 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:46:58.003 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:46:58.005 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:46:58.005 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:58.006 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:58.006 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:58.006 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:58.007 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:58.007 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:58.007 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:58.007 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:58.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:46:58.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:46:58.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:46:58.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:46:58.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:46:58.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:58.009 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:58.009 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:58.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:58.011 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:58.011 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:58.012 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:58.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:46:58.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:58.014 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:58.015 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:58.015 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:58.015 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:58.016 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:58.016 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:58.017 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:58.017 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:58.017 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:58.018 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:58.018 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:58.019 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:58.019 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:58.020 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:58.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:58.021 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:58.021 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:58.022 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:58.022 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:58.023 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:58.023 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:58.024 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:58.024 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:58.024 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:58.024 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:58.025 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.025 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:58.025 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:58.025 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:58.026 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:58.026 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:58.026 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:58.026 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:58.027 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:58.027 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:58.027 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:58.027 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:58.027 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:46:58.028 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:46:58.028 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.028 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:58.028 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:46:58.029 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.029 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.029 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:46:58.030 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:46:58.030 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:46:58.030 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:46:58.030 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:46:58.031 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:46:58.031 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:46:58.031 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:46:58.031 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:46:58.032 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:46:58.032 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:46:58.032 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:46:58.032 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:46:58.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:46:58.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:46:58.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:46:58.033 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:46:58.033 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:46:58.034 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:46:58.034 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:46:58.035 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:46:58.035 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:46:58.036 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:46:58.036 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.036 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:46:58.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:46:58.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:46:58.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:46:58.038 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:46:58.038 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:46:58.039 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:46:58.039 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:46:58.040 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:46:58.040 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:46:58.041 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:46:58.041 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:46:58.042 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:46:58.042 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:46:58.043 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:46:58.043 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:46:58.044 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:46:58.044 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:46:58.044 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:46:58.045 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:46:58.045 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:46:58.045 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:46:58.046 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:46:58.047 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:46:58.047 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:46:58.048 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:46:58.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:46:58.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.048 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:46:58.049 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:46:58.049 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:46:58.049 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:46:58.049 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:46:58.050 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:46:58.050 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:46:58.050 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:46:58.050 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:46:58.051 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:46:58.051 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:46:58.051 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:46:58.051 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:46:58.051 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:46:58.052 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.052 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.052 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:58.052 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:46:58.053 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.053 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:58.053 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:46:58.053 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.053 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.054 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.054 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.054 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.055 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.055 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.055 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:58.055 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.056 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.056 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:46:58.057 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.057 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.058 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:46:58.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.058 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.059 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.059 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.060 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:46:58.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.061 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:46:58.061 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.062 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.062 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.062 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:46:58.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.064 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.064 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.064 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:46:58.065 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.065 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.065 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.066 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:46:58.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.067 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:46:58.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:46:58.069 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.069 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.069 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.070 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:46:58.070 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:46:58.070 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:46:58.070 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:46:58.071 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.071 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.071 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.071 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.072 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:58.072 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:46:58.072 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.073 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.073 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.073 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.073 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.074 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.074 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:58.074 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.074 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.075 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.075 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.075 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.075 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:46:58.076 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.076 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.077 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:46:58.077 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.077 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.078 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.078 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.078 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.078 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.079 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:46:58.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.080 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.080 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:46:58.080 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.080 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.081 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.081 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.081 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.082 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:46:58.082 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.083 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.083 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:46:58.084 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.084 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.084 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.085 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.086 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:46:58.087 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.088 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:46:58.088 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:46:58.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.089 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.089 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:46:58.090 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:46:58.090 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:46:58.091 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:46:58.091 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.091 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.091 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.092 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.092 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:58.093 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:46:58.093 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.093 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.093 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.094 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.094 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.094 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.095 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.095 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-12T08:46:58.095 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.095 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.095 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.096 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.096 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.096 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:46:58.097 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.097 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:46:58.098 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.098 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.098 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.099 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.099 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.099 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:46:58.100 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.100 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.101 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:46:58.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.101 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.101 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.102 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.102 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.102 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.102 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.103 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:46:58.103 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.103 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.104 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.104 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.104 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:46:58.104 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.104 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.105 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.105 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.105 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.106 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:46:58.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:46:58.109 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:46:58.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.111 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:46:58.111 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:46:58.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:46:58.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:46:58.112 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.112 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.112 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.112 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.113 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.113 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:58.114 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:46:58.114 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.115 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.115 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.115 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.116 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.116 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.117 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.117 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-12T08:46:58.118 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.118 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.119 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.119 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.120 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.120 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:46:58.121 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:46:58.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.125 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.125 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.126 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.126 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.126 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.127 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.127 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:46:58.128 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.128 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.128 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.129 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:46:58.130 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.130 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.131 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.131 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.132 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.133 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:46:58.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:46:58.136 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.136 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.137 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.137 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:46:58.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:46:58.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:46:58.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.141 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:46:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:46:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:46:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:46:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:46:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:46:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.149 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.149 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:46:58.150 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.150 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.150 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.150 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.151 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.152 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:46:58.152 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.154 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.155 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:46:58.155 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.156 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.156 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.157 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.158 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.159 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.159 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:46:58.162 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.163 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.163 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.164 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:46:58.164 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.165 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.165 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.166 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.166 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.167 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.167 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:46:58.169 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.170 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:46:58.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:46:58.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.173 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.175 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:46:58.175 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:46:58.176 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:46:58.176 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:46:58.177 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.177 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.178 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.178 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.180 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:58.181 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:46:58.181 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.181 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.182 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.182 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.183 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.183 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.183 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.184 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-12T08:46:58.187 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.187 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.188 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:46:58.189 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.190 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.190 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.191 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.191 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:46:58.192 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.192 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.193 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.193 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.193 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.194 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.195 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:46:58.196 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.196 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.196 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.197 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:46:58.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.199 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.200 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.200 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.201 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.202 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.202 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:46:58.203 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.203 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.204 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:46:58.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.205 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.206 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.207 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:46:58.210 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:46:58.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:46:58.212 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.213 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.215 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:46:58.216 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:46:58.216 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:46:58.219 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:46:58.219 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.220 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.220 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.221 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.223 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.223 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:58.223 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:46:58.224 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.224 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.225 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.225 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.227 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.228 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:46:58.228 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.229 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.229 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:46:58.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.233 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.233 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.234 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:46:58.234 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.235 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.235 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.236 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.238 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.238 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:46:58.239 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.241 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:46:58.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.243 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.243 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.243 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:46:58.245 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:46:58.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.246 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.246 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.246 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.247 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.247 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:46:58.248 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.248 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:46:58.249 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:46:58.249 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.249 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.250 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.250 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:46:58.250 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:46:58.250 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:46:58.251 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:46:58.251 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.251 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.251 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.252 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:58.253 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:46:58.253 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:46:58.253 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:46:58.254 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:46:58.254 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:46:58.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:46:58.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:46:58.254 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:46:58.255 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-12T08:46:58.255 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:46:58.255 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:46:58.255 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:46:58.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.256 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.256 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:46:58.257 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.257 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.257 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.257 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:46:58.258 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.258 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.258 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.258 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.259 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:46:58.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:46:58.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.260 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.260 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.261 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:46:58.261 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.261 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.261 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.262 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.262 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:46:58.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.262 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.262 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:46:58.263 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:46:58.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.264 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.265 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.265 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:46:58.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:46:58.266 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.266 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:46:58.267 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:46:58.267 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:46:58.268 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:46:58.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:46:58.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.268 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.269 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:46:58.269 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:46:58.269 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:46:58.270 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:46:58.270 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:46:58.270 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:46:58.270 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:46:58.271 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:46:58.271 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.271 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:58.271 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:46:58.271 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.272 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:46:58.272 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.272 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.272 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.273 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.273 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.273 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:46:58.273 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.273 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.274 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:46:58.274 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.275 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.275 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.275 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.275 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.276 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:46:58.276 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.276 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.277 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:46:58.277 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.277 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.277 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.277 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.278 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.278 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:46:58.278 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.279 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.279 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:46:58.279 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.280 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.280 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.280 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.280 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:46:58.281 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.282 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:46:58.282 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.282 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.283 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.283 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.283 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.283 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:46:58.284 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.284 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:46:58.285 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.285 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.285 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.285 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.286 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.286 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:46:58.286 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.287 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:46:58.287 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.287 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.287 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.289 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.289 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.289 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:46:58.289 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.290 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.290 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:46:58.290 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:46:58.290 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:46:58.291 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:46:58.291 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:46:58.291 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:46:58.291 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:46:58.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:46:58.292 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:46:58.292 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.292 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:46:58.292 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:46:58.293 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:46:58.293 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:46:58.293 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:46:58.294 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:46:58.294 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/3698454378": "2022-07-13T08:46:51.165308+0000", 2022-07-12T08:46:58.294 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/3698454378": "2022-07-13T08:46:51.165308+0000" 2022-07-12T08:46:58.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.295 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:46:58.295 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:46:58.295 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:46:58.295 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:46:58.296 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:46:58.296 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:46:58.296 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:46:58.297 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:46:58.297 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:46:58.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:46:58.298 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:46:58.298 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:46:58.298 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:46:58.298 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:46:58.299 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:46:58.299 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:46:58.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5150053940 msgr2=0x7f5150055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5150053940 0x7f5150055e00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f516801c000 tx=0x7f5168015000).stop 2022-07-12T08:46:58.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 msgr2=0x7f5178063f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 0x7f5178063f10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f5178148150 tx=0x7f5170007f50).stop 2022-07-12T08:46:58.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 shutdown_connections 2022-07-12T08:46:58.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5178147000 0x7f5178063f10 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f51780b5aa0 0x7f5178063a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5150053940 0x7f5150055e00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 --2- 172.21.15.61:0/317175453 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517805bfc0 0x7f51780b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 >> 172.21.15.61:0/317175453 conn(0x7f5178063520 msgr2=0x7f51780b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:58.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 shutdown_connections 2022-07-12T08:46:58.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:57.996+0000 7f517e445700 1 -- 172.21.15.61:0/317175453 wait complete. 2022-07-12T08:46:58.307 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:58.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 Processor -- start 2022-07-12T08:46:58.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 -- start start 2022-07-12T08:46:58.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b24e0 0x7f39cc0ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f39cc0b0320 0x7f39cc142610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39cc0b28c0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39cc0aefc0 con 0x7f39cc0b0320 2022-07-12T08:46:58.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d2d67700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc0b0f80 con 0x7f39cc0b24e0 2022-07-12T08:46:58.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d1564700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f39cc0b0ae0 0x7f39cc0ae8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45192/0 (socket says 172.21.15.61:45192) 2022-07-12T08:46:58.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.420+0000 7f39d1564700 1 -- 172.21.15.61:0/2023739642 learned_addr learned my addr 172.21.15.61:0/2023739642 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:58.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d1d65700 1 --2- 172.21.15.61:0/2023739642 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b24e0 0x7f39cc0ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2566700 1 --2- 172.21.15.61:0/2023739642 >> v2:172.21.15.111:3300/0 conn(0x7f39cc0b0320 0x7f39cc142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d0d63700 1 -- 172.21.15.61:0/2023739642 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655788440 0 0) 0x7f39cc0b28c0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d0d63700 1 -- 172.21.15.61:0/2023739642 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b8003260 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d1d65700 1 -- 172.21.15.61:0/2023739642 >> v2:172.21.15.111:3300/0 conn(0x7f39cc0b0320 msgr2=0x7f39cc142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:58.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d1d65700 1 --2- 172.21.15.61:0/2023739642 >> v2:172.21.15.111:3300/0 conn(0x7f39cc0b0320 0x7f39cc142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d1d65700 1 -- 172.21.15.61:0/2023739642 >> v1:172.21.15.61:6789/0 conn(0x7f39cc0b0ae0 legacy=0x7f39cc0ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d0d63700 1 -- 172.21.15.61:0/2023739642 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095398664 0 0) 0x7f39b8003260 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d1d65700 1 -- 172.21.15.61:0/2023739642 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0b1220 con 0x7f39cc0b24e0 2022-07-12T08:46:58.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2566700 1 --2- 172.21.15.61:0/2023739642 >> v2:172.21.15.111:3300/0 conn(0x7f39cc0b0320 0x7f39cc142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:46:58.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d1d65700 1 --2- 172.21.15.61:0/2023739642 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b24e0 0x7f39cc0ae3c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f39c0001820 tx=0x7f39bc006df0).ready entity=mon.2 client_cookie=9b0bd18d5da865fc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:58.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d0d63700 1 -- 172.21.15.61:0/2023739642 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39c0001680 con 0x7f39cc0b24e0 2022-07-12T08:46:58.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d0d63700 1 -- 172.21.15.61:0/2023739642 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc000e90 con 0x7f39cc0b24e0 2022-07-12T08:46:58.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d0d63700 1 -- 172.21.15.61:0/2023739642 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39bc012460 con 0x7f39cc0b24e0 2022-07-12T08:46:58.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 -- 172.21.15.61:0/2023739642 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b24e0 msgr2=0x7f39cc0ae3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 --2- 172.21.15.61:0/2023739642 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b24e0 0x7f39cc0ae3c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f39c0001820 tx=0x7f39bc006df0).stop 2022-07-12T08:46:58.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 -- 172.21.15.61:0/2023739642 shutdown_connections 2022-07-12T08:46:58.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 --2- 172.21.15.61:0/2023739642 >> v2:172.21.15.111:3300/0 conn(0x7f39cc0b0320 0x7f39cc142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 --2- 172.21.15.61:0/2023739642 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b24e0 0x7f39cc0ae3c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 -- 172.21.15.61:0/2023739642 >> 172.21.15.61:0/2023739642 conn(0x7f39cc05f140 msgr2=0x7f39cc05f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:58.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 -- 172.21.15.61:0/2023739642 shutdown_connections 2022-07-12T08:46:58.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 -- 172.21.15.61:0/2023739642 wait complete. 2022-07-12T08:46:58.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 Processor -- start 2022-07-12T08:46:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.424+0000 7f39d2d67700 1 -- start start 2022-07-12T08:46:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b0320 0x7f39cc13cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 0x7f39cc13e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc0b24e0 0x7f39cc134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc13a930 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc142ac0 con 0x7f39cc0b24e0 2022-07-12T08:46:58.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39cc13add0 con 0x7f39cc0b0320 2022-07-12T08:46:58.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 0x7f39cc13e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 0x7f39cc13e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56982/0 (socket says 172.21.15.61:56982) 2022-07-12T08:46:58.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 -- 172.21.15.61:0/1977000624 learned_addr learned my addr 172.21.15.61:0/1977000624 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:58.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1d65700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b0320 0x7f39cc13cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2566700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc0b24e0 0x7f39cc134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 -- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b0320 msgr2=0x7f39cc13cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b0320 0x7f39cc13cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 -- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc0b24e0 msgr2=0x7f39cc134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc0b24e0 0x7f39cc134c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 -- 172.21.15.61:0/1977000624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c4009470 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2566700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc0b24e0 0x7f39cc134c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:58.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d1564700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 0x7f39cc13e4f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f39c00018e0 tx=0x7f39c00012e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:58.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39c0007170 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39c00072d0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc00b180 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39c0004920 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.428+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39cc135f10 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.432+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f39c0004a80 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.432+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39cc057460 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.432+0000 7f39caffd700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39ac053920 0x7f39ac055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.432+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f39c006b6f0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.432+0000 7f39d1d65700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39ac053920 0x7f39ac055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.432+0000 7f39d1d65700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39ac053920 0x7f39ac055de0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f39bc01a000 tx=0x7f39bc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:58.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.436+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f39c00428d0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.604+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39cc0aecd0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39caffd700 1 -- 172.21.15.61:0/1977000624 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f39c0045ef0 con 0x7f39cc0b0ae0 2022-07-12T08:46:58.611 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:58.612 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28},{"gid":6219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2990032756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2990032756},{"type":"v1","addr":"172.21.15.61:6833","nonce":2990032756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[]} 2022-07-12T08:46:58.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39ac053920 msgr2=0x7f39ac055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39ac053920 0x7f39ac055de0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f39bc01a000 tx=0x7f39bc014000).stop 2022-07-12T08:46:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 msgr2=0x7f39cc13e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 0x7f39cc13e4f0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f39c00018e0 tx=0x7f39c00012e0).stop 2022-07-12T08:46:58.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 shutdown_connections 2022-07-12T08:46:58.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f39ac053920 0x7f39ac055de0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39cc0b24e0 0x7f39cc134c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f39cc0b0ae0 0x7f39cc13e4f0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 --2- 172.21.15.61:0/1977000624 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f39cc0b0320 0x7f39cc13cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 >> 172.21.15.61:0/1977000624 conn(0x7f39cc05f140 msgr2=0x7f39cc0ae2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:58.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 shutdown_connections 2022-07-12T08:46:58.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.608+0000 7f39d2d67700 1 -- 172.21.15.61:0/1977000624 wait complete. 2022-07-12T08:46:58.621 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 33 2022-07-12T08:46:58.631 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:46:58.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 Processor -- start 2022-07-12T08:46:58.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- start start 2022-07-12T08:46:58.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b5f20 0x7fbb48147aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb481440f0 0x7fbb48144540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb48147360 con 0x7fbb480b3e00 2022-07-12T08:46:58.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb48147170 con 0x7fbb481440f0 2022-07-12T08:46:58.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480b0b10 con 0x7fbb480b5f20 2022-07-12T08:46:58.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4d77f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbb480b3e00 0x7fbb480b2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45204/0 (socket says 172.21.15.61:45204) 2022-07-12T08:46:58.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4d77f700 1 -- 172.21.15.61:0/1169268369 learned_addr learned my addr 172.21.15.61:0/1169268369 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:58.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4e781700 1 --2- 172.21.15.61:0/1169268369 >> v2:172.21.15.61:3300/0 conn(0x7fbb481440f0 0x7fbb48144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4df80700 1 --2- 172.21.15.61:0/1169268369 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b5f20 0x7fbb48147aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4cf7e700 1 -- 172.21.15.61:0/1169268369 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66030178 0 0) 0x7fbb48147360 con 0x7fbb480b3e00 2022-07-12T08:46:58.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4cf7e700 1 -- 172.21.15.61:0/1169268369 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb2c003260 con 0x7fbb480b3e00 2022-07-12T08:46:58.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4e781700 1 -- 172.21.15.61:0/1169268369 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b5f20 msgr2=0x7fbb48147aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:58.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4e781700 1 --2- 172.21.15.61:0/1169268369 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b5f20 0x7fbb48147aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4e781700 1 -- 172.21.15.61:0/1169268369 >> v1:172.21.15.61:6789/0 conn(0x7fbb480b3e00 legacy=0x7fbb480b2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4e781700 1 -- 172.21.15.61:0/1169268369 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48147600 con 0x7fbb481440f0 2022-07-12T08:46:58.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4e781700 1 --2- 172.21.15.61:0/1169268369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb48144540 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fbb400032f0 tx=0x7fbb400091d0).ready entity=mon.0 client_cookie=63a924865c96cf34 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:58.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4cf7e700 1 -- 172.21.15.61:0/1169268369 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb4000b600 con 0x7fbb481440f0 2022-07-12T08:46:58.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4cf7e700 1 -- 172.21.15.61:0/1169268369 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4000b760 con 0x7fbb481440f0 2022-07-12T08:46:58.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb4cf7e700 1 -- 172.21.15.61:0/1169268369 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb4000b8d0 con 0x7fbb481440f0 2022-07-12T08:46:58.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- 172.21.15.61:0/1169268369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 msgr2=0x7fbb48144540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 --2- 172.21.15.61:0/1169268369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb48144540 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fbb400032f0 tx=0x7fbb400091d0).stop 2022-07-12T08:46:58.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- 172.21.15.61:0/1169268369 shutdown_connections 2022-07-12T08:46:58.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 --2- 172.21.15.61:0/1169268369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb48144540 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 --2- 172.21.15.61:0/1169268369 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b5f20 0x7fbb48147aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- 172.21.15.61:0/1169268369 >> 172.21.15.61:0/1169268369 conn(0x7fbb4805f9f0 msgr2=0x7fbb48066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:58.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- 172.21.15.61:0/1169268369 shutdown_connections 2022-07-12T08:46:58.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.744+0000 7fbb501e4700 1 -- 172.21.15.61:0/1169268369 wait complete. 2022-07-12T08:46:58.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 Processor -- start 2022-07-12T08:46:58.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 -- start start 2022-07-12T08:46:58.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b3e00 0x7fbb480b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb480b5f20 0x7fbb481564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb481569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48156ec0 con 0x7fbb481440f0 2022-07-12T08:46:58.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480aee80 con 0x7fbb480b5f20 2022-07-12T08:46:58.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb501e4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480af0b0 con 0x7fbb480b3e00 2022-07-12T08:46:58.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb481569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb481569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:56996/0 (socket says 172.21.15.61:56996) 2022-07-12T08:46:58.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 -- 172.21.15.61:0/936560719 learned_addr learned my addr 172.21.15.61:0/936560719 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:58.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4df80700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b3e00 0x7fbb480b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4d77f700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb480b5f20 0x7fbb481564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 -- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b3e00 msgr2=0x7fbb480b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b3e00 0x7fbb480b77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 -- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb480b5f20 msgr2=0x7fbb481564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.748+0000 7fbb4e781700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb480b5f20 0x7fbb481564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb4e781700 1 -- 172.21.15.61:0/936560719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40009470 con 0x7fbb481440f0 2022-07-12T08:46:58.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb4e781700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb481569d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fbb480624b0 tx=0x7fbb40016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:58.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb4000b600 con 0x7fbb481440f0 2022-07-12T08:46:58.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb40011e10 con 0x7fbb481440f0 2022-07-12T08:46:58.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb400205f0 con 0x7fbb481440f0 2022-07-12T08:46:58.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb480af350 con 0x7fbb481440f0 2022-07-12T08:46:58.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb480b7d40 con 0x7fbb481440f0 2022-07-12T08:46:58.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb48057460 con 0x7fbb481440f0 2022-07-12T08:46:58.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbb40027030 con 0x7fbb481440f0 2022-07-12T08:46:58.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb3effd700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb30053920 0x7fbb30055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:58.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fbb4006cbe0 con 0x7fbb481440f0 2022-07-12T08:46:58.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.752+0000 7fbb4df80700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb30053920 0x7fbb30055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:58.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.756+0000 7fbb4df80700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb30053920 0x7fbb30055de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fbb4400b000 tx=0x7fbb44006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:58.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.760+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb40043cb0 con 0x7fbb481440f0 2022-07-12T08:46:58.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.916+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb480afbd0 con 0x7fbb481440f0 2022-07-12T08:46:58.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb3effd700 1 -- 172.21.15.61:0/936560719 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+1212 (secure 0 0 0) 0x7fbb400254d0 con 0x7fbb481440f0 2022-07-12T08:46:58.922 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:46:58.922 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5884,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/3811247756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3811247756},{"type":"v1","addr":"172.21.15.111:6835","nonce":3811247756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":28},{"gid":6219,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2990032756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2990032756},{"type":"v1","addr":"172.21.15.61:6833","nonce":2990032756}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[]} 2022-07-12T08:46:58.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb30053920 msgr2=0x7fbb30055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb30053920 0x7fbb30055de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fbb4400b000 tx=0x7fbb44006df0).stop 2022-07-12T08:46:58.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 msgr2=0x7fbb481569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:58.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb481569d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7fbb480624b0 tx=0x7fbb40016000).stop 2022-07-12T08:46:58.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 shutdown_connections 2022-07-12T08:46:58.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb481440f0 0x7fbb481569d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb30053920 0x7fbb30055de0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb480b5f20 0x7fbb481564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 --2- 172.21.15.61:0/936560719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb480b3e00 0x7fbb480b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 >> 172.21.15.61:0/936560719 conn(0x7fbb4805f9f0 msgr2=0x7fbb480b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:58.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 shutdown_connections 2022-07-12T08:46:58.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:58.920+0000 7fbb501e4700 1 -- 172.21.15.61:0/936560719 wait complete. 2022-07-12T08:46:58.931 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 33 2022-07-12T08:46:58.938 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:46:58.938 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:46:58.939 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:46:58.939 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:46:58.940 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:46:58.940 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:46:58.941 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:46:58.941 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:46:58.954 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:46:58.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:46:58.959 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:46:58.960 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:46:58.962 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:46:58.962 DEBUG:teuthology.parallel:result is None 2022-07-12T08:46:58.965 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:46:58.966 DEBUG:teuthology.parallel:result is None 2022-07-12T08:46:58.967 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:46:59.041 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:46:59.050 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:46:59.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 Processor -- start 2022-07-12T08:46:59.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- start start 2022-07-12T08:46:59.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae540ae500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae540acee0 con 0x7fae540b0120 2022-07-12T08:46:59.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae540ad180 con 0x7fae5413ec70 2022-07-12T08:46:59.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae540acb60 con 0x7fae540b1320 2022-07-12T08:46:59.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae53fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fae540b0120 0x7fae540ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45234/0 (socket says 172.21.15.61:45234) 2022-07-12T08:46:59.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae53fff700 1 -- 172.21.15.61:0/1722829009 learned_addr learned my addr 172.21.15.61:0/1722829009 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:46:59.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae58f59700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae540ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502217933 0 0) 0x7fae540acee0 con 0x7fae540b0120 2022-07-12T08:46:59.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae38003260 con 0x7fae540b0120 2022-07-12T08:46:59.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1929850754 0 0) 0x7fae38003260 con 0x7fae540b0120 2022-07-12T08:46:59.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 msgr2=0x7fae540ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:59.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae540ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:59.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 >> v1:172.21.15.111:6789/0 conn(0x7fae5413ec70 legacy=0x7fae5413f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:59.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae540acca0 con 0x7fae540b0120 2022-07-12T08:46:59.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fae40001660 con 0x7fae540b0120 2022-07-12T08:46:59.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 >> v1:172.21.15.61:6789/0 conn(0x7fae540b0120 legacy=0x7fae540ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:59.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae38003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae38003c70 0x7fae38004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae380049f0 0x7fae38004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae540acb60 con 0x7fae380049f0 2022-07-12T08:46:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3800b750 con 0x7fae38003c70 2022-07-12T08:46:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3800b910 con 0x7fae540b1320 2022-07-12T08:46:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae380049f0 0x7fae38004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae53fff700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae38003c70 0x7fae38004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae58f59700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae38003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 -- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 msgr2=0x7fae38003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:59.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae38003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:59.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 -- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae38003c70 msgr2=0x7fae38004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:46:59.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae38003c70 0x7fae38004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:59.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 -- 172.21.15.61:0/1722829009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38003260 con 0x7fae380049f0 2022-07-12T08:46:59.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae58f59700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae540b1320 0x7fae38003780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:46:59.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae380049f0 0x7fae38004e70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fae4c0013a0 tx=0x7fae4c007d60).ready entity=mon.0 client_cookie=c0b2c6512f2c1c90 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:59.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae4c0016c0 con 0x7fae380049f0 2022-07-12T08:46:59.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c006890 con 0x7fae380049f0 2022-07-12T08:46:59.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae537fe700 1 -- 172.21.15.61:0/1722829009 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae4c006bf0 con 0x7fae380049f0 2022-07-12T08:46:59.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- 172.21.15.61:0/1722829009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae380049f0 msgr2=0x7fae38004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:59.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 --2- 172.21.15.61:0/1722829009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae380049f0 0x7fae38004e70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7fae4c0013a0 tx=0x7fae4c007d60).stop 2022-07-12T08:46:59.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae5975a700 1 -- 172.21.15.61:0/1722829009 reap_dead start 2022-07-12T08:46:59.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- 172.21.15.61:0/1722829009 shutdown_connections 2022-07-12T08:46:59.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.076+0000 7fae59f5b700 1 -- 172.21.15.61:0/1722829009 >> 172.21.15.61:0/1722829009 conn(0x7fae540b5780 msgr2=0x7fae540b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:46:59.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- 172.21.15.61:0/1722829009 shutdown_connections 2022-07-12T08:46:59.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- 172.21.15.61:0/1722829009 wait complete. 2022-07-12T08:46:59.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 Processor -- start 2022-07-12T08:46:59.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- start start 2022-07-12T08:46:59.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae5410ecb0 0x7fae540641d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 0x7fae540646c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae54066f10 0x7fae54064bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5410f8d0 con 0x7fae54066000 2022-07-12T08:46:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5414e330 con 0x7fae5410ecb0 2022-07-12T08:46:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5410f090 con 0x7fae54066f10 2022-07-12T08:46:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 0x7fae540646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 0x7fae540646c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57026/0 (socket says 172.21.15.61:57026) 2022-07-12T08:46:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 -- 172.21.15.61:0/2406762784 learned_addr learned my addr 172.21.15.61:0/2406762784 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:46:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae58f59700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae5410ecb0 0x7fae540641d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae5975a700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae54066f10 0x7fae54064bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 -- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae54066f10 msgr2=0x7fae54064bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae54066f10 0x7fae54064bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 -- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae5410ecb0 msgr2=0x7fae540641d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:46:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae5410ecb0 0x7fae540641d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:46:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 -- 172.21.15.61:0/2406762784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae540674d0 con 0x7fae54066000 2022-07-12T08:46:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae53fff700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 0x7fae540646c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fae5410f450 tx=0x7fae540afcf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae40015630 con 0x7fae54066000 2022-07-12T08:46:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae40010040 con 0x7fae54066000 2022-07-12T08:46:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae540664a0 con 0x7fae54066000 2022-07-12T08:46:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae40015980 con 0x7fae54066000 2022-07-12T08:46:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.080+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5414ea80 con 0x7fae54066000 2022-07-12T08:46:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.084+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae54057460 con 0x7fae54066000 2022-07-12T08:46:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.084+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fae40015ae0 con 0x7fae54066000 2022-07-12T08:46:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.084+0000 7fae51ffb700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fae340539b0 0x7fae34055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:46:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.084+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fae4006b4e0 con 0x7fae54066000 2022-07-12T08:46:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.084+0000 7fae58f59700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fae340539b0 0x7fae34055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:46:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.084+0000 7fae58f59700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fae340539b0 0x7fae34055e70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fae4800b000 tx=0x7fae48008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:46:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.088+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae40042720 con 0x7fae54066000 2022-07-12T08:46:59.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2022-07-12T08:46:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:46:59.228+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fae540acca0 con 0x7fae54066000 2022-07-12T08:47:00.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae51ffb700 1 -- 172.21.15.61:0/2406762784 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7fae540acca0 con 0x7fae54066000 2022-07-12T08:47:00.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fae340539b0 msgr2=0x7fae34055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fae340539b0 0x7fae34055e70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fae4800b000 tx=0x7fae48008040).stop 2022-07-12T08:47:00.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 msgr2=0x7fae540646c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 0x7fae540646c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fae5410f450 tx=0x7fae540afcf0).stop 2022-07-12T08:47:00.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 shutdown_connections 2022-07-12T08:47:00.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fae54066f10 0x7fae54064bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae54066000 0x7fae540646c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fae340539b0 0x7fae34055e70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 --2- 172.21.15.61:0/2406762784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae5410ecb0 0x7fae540641d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 >> 172.21.15.61:0/2406762784 conn(0x7fae540b5780 msgr2=0x7fae540603e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:00.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 shutdown_connections 2022-07-12T08:47:00.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.232+0000 7fae59f5b700 1 -- 172.21.15.61:0/2406762784 wait complete. 2022-07-12T08:47:00.241 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:47:00.250 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:47:00.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:47:00.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 Processor -- start 2022-07-12T08:47:00.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 -- start start 2022-07-12T08:47:00.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4f28067130 0x7f4f28144810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f280aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f280b17d0 con 0x7f4f280b01b0 2022-07-12T08:47:00.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f28141210 con 0x7f4f28067130 2022-07-12T08:47:00.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2e3b1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28144110 con 0x7f4f280ac350 2022-07-12T08:47:00.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f27fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4f280b01b0 0x7f4f28140b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45252/0 (socket says 172.21.15.61:45252) 2022-07-12T08:47:00.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f27fff700 1 -- 172.21.15.61:0/717888847 learned_addr learned my addr 172.21.15.61:0/717888847 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:00.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f277fe700 1 --2- 172.21.15.61:0/717888847 >> v2:172.21.15.61:3300/0 conn(0x7f4f28067130 0x7f4f28144810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f2c94e700 1 --2- 172.21.15.61:0/717888847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f280aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f277fe700 1 -- 172.21.15.61:0/717888847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 msgr2=0x7f4f280aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:00.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f277fe700 1 --2- 172.21.15.61:0/717888847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f280aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f277fe700 1 -- 172.21.15.61:0/717888847 >> v1:172.21.15.61:6789/0 conn(0x7f4f280b01b0 legacy=0x7f4f28140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f26ffd700 1 -- 172.21.15.61:0/717888847 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519564950 0 0) 0x7f4f280b17d0 con 0x7f4f280b01b0 2022-07-12T08:47:00.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f277fe700 1 -- 172.21.15.61:0/717888847 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f280b0590 con 0x7f4f28067130 2022-07-12T08:47:00.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f277fe700 1 --2- 172.21.15.61:0/717888847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f28144810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f4f10002f70 tx=0x7f4f10006df0).ready entity=mon.0 client_cookie=fb1ab7fa41e2319e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f26ffd700 1 -- 172.21.15.61:0/717888847 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f1000d570 con 0x7f4f28067130 2022-07-12T08:47:00.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f26ffd700 1 -- 172.21.15.61:0/717888847 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f10000e90 con 0x7f4f28067130 2022-07-12T08:47:00.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.356+0000 7f4f26ffd700 1 -- 172.21.15.61:0/717888847 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f1000d8c0 con 0x7f4f28067130 2022-07-12T08:47:00.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/717888847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 msgr2=0x7f4f28144810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/717888847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f28144810 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f4f10002f70 tx=0x7f4f10006df0).stop 2022-07-12T08:47:00.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/717888847 shutdown_connections 2022-07-12T08:47:00.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/717888847 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f280aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/717888847 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f28144810 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/717888847 >> 172.21.15.61:0/717888847 conn(0x7f4f280b54d0 msgr2=0x7f4f280b58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:00.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/717888847 shutdown_connections 2022-07-12T08:47:00.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/717888847 wait complete. 2022-07-12T08:47:00.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.360+0000 7f4f2e3b1700 1 Processor -- start 2022-07-12T08:47:00.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 -- start start 2022-07-12T08:47:00.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f2809e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f28064250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f280b01b0 0x7f4f280a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28066950 con 0x7f4f28067130 2022-07-12T08:47:00.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28066bf0 con 0x7f4f280b01b0 2022-07-12T08:47:00.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28140260 con 0x7f4f280ac350 2022-07-12T08:47:00.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f2809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f2809e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57040/0 (socket says 172.21.15.61:57040) 2022-07-12T08:47:00.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 -- 172.21.15.61:0/982527572 learned_addr learned my addr 172.21.15.61:0/982527572 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:00.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2c94e700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f280b01b0 0x7f4f280a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f277fe700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f28064250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 -- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 msgr2=0x7f4f28064250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f28064250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 -- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f280b01b0 msgr2=0x7f4f280a9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f280b01b0 0x7f4f280a9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 -- 172.21.15.61:0/982527572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1000b180 con 0x7f4f28067130 2022-07-12T08:47:00.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f27fff700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f2809e440 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f4f18001360 tx=0x7f4f18001490).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f18005ba0 con 0x7f4f28067130 2022-07-12T08:47:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f18005d00 con 0x7f4f28067130 2022-07-12T08:47:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2813f950 con 0x7f4f28067130 2022-07-12T08:47:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f180072c0 con 0x7f4f28067130 2022-07-12T08:47:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.364+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2813ff70 con 0x7f4f28067130 2022-07-12T08:47:00.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.368+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4f1801a060 con 0x7f4f28067130 2022-07-12T08:47:00.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.368+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f28066db0 con 0x7f4f28067130 2022-07-12T08:47:00.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.368+0000 7f4f257fa700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4f0c053af0 0x7f4f0c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.368+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4f1806b740 con 0x7f4f28067130 2022-07-12T08:47:00.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.368+0000 7f4f277fe700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4f0c053af0 0x7f4f0c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.368+0000 7f4f277fe700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4f0c053af0 0x7f4f0c055fb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4f1001a000 tx=0x7f4f10014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.372+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f18042840 con 0x7f4f28067130 2022-07-12T08:47:00.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.544+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4f28057460 con 0x7f4f28067130 2022-07-12T08:47:00.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.544+0000 7f4f257fa700 1 -- 172.21.15.61:0/982527572 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f4f18018030 con 0x7f4f28067130 2022-07-12T08:47:00.548 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:00.548 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:47:00.548 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:47:00.549 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.549 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:47:00.549 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:47:00.549 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:47:00.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.550 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.550 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.551 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:47:00.551 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:47:00.552 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:47:00.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.553 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.553 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.554 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:47:00.554 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:47:00.554 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.554 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.555 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.555 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.555 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.555 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.555 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.556 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:47:00.556 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:47:00.556 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.557 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.557 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.557 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.557 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.557 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.558 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:47:00.558 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:47:00.558 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.558 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.559 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.559 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.559 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.559 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.559 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.560 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:47:00.560 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:47:00.560 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.561 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.561 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.561 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.561 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.561 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.562 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:47:00.562 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:47:00.562 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.563 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.563 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.563 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.563 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.563 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.564 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.564 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:47:00.564 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:47:00.564 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.564 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.565 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.565 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.565 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.565 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.566 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.566 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:47:00.568 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:47:00.569 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.569 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.570 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.570 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.572 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.572 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.579 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.579 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:47:00.580 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:47:00.580 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.581 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.582 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.582 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.583 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.583 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:47:00.584 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:47:00.584 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.584 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:47:00.585 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:47:00.585 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:47:00.585 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.585 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.585 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.586 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:47:00.586 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:47:00.586 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.586 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:47:00.587 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:47:00.587 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:47:00.587 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.588 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.588 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.588 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:47:00.589 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:47:00.589 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.589 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:47:00.589 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:47:00.590 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:47:00.590 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.591 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.591 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.591 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:47:00.591 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:47:00.592 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:47:00.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:47:00.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:47:00.593 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.593 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.593 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.594 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.594 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:47:00.594 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:47:00.595 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.595 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:47:00.595 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:47:00.595 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.596 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.596 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:00.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:00.596 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:47:00.597 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:47:00.597 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:47:00.597 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:47:00.597 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:47:00.597 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:47:00.598 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.598 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:00.598 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:00.598 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:47:00.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4f0c053af0 msgr2=0x7f4f0c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4f0c053af0 0x7f4f0c055fb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f4f1001a000 tx=0x7f4f10014000).stop 2022-07-12T08:47:00.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 msgr2=0x7f4f2809e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f2809e440 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f4f18001360 tx=0x7f4f18001490).stop 2022-07-12T08:47:00.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 shutdown_connections 2022-07-12T08:47:00.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4f0c053af0 0x7f4f0c055fb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f280b01b0 0x7f4f280a9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4f280ac350 0x7f4f28064250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 --2- 172.21.15.61:0/982527572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f28067130 0x7f4f2809e440 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 >> 172.21.15.61:0/982527572 conn(0x7f4f280b54d0 msgr2=0x7f4f280b58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:00.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 shutdown_connections 2022-07-12T08:47:00.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.548+0000 7f4f2e3b1700 1 -- 172.21.15.61:0/982527572 wait complete. 2022-07-12T08:47:00.606 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:00.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 Processor -- start 2022-07-12T08:47:00.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- start start 2022-07-12T08:47:00.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f842c0b2ee0 0x7f842c0b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f842c0b0b10 0x7f842c1403d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842c0b32c0 con 0x7f842c1408c0 2022-07-12T08:47:00.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842c060b50 con 0x7f842c0b2ee0 2022-07-12T08:47:00.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f842c0a93f0 con 0x7f842c0b0b10 2022-07-12T08:47:00.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8431ff4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f842c0b2ee0 0x7f842c0b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f84317f3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f842c0b0b10 0x7f842c1403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8431ff4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f842c0b2ee0 0x7f842c0b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57050/0 (socket says 172.21.15.61:57050) 2022-07-12T08:47:00.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8431ff4700 1 -- 172.21.15.61:0/2113438981 learned_addr learned my addr 172.21.15.61:0/2113438981 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:00.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f84317f3700 1 -- 172.21.15.61:0/2113438981 >> v2:172.21.15.61:3300/0 conn(0x7f842c0b2ee0 msgr2=0x7f842c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:00.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f84317f3700 1 --2- 172.21.15.61:0/2113438981 >> v2:172.21.15.61:3300/0 conn(0x7f842c0b2ee0 0x7f842c0b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f84317f3700 1 -- 172.21.15.61:0/2113438981 >> v1:172.21.15.111:6789/0 conn(0x7f842c1408c0 legacy=0x7f842c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f84317f3700 1 -- 172.21.15.61:0/2113438981 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c0a9690 con 0x7f842c0b0b10 2022-07-12T08:47:00.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f84317f3700 1 --2- 172.21.15.61:0/2113438981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c1403d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f84240032f0 tx=0x7f84240091d0).ready entity=mon.1 client_cookie=408e350aee40fd59 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8430ff2700 1 -- 172.21.15.61:0/2113438981 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f842400b680 con 0x7f842c0b0b10 2022-07-12T08:47:00.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8430ff2700 1 -- 172.21.15.61:0/2113438981 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842400b7e0 con 0x7f842c0b0b10 2022-07-12T08:47:00.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8430ff2700 1 -- 172.21.15.61:0/2113438981 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f842400b950 con 0x7f842c0b0b10 2022-07-12T08:47:00.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- 172.21.15.61:0/2113438981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 msgr2=0x7f842c1403d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 --2- 172.21.15.61:0/2113438981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c1403d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f84240032f0 tx=0x7f84240091d0).stop 2022-07-12T08:47:00.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- 172.21.15.61:0/2113438981 shutdown_connections 2022-07-12T08:47:00.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 --2- 172.21.15.61:0/2113438981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c1403d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 --2- 172.21.15.61:0/2113438981 >> v2:172.21.15.61:3300/0 conn(0x7f842c0b2ee0 0x7f842c0b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- 172.21.15.61:0/2113438981 >> 172.21.15.61:0/2113438981 conn(0x7f842c063ab0 msgr2=0x7f842c063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:00.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- 172.21.15.61:0/2113438981 shutdown_connections 2022-07-12T08:47:00.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.712+0000 7f8434258700 1 -- 172.21.15.61:0/2113438981 wait complete. 2022-07-12T08:47:00.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 Processor -- start 2022-07-12T08:47:00.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 -- start start 2022-07-12T08:47:00.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c0b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 0x7f842c0ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f842c1408c0 0x7f842c0ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c066690 con 0x7f842c0b2ee0 2022-07-12T08:47:00.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842c067760 con 0x7f842c0b0b10 2022-07-12T08:47:00.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8434258700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842c065660 con 0x7f842c1408c0 2022-07-12T08:47:00.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 0x7f842c0ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8431ff4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c0b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 0x7f842c0ac480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57056/0 (socket says 172.21.15.61:57056) 2022-07-12T08:47:00.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 -- 172.21.15.61:0/3483206636 learned_addr learned my addr 172.21.15.61:0/3483206636 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:00.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84327f5700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f842c1408c0 0x7f842c0ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 -- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f842c1408c0 msgr2=0x7f842c0ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f842c1408c0 0x7f842c0ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 -- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 msgr2=0x7f842c0b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c0b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 -- 172.21.15.61:0/3483206636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8424009470 con 0x7f842c0b2ee0 2022-07-12T08:47:00.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f84317f3700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 0x7f842c0ac480 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f84240032f0 tx=0x7f8424016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.716+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8424011e50 con 0x7f842c0b2ee0 2022-07-12T08:47:00.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8424021440 con 0x7f842c0b2ee0 2022-07-12T08:47:00.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f842c065900 con 0x7f842c0b2ee0 2022-07-12T08:47:00.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84240217a0 con 0x7f842c0b2ee0 2022-07-12T08:47:00.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f842c0673b0 con 0x7f842c0b2ee0 2022-07-12T08:47:00.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8424029070 con 0x7f842c0b2ee0 2022-07-12T08:47:00.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f842c057460 con 0x7f842c0b2ee0 2022-07-12T08:47:00.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8422ffd700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f840c053a40 0x7f840c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f842406cbb0 con 0x7f842c0b2ee0 2022-07-12T08:47:00.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8431ff4700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f840c053a40 0x7f840c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.720+0000 7f8431ff4700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f840c053a40 0x7f840c055f00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f842800d000 tx=0x7f8428006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.728+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8424073080 con 0x7f842c0b2ee0 2022-07-12T08:47:00.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.856+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f842c0679a0 con 0x7f842c0b2ee0 2022-07-12T08:47:00.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8422ffd700 1 -- 172.21.15.61:0/3483206636 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+13230 (secure 0 0 0) 0x7f8424015070 con 0x7f842c0b2ee0 2022-07-12T08:47:00.861 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:00.862 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":67,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:00.223388+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:00.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f840c053a40 msgr2=0x7f840c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f840c053a40 0x7f840c055f00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f842800d000 tx=0x7f8428006f40).stop 2022-07-12T08:47:00.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 msgr2=0x7f842c0ac480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 0x7f842c0ac480 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f84240032f0 tx=0x7f8424016000).stop 2022-07-12T08:47:00.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 shutdown_connections 2022-07-12T08:47:00.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f842c1408c0 0x7f842c0ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f842c0b2ee0 0x7f842c0ac480 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f840c053a40 0x7f840c055f00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 --2- 172.21.15.61:0/3483206636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f842c0b0b10 0x7f842c0b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 >> 172.21.15.61:0/3483206636 conn(0x7f842c063ab0 msgr2=0x7f842c0b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:00.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 shutdown_connections 2022-07-12T08:47:00.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.860+0000 7f8434258700 1 -- 172.21.15.61:0/3483206636 wait complete. 2022-07-12T08:47:00.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:47:00.877 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:47:00.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 Processor -- start 2022-07-12T08:47:00.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 -- start start 2022-07-12T08:47:00.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c14b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f841c0b1000 0x7f841c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:00.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841c099010 con 0x7f841c0afdc0 2022-07-12T08:47:00.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841c060ff0 con 0x7f841c0b1000 2022-07-12T08:47:00.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f84209e8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c14a930 con 0x7f841c060c10 2022-07-12T08:47:00.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f841c0b1000 0x7f841c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f841c0b1000 0x7f841c0af8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57066/0 (socket says 172.21.15.61:57066) 2022-07-12T08:47:00.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 -- 172.21.15.61:0/339922791 learned_addr learned my addr 172.21.15.61:0/339922791 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:00.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841ad9d700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c14b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:00.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 -- 172.21.15.61:0/339922791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 msgr2=0x7f841c14b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:00.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c14b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 -- 172.21.15.61:0/339922791 >> v1:172.21.15.111:6789/0 conn(0x7f841c0afdc0 legacy=0x7f841c09a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 -- 172.21.15.61:0/339922791 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841c14abd0 con 0x7f841c0b1000 2022-07-12T08:47:00.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841ad9d700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c14b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:47:00.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f841a59c700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c0af8d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f840c002f70 tx=0x7f840c006eb0).ready entity=mon.0 client_cookie=4e9adb16f6e170e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:00.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f8419d9b700 1 -- 172.21.15.61:0/339922791 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f840c00d5c0 con 0x7f841c0b1000 2022-07-12T08:47:00.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f8419d9b700 1 -- 172.21.15.61:0/339922791 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840c000e90 con 0x7f841c0b1000 2022-07-12T08:47:00.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.984+0000 7f8419d9b700 1 -- 172.21.15.61:0/339922791 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f840c00d950 con 0x7f841c0b1000 2022-07-12T08:47:00.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 -- 172.21.15.61:0/339922791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 msgr2=0x7f841c0af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:00.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c0af8d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f840c002f70 tx=0x7f840c006eb0).stop 2022-07-12T08:47:00.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 -- 172.21.15.61:0/339922791 shutdown_connections 2022-07-12T08:47:00.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c0af8d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 --2- 172.21.15.61:0/339922791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c14b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:00.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 -- 172.21.15.61:0/339922791 >> 172.21.15.61:0/339922791 conn(0x7f841c05bd10 msgr2=0x7f841c05c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:00.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 -- 172.21.15.61:0/339922791 shutdown_connections 2022-07-12T08:47:00.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 -- 172.21.15.61:0/339922791 wait complete. 2022-07-12T08:47:01.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 Processor -- start 2022-07-12T08:47:01.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.988+0000 7f84209e8700 1 -- start start 2022-07-12T08:47:01.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f841c0afdc0 0x7f841c09a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c066c70 con 0x7f841c0b1000 2022-07-12T08:47:01.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841c064c80 con 0x7f841c0afdc0 2022-07-12T08:47:01.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841c064f20 con 0x7f841c060c10 2022-07-12T08:47:01.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841ad9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841ad9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39264/0 (socket says 172.21.15.61:39264) 2022-07-12T08:47:01.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57072/0 (socket says 172.21.15.61:57072) 2022-07-12T08:47:01.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841ad9d700 1 -- 172.21.15.61:0/3657876331 learned_addr learned my addr 172.21.15.61:0/3657876331 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:01.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841a59c700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f841c0afdc0 0x7f841c09a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 -- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 msgr2=0x7f841c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 -- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f841c0afdc0 msgr2=0x7f841c09a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f841c0afdc0 0x7f841c09a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 -- 172.21.15.61:0/3657876331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c00b1e0 con 0x7f841c0b1000 2022-07-12T08:47:01.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f841b59e700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c159bd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f841c062570 tx=0x7f8414007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f841400f470 con 0x7f841c0b1000 2022-07-12T08:47:01.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841400f5d0 con 0x7f841c0b1000 2022-07-12T08:47:01.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841c159730 con 0x7f841c0b1000 2022-07-12T08:47:01.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f841400f930 con 0x7f841c0b1000 2022-07-12T08:47:01.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.992+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841c0645b0 con 0x7f841c0b1000 2022-07-12T08:47:01.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.996+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841c1598c0 con 0x7f841c0b1000 2022-07-12T08:47:01.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.996+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8414010070 con 0x7f841c0b1000 2022-07-12T08:47:01.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.996+0000 7f840bfff700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83f80539b0 0x7f83f8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.996+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f841406b0e0 con 0x7f841c0b1000 2022-07-12T08:47:01.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.996+0000 7f841ad9d700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83f80539b0 0x7f83f8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:00.996+0000 7f841ad9d700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83f80539b0 0x7f83f8055e70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f841000d000 tx=0x7f8410008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.004+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8414042260 con 0x7f841c0b1000 2022-07-12T08:47:01.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.128+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f841c057460 con 0x7f841c0b1000 2022-07-12T08:47:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.128+0000 7f840bfff700 1 -- 172.21.15.61:0/3657876331 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v67) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8414045880 con 0x7f841c0b1000 2022-07-12T08:47:01.132 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:47:01.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83f80539b0 msgr2=0x7f83f8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83f80539b0 0x7f83f8055e70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f841000d000 tx=0x7f8410008040).stop 2022-07-12T08:47:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 msgr2=0x7f841c159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c159bd0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f841c062570 tx=0x7f8414007eb0).stop 2022-07-12T08:47:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 shutdown_connections 2022-07-12T08:47:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f841c0b1000 0x7f841c159bd0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f83f80539b0 0x7f83f8055e70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f841c0afdc0 0x7f841c09a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 --2- 172.21.15.61:0/3657876331 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f841c060c10 0x7f841c157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 >> 172.21.15.61:0/3657876331 conn(0x7f841c05bd10 msgr2=0x7f841c0b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:01.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 shutdown_connections 2022-07-12T08:47:01.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.132+0000 7f84209e8700 1 -- 172.21.15.61:0/3657876331 wait complete. 2022-07-12T08:47:01.146 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:47:01.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 Processor -- start 2022-07-12T08:47:01.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 -- start start 2022-07-12T08:47:01.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcc340b79e0 0x7fcc340b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcc34140840 0x7fcc341443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc340b1040 con 0x7fcc340b2180 2022-07-12T08:47:01.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc340663a0 con 0x7fcc340b79e0 2022-07-12T08:47:01.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3c5c8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc340b1df0 con 0x7fcc34140840 2022-07-12T08:47:01.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc3a364700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcc340b79e0 0x7fcc340b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc39b63700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcc340b2180 0x7fcc34140130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45296/0 (socket says 172.21.15.61:45296) 2022-07-12T08:47:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.248+0000 7fcc39b63700 1 -- 172.21.15.61:0/3889249938 learned_addr learned my addr 172.21.15.61:0/3889249938 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3ab65700 1 --2- 172.21.15.61:0/3889249938 >> v2:172.21.15.111:3300/0 conn(0x7fcc34140840 0x7fcc341443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3a364700 1 -- 172.21.15.61:0/3889249938 >> v2:172.21.15.111:3300/0 conn(0x7fcc34140840 msgr2=0x7fcc341443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3a364700 1 --2- 172.21.15.61:0/3889249938 >> v2:172.21.15.111:3300/0 conn(0x7fcc34140840 0x7fcc341443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3a364700 1 -- 172.21.15.61:0/3889249938 >> v1:172.21.15.61:6789/0 conn(0x7fcc340b2180 legacy=0x7fcc34140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3a364700 1 -- 172.21.15.61:0/3889249938 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc340b1270 con 0x7fcc340b79e0 2022-07-12T08:47:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3a364700 1 --2- 172.21.15.61:0/3889249938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc340b7dc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7fcc340ab060 tx=0x7fcc240091a0).ready entity=mon.0 client_cookie=2e00b71929e76ab6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39362700 1 -- 172.21.15.61:0/3889249938 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc24009ea0 con 0x7fcc340b79e0 2022-07-12T08:47:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39362700 1 -- 172.21.15.61:0/3889249938 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2400b830 con 0x7fcc340b79e0 2022-07-12T08:47:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39362700 1 -- 172.21.15.61:0/3889249938 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc2400b9a0 con 0x7fcc340b79e0 2022-07-12T08:47:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/3889249938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 msgr2=0x7fcc340b7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/3889249938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc340b7dc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7fcc340ab060 tx=0x7fcc240091a0).stop 2022-07-12T08:47:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/3889249938 shutdown_connections 2022-07-12T08:47:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/3889249938 >> v2:172.21.15.111:3300/0 conn(0x7fcc34140840 0x7fcc341443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/3889249938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc340b7dc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/3889249938 >> 172.21.15.61:0/3889249938 conn(0x7fcc340656e0 msgr2=0x7fcc340b72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/3889249938 shutdown_connections 2022-07-12T08:47:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/3889249938 wait complete. 2022-07-12T08:47:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 Processor -- start 2022-07-12T08:47:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- start start 2022-07-12T08:47:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcc340b2180 0x7fcc341427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc34151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc34140840 0x7fcc34155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc34145440 con 0x7fcc340b79e0 2022-07-12T08:47:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc341554e0 con 0x7fcc34140840 2022-07-12T08:47:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3c5c8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340b0760 con 0x7fcc340b2180 2022-07-12T08:47:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc34151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc34151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57082/0 (socket says 172.21.15.61:57082) 2022-07-12T08:47:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 -- 172.21.15.61:0/2502727202 learned_addr learned my addr 172.21.15.61:0/2502727202 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3ab65700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc34140840 0x7fcc34155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3a364700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcc340b2180 0x7fcc341427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 -- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcc340b2180 msgr2=0x7fcc341427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcc340b2180 0x7fcc341427a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 -- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc34140840 msgr2=0x7fcc34155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc34140840 0x7fcc34155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc39b63700 1 -- 172.21.15.61:0/2502727202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24009420 con 0x7fcc340b79e0 2022-07-12T08:47:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.252+0000 7fcc3ab65700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc34140840 0x7fcc34155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc39b63700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc34151e60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7fcc34067000 tx=0x7fcc28007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc2800ce30 con 0x7fcc340b79e0 2022-07-12T08:47:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc280072e0 con 0x7fcc340b79e0 2022-07-12T08:47:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc34155710 con 0x7fcc340b79e0 2022-07-12T08:47:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc28005800 con 0x7fcc340b79e0 2022-07-12T08:47:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc340b0a00 con 0x7fcc340b79e0 2022-07-12T08:47:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcc28010070 con 0x7fcc340b79e0 2022-07-12T08:47:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc341558a0 con 0x7fcc340b79e0 2022-07-12T08:47:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc337fe700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcc14053af0 0x7fcc14055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcc2806b1f0 con 0x7fcc340b79e0 2022-07-12T08:47:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc3a364700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcc14053af0 0x7fcc14055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.256+0000 7fcc3a364700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcc14053af0 0x7fcc14055fb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fcc24019000 tx=0x7fcc24016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.264+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcc28042400 con 0x7fcc340b79e0 2022-07-12T08:47:01.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.388+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcc34057460 con 0x7fcc340b79e0 2022-07-12T08:47:01.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.388+0000 7fcc337fe700 1 -- 172.21.15.61:0/2502727202 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v67) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcc28045a20 con 0x7fcc340b79e0 2022-07-12T08:47:01.393 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:01.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcc14053af0 msgr2=0x7fcc14055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcc14053af0 0x7fcc14055fb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fcc24019000 tx=0x7fcc24016000).stop 2022-07-12T08:47:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 msgr2=0x7fcc34151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc34151e60 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7fcc34067000 tx=0x7fcc28007c40).stop 2022-07-12T08:47:01.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 shutdown_connections 2022-07-12T08:47:01.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcc14053af0 0x7fcc14055fb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc34140840 0x7fcc34155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcc340b79e0 0x7fcc34151e60 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 --2- 172.21.15.61:0/2502727202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcc340b2180 0x7fcc341427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 >> 172.21.15.61:0/2502727202 conn(0x7fcc340656e0 msgr2=0x7fcc341485a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:01.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 shutdown_connections 2022-07-12T08:47:01.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.392+0000 7fcc3c5c8700 1 -- 172.21.15.61:0/2502727202 wait complete. 2022-07-12T08:47:01.406 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:47:01.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 Processor -- start 2022-07-12T08:47:01.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 -- start start 2022-07-12T08:47:01.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5b40b6780 0x7fb5b40b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb5b40b25c0 0x7fb5b413f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b41436b0 0x7fb5b4065a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b40668f0 con 0x7fb5b40b6780 2022-07-12T08:47:01.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5b4142f70 con 0x7fb5b40b25c0 2022-07-12T08:47:01.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.504+0000 7fb5b95b0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b4143210 con 0x7fb5b41436b0 2022-07-12T08:47:01.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5b40b6780 0x7fb5b40b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5b40b6780 0x7fb5b40b6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57090/0 (socket says 172.21.15.61:57090) 2022-07-12T08:47:01.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2251524402 learned_addr learned my addr 172.21.15.61:0/2251524402 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:01.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b37fe700 1 --2- 172.21.15.61:0/2251524402 >> v2:172.21.15.111:3300/0 conn(0x7fb5b40b25c0 0x7fb5b413f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b8daf700 1 --2- 172.21.15.61:0/2251524402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b41436b0 0x7fb5b4065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2251524402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b41436b0 msgr2=0x7fb5b4065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:01.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 --2- 172.21.15.61:0/2251524402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b41436b0 0x7fb5b4065a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2251524402 >> v2:172.21.15.111:3300/0 conn(0x7fb5b40b25c0 msgr2=0x7fb5b413f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:01.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 --2- 172.21.15.61:0/2251524402 >> v2:172.21.15.111:3300/0 conn(0x7fb5b40b25c0 0x7fb5b413f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2251524402 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b40607a0 con 0x7fb5b40b6780 2022-07-12T08:47:01.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b37fe700 1 --2- 172.21.15.61:0/2251524402 >> v2:172.21.15.111:3300/0 conn(0x7fb5b40b25c0 0x7fb5b413f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:01.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b3fff700 1 --2- 172.21.15.61:0/2251524402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b6780 0x7fb5b40b6b60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fb5b4143590 tx=0x7fb5a4009290).ready entity=mon.0 client_cookie=2fb770eea9d164af server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b2ffd700 1 -- 172.21.15.61:0/2251524402 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a400b860 con 0x7fb5b40b6780 2022-07-12T08:47:01.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b2ffd700 1 -- 172.21.15.61:0/2251524402 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a400b9c0 con 0x7fb5b40b6780 2022-07-12T08:47:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b2ffd700 1 -- 172.21.15.61:0/2251524402 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a400bb30 con 0x7fb5b40b6780 2022-07-12T08:47:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2251524402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b6780 msgr2=0x7fb5b40b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2251524402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b6780 0x7fb5b40b6b60 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fb5b4143590 tx=0x7fb5a4009290).stop 2022-07-12T08:47:01.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2251524402 shutdown_connections 2022-07-12T08:47:01.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2251524402 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b41436b0 0x7fb5b4065a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2251524402 >> v2:172.21.15.111:3300/0 conn(0x7fb5b40b25c0 0x7fb5b413f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2251524402 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b6780 0x7fb5b40b6b60 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2251524402 >> 172.21.15.61:0/2251524402 conn(0x7fb5b4064570 msgr2=0x7fb5b40b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:01.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2251524402 shutdown_connections 2022-07-12T08:47:01.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2251524402 wait complete. 2022-07-12T08:47:01.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 Processor -- start 2022-07-12T08:47:01.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.508+0000 7fb5b95b0700 1 -- start start 2022-07-12T08:47:01.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 0x7fb5b4128e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b40b6780 0x7fb5b4152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41436b0 0x7fb5b4152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b413e5a0 con 0x7fb5b40b25c0 2022-07-12T08:47:01.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b413e840 con 0x7fb5b41436b0 2022-07-12T08:47:01.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b413eae0 con 0x7fb5b40b6780 2022-07-12T08:47:01.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 0x7fb5b4128e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 0x7fb5b4128e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57096/0 (socket says 172.21.15.61:57096) 2022-07-12T08:47:01.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2391232408 learned_addr learned my addr 172.21.15.61:0/2391232408 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:01.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b37fe700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b40b6780 0x7fb5b4152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b8daf700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41436b0 0x7fb5b4152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b40b6780 msgr2=0x7fb5b4152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b40b6780 0x7fb5b4152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41436b0 msgr2=0x7fb5b4152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41436b0 0x7fb5b4152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 -- 172.21.15.61:0/2391232408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4009530 con 0x7fb5b40b25c0 2022-07-12T08:47:01.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b8daf700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41436b0 0x7fb5b4152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:01.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b3fff700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 0x7fb5b4128e00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fb5b4143590 tx=0x7fb5a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a400b860 con 0x7fb5b40b25c0 2022-07-12T08:47:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a40032d0 con 0x7fb5b40b25c0 2022-07-12T08:47:01.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b405fec0 con 0x7fb5b40b25c0 2022-07-12T08:47:01.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5a4020620 con 0x7fb5b40b25c0 2022-07-12T08:47:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b4151b10 con 0x7fb5b40b25c0 2022-07-12T08:47:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b4057460 con 0x7fb5b40b25c0 2022-07-12T08:47:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb5a4026020 con 0x7fb5b40b25c0 2022-07-12T08:47:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.512+0000 7fb5b17fa700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5980539b0 0x7fb598055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.516+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb5a406ca00 con 0x7fb5b40b25c0 2022-07-12T08:47:01.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.516+0000 7fb5b37fe700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5980539b0 0x7fb598055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.516+0000 7fb5b37fe700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5980539b0 0x7fb598055e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fb59c00c000 tx=0x7fb59c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.520+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5a4043c90 con 0x7fb5b40b25c0 2022-07-12T08:47:01.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb5b4067b80 con 0x7fb5b40b25c0 2022-07-12T08:47:01.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b17fa700 1 -- 172.21.15.61:0/2391232408 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+143 (secure 0 0 0) 0x7fb5a4015070 con 0x7fb5b40b25c0 2022-07-12T08:47:01.654 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:01.654 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:47:01.655 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:01.655 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:47:01.655 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:47:01.656 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:01.657 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:01.657 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:47:01.658 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:47:01.660 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:01.660 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:47:01.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5980539b0 msgr2=0x7fb598055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5980539b0 0x7fb598055e70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fb59c00c000 tx=0x7fb59c008040).stop 2022-07-12T08:47:01.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 msgr2=0x7fb5b4128e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 0x7fb5b4128e00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fb5b4143590 tx=0x7fb5a4016000).stop 2022-07-12T08:47:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 shutdown_connections 2022-07-12T08:47:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb5980539b0 0x7fb598055e70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb5b41436b0 0x7fb5b4152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb5b40b6780 0x7fb5b4152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 --2- 172.21.15.61:0/2391232408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5b40b25c0 0x7fb5b4128e00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 >> 172.21.15.61:0/2391232408 conn(0x7fb5b4064570 msgr2=0x7fb5b40b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 shutdown_connections 2022-07-12T08:47:01.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.652+0000 7fb5b95b0700 1 -- 172.21.15.61:0/2391232408 wait complete. 2022-07-12T08:47:01.677 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:47:01.678 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:47:01.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.784+0000 7f7bf8ffd700 1 Processor -- start 2022-07-12T08:47:01.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.784+0000 7f7bf8ffd700 1 -- start start 2022-07-12T08:47:01.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7bf40b5920 0x7f7bf414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7bf405faf0 0x7f7bf405ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf405c3e0 con 0x7f7bf40b5920 2022-07-12T08:47:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf40b5d00 con 0x7f7bf405faf0 2022-07-12T08:47:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf405c610 con 0x7f7bf40b0280 2022-07-12T08:47:01.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7bf40b5920 0x7f7bf414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7bf40b5920 0x7f7bf414a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57104/0 (socket says 172.21.15.61:57104) 2022-07-12T08:47:01.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 -- 172.21.15.61:0/722900063 learned_addr learned my addr 172.21.15.61:0/722900063 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:01.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf2ffd700 1 --2- 172.21.15.61:0/722900063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf3fff700 1 --2- 172.21.15.61:0/722900063 >> v2:172.21.15.111:3300/0 conn(0x7f7bf405faf0 0x7f7bf405ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 -- 172.21.15.61:0/722900063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 msgr2=0x7f7bf414aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:01.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 --2- 172.21.15.61:0/722900063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf414aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 -- 172.21.15.61:0/722900063 >> v2:172.21.15.111:3300/0 conn(0x7f7bf405faf0 msgr2=0x7f7bf405ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:01.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 --2- 172.21.15.61:0/722900063 >> v2:172.21.15.111:3300/0 conn(0x7f7bf405faf0 0x7f7bf405ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 -- 172.21.15.61:0/722900063 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf405c8b0 con 0x7f7bf40b5920 2022-07-12T08:47:01.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf37fe700 1 --2- 172.21.15.61:0/722900063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf414a9e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f7be8002f70 tx=0x7f7be8006eb0).ready entity=mon.0 client_cookie=1caa16f8d2ec6e08 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf27fc700 1 -- 172.21.15.61:0/722900063 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be800d570 con 0x7f7bf40b5920 2022-07-12T08:47:01.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf27fc700 1 -- 172.21.15.61:0/722900063 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8000e90 con 0x7f7bf40b5920 2022-07-12T08:47:01.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf27fc700 1 -- 172.21.15.61:0/722900063 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be800d900 con 0x7f7bf40b5920 2022-07-12T08:47:01.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/722900063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 msgr2=0x7f7bf414a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/722900063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf414a9e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f7be8002f70 tx=0x7f7be8006eb0).stop 2022-07-12T08:47:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/722900063 shutdown_connections 2022-07-12T08:47:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/722900063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf414aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/722900063 >> v2:172.21.15.111:3300/0 conn(0x7f7bf405faf0 0x7f7bf405ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/722900063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf414a9e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/722900063 >> 172.21.15.61:0/722900063 conn(0x7f7bf4063340 msgr2=0x7f7bf40b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/722900063 shutdown_connections 2022-07-12T08:47:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.788+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/722900063 wait complete. 2022-07-12T08:47:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 Processor -- start 2022-07-12T08:47:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 -- start start 2022-07-12T08:47:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf405faf0 0x7f7bf4098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf40ac970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf40ace60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf414a410 con 0x7f7bf40b5920 2022-07-12T08:47:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf4149ed0 con 0x7f7bf405faf0 2022-07-12T08:47:01.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf414a170 con 0x7f7bf40b0280 2022-07-12T08:47:01.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf40ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf40ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57114/0 (socket says 172.21.15.61:57114) 2022-07-12T08:47:01.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 -- 172.21.15.61:0/2662661135 learned_addr learned my addr 172.21.15.61:0/2662661135 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:01.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf2ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf40ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf37fe700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf405faf0 0x7f7bf4098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 -- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 msgr2=0x7f7bf40ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf40ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 -- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf405faf0 msgr2=0x7f7bf4098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:01.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf405faf0 0x7f7bf4098a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:01.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 -- 172.21.15.61:0/2662661135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be800b1e0 con 0x7f7bf40b5920 2022-07-12T08:47:01.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf2ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf40ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:01.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf3fff700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf40ace60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f7bec00c000 tx=0x7f7bec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bec011750 con 0x7f7bf40b5920 2022-07-12T08:47:01.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec0118b0 con 0x7f7bf40b5920 2022-07-12T08:47:01.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf40ad350 con 0x7f7bf40b5920 2022-07-12T08:47:01.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bec011c10 con 0x7f7bf40b5920 2022-07-12T08:47:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.792+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40af180 con 0x7f7bf40b5920 2022-07-12T08:47:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.796+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf4148600 con 0x7f7bf40b5920 2022-07-12T08:47:01.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.796+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7bec011d70 con 0x7f7bf40b5920 2022-07-12T08:47:01.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.796+0000 7f7bf0ff9700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7bd40539b0 0x7f7bd4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:01.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.796+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7bec06bea0 con 0x7f7bf40b5920 2022-07-12T08:47:01.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.796+0000 7f7bf37fe700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7bd40539b0 0x7f7bd4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:01.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.796+0000 7f7bf37fe700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7bd40539b0 0x7f7bd4055e70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7be801c000 tx=0x7f7be8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:01.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.804+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7bf4148600 con 0x7f7bf40b5920 2022-07-12T08:47:01.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:01.932+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f7bf4148600 con 0x7f7bf40b5920 2022-07-12T08:47:02.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.248+0000 7f7bf0ff9700 1 -- 172.21.15.61:0/2662661135 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v68) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7bf4148600 con 0x7f7bf40b5920 2022-07-12T08:47:02.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7bd40539b0 msgr2=0x7f7bd4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:02.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7bd40539b0 0x7f7bd4055e70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7be801c000 tx=0x7f7be8015000).stop 2022-07-12T08:47:02.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 msgr2=0x7f7bf40ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:02.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf40ace60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f7bec00c000 tx=0x7f7bec006df0).stop 2022-07-12T08:47:02.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 shutdown_connections 2022-07-12T08:47:02.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bf40b5920 0x7f7bf40ace60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bf40b0280 0x7f7bf40ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7bd40539b0 0x7f7bd4055e70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.252+0000 7f7bf8ffd700 1 --2- 172.21.15.61:0/2662661135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bf405faf0 0x7f7bf4098a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.256+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 >> 172.21.15.61:0/2662661135 conn(0x7f7bf4063340 msgr2=0x7f7bf40b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:02.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.256+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 shutdown_connections 2022-07-12T08:47:02.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.256+0000 7f7bf8ffd700 1 -- 172.21.15.61:0/2662661135 wait complete. 2022-07-12T08:47:02.256 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:47:02.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:47:02.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 Processor -- start 2022-07-12T08:47:02.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 -- start start 2022-07-12T08:47:02.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b0df0 0x7f56e8146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56e8065490 0x7f56e80b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f56e80b6550 0x7f56e813fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e805f1e0 con 0x7f56e80b6550 2022-07-12T08:47:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e80b11d0 con 0x7f56e8065490 2022-07-12T08:47:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56ecdc5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e8060170 con 0x7f56e80b0df0 2022-07-12T08:47:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f56e80b6550 0x7f56e813fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b0df0 0x7f56e8146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56e8065490 0x7f56e80b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f56e80b6550 0x7f56e813fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57122/0 (socket says 172.21.15.61:57122) 2022-07-12T08:47:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e6ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56e8065490 0x7f56e80b6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:35694/0 (socket says 172.21.15.61:35694) 2022-07-12T08:47:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b0df0 0x7f56e8146140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39314/0 (socket says 172.21.15.61:39314) 2022-07-12T08:47:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 -- 172.21.15.61:0/3996193167 learned_addr learned my addr 172.21.15.61:0/3996193167 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 -- 172.21.15.61:0/3996193167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b0df0 msgr2=0x7f56e8146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:02.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 --2- 172.21.15.61:0/3996193167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b0df0 0x7f56e8146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 -- 172.21.15.61:0/3996193167 >> v2:172.21.15.111:3300/0 conn(0x7f56e8065490 msgr2=0x7f56e80b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:02.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 --2- 172.21.15.61:0/3996193167 >> v2:172.21.15.111:3300/0 conn(0x7f56e8065490 0x7f56e80b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.380+0000 7f56e7fff700 1 -- 172.21.15.61:0/3996193167 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e805f410 con 0x7f56e80b6550 2022-07-12T08:47:02.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e7fff700 1 --2- 172.21.15.61:0/3996193167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b6550 0x7f56e813fcb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f56e00032f0 tx=0x7f56e0009290).ready entity=mon.0 client_cookie=c00152262d6e866c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:02.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e67fc700 1 -- 172.21.15.61:0/3996193167 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56e000b680 con 0x7f56e80b6550 2022-07-12T08:47:02.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e67fc700 1 -- 172.21.15.61:0/3996193167 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e000b7e0 con 0x7f56e80b6550 2022-07-12T08:47:02.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e67fc700 1 -- 172.21.15.61:0/3996193167 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56e000b950 con 0x7f56e80b6550 2022-07-12T08:47:02.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3996193167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b6550 msgr2=0x7f56e813fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:02.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3996193167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b6550 0x7f56e813fcb0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f56e00032f0 tx=0x7f56e0009290).stop 2022-07-12T08:47:02.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3996193167 shutdown_connections 2022-07-12T08:47:02.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3996193167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b6550 0x7f56e813fcb0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3996193167 >> v2:172.21.15.111:3300/0 conn(0x7f56e8065490 0x7f56e80b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3996193167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b0df0 0x7f56e8146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3996193167 >> 172.21.15.61:0/3996193167 conn(0x7f56e8063ad0 msgr2=0x7f56e8063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:02.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3996193167 shutdown_connections 2022-07-12T08:47:02.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3996193167 wait complete. 2022-07-12T08:47:02.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 Processor -- start 2022-07-12T08:47:02.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- start start 2022-07-12T08:47:02.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 0x7f56e813e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b6550 0x7f56e8155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56e813fa80 0x7f56e8155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80b5930 con 0x7f56e80b0df0 2022-07-12T08:47:02.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80aeb10 con 0x7f56e813fa80 2022-07-12T08:47:02.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56ecdc5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e81552f0 con 0x7f56e80b6550 2022-07-12T08:47:02.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 0x7f56e813e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 0x7f56e813e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57124/0 (socket says 172.21.15.61:57124) 2022-07-12T08:47:02.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e77fe700 1 -- 172.21.15.61:0/3183008719 learned_addr learned my addr 172.21.15.61:0/3183008719 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:02.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e7fff700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56e813fa80 0x7f56e8155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.384+0000 7f56e6ffd700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b6550 0x7f56e8155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e77fe700 1 -- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b6550 msgr2=0x7f56e8155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e77fe700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b6550 0x7f56e8155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e77fe700 1 -- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56e813fa80 msgr2=0x7f56e8155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e77fe700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56e813fa80 0x7f56e8155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:02.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e77fe700 1 -- 172.21.15.61:0/3183008719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e0009530 con 0x7f56e80b0df0 2022-07-12T08:47:02.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e6ffd700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b6550 0x7f56e8155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:02.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e77fe700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 0x7f56e813e080 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f56d800c000 tx=0x7f56d8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:02.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56d8010780 con 0x7f56e80b0df0 2022-07-12T08:47:02.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d800b3b0 con 0x7f56e80b0df0 2022-07-12T08:47:02.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e80ae820 con 0x7f56e80b0df0 2022-07-12T08:47:02.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56d8010ad0 con 0x7f56e80b0df0 2022-07-12T08:47:02.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e80b4190 con 0x7f56e80b0df0 2022-07-12T08:47:02.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e80aecd0 con 0x7f56e80b0df0 2022-07-12T08:47:02.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f56d8010c30 con 0x7f56e80b0df0 2022-07-12T08:47:02.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e4ff9700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56c8053ad0 0x7f56c8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:02.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.388+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f56d806c050 con 0x7f56e80b0df0 2022-07-12T08:47:02.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.392+0000 7f56e6ffd700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56c8053ad0 0x7f56c8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:02.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.392+0000 7f56e6ffd700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56c8053ad0 0x7f56c8055f90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f56dc00b000 tx=0x7f56dc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:02.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.400+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56d80430e0 con 0x7f56e80b0df0 2022-07-12T08:47:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:02.528+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f56e8057460 con 0x7f56e80b0df0 2022-07-12T08:47:02.972 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:02.981 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:02.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:02.993 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:02.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:03.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.064+0000 7f56e4ff9700 1 -- 172.21.15.61:0/3183008719 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v69) v1 ==== 125+0+0 (secure 0 0 0) 0x7f56d8046700 con 0x7f56e80b0df0 2022-07-12T08:47:03.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56c8053ad0 msgr2=0x7f56c8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56c8053ad0 0x7f56c8055f90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f56dc00b000 tx=0x7f56dc008040).stop 2022-07-12T08:47:03.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 msgr2=0x7f56e813e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 0x7f56e813e080 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f56d800c000 tx=0x7f56d8006df0).stop 2022-07-12T08:47:03.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 shutdown_connections 2022-07-12T08:47:03.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56c8053ad0 0x7f56c8055f90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56e813fa80 0x7f56e8155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56e80b6550 0x7f56e8155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 --2- 172.21.15.61:0/3183008719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56e80b0df0 0x7f56e813e080 secure :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0x7f56d800c000 tx=0x7f56d8006df0).stop 2022-07-12T08:47:03.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.068+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 >> 172.21.15.61:0/3183008719 conn(0x7f56e8063ad0 msgr2=0x7f56e8063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.072+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 shutdown_connections 2022-07-12T08:47:03.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.072+0000 7f56ecdc5700 1 -- 172.21.15.61:0/3183008719 wait complete. 2022-07-12T08:47:03.075 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:47:03.085 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:47:03.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 Processor -- start 2022-07-12T08:47:03.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 -- start start 2022-07-12T08:47:03.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8444066fc0 0x7f84440b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f84440b45d0 0x7f84440b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84440b2940 0x7f844405c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84440b2f20 con 0x7f84440b2940 2022-07-12T08:47:03.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84440b3170 con 0x7f84440b45d0 2022-07-12T08:47:03.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f844a191700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84440b4170 con 0x7f8444066fc0 2022-07-12T08:47:03.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f8443fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84440b2940 0x7f844405c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f8443fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84440b2940 0x7f844405c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57136/0 (socket says 172.21.15.61:57136) 2022-07-12T08:47:03.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f8443fff700 1 -- 172.21.15.61:0/2561127217 learned_addr learned my addr 172.21.15.61:0/2561127217 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:03.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f8442ffd700 1 --2- 172.21.15.61:0/2561127217 >> v2:172.21.15.111:3300/0 conn(0x7f84440b45d0 0x7f84440b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f84437fe700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8444066fc0 0x7f84440b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.204+0000 7f8443fff700 1 -- 172.21.15.61:0/2561127217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8444066fc0 msgr2=0x7f84440b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:03.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f8443fff700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8444066fc0 0x7f84440b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f8443fff700 1 -- 172.21.15.61:0/2561127217 >> v2:172.21.15.111:3300/0 conn(0x7f84440b45d0 msgr2=0x7f84440b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:03.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f8443fff700 1 --2- 172.21.15.61:0/2561127217 >> v2:172.21.15.111:3300/0 conn(0x7f84440b45d0 0x7f84440b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f8443fff700 1 -- 172.21.15.61:0/2561127217 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84440b56e0 con 0x7f84440b2940 2022-07-12T08:47:03.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f84437fe700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8444066fc0 0x7f84440b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:03.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f8443fff700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84440b2940 0x7f844405c090 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f843c0032f0 tx=0x7f843c009290).ready entity=mon.0 client_cookie=818b7fb7d4b2b28 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f84427fc700 1 -- 172.21.15.61:0/2561127217 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f843c00b870 con 0x7f84440b2940 2022-07-12T08:47:03.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f84427fc700 1 -- 172.21.15.61:0/2561127217 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c00b9d0 con 0x7f84440b2940 2022-07-12T08:47:03.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f84427fc700 1 -- 172.21.15.61:0/2561127217 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f843c00bb40 con 0x7f84440b2940 2022-07-12T08:47:03.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 -- 172.21.15.61:0/2561127217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84440b2940 msgr2=0x7f844405c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84440b2940 0x7f844405c090 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 rx=0x7f843c0032f0 tx=0x7f843c009290).stop 2022-07-12T08:47:03.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 -- 172.21.15.61:0/2561127217 shutdown_connections 2022-07-12T08:47:03.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84440b2940 0x7f844405c090 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 --2- 172.21.15.61:0/2561127217 >> v2:172.21.15.111:3300/0 conn(0x7f84440b45d0 0x7f84440b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 --2- 172.21.15.61:0/2561127217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8444066fc0 0x7f84440b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 -- 172.21.15.61:0/2561127217 >> 172.21.15.61:0/2561127217 conn(0x7f84440b94d0 msgr2=0x7f84440b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 -- 172.21.15.61:0/2561127217 shutdown_connections 2022-07-12T08:47:03.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.208+0000 7f844a191700 1 -- 172.21.15.61:0/2561127217 wait complete. 2022-07-12T08:47:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 Processor -- start 2022-07-12T08:47:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 -- start start 2022-07-12T08:47:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 0x7f84440acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84440b2940 0x7f84440b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84440b45d0 0x7f84440afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440b8930 con 0x7f8444066fc0 2022-07-12T08:47:03.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440b8bd0 con 0x7f84440b45d0 2022-07-12T08:47:03.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84440af880 con 0x7f84440b2940 2022-07-12T08:47:03.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 0x7f84440acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 0x7f84440acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57138/0 (socket says 172.21.15.61:57138) 2022-07-12T08:47:03.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 -- 172.21.15.61:0/1539612242 learned_addr learned my addr 172.21.15.61:0/1539612242 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:03.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f8442ffd700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84440b2940 0x7f84440b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f8443fff700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84440b45d0 0x7f84440afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 -- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84440b2940 msgr2=0x7f84440b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84440b2940 0x7f84440b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 -- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84440b45d0 msgr2=0x7f84440afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84440b45d0 0x7f84440afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 -- 172.21.15.61:0/1539612242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c009530 con 0x7f8444066fc0 2022-07-12T08:47:03.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f8443fff700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84440b45d0 0x7f84440afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:47:03.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f84437fe700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 0x7f84440acea0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f842c006940 tx=0x7f842c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f842c011780 con 0x7f8444066fc0 2022-07-12T08:47:03.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c0118e0 con 0x7f8444066fc0 2022-07-12T08:47:03.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f842c011c40 con 0x7f8444066fc0 2022-07-12T08:47:03.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84440b0210 con 0x7f8444066fc0 2022-07-12T08:47:03.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.212+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8444148fa0 con 0x7f8444066fc0 2022-07-12T08:47:03.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.216+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84440b8780 con 0x7f8444066fc0 2022-07-12T08:47:03.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.216+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f842c012420 con 0x7f8444066fc0 2022-07-12T08:47:03.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.216+0000 7f8440ff9700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84280539b0 0x7f8428055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.216+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f842c06bd70 con 0x7f8444066fc0 2022-07-12T08:47:03.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.216+0000 7f8442ffd700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84280539b0 0x7f8428055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.216+0000 7f8442ffd700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84280539b0 0x7f8428055e70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f843400b000 tx=0x7f8434008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.224+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f842c042ce0 con 0x7f8444066fc0 2022-07-12T08:47:03.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.392+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f84440afa40 con 0x7f8444066fc0 2022-07-12T08:47:03.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.404+0000 7f8440ff9700 1 -- 172.21.15.61:0/1539612242 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 13 and data pool 14 v36) v1 ==== 189+0+0 (secure 0 0 0) 0x7f842c046300 con 0x7f8444066fc0 2022-07-12T08:47:03.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84280539b0 msgr2=0x7f8428055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84280539b0 0x7f8428055e70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f843400b000 tx=0x7f8434008040).stop 2022-07-12T08:47:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 msgr2=0x7f84440acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 0x7f84440acea0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f842c006940 tx=0x7f842c008040).stop 2022-07-12T08:47:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 shutdown_connections 2022-07-12T08:47:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84280539b0 0x7f8428055e70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84440b45d0 0x7f84440afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84440b2940 0x7f84440b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 --2- 172.21.15.61:0/1539612242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8444066fc0 0x7f84440acea0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 >> 172.21.15.61:0/1539612242 conn(0x7f84440b94d0 msgr2=0x7f84441442b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 shutdown_connections 2022-07-12T08:47:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.412+0000 7f844a191700 1 -- 172.21.15.61:0/1539612242 wait complete. 2022-07-12T08:47:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 13 and data pool 14 2022-07-12T08:47:03.425 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:03.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.520+0000 7f78f3fff700 1 Processor -- start 2022-07-12T08:47:03.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.520+0000 7f78f3fff700 1 -- start start 2022-07-12T08:47:03.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f78f40b6ac0 0x7f78f4146220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f3fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40b6ea0 con 0x7f78f41420b0 2022-07-12T08:47:03.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f3fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f4060200 con 0x7f78f405f550 2022-07-12T08:47:03.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f3fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f4146710 con 0x7f78f40b6ac0 2022-07-12T08:47:03.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f78f40b6ac0 0x7f78f4146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f78f41420b0 0x7f78f4142510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45378/0 (socket says 172.21.15.61:45378) 2022-07-12T08:47:03.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f37fe700 1 -- 172.21.15.61:0/2131789103 learned_addr learned my addr 172.21.15.61:0/2131789103 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:03.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f2ffd700 1 -- 172.21.15.61:0/2131789103 >> v1:172.21.15.111:6789/0 conn(0x7f78f405f550 legacy=0x7f78f4060430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f1ffb700 1 -- 172.21.15.61:0/2131789103 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968853721 0 0) 0x7f78f40b6ea0 con 0x7f78f41420b0 2022-07-12T08:47:03.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f2ffd700 1 -- 172.21.15.61:0/2131789103 >> v1:172.21.15.61:6789/0 conn(0x7f78f41420b0 legacy=0x7f78f4142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f2ffd700 1 -- 172.21.15.61:0/2131789103 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f40b1360 con 0x7f78f40b6ac0 2022-07-12T08:47:03.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f2ffd700 1 --2- 172.21.15.61:0/2131789103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f40b6ac0 0x7f78f4146220 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f78e4002f70 tx=0x7f78e4006df0).ready entity=mon.0 client_cookie=ad70da82e273755 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f1ffb700 1 -- 172.21.15.61:0/2131789103 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e400d610 con 0x7f78f40b6ac0 2022-07-12T08:47:03.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f1ffb700 1 -- 172.21.15.61:0/2131789103 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4000e90 con 0x7f78f40b6ac0 2022-07-12T08:47:03.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.524+0000 7f78f1ffb700 1 -- 172.21.15.61:0/2131789103 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e400d960 con 0x7f78f40b6ac0 2022-07-12T08:47:03.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- 172.21.15.61:0/2131789103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f40b6ac0 msgr2=0x7f78f4146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 --2- 172.21.15.61:0/2131789103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f40b6ac0 0x7f78f4146220 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f78e4002f70 tx=0x7f78e4006df0).stop 2022-07-12T08:47:03.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- 172.21.15.61:0/2131789103 shutdown_connections 2022-07-12T08:47:03.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 --2- 172.21.15.61:0/2131789103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f40b6ac0 0x7f78f4146220 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- 172.21.15.61:0/2131789103 >> 172.21.15.61:0/2131789103 conn(0x7f78f4064570 msgr2=0x7f78f40b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- 172.21.15.61:0/2131789103 shutdown_connections 2022-07-12T08:47:03.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- 172.21.15.61:0/2131789103 wait complete. 2022-07-12T08:47:03.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 Processor -- start 2022-07-12T08:47:03.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- start start 2022-07-12T08:47:03.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 0x7f78f4066220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f78f40b6ac0 0x7f78f4155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78f41420b0 0x7f78f4155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4149250 con 0x7f78f405f550 2022-07-12T08:47:03.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f41494f0 con 0x7f78f41420b0 2022-07-12T08:47:03.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f3fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4149790 con 0x7f78f40b6ac0 2022-07-12T08:47:03.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 0x7f78f4066220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 0x7f78f4066220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57162/0 (socket says 172.21.15.61:57162) 2022-07-12T08:47:03.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 -- 172.21.15.61:0/1005285742 learned_addr learned my addr 172.21.15.61:0/1005285742 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:03.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f37fe700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78f41420b0 0x7f78f4155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f27fc700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f78f40b6ac0 0x7f78f4155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 -- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f78f40b6ac0 msgr2=0x7f78f4155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f78f40b6ac0 0x7f78f4155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 -- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78f41420b0 msgr2=0x7f78f4155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78f41420b0 0x7f78f4155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.528+0000 7f78f2ffd700 1 -- 172.21.15.61:0/1005285742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e400b180 con 0x7f78f405f550 2022-07-12T08:47:03.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78f2ffd700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 0x7f78f4066220 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f78e4002f70 tx=0x7f78e4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e400de90 con 0x7f78f405f550 2022-07-12T08:47:03.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e4006a60 con 0x7f78f405f550 2022-07-12T08:47:03.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f4149a30 con 0x7f78f405f550 2022-07-12T08:47:03.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78e4011d30 con 0x7f78f405f550 2022-07-12T08:47:03.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f414a110 con 0x7f78f405f550 2022-07-12T08:47:03.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f41487d0 con 0x7f78f405f550 2022-07-12T08:47:03.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f78e4025030 con 0x7f78f405f550 2022-07-12T08:47:03.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78e3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78d0053920 0x7f78d0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f78e406d060 con 0x7f78f405f550 2022-07-12T08:47:03.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.532+0000 7f78f27fc700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78d0053920 0x7f78d0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.536+0000 7f78f27fc700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78d0053920 0x7f78d0055de0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f78e800d000 tx=0x7f78e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.540+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f78f41487d0 con 0x7f78f405f550 2022-07-12T08:47:03.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.668+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78f41487d0 con 0x7f78f405f550 2022-07-12T08:47:03.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.668+0000 7f78e3fff700 1 -- 172.21.15.61:0/1005285742 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f78f41487d0 con 0x7f78f405f550 2022-07-12T08:47:03.669 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:03.669 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":70,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:03.397152+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:47:01.995190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-12T08:47:02.575389+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:03.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78d0053920 msgr2=0x7f78d0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78d0053920 0x7f78d0055de0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f78e800d000 tx=0x7f78e8009040).stop 2022-07-12T08:47:03.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 msgr2=0x7f78f4066220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 0x7f78f4066220 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f78e4002f70 tx=0x7f78e4015000).stop 2022-07-12T08:47:03.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 shutdown_connections 2022-07-12T08:47:03.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78d0053920 0x7f78d0055de0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78f41420b0 0x7f78f4155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f78f40b6ac0 0x7f78f4155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 --2- 172.21.15.61:0/1005285742 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78f405f550 0x7f78f4066220 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 >> 172.21.15.61:0/1005285742 conn(0x7f78f4064570 msgr2=0x7f78f41469b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 shutdown_connections 2022-07-12T08:47:03.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.672+0000 7f78f3fff700 1 -- 172.21.15.61:0/1005285742 wait complete. 2022-07-12T08:47:03.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:03.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.788+0000 7f5069922700 1 Processor -- start 2022-07-12T08:47:03.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.788+0000 7f5069922700 1 -- start start 2022-07-12T08:47:03.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f506405c050 0x7f50640ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f506414a8f0 0x7f50641443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640ad2f0 con 0x7f50640b01f0 2022-07-12T08:47:03.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50640b0a40 con 0x7f506414a8f0 2022-07-12T08:47:03.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f506405c430 con 0x7f506405c050 2022-07-12T08:47:03.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50627fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f50640b01f0 0x7f506405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45388/0 (socket says 172.21.15.61:45388) 2022-07-12T08:47:03.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50627fc700 1 -- 172.21.15.61:0/4081527104 learned_addr learned my addr 172.21.15.61:0/4081527104 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:03.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50637fe700 1 --2- 172.21.15.61:0/4081527104 >> v2:172.21.15.61:3300/0 conn(0x7f506414a8f0 0x7f50641443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5062ffd700 1 --2- 172.21.15.61:0/4081527104 >> v2:172.21.15.111:3300/0 conn(0x7f506405c050 0x7f50640ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5061ffb700 1 -- 172.21.15.61:0/4081527104 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192271546 0 0) 0x7f50640ad2f0 con 0x7f50640b01f0 2022-07-12T08:47:03.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5061ffb700 1 -- 172.21.15.61:0/4081527104 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5048003260 con 0x7f50640b01f0 2022-07-12T08:47:03.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50637fe700 1 -- 172.21.15.61:0/4081527104 >> v2:172.21.15.111:3300/0 conn(0x7f506405c050 msgr2=0x7f50640ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:03.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50637fe700 1 --2- 172.21.15.61:0/4081527104 >> v2:172.21.15.111:3300/0 conn(0x7f506405c050 0x7f50640ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5061ffb700 1 -- 172.21.15.61:0/4081527104 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104900963 0 0) 0x7f5048003260 con 0x7f50640b01f0 2022-07-12T08:47:03.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50637fe700 1 -- 172.21.15.61:0/4081527104 >> v1:172.21.15.61:6789/0 conn(0x7f50640b01f0 legacy=0x7f506405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50637fe700 1 -- 172.21.15.61:0/4081527104 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640b2420 con 0x7f506414a8f0 2022-07-12T08:47:03.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f50637fe700 1 --2- 172.21.15.61:0/4081527104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506414a8f0 0x7f50641443c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f5054001b80 tx=0x7f505c006df0).ready entity=mon.0 client_cookie=725516499701cb44 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5061ffb700 1 -- 172.21.15.61:0/4081527104 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50540019e0 con 0x7f506414a8f0 2022-07-12T08:47:03.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5061ffb700 1 -- 172.21.15.61:0/4081527104 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c000e90 con 0x7f506414a8f0 2022-07-12T08:47:03.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5061ffb700 1 -- 172.21.15.61:0/4081527104 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f505c012460 con 0x7f506414a8f0 2022-07-12T08:47:03.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- 172.21.15.61:0/4081527104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506414a8f0 msgr2=0x7f50641443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 --2- 172.21.15.61:0/4081527104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506414a8f0 0x7f50641443c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f5054001b80 tx=0x7f505c006df0).stop 2022-07-12T08:47:03.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- 172.21.15.61:0/4081527104 shutdown_connections 2022-07-12T08:47:03.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 --2- 172.21.15.61:0/4081527104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506414a8f0 0x7f50641443c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 --2- 172.21.15.61:0/4081527104 >> v2:172.21.15.111:3300/0 conn(0x7f506405c050 0x7f50640ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- 172.21.15.61:0/4081527104 >> 172.21.15.61:0/4081527104 conn(0x7f5064062c00 msgr2=0x7f5064062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- 172.21.15.61:0/4081527104 shutdown_connections 2022-07-12T08:47:03.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.792+0000 7f5069922700 1 -- 172.21.15.61:0/4081527104 wait complete. 2022-07-12T08:47:03.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 Processor -- start 2022-07-12T08:47:03.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 -- start start 2022-07-12T08:47:03.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 0x7f506414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50640b01f0 0x7f50640aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f506414a8f0 0x7f5064149490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506409e5a0 con 0x7f506405c050 2022-07-12T08:47:03.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641487f0 con 0x7f50640b01f0 2022-07-12T08:47:03.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5064148a90 con 0x7f506414a8f0 2022-07-12T08:47:03.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 0x7f506414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 0x7f506414df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57176/0 (socket says 172.21.15.61:57176) 2022-07-12T08:47:03.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 -- 172.21.15.61:0/282611135 learned_addr learned my addr 172.21.15.61:0/282611135 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:03.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f50627fc700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50640b01f0 0x7f50640aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f50637fe700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f506414a8f0 0x7f5064149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 -- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f506414a8f0 msgr2=0x7f5064149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f506414a8f0 0x7f5064149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 -- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50640b01f0 msgr2=0x7f50640aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50640b01f0 0x7f50640aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 -- 172.21.15.61:0/282611135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c00b180 con 0x7f506405c050 2022-07-12T08:47:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f50637fe700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f506414a8f0 0x7f5064149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5062ffd700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 0x7f506414df20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f505800c000 tx=0x7f5058006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5058010780 con 0x7f506405c050 2022-07-12T08:47:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505800b3b0 con 0x7f506405c050 2022-07-12T08:47:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506414a2c0 con 0x7f506405c050 2022-07-12T08:47:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5058010ad0 con 0x7f506405c050 2022-07-12T08:47:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.796+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5064149980 con 0x7f506405c050 2022-07-12T08:47:03.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.800+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064148c50 con 0x7f506405c050 2022-07-12T08:47:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.800+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5058010c30 con 0x7f506405c050 2022-07-12T08:47:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.800+0000 7f5053fff700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5040053a60 0x7f5040055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.800+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f505806c1c0 con 0x7f506405c050 2022-07-12T08:47:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.800+0000 7f50627fc700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5040053a60 0x7f5040055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:03.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.800+0000 7f50627fc700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5040053a60 0x7f5040055f20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f505400d000 tx=0x7f5054001740).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:03.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.804+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5058043080 con 0x7f506405c050 2022-07-12T08:47:03.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.932+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5064057460 con 0x7f506405c050 2022-07-12T08:47:03.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5053fff700 1 -- 172.21.15.61:0/282611135 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f50580466a0 con 0x7f506405c050 2022-07-12T08:47:03.941 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:03.942 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":70,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:03.397152+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:47:01.995190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-12T08:47:02.575389+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:03.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5040053a60 msgr2=0x7f5040055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5040053a60 0x7f5040055f20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f505400d000 tx=0x7f5054001740).stop 2022-07-12T08:47:03.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 msgr2=0x7f506414df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:03.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 0x7f506414df20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f505800c000 tx=0x7f5058006df0).stop 2022-07-12T08:47:03.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 shutdown_connections 2022-07-12T08:47:03.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f506414a8f0 0x7f5064149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5040053a60 0x7f5040055f20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50640b01f0 0x7f50640aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 --2- 172.21.15.61:0/282611135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f506405c050 0x7f506414df20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:03.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 >> 172.21.15.61:0/282611135 conn(0x7f5064062c00 msgr2=0x7f50640b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:03.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 shutdown_connections 2022-07-12T08:47:03.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:03.936+0000 7f5069922700 1 -- 172.21.15.61:0/282611135 wait complete. 2022-07-12T08:47:03.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:47:04.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 Processor -- start 2022-07-12T08:47:04.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 -- start start 2022-07-12T08:47:04.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec240b9db0 0x7fec240b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b3ea0 0x7fec2405c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec2405c5f0 0x7fec240b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2405bd10 con 0x7fec240b9db0 2022-07-12T08:47:04.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec240b3120 con 0x7fec2405c5f0 2022-07-12T08:47:04.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec29712700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec240b33c0 con 0x7fec240b3ea0 2022-07-12T08:47:04.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec240b9db0 0x7fec240b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec240b9db0 0x7fec240b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57184/0 (socket says 172.21.15.61:57184) 2022-07-12T08:47:04.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 -- 172.21.15.61:0/1979567883 learned_addr learned my addr 172.21.15.61:0/1979567883 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:04.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec1bfff700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b3ea0 0x7fec2405c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec237fe700 1 --2- 172.21.15.61:0/1979567883 >> v2:172.21.15.111:3300/0 conn(0x7fec2405c5f0 0x7fec240b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 -- 172.21.15.61:0/1979567883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b3ea0 msgr2=0x7fec2405c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:04.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b3ea0 0x7fec2405c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 -- 172.21.15.61:0/1979567883 >> v2:172.21.15.111:3300/0 conn(0x7fec2405c5f0 msgr2=0x7fec240b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:04.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 --2- 172.21.15.61:0/1979567883 >> v2:172.21.15.111:3300/0 conn(0x7fec2405c5f0 0x7fec240b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 -- 172.21.15.61:0/1979567883 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec240ba190 con 0x7fec240b9db0 2022-07-12T08:47:04.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec1bfff700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b3ea0 0x7fec2405c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:04.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.068+0000 7fec22ffd700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b9db0 0x7fec240b2a40 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fec14002f70 tx=0x7fec14006eb0).ready entity=mon.0 client_cookie=3397dc109f82c67d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:04.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec227fc700 1 -- 172.21.15.61:0/1979567883 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec1400d570 con 0x7fec240b9db0 2022-07-12T08:47:04.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec227fc700 1 -- 172.21.15.61:0/1979567883 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec14000e90 con 0x7fec240b9db0 2022-07-12T08:47:04.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec227fc700 1 -- 172.21.15.61:0/1979567883 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec1400d900 con 0x7fec240b9db0 2022-07-12T08:47:04.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 -- 172.21.15.61:0/1979567883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b9db0 msgr2=0x7fec240b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:04.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b9db0 0x7fec240b2a40 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7fec14002f70 tx=0x7fec14006eb0).stop 2022-07-12T08:47:04.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 -- 172.21.15.61:0/1979567883 shutdown_connections 2022-07-12T08:47:04.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b3ea0 0x7fec2405c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 --2- 172.21.15.61:0/1979567883 >> v2:172.21.15.111:3300/0 conn(0x7fec2405c5f0 0x7fec240b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 --2- 172.21.15.61:0/1979567883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b9db0 0x7fec240b2a40 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 -- 172.21.15.61:0/1979567883 >> 172.21.15.61:0/1979567883 conn(0x7fec24067960 msgr2=0x7fec240b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:04.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 -- 172.21.15.61:0/1979567883 shutdown_connections 2022-07-12T08:47:04.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 -- 172.21.15.61:0/1979567883 wait complete. 2022-07-12T08:47:04.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 Processor -- start 2022-07-12T08:47:04.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.072+0000 7fec29712700 1 -- start start 2022-07-12T08:47:04.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec2405c5f0 0x7fec24155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 0x7fec240af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b9db0 0x7fec24141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24152720 con 0x7fec240b3ea0 2022-07-12T08:47:04.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24146eb0 con 0x7fec2405c5f0 2022-07-12T08:47:04.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec24145ad0 con 0x7fec240b9db0 2022-07-12T08:47:04.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 0x7fec240af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 0x7fec240af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57192/0 (socket says 172.21.15.61:57192) 2022-07-12T08:47:04.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 -- 172.21.15.61:0/3810363716 learned_addr learned my addr 172.21.15.61:0/3810363716 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:04.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec22ffd700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec2405c5f0 0x7fec24155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec237fe700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b9db0 0x7fec24141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 -- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b9db0 msgr2=0x7fec24141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:04.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b9db0 0x7fec24141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 -- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec2405c5f0 msgr2=0x7fec24155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:04.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec2405c5f0 0x7fec24155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:04.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 -- 172.21.15.61:0/3810363716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c00b180 con 0x7fec240b3ea0 2022-07-12T08:47:04.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec22ffd700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec2405c5f0 0x7fec24155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:04.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec1bfff700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 0x7fec240af840 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fec0c00c000 tx=0x7fec0c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:04.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec0c00ca40 con 0x7fec240b3ea0 2022-07-12T08:47:04.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00cba0 con 0x7fec240b3ea0 2022-07-12T08:47:04.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1400b1e0 con 0x7fec240b3ea0 2022-07-12T08:47:04.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec0c0123f0 con 0x7fec240b3ea0 2022-07-12T08:47:04.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec24146a10 con 0x7fec240b3ea0 2022-07-12T08:47:04.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec24152b90 con 0x7fec240b3ea0 2022-07-12T08:47:04.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fec0c012550 con 0x7fec240b3ea0 2022-07-12T08:47:04.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.076+0000 7fec20ff9700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec080539b0 0x7fec08055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:04.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.080+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fec0c06c590 con 0x7fec240b3ea0 2022-07-12T08:47:04.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.080+0000 7fec22ffd700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec080539b0 0x7fec08055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:04.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.080+0000 7fec22ffd700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec080539b0 0x7fec08055e70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fec1401a000 tx=0x7fec14015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:04.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.084+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec0c043490 con 0x7fec240b3ea0 2022-07-12T08:47:04.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:04.252+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fec24057460 con 0x7fec240b3ea0 2022-07-12T08:47:05.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.160+0000 7fec20ff9700 1 -- 172.21.15.61:0/3810363716 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v38) v1 ==== 118+0+0 (secure 0 0 0) 0x7fec0c046ab0 con 0x7fec240b3ea0 2022-07-12T08:47:05.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec080539b0 msgr2=0x7fec08055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec080539b0 0x7fec08055e70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fec1401a000 tx=0x7fec14015000).stop 2022-07-12T08:47:05.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 msgr2=0x7fec240af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 0x7fec240af840 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 rx=0x7fec0c00c000 tx=0x7fec0c006df0).stop 2022-07-12T08:47:05.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 shutdown_connections 2022-07-12T08:47:05.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec240b9db0 0x7fec24141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec240b3ea0 0x7fec240af840 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec080539b0 0x7fec08055e70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 --2- 172.21.15.61:0/3810363716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec2405c5f0 0x7fec24155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 >> 172.21.15.61:0/3810363716 conn(0x7fec24067960 msgr2=0x7fec24142f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:05.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 shutdown_connections 2022-07-12T08:47:05.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.164+0000 7fec29712700 1 -- 172.21.15.61:0/3810363716 wait complete. 2022-07-12T08:47:05.180 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:05.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 Processor -- start 2022-07-12T08:47:05.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- start start 2022-07-12T08:47:05.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c05cec0 0x7f704c147090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f704c0abfe0 0x7f704c0ab320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704c147660 con 0x7f704c0ab810 2022-07-12T08:47:05.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704c147900 con 0x7f704c0abfe0 2022-07-12T08:47:05.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c147ba0 con 0x7f704c05cec0 2022-07-12T08:47:05.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f704c0abfe0 0x7f704c0ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f704c0abfe0 0x7f704c0ab320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57198/0 (socket says 172.21.15.61:57198) 2022-07-12T08:47:05.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 -- 172.21.15.61:0/366339813 learned_addr learned my addr 172.21.15.61:0/366339813 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:05.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f7051e4c700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c05cec0 0x7f704c147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 -- 172.21.15.61:0/366339813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c05cec0 msgr2=0x7f704c147090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c05cec0 0x7f704c147090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 -- 172.21.15.61:0/366339813 >> v1:172.21.15.111:6789/0 conn(0x7f704c0ab810 legacy=0x7f704c060010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:05.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 -- 172.21.15.61:0/366339813 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c148040 con 0x7f704c0abfe0 2022-07-12T08:47:05.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f7051e4c700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c05cec0 0x7f704c147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:47:05.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f705164b700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c0abfe0 0x7f704c0ab320 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f704c0b2990 tx=0x7f7048006eb0).ready entity=mon.0 client_cookie=3eae2fa140ba2cc7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f7050e4a700 1 -- 172.21.15.61:0/366339813 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f704800d570 con 0x7f704c0abfe0 2022-07-12T08:47:05.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f7050e4a700 1 -- 172.21.15.61:0/366339813 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7048000e90 con 0x7f704c0abfe0 2022-07-12T08:47:05.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f7050e4a700 1 -- 172.21.15.61:0/366339813 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f704800d8c0 con 0x7f704c0abfe0 2022-07-12T08:47:05.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- 172.21.15.61:0/366339813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c0abfe0 msgr2=0x7f704c0ab320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c0abfe0 0x7f704c0ab320 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f704c0b2990 tx=0x7f7048006eb0).stop 2022-07-12T08:47:05.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- 172.21.15.61:0/366339813 shutdown_connections 2022-07-12T08:47:05.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c0abfe0 0x7f704c0ab320 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 --2- 172.21.15.61:0/366339813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c05cec0 0x7f704c147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- 172.21.15.61:0/366339813 >> 172.21.15.61:0/366339813 conn(0x7f704c0b5f60 msgr2=0x7f704c0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:05.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- 172.21.15.61:0/366339813 shutdown_connections 2022-07-12T08:47:05.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.292+0000 7f70540b0700 1 -- 172.21.15.61:0/366339813 wait complete. 2022-07-12T08:47:05.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 Processor -- start 2022-07-12T08:47:05.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 -- start start 2022-07-12T08:47:05.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 0x7f704c14e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c0ab810 0x7f704c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c0abfe0 0x7f704c0952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c097a50 con 0x7f704c05cec0 2022-07-12T08:47:05.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704c094ec0 con 0x7f704c0abfe0 2022-07-12T08:47:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f70540b0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704c0af230 con 0x7f704c0ab810 2022-07-12T08:47:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f7051e4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 0x7f704c14e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f7051e4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 0x7f704c14e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57204/0 (socket says 172.21.15.61:57204) 2022-07-12T08:47:05.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.296+0000 7f7051e4c700 1 -- 172.21.15.61:0/711237015 learned_addr learned my addr 172.21.15.61:0/711237015 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:05.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f705164b700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c0ab810 0x7f704c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f705264d700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c0abfe0 0x7f704c0952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f7051e4c700 1 -- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c0ab810 msgr2=0x7f704c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f7051e4c700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c0ab810 0x7f704c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f7051e4c700 1 -- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c0abfe0 msgr2=0x7f704c0952f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f7051e4c700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c0abfe0 0x7f704c0952f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f7051e4c700 1 -- 172.21.15.61:0/711237015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704800b1e0 con 0x7f704c05cec0 2022-07-12T08:47:05.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f7051e4c700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 0x7f704c14e020 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f7040007000 tx=0x7f7040008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7040010580 con 0x7f704c05cec0 2022-07-12T08:47:05.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70400106e0 con 0x7f704c05cec0 2022-07-12T08:47:05.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704c14e510 con 0x7f704c05cec0 2022-07-12T08:47:05.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7040010a40 con 0x7f704c05cec0 2022-07-12T08:47:05.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704c0ace40 con 0x7f704c05cec0 2022-07-12T08:47:05.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704c0ae560 con 0x7f704c05cec0 2022-07-12T08:47:05.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7040010ba0 con 0x7f704c05cec0 2022-07-12T08:47:05.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f703effd700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f702c053a40 0x7f702c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f704006be70 con 0x7f704c05cec0 2022-07-12T08:47:05.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.300+0000 7f705164b700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f702c053a40 0x7f702c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.304+0000 7f705164b700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f702c053a40 0x7f702c055f00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f704801c000 tx=0x7f7048015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.308+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7040042d70 con 0x7f704c05cec0 2022-07-12T08:47:05.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.468+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f704c057460 con 0x7f704c05cec0 2022-07-12T08:47:05.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.472+0000 7f703effd700 1 -- 172.21.15.61:0/711237015 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f7040046390 con 0x7f704c05cec0 2022-07-12T08:47:05.479 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:05.479 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:05.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.472+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f702c053a40 msgr2=0x7f702c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.472+0000 7f70540b0700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f702c053a40 0x7f702c055f00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f704801c000 tx=0x7f7048015000).stop 2022-07-12T08:47:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.472+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 msgr2=0x7f704c14e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.472+0000 7f70540b0700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 0x7f704c14e020 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f7040007000 tx=0x7f7040008040).stop 2022-07-12T08:47:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 shutdown_connections 2022-07-12T08:47:05.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f702c053a40 0x7f702c055f00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f704c0abfe0 0x7f704c0952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f704c0ab810 0x7f704c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 --2- 172.21.15.61:0/711237015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f704c05cec0 0x7f704c14e020 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 >> 172.21.15.61:0/711237015 conn(0x7f704c0b5f60 msgr2=0x7f704c14cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:05.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 shutdown_connections 2022-07-12T08:47:05.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.476+0000 7f70540b0700 1 -- 172.21.15.61:0/711237015 wait complete. 2022-07-12T08:47:05.489 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:05.492 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:05.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.600+0000 7f8add804700 1 Processor -- start 2022-07-12T08:47:05.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.600+0000 7f8add804700 1 -- start start 2022-07-12T08:47:05.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add804700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8ad805f480 0x7f8ad805f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add804700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad805fd50 con 0x7f8ad813ce20 2022-07-12T08:47:05.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add804700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad8140680 con 0x7f8ad8140160 2022-07-12T08:47:05.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add804700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad813d260 con 0x7f8ad805f480 2022-07-12T08:47:05.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add003700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8ad813ce20 0x7f8ad8141410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45432/0 (socket says 172.21.15.61:45432) 2022-07-12T08:47:05.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add003700 1 -- 172.21.15.61:0/915669146 learned_addr learned my addr 172.21.15.61:0/915669146 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:05.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 --2- 172.21.15.61:0/915669146 >> v2:172.21.15.111:3300/0 conn(0x7f8ad805f480 0x7f8ad805f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522327263 0 0) 0x7f8ad805fd50 con 0x7f8ad813ce20 2022-07-12T08:47:05.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8abc003260 con 0x7f8ad813ce20 2022-07-12T08:47:05.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584265925 0 0) 0x7f8ad8140680 con 0x7f8ad8140160 2022-07-12T08:47:05.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad805fd50 con 0x7f8ad8140160 2022-07-12T08:47:05.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57759995 0 0) 0x7f8abc003260 con 0x7f8ad813ce20 2022-07-12T08:47:05.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 >> v2:172.21.15.111:3300/0 conn(0x7f8ad805f480 msgr2=0x7f8ad805f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:05.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 --2- 172.21.15.61:0/915669146 >> v2:172.21.15.111:3300/0 conn(0x7f8ad805f480 0x7f8ad805f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 >> v1:172.21.15.111:6789/0 conn(0x7f8ad8140160 legacy=0x7f8ad813c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad8140cd0 con 0x7f8ad813ce20 2022-07-12T08:47:05.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f8ad0001a00 con 0x7f8ad813ce20 2022-07-12T08:47:05.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 --2- 172.21.15.61:0/915669146 >> v2:172.21.15.111:3300/0 conn(0x7f8ad805f480 0x7f8ad805f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:05.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 >> v1:172.21.15.61:6789/0 conn(0x7f8ad813ce20 legacy=0x7f8ad8141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8abc0038e0 0x7f8abc003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc004500 0x7f8abc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8abc004e70 0x7f8abc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad813d260 con 0x7f8abc0038e0 2022-07-12T08:47:05.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc00bcf0 con 0x7f8abc004e70 2022-07-12T08:47:05.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc00e0c0 con 0x7f8abc004500 2022-07-12T08:47:05.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8abc0038e0 0x7f8abc003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad7fff700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc004500 0x7f8abc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add003700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8abc004e70 0x7f8abc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 -- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc004500 msgr2=0x7f8abc004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc004500 0x7f8abc004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 -- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8abc004e70 msgr2=0x7f8abc00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8abc004e70 0x7f8abc00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8add003700 1 -- 172.21.15.61:0/915669146 reap_dead start 2022-07-12T08:47:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 -- 172.21.15.61:0/915669146 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc003260 con 0x7f8abc0038e0 2022-07-12T08:47:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad7fff700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc004500 0x7f8abc004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8adc802700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8abc0038e0 0x7f8abc003ce0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f8acc00c000 tx=0x7f8acc006df0).ready entity=mon.0 client_cookie=bc9e217370d6948d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8acc00e800 con 0x7f8abc0038e0 2022-07-12T08:47:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8acc00e960 con 0x7f8abc0038e0 2022-07-12T08:47:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.604+0000 7f8ad77fe700 1 -- 172.21.15.61:0/915669146 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8acc00ecc0 con 0x7f8abc0038e0 2022-07-12T08:47:05.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- 172.21.15.61:0/915669146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8abc0038e0 msgr2=0x7f8abc003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8abc0038e0 0x7f8abc003ce0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f8acc00c000 tx=0x7f8acc006df0).stop 2022-07-12T08:47:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- 172.21.15.61:0/915669146 shutdown_connections 2022-07-12T08:47:05.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 --2- 172.21.15.61:0/915669146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8abc0038e0 0x7f8abc003ce0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- 172.21.15.61:0/915669146 >> 172.21.15.61:0/915669146 conn(0x7f8ad8063a60 msgr2=0x7f8ad8063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:05.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- 172.21.15.61:0/915669146 shutdown_connections 2022-07-12T08:47:05.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- 172.21.15.61:0/915669146 wait complete. 2022-07-12T08:47:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 Processor -- start 2022-07-12T08:47:05.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- start start 2022-07-12T08:47:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc0038e0 0x7f8ad80aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ad81423e0 0x7f8ad8065570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ad814d800 0x7f8ad814dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81430d0 con 0x7f8ad81423e0 2022-07-12T08:47:05.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad80b3280 con 0x7f8ad814d800 2022-07-12T08:47:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8add804700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad8064e30 con 0x7f8abc0038e0 2022-07-12T08:47:05.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.608+0000 7f8ad7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ad81423e0 0x7f8ad8065570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8ad7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ad81423e0 0x7f8ad8065570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57226/0 (socket says 172.21.15.61:57226) 2022-07-12T08:47:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8ad7fff700 1 -- 172.21.15.61:0/1714000478 learned_addr learned my addr 172.21.15.61:0/1714000478 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ad814d800 0x7f8ad814dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8adc802700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc0038e0 0x7f8ad80aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 -- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc0038e0 msgr2=0x7f8ad80aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc0038e0 0x7f8ad80aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 -- 172.21.15.61:0/1714000478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ad81423e0 msgr2=0x7f8ad8065570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ad81423e0 0x7f8ad8065570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 -- 172.21.15.61:0/1714000478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad80650d0 con 0x7f8ad814d800 2022-07-12T08:47:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add003700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ad814d800 0x7f8ad814dc80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8ad80645b0 tx=0x7f8ad8060640).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ad000fa30 con 0x7f8ad814d800 2022-07-12T08:47:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad000fb90 con 0x7f8ad814d800 2022-07-12T08:47:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad80b2f00 con 0x7f8ad814d800 2022-07-12T08:47:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ad00077f0 con 0x7f8ad814d800 2022-07-12T08:47:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad8065c00 con 0x7f8ad814d800 2022-07-12T08:47:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad8057460 con 0x7f8ad814d800 2022-07-12T08:47:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.612+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8ad001c360 con 0x7f8ad814d800 2022-07-12T08:47:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.616+0000 7f8ad5ffb700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ab8053a60 0x7f8ab8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.616+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8ad006b900 con 0x7f8ad814d800 2022-07-12T08:47:05.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.616+0000 7f8adc802700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ab8053a60 0x7f8ab8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.616+0000 7f8adc802700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ab8053a60 0x7f8ab8055f20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f8acc01d000 tx=0x7f8acc019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.624+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ad0042670 con 0x7f8ad814d800 2022-07-12T08:47:05.678 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2022-07-12T08:47:05.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.748+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ad8065c00 con 0x7f8ad814d800 2022-07-12T08:47:05.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8ad5ffb700 1 -- 172.21.15.61:0/1714000478 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f8ad8065c00 con 0x7f8ad814d800 2022-07-12T08:47:05.750 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:05.751 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":71,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:04.401539+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:47:01.995190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-12T08:47:02.575389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:05.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ab8053a60 msgr2=0x7f8ab8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ab8053a60 0x7f8ab8055f20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f8acc01d000 tx=0x7f8acc019000).stop 2022-07-12T08:47:05.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ad814d800 msgr2=0x7f8ad814dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ad814d800 0x7f8ad814dc80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f8ad80645b0 tx=0x7f8ad8060640).stop 2022-07-12T08:47:05.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 shutdown_connections 2022-07-12T08:47:05.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ab8053a60 0x7f8ab8055f20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ad814d800 0x7f8ad814dc80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ad81423e0 0x7f8ad8065570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 --2- 172.21.15.61:0/1714000478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8abc0038e0 0x7f8ad80aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 >> 172.21.15.61:0/1714000478 conn(0x7f8ad8063a60 msgr2=0x7f8ad8063e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:05.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 shutdown_connections 2022-07-12T08:47:05.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.752+0000 7f8add804700 1 -- 172.21.15.61:0/1714000478 wait complete. 2022-07-12T08:47:05.764 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:47:05.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 Processor -- start 2022-07-12T08:47:05.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 -- start start 2022-07-12T08:47:05.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f857005cd20 0x7f857005d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85701465a0 0x7f85701469e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857014a560 con 0x7f8570060e80 2022-07-12T08:47:05.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8570097d40 con 0x7f857005cd20 2022-07-12T08:47:05.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857651f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857014b340 con 0x7f85701465a0 2022-07-12T08:47:05.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f8575d1e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85701465a0 0x7f85701469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f857551d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f857005cd20 0x7f857005d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f8574d1c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8570060e80 0x7f857014ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59054/0 (socket says 172.21.15.61:59054) 2022-07-12T08:47:05.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f8574d1c700 1 -- 172.21.15.61:0/3124691762 learned_addr learned my addr 172.21.15.61:0/3124691762 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:05.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f8567fff700 1 -- 172.21.15.61:0/3124691762 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109120829 0 0) 0x7f857014a560 con 0x7f8570060e80 2022-07-12T08:47:05.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.880+0000 7f8567fff700 1 -- 172.21.15.61:0/3124691762 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8554003260 con 0x7f8570060e80 2022-07-12T08:47:05.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8575d1e700 1 -- 172.21.15.61:0/3124691762 >> v2:172.21.15.111:3300/0 conn(0x7f857005cd20 msgr2=0x7f857005d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:05.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8575d1e700 1 --2- 172.21.15.61:0/3124691762 >> v2:172.21.15.111:3300/0 conn(0x7f857005cd20 0x7f857005d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8575d1e700 1 -- 172.21.15.61:0/3124691762 >> v1:172.21.15.111:6789/0 conn(0x7f8570060e80 legacy=0x7f857014ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8575d1e700 1 -- 172.21.15.61:0/3124691762 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85700aafa0 con 0x7f85701465a0 2022-07-12T08:47:05.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857551d700 1 --2- 172.21.15.61:0/3124691762 >> v2:172.21.15.111:3300/0 conn(0x7f857005cd20 0x7f857005d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:05.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8575d1e700 1 --2- 172.21.15.61:0/3124691762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85701465a0 0x7f85701469e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8568002f70 tx=0x7f8568006df0).ready entity=mon.2 client_cookie=fba42be93ff10241 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8567fff700 1 -- 172.21.15.61:0/3124691762 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856800d4a0 con 0x7f85701465a0 2022-07-12T08:47:05.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8567fff700 1 -- 172.21.15.61:0/3124691762 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568000e90 con 0x7f85701465a0 2022-07-12T08:47:05.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f8567fff700 1 -- 172.21.15.61:0/3124691762 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856800d7f0 con 0x7f85701465a0 2022-07-12T08:47:05.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 -- 172.21.15.61:0/3124691762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85701465a0 msgr2=0x7f85701469e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 --2- 172.21.15.61:0/3124691762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85701465a0 0x7f85701469e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8568002f70 tx=0x7f8568006df0).stop 2022-07-12T08:47:05.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 -- 172.21.15.61:0/3124691762 shutdown_connections 2022-07-12T08:47:05.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 --2- 172.21.15.61:0/3124691762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85701465a0 0x7f85701469e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 --2- 172.21.15.61:0/3124691762 >> v2:172.21.15.111:3300/0 conn(0x7f857005cd20 0x7f857005d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 -- 172.21.15.61:0/3124691762 >> 172.21.15.61:0/3124691762 conn(0x7f85700b1a50 msgr2=0x7f85700b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:05.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 -- 172.21.15.61:0/3124691762 shutdown_connections 2022-07-12T08:47:05.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 -- 172.21.15.61:0/3124691762 wait complete. 2022-07-12T08:47:05.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 Processor -- start 2022-07-12T08:47:05.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 -- start start 2022-07-12T08:47:05.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f857005cd20 0x7f85700b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.884+0000 7f857651f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8570060e80 0x7f857009a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 0x7f8570098310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8570159fb0 con 0x7f85701465a0 2022-07-12T08:47:05.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85700ac5f0 con 0x7f857005cd20 2022-07-12T08:47:05.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85700b4360 con 0x7f8570060e80 2022-07-12T08:47:05.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 0x7f8570098310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 0x7f8570098310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57240/0 (socket says 172.21.15.61:57240) 2022-07-12T08:47:05.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 -- 172.21.15.61:0/1066206158 learned_addr learned my addr 172.21.15.61:0/1066206158 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:05.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857551d700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f857005cd20 0x7f85700b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8574d1c700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8570060e80 0x7f857009a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 -- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8570060e80 msgr2=0x7f857009a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8570060e80 0x7f857009a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 -- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f857005cd20 msgr2=0x7f85700b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:05.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f857005cd20 0x7f85700b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:05.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 -- 172.21.15.61:0/1066206158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856000b180 con 0x7f85701465a0 2022-07-12T08:47:05.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f8575d1e700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 0x7f8570098310 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f8568002f70 tx=0x7f8568014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8568016070 con 0x7f85701465a0 2022-07-12T08:47:05.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856800b180 con 0x7f85701465a0 2022-07-12T08:47:05.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568013850 con 0x7f85701465a0 2022-07-12T08:47:05.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8568013bb0 con 0x7f85701465a0 2022-07-12T08:47:05.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857009ad50 con 0x7f85701465a0 2022-07-12T08:47:05.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8570159df0 con 0x7f85701465a0 2022-07-12T08:47:05.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f856800f030 con 0x7f85701465a0 2022-07-12T08:47:05.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f85667fc700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8558053a40 0x7f8558055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:05.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.888+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f856806cab0 con 0x7f85701465a0 2022-07-12T08:47:05.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.892+0000 7f857551d700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8558053a40 0x7f8558055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:05.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.892+0000 7f857551d700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8558053a40 0x7f8558055f00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f856000b000 tx=0x7f8560006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:05.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:05.896+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8568043970 con 0x7f85701465a0 2022-07-12T08:47:06.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.060+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f85700ac7b0 con 0x7f85701465a0 2022-07-12T08:47:06.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.068+0000 7f85667fc700 1 -- 172.21.15.61:0/1066206158 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8568046f90 con 0x7f85701465a0 2022-07-12T08:47:06.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8558053a40 msgr2=0x7f8558055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8558053a40 0x7f8558055f00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f856000b000 tx=0x7f8560006df0).stop 2022-07-12T08:47:06.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 msgr2=0x7f8570098310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 0x7f8570098310 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f8568002f70 tx=0x7f8568014000).stop 2022-07-12T08:47:06.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 shutdown_connections 2022-07-12T08:47:06.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85701465a0 0x7f8570098310 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8570060e80 0x7f857009a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8558053a40 0x7f8558055f00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 --2- 172.21.15.61:0/1066206158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f857005cd20 0x7f85700b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 >> 172.21.15.61:0/1066206158 conn(0x7f85700b1a50 msgr2=0x7f8570146980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:06.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 shutdown_connections 2022-07-12T08:47:06.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.072+0000 7f857651f700 1 -- 172.21.15.61:0/1066206158 wait complete. 2022-07-12T08:47:06.088 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:47:06.090 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:47:06.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.232+0000 7facd9c24700 1 Processor -- start 2022-07-12T08:47:06.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.232+0000 7facd9c24700 1 -- start start 2022-07-12T08:47:06.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7facd405fdf0 0x7facd40b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd405ccc0 con 0x7facd405c8e0 2022-07-12T08:47:06.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd405f660 con 0x7facd4146100 2022-07-12T08:47:06.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd40b2040 con 0x7facd405fdf0 2022-07-12T08:47:06.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd8c22700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7facd405c8e0 0x7facd40adb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45464/0 (socket says 172.21.15.61:45464) 2022-07-12T08:47:06.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd8c22700 1 -- 172.21.15.61:0/168863524 learned_addr learned my addr 172.21.15.61:0/168863524 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:06.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd3fff700 1 --2- 172.21.15.61:0/168863524 >> v2:172.21.15.61:3300/0 conn(0x7facd405fdf0 0x7facd40b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd37fe700 1 -- 172.21.15.61:0/168863524 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263022884 0 0) 0x7facd405ccc0 con 0x7facd405c8e0 2022-07-12T08:47:06.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd37fe700 1 -- 172.21.15.61:0/168863524 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8003260 con 0x7facd405c8e0 2022-07-12T08:47:06.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd3fff700 1 -- 172.21.15.61:0/168863524 >> v1:172.21.15.111:6789/0 conn(0x7facd4146100 legacy=0x7facd413fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd3fff700 1 -- 172.21.15.61:0/168863524 >> v1:172.21.15.61:6789/0 conn(0x7facd405c8e0 legacy=0x7facd40adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd3fff700 1 -- 172.21.15.61:0/168863524 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40ad670 con 0x7facd405fdf0 2022-07-12T08:47:06.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd3fff700 1 --2- 172.21.15.61:0/168863524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd405fdf0 0x7facd40b13f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7facc0002f70 tx=0x7facc0006df0).ready entity=mon.0 client_cookie=8f43ff1db5363586 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd37fe700 1 -- 172.21.15.61:0/168863524 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facc000d610 con 0x7facd405fdf0 2022-07-12T08:47:06.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd37fe700 1 -- 172.21.15.61:0/168863524 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc0000e90 con 0x7facd405fdf0 2022-07-12T08:47:06.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd37fe700 1 -- 172.21.15.61:0/168863524 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facc000d960 con 0x7facd405fdf0 2022-07-12T08:47:06.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- 172.21.15.61:0/168863524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd405fdf0 msgr2=0x7facd40b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 --2- 172.21.15.61:0/168863524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd405fdf0 0x7facd40b13f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7facc0002f70 tx=0x7facc0006df0).stop 2022-07-12T08:47:06.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- 172.21.15.61:0/168863524 shutdown_connections 2022-07-12T08:47:06.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 --2- 172.21.15.61:0/168863524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd405fdf0 0x7facd40b13f0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- 172.21.15.61:0/168863524 >> 172.21.15.61:0/168863524 conn(0x7facd4063a60 msgr2=0x7facd4063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:06.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- 172.21.15.61:0/168863524 shutdown_connections 2022-07-12T08:47:06.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- 172.21.15.61:0/168863524 wait complete. 2022-07-12T08:47:06.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 Processor -- start 2022-07-12T08:47:06.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.236+0000 7facd9c24700 1 -- start start 2022-07-12T08:47:06.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facd405c8e0 0x7facd40674c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7facd4146100 0x7facd4113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 0x7facd413d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4152060 con 0x7facd4113b60 2022-07-12T08:47:06.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4152300 con 0x7facd405c8e0 2022-07-12T08:47:06.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd41525a0 con 0x7facd4146100 2022-07-12T08:47:06.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 0x7facd413d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7facd4146100 0x7facd4113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd8c22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facd405c8e0 0x7facd40674c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 0x7facd413d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57256/0 (socket says 172.21.15.61:57256) 2022-07-12T08:47:06.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7facd4146100 0x7facd4113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39448/0 (socket says 172.21.15.61:39448) 2022-07-12T08:47:06.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 -- 172.21.15.61:0/2395795167 learned_addr learned my addr 172.21.15.61:0/2395795167 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:06.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 -- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7facd4146100 msgr2=0x7facd4113670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7facd4146100 0x7facd4113670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 -- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facd405c8e0 msgr2=0x7facd40674c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facd405c8e0 0x7facd40674c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 -- 172.21.15.61:0/2395795167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc000b180 con 0x7facd4113b60 2022-07-12T08:47:06.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9423700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 0x7facd413d320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7facd40634e0 tx=0x7faccc0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faccc006aa0 con 0x7facd4113b60 2022-07-12T08:47:06.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc006c00 con 0x7facd4113b60 2022-07-12T08:47:06.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faccc007030 con 0x7facd4113b60 2022-07-12T08:47:06.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd4151d10 con 0x7facd4113b60 2022-07-12T08:47:06.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd41427f0 con 0x7facd4113b60 2022-07-12T08:47:06.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.240+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd4151ea0 con 0x7facd4113b60 2022-07-12T08:47:06.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.244+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faccc010030 con 0x7facd4113b60 2022-07-12T08:47:06.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.244+0000 7facd1ffb700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7facb4053ad0 0x7facb4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.244+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faccc06bcd0 con 0x7facd4113b60 2022-07-12T08:47:06.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.244+0000 7facd8c22700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7facb4053ad0 0x7facb4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.244+0000 7facd8c22700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7facb4053ad0 0x7facb4055f90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7facc800d000 tx=0x7facc8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.252+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faccc042c30 con 0x7facd4113b60 2022-07-12T08:47:06.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.420+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7facd4152760 con 0x7facd4113b60 2022-07-12T08:47:06.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd1ffb700 1 -- 172.21.15.61:0/2395795167 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7faccc046250 con 0x7facd4113b60 2022-07-12T08:47:06.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7facb4053ad0 msgr2=0x7facb4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7facb4053ad0 0x7facb4055f90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7facc800d000 tx=0x7facc8009040).stop 2022-07-12T08:47:06.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 msgr2=0x7facd413d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 0x7facd413d320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7facd40634e0 tx=0x7faccc0013a0).stop 2022-07-12T08:47:06.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 shutdown_connections 2022-07-12T08:47:06.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7facd4113b60 0x7facd413d320 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7facd4146100 0x7facd4113670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7facb4053ad0 0x7facb4055f90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 --2- 172.21.15.61:0/2395795167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facd405c8e0 0x7facd40674c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 >> 172.21.15.61:0/2395795167 conn(0x7facd4063a60 msgr2=0x7facd413e510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:06.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 shutdown_connections 2022-07-12T08:47:06.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.428+0000 7facd9c24700 1 -- 172.21.15.61:0/2395795167 wait complete. 2022-07-12T08:47:06.443 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:47:06.444 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:47:06.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 Processor -- start 2022-07-12T08:47:06.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 -- start start 2022-07-12T08:47:06.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fade00b4210 0x7fade005bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fade00acb60 0x7fade00ada30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00b2050 con 0x7fade00adf20 2022-07-12T08:47:06.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade005cea0 con 0x7fade00acb60 2022-07-12T08:47:06.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade74d2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade005c200 con 0x7fade00b4210 2022-07-12T08:47:06.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fade00acb60 0x7fade00ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fade00acb60 0x7fade00ada30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57262/0 (socket says 172.21.15.61:57262) 2022-07-12T08:47:06.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 -- 172.21.15.61:0/3442457398 learned_addr learned my addr 172.21.15.61:0/3442457398 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:06.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade526e700 1 --2- 172.21.15.61:0/3442457398 >> v2:172.21.15.111:3300/0 conn(0x7fade00b4210 0x7fade005bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 -- 172.21.15.61:0/3442457398 >> v2:172.21.15.111:3300/0 conn(0x7fade00b4210 msgr2=0x7fade005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:06.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 --2- 172.21.15.61:0/3442457398 >> v2:172.21.15.111:3300/0 conn(0x7fade00b4210 0x7fade005bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 -- 172.21.15.61:0/3442457398 >> v1:172.21.15.111:6789/0 conn(0x7fade00adf20 legacy=0x7fade0146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 -- 172.21.15.61:0/3442457398 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00ae360 con 0x7fade00acb60 2022-07-12T08:47:06.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fade4a6d700 1 --2- 172.21.15.61:0/3442457398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00acb60 0x7fade00ada30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fadd0002f70 tx=0x7fadd0006eb0).ready entity=mon.0 client_cookie=8cfb51bcdb7bb3ca server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fadd7fff700 1 -- 172.21.15.61:0/3442457398 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd000d5c0 con 0x7fade00acb60 2022-07-12T08:47:06.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fadd7fff700 1 -- 172.21.15.61:0/3442457398 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0000e90 con 0x7fade00acb60 2022-07-12T08:47:06.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.572+0000 7fadd7fff700 1 -- 172.21.15.61:0/3442457398 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd000d950 con 0x7fade00acb60 2022-07-12T08:47:06.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- 172.21.15.61:0/3442457398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00acb60 msgr2=0x7fade00ada30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 --2- 172.21.15.61:0/3442457398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00acb60 0x7fade00ada30 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fadd0002f70 tx=0x7fadd0006eb0).stop 2022-07-12T08:47:06.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- 172.21.15.61:0/3442457398 shutdown_connections 2022-07-12T08:47:06.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 --2- 172.21.15.61:0/3442457398 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00acb60 0x7fade00ada30 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 --2- 172.21.15.61:0/3442457398 >> v2:172.21.15.111:3300/0 conn(0x7fade00b4210 0x7fade005bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- 172.21.15.61:0/3442457398 >> 172.21.15.61:0/3442457398 conn(0x7fade0064e10 msgr2=0x7fade00651f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:06.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- 172.21.15.61:0/3442457398 shutdown_connections 2022-07-12T08:47:06.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- 172.21.15.61:0/3442457398 wait complete. 2022-07-12T08:47:06.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 Processor -- start 2022-07-12T08:47:06.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- start start 2022-07-12T08:47:06.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fade00acb60 0x7fade0066520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fade00adf20 0x7fade0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 0x7fade0148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade0149870 con 0x7fade00b4210 2022-07-12T08:47:06.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade014a590 con 0x7fade00acb60 2022-07-12T08:47:06.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade74d2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade0066b50 con 0x7fade00adf20 2022-07-12T08:47:06.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade5a6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 0x7fade0148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade526e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fade00acb60 0x7fade0066520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade5a6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 0x7fade0148a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57270/0 (socket says 172.21.15.61:57270) 2022-07-12T08:47:06.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade4a6d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fade00adf20 0x7fade0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.576+0000 7fade5a6f700 1 -- 172.21.15.61:0/1723167754 learned_addr learned my addr 172.21.15.61:0/1723167754 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:06.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade526e700 1 -- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fade00adf20 msgr2=0x7fade0159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade526e700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fade00adf20 0x7fade0159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade526e700 1 -- 172.21.15.61:0/1723167754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 msgr2=0x7fade0148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade526e700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 0x7fade0148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade526e700 1 -- 172.21.15.61:0/1723167754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd000b1e0 con 0x7fade00acb60 2022-07-12T08:47:06.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade5a6f700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 0x7fade0148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:06.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade526e700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fade00acb60 0x7fade0066520 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7faddc00c000 tx=0x7faddc006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faddc011750 con 0x7fade00acb60 2022-07-12T08:47:06.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faddc0118b0 con 0x7fade00acb60 2022-07-12T08:47:06.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faddc011c10 con 0x7fade00acb60 2022-07-12T08:47:06.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade0066df0 con 0x7fade00acb60 2022-07-12T08:47:06.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade0148f00 con 0x7fade00acb60 2022-07-12T08:47:06.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade0057460 con 0x7fade00acb60 2022-07-12T08:47:06.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.580+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faddc011d70 con 0x7fade00acb60 2022-07-12T08:47:06.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.584+0000 7fadd67fc700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fadc8053af0 0x7fadc8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.584+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faddc06c240 con 0x7fade00acb60 2022-07-12T08:47:06.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.584+0000 7fade4a6d700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fadc8053af0 0x7fadc8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.584+0000 7fade4a6d700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fadc8053af0 0x7fadc8055fb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fadd001c000 tx=0x7fadd0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.592+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faddc043030 con 0x7fade00acb60 2022-07-12T08:47:06.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.760+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fade00447a0 con 0x7fade00acb60 2022-07-12T08:47:06.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.768+0000 7fadd67fc700 1 -- 172.21.15.61:0/1723167754 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7faddc046650 con 0x7fade00acb60 2022-07-12T08:47:06.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.768+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fadc8053af0 msgr2=0x7fadc8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.768+0000 7fade74d2700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fadc8053af0 0x7fadc8055fb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fadd001c000 tx=0x7fadd0015000).stop 2022-07-12T08:47:06.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.768+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fade00acb60 msgr2=0x7fade0066520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.768+0000 7fade74d2700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fade00acb60 0x7fade0066520 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7faddc00c000 tx=0x7faddc006df0).stop 2022-07-12T08:47:06.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 shutdown_connections 2022-07-12T08:47:06.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade00b4210 0x7fade0148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fade00adf20 0x7fade0159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fadc8053af0 0x7fadc8055fb0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 --2- 172.21.15.61:0/1723167754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fade00acb60 0x7fade0066520 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 >> 172.21.15.61:0/1723167754 conn(0x7fade0064e10 msgr2=0x7fade00b6590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:06.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 shutdown_connections 2022-07-12T08:47:06.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.772+0000 7fade74d2700 1 -- 172.21.15.61:0/1723167754 wait complete. 2022-07-12T08:47:06.784 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:47:06.786 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:06.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 Processor -- start 2022-07-12T08:47:06.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 -- start start 2022-07-12T08:47:06.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f76f80af280 0x7f76f8060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f809bce0 0x7f76f80af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f80aecb0 con 0x7f76f80b3f60 2022-07-12T08:47:06.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f8143b00 con 0x7f76f80af280 2022-07-12T08:47:06.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76fcc6f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81435c0 con 0x7f76f809bce0 2022-07-12T08:47:06.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f76f80af280 0x7f76f8060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f76f80b3f60 0x7f76f80b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59098/0 (socket says 172.21.15.61:59098) 2022-07-12T08:47:06.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f7fff700 1 -- 172.21.15.61:0/842713150 learned_addr learned my addr 172.21.15.61:0/842713150 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:06.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f6ffd700 1 --2- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f809bce0 0x7f76f80af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f77fe700 1 -- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f809bce0 msgr2=0x7f76f80af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:06.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f77fe700 1 --2- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f809bce0 0x7f76f80af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f77fe700 1 -- 172.21.15.61:0/842713150 >> v1:172.21.15.111:6789/0 conn(0x7f76f80b3f60 legacy=0x7f76f80b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.892+0000 7f76f77fe700 1 -- 172.21.15.61:0/842713150 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f8143860 con 0x7f76f80af280 2022-07-12T08:47:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76f77fe700 1 --2- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80af280 0x7f76f8060680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8006d90).ready entity=mon.1 client_cookie=bcc504c7626846fd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76f67fc700 1 -- 172.21.15.61:0/842713150 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e800d4a0 con 0x7f76f80af280 2022-07-12T08:47:06.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76f67fc700 1 -- 172.21.15.61:0/842713150 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e8000e90 con 0x7f76f80af280 2022-07-12T08:47:06.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76f67fc700 1 -- 172.21.15.61:0/842713150 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e800d7f0 con 0x7f76f80af280 2022-07-12T08:47:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 -- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80af280 msgr2=0x7f76f8060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80af280 0x7f76f8060680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8006d90).stop 2022-07-12T08:47:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 -- 172.21.15.61:0/842713150 shutdown_connections 2022-07-12T08:47:06.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f809bce0 0x7f76f80af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/842713150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80af280 0x7f76f8060680 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 -- 172.21.15.61:0/842713150 >> 172.21.15.61:0/842713150 conn(0x7f76f80b92c0 msgr2=0x7f76f80b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:06.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 -- 172.21.15.61:0/842713150 shutdown_connections 2022-07-12T08:47:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 -- 172.21.15.61:0/842713150 wait complete. 2022-07-12T08:47:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 Processor -- start 2022-07-12T08:47:06.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 -- start start 2022-07-12T08:47:06.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 0x7f76f8155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f80af280 0x7f76f8147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.896+0000 7f76fcc6f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80b3f60 0x7f76f809c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76fcc6f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80b0880 con 0x7f76f809bce0 2022-07-12T08:47:06.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76fcc6f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80b24b0 con 0x7f76f80b3f60 2022-07-12T08:47:06.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76fcc6f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8142690 con 0x7f76f80af280 2022-07-12T08:47:06.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 0x7f76f8155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 0x7f76f8155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57280/0 (socket says 172.21.15.61:57280) 2022-07-12T08:47:06.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80b3f60 0x7f76f809c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 -- 172.21.15.61:0/1998089410 learned_addr learned my addr 172.21.15.61:0/1998089410 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:06.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f6ffd700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f80af280 0x7f76f8147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 -- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f80af280 msgr2=0x7f76f8147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f80af280 0x7f76f8147b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 -- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80b3f60 msgr2=0x7f76f809c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:06.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80b3f60 0x7f76f809c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:06.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 -- 172.21.15.61:0/1998089410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e800b120 con 0x7f76f809bce0 2022-07-12T08:47:06.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f77fe700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 0x7f76f8155420 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8011840 con 0x7f76f809bce0 2022-07-12T08:47:06.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e80119a0 con 0x7f76f809bce0 2022-07-12T08:47:06.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f80b1340 con 0x7f76f809bce0 2022-07-12T08:47:06.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8011d00 con 0x7f76f809bce0 2022-07-12T08:47:06.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f809ce60 con 0x7f76f809bce0 2022-07-12T08:47:06.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.900+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f809d3c0 con 0x7f76f809bce0 2022-07-12T08:47:06.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.904+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f76e800f070 con 0x7f76f809bce0 2022-07-12T08:47:06.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.904+0000 7f76f4ff9700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f76dc053a40 0x7f76dc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:06.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.904+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f76e806d380 con 0x7f76f809bce0 2022-07-12T08:47:06.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.904+0000 7f76f6ffd700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f76dc053a40 0x7f76dc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:06.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.904+0000 7f76f6ffd700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f76dc053a40 0x7f76dc055f00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f76e000b000 tx=0x7f76e0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:06.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:06.908+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76e8044320 con 0x7f76f809bce0 2022-07-12T08:47:07.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.072+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76f8057460 con 0x7f76f809bce0 2022-07-12T08:47:07.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.072+0000 7f76f4ff9700 1 -- 172.21.15.61:0/1998089410 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f76e8047900 con 0x7f76f809bce0 2022-07-12T08:47:07.078 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:07.079 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:07.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f76dc053a40 msgr2=0x7f76dc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f76dc053a40 0x7f76dc055f00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f76e000b000 tx=0x7f76e0006df0).stop 2022-07-12T08:47:07.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 msgr2=0x7f76f8155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 0x7f76f8155420 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8014000).stop 2022-07-12T08:47:07.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 shutdown_connections 2022-07-12T08:47:07.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f76dc053a40 0x7f76dc055f00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76f80b3f60 0x7f76f809c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f76f80af280 0x7f76f8147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 --2- 172.21.15.61:0/1998089410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76f809bce0 0x7f76f8155420 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 >> 172.21.15.61:0/1998089410 conn(0x7f76f80b92c0 msgr2=0x7f76f80b8190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:07.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 shutdown_connections 2022-07-12T08:47:07.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.076+0000 7f76fcc6f700 1 -- 172.21.15.61:0/1998089410 wait complete. 2022-07-12T08:47:07.084 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:07.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:47:03.397499+0000', 'modified': '2022-07-12T08:47:05.156395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6196}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6196': {'gid': 6196, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3265139715', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3265139715}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3265139715}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:47:07.301 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:47:07.302 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:47:07.423 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6196,"want_state":"up:active","state":"up:active","rank_uptime":4.0136324219999997,"mdsmap_epoch":38,"osdmap_epoch":71,"osdmap_epoch_barrier":71,"uptime":8.3616975890000003} 2022-07-12T08:47:07.424 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6196, "mdsmap_epoch": 38, "osdmap_epoch": 71, "osdmap_epoch_barrier": 71, "rank_uptime": 4.013632422, "state": "up:active", "uptime": 8.361697589, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:47:07.425 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:07.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 Processor -- start 2022-07-12T08:47:07.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 -- start start 2022-07-12T08:47:07.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71500ab670 0x7f715005f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f71500afa60 0x7f71500afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71500abc70 con 0x7f71500ab290 2022-07-12T08:47:07.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71500aba90 con 0x7f71500ab670 2022-07-12T08:47:07.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f7154d17700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715005c720 con 0x7f71500afa60 2022-07-12T08:47:07.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71500ab670 0x7f715005f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f71500ab290 0x7f715005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45506/0 (socket says 172.21.15.61:45506) 2022-07-12T08:47:07.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714f7fe700 1 -- 172.21.15.61:0/534256937 learned_addr learned my addr 172.21.15.61:0/534256937 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:07.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714ffff700 1 --2- 172.21.15.61:0/534256937 >> v2:172.21.15.111:3300/0 conn(0x7f71500afa60 0x7f71500afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714effd700 1 -- 172.21.15.61:0/534256937 >> v2:172.21.15.111:3300/0 conn(0x7f71500afa60 msgr2=0x7f71500afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:07.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714effd700 1 --2- 172.21.15.61:0/534256937 >> v2:172.21.15.111:3300/0 conn(0x7f71500afa60 0x7f71500afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714effd700 1 -- 172.21.15.61:0/534256937 >> v1:172.21.15.61:6789/0 conn(0x7f71500ab290 legacy=0x7f715005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714effd700 1 -- 172.21.15.61:0/534256937 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715005c9c0 con 0x7f71500ab670 2022-07-12T08:47:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.532+0000 7f714effd700 1 --2- 172.21.15.61:0/534256937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500ab670 0x7f715005f140 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f7144002f70 tx=0x7f7144006df0).ready entity=mon.0 client_cookie=8946b7f15e9418aa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f714e7fc700 1 -- 172.21.15.61:0/534256937 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f714400d5c0 con 0x7f71500ab670 2022-07-12T08:47:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f714e7fc700 1 -- 172.21.15.61:0/534256937 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7144000e90 con 0x7f71500ab670 2022-07-12T08:47:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f714e7fc700 1 -- 172.21.15.61:0/534256937 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f714400d910 con 0x7f71500ab670 2022-07-12T08:47:07.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- 172.21.15.61:0/534256937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500ab670 msgr2=0x7f715005f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 --2- 172.21.15.61:0/534256937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500ab670 0x7f715005f140 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f7144002f70 tx=0x7f7144006df0).stop 2022-07-12T08:47:07.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- 172.21.15.61:0/534256937 shutdown_connections 2022-07-12T08:47:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 --2- 172.21.15.61:0/534256937 >> v2:172.21.15.111:3300/0 conn(0x7f71500afa60 0x7f71500afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 --2- 172.21.15.61:0/534256937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500ab670 0x7f715005f140 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- 172.21.15.61:0/534256937 >> 172.21.15.61:0/534256937 conn(0x7f71500b1950 msgr2=0x7f71500b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- 172.21.15.61:0/534256937 shutdown_connections 2022-07-12T08:47:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- 172.21.15.61:0/534256937 wait complete. 2022-07-12T08:47:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 Processor -- start 2022-07-12T08:47:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- start start 2022-07-12T08:47:07.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71500ab290 0x7f7150149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71500ab670 0x7f7150066870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 0x7f7150155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715014a210 con 0x7f71500afa60 2022-07-12T08:47:07.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150148720 con 0x7f71500ab670 2022-07-12T08:47:07.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.536+0000 7f7154d17700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71501489c0 con 0x7f71500ab290 2022-07-12T08:47:07.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 0x7f7150155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 0x7f7150155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57298/0 (socket says 172.21.15.61:57298) 2022-07-12T08:47:07.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 -- 172.21.15.61:0/2708430543 learned_addr learned my addr 172.21.15.61:0/2708430543 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:07.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714effd700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71500ab670 0x7f7150066870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714f7fe700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71500ab290 0x7f7150149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 -- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71500ab290 msgr2=0x7f7150149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71500ab290 0x7f7150149100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 -- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71500ab670 msgr2=0x7f7150066870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71500ab670 0x7f7150066870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 -- 172.21.15.61:0/2708430543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714400b180 con 0x7f71500afa60 2022-07-12T08:47:07.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714effd700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71500ab670 0x7f7150066870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:07.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714f7fe700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71500ab290 0x7f7150149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:07.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714ffff700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 0x7f7150155e20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f714000a000 tx=0x7f7140008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:07.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7140011780 con 0x7f71500afa60 2022-07-12T08:47:07.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71400118e0 con 0x7f71500afa60 2022-07-12T08:47:07.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7140011c40 con 0x7f71500afa60 2022-07-12T08:47:07.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7150148c60 con 0x7f71500afa60 2022-07-12T08:47:07.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7150066d60 con 0x7f71500afa60 2022-07-12T08:47:07.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71500678b0 con 0x7f71500afa60 2022-07-12T08:47:07.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.540+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7140012420 con 0x7f71500afa60 2022-07-12T08:47:07.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.544+0000 7f714cff9700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7130053a40 0x7f7130055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.544+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f714006bd40 con 0x7f71500afa60 2022-07-12T08:47:07.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.544+0000 7f714f7fe700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7130053a40 0x7f7130055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.544+0000 7f714f7fe700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7130053a40 0x7f7130055f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f714800b000 tx=0x7f7148009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:07.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.548+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71500678b0 con 0x7f71500afa60 2022-07-12T08:47:07.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.672+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71500678b0 con 0x7f71500afa60 2022-07-12T08:47:07.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.672+0000 7f714cff9700 1 -- 172.21.15.61:0/2708430543 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f71500678b0 con 0x7f71500afa60 2022-07-12T08:47:07.676 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:07.677 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":71,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:04.401539+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:47:01.995190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-12T08:47:02.575389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:07.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7130053a40 msgr2=0x7f7130055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7130053a40 0x7f7130055f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f714800b000 tx=0x7f7148009040).stop 2022-07-12T08:47:07.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 msgr2=0x7f7150155e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 0x7f7150155e20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f714000a000 tx=0x7f7140008040).stop 2022-07-12T08:47:07.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 shutdown_connections 2022-07-12T08:47:07.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71500afa60 0x7f7150155e20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7130053a40 0x7f7130055f00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f71500ab670 0x7f7150066870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 --2- 172.21.15.61:0/2708430543 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f71500ab290 0x7f7150149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 >> 172.21.15.61:0/2708430543 conn(0x7f71500b1950 msgr2=0x7f71500afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:07.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 shutdown_connections 2022-07-12T08:47:07.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.676+0000 7f7154d17700 1 -- 172.21.15.61:0/2708430543 wait complete. 2022-07-12T08:47:07.692 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:47:01.995190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:47:02.575389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:47:07.693 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:47:07.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 Processor -- start 2022-07-12T08:47:07.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 -- start start 2022-07-12T08:47:07.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b005d060 0x7fe6b00b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6b00afb50 0x7fe6b0143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b005d440 con 0x7fe6b0142ab0 2022-07-12T08:47:07.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b0143d80 con 0x7fe6b00afb50 2022-07-12T08:47:07.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6a1a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b00674b0 con 0x7fe6b005d060 2022-07-12T08:47:07.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b6219700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe6b00afb50 0x7fe6b0143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b5217700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe6b0142ab0 0x7fe6b0142f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59122/0 (socket says 172.21.15.61:59122) 2022-07-12T08:47:07.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b5217700 1 -- 172.21.15.61:0/3624102922 learned_addr learned my addr 172.21.15.61:0/3624102922 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:07.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.800+0000 7fe6b5a18700 1 --2- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b005d060 0x7fe6b00b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6219700 1 -- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b005d060 msgr2=0x7fe6b00b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:07.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6219700 1 --2- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b005d060 0x7fe6b00b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6219700 1 -- 172.21.15.61:0/3624102922 >> v1:172.21.15.111:6789/0 conn(0x7fe6b0142ab0 legacy=0x7fe6b0142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6219700 1 -- 172.21.15.61:0/3624102922 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0067750 con 0x7fe6b00afb50 2022-07-12T08:47:07.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6219700 1 --2- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b00afb50 0x7fe6b0143620 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe6a80032f0 tx=0x7fe6a80091d0).ready entity=mon.1 client_cookie=23423bdb95c7476a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:07.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b4a16700 1 -- 172.21.15.61:0/3624102922 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a800b6c0 con 0x7fe6b00afb50 2022-07-12T08:47:07.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b4a16700 1 -- 172.21.15.61:0/3624102922 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a800b820 con 0x7fe6b00afb50 2022-07-12T08:47:07.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b4a16700 1 -- 172.21.15.61:0/3624102922 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a800b990 con 0x7fe6b00afb50 2022-07-12T08:47:07.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b00afb50 msgr2=0x7fe6b0143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b00afb50 0x7fe6b0143620 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fe6a80032f0 tx=0x7fe6a80091d0).stop 2022-07-12T08:47:07.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3624102922 shutdown_connections 2022-07-12T08:47:07.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b00afb50 0x7fe6b0143620 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3624102922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b005d060 0x7fe6b00b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3624102922 >> 172.21.15.61:0/3624102922 conn(0x7fe6b00b6900 msgr2=0x7fe6b0066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:07.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3624102922 shutdown_connections 2022-07-12T08:47:07.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3624102922 wait complete. 2022-07-12T08:47:07.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 Processor -- start 2022-07-12T08:47:07.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.804+0000 7fe6b6a1a700 1 -- start start 2022-07-12T08:47:07.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b005d060 0x7fe6b0152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b00afb50 0x7fe6b01557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 0x7fe6b0143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b009c700 con 0x7fe6b009c940 2022-07-12T08:47:07.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0156020 con 0x7fe6b005d060 2022-07-12T08:47:07.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b0152fc0 con 0x7fe6b00afb50 2022-07-12T08:47:07.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 0x7fe6b0143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 0x7fe6b0143fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57312/0 (socket says 172.21.15.61:57312) 2022-07-12T08:47:07.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 -- 172.21.15.61:0/3772128296 learned_addr learned my addr 172.21.15.61:0/3772128296 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:07.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b5a18700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b005d060 0x7fe6b0152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b5217700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b00afb50 0x7fe6b01557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 -- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b00afb50 msgr2=0x7fe6b01557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b00afb50 0x7fe6b01557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 -- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b005d060 msgr2=0x7fe6b0152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b005d060 0x7fe6b0152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 -- 172.21.15.61:0/3772128296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a8009470 con 0x7fe6b009c940 2022-07-12T08:47:07.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6219700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 0x7fe6b0143fc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fe6a80032f0 tx=0x7fe6a8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:07.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a800b6c0 con 0x7fe6b009c940 2022-07-12T08:47:07.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a8021630 con 0x7fe6b009c940 2022-07-12T08:47:07.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b0155cc0 con 0x7fe6b009c940 2022-07-12T08:47:07.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a8021990 con 0x7fe6b009c940 2022-07-12T08:47:07.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b009d460 con 0x7fe6b009c940 2022-07-12T08:47:07.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.808+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe6a800f070 con 0x7fe6b009c940 2022-07-12T08:47:07.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.812+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b0156190 con 0x7fe6b009c940 2022-07-12T08:47:07.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.812+0000 7fe69effd700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6900539d0 0x7fe690055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:07.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.812+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe6a806c580 con 0x7fe6b009c940 2022-07-12T08:47:07.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.812+0000 7fe6b5a18700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6900539d0 0x7fe690055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:07.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.812+0000 7fe6b5a18700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6900539d0 0x7fe690055e90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe6a000b000 tx=0x7fe6a0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:07.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.816+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6b0156190 con 0x7fe6b009c940 2022-07-12T08:47:07.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe6b0156190 con 0x7fe6b009c940 2022-07-12T08:47:07.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe69effd700 1 -- 172.21.15.61:0/3772128296 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v71) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe6b0156190 con 0x7fe6b009c940 2022-07-12T08:47:07.944 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:47:07.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6900539d0 msgr2=0x7fe690055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6900539d0 0x7fe690055e90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe6a000b000 tx=0x7fe6a0006df0).stop 2022-07-12T08:47:07.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 msgr2=0x7fe6b0143fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:07.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 0x7fe6b0143fc0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7fe6a80032f0 tx=0x7fe6a8016000).stop 2022-07-12T08:47:07.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 shutdown_connections 2022-07-12T08:47:07.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6b009c940 0x7fe6b0143fc0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe6b00afb50 0x7fe6b01557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe6900539d0 0x7fe690055e90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 --2- 172.21.15.61:0/3772128296 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe6b005d060 0x7fe6b0152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:07.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.943+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 >> 172.21.15.61:0/3772128296 conn(0x7fe6b00b6900 msgr2=0x7fe6b013d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:07.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.947+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 shutdown_connections 2022-07-12T08:47:07.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:07.947+0000 7fe6b6a1a700 1 -- 172.21.15.61:0/3772128296 wait complete. 2022-07-12T08:47:07.961 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:47:08.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.075+0000 7f5b2ec2b700 1 Processor -- start 2022-07-12T08:47:08.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.075+0000 7f5b2ec2b700 1 -- start start 2022-07-12T08:47:08.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b280b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5b280b05f0 0x7f5b28142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b280b3560 con 0x7f5b280b3730 2022-07-12T08:47:08.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b280b5b80 con 0x7f5b280b05f0 2022-07-12T08:47:08.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280b5e20 con 0x7f5b280b5480 2022-07-12T08:47:08.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5b280b05f0 0x7f5b28142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5b280b05f0 0x7f5b28142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57320/0 (socket says 172.21.15.61:57320) 2022-07-12T08:47:08.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 -- 172.21.15.61:0/1492033270 learned_addr learned my addr 172.21.15.61:0/1492033270 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:08.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2c9c7700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b280b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 -- 172.21.15.61:0/1492033270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 msgr2=0x7f5b280b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b277fe700 1 -- 172.21.15.61:0/1492033270 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602467695 0 0) 0x7f5b280b3560 con 0x7f5b280b3730 2022-07-12T08:47:08.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b280b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 -- 172.21.15.61:0/1492033270 >> v1:172.21.15.111:6789/0 conn(0x7f5b280b3730 legacy=0x7f5b280afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 -- 172.21.15.61:0/1492033270 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b280b5860 con 0x7f5b280b05f0 2022-07-12T08:47:08.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2c9c7700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b280b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:08.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2d1c8700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b05f0 0x7f5b28142f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f5b200032f0 tx=0x7f5b20009290).ready entity=mon.0 client_cookie=b0967afc0c71432b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b277fe700 1 -- 172.21.15.61:0/1492033270 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b2000b8a0 con 0x7f5b280b05f0 2022-07-12T08:47:08.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b277fe700 1 -- 172.21.15.61:0/1492033270 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000ba00 con 0x7f5b280b05f0 2022-07-12T08:47:08.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b277fe700 1 -- 172.21.15.61:0/1492033270 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b2000bb70 con 0x7f5b280b05f0 2022-07-12T08:47:08.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/1492033270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b05f0 msgr2=0x7f5b28142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b05f0 0x7f5b28142f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f5b200032f0 tx=0x7f5b20009290).stop 2022-07-12T08:47:08.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/1492033270 shutdown_connections 2022-07-12T08:47:08.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b05f0 0x7f5b28142f90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/1492033270 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b280b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/1492033270 >> 172.21.15.61:0/1492033270 conn(0x7f5b280b96b0 msgr2=0x7f5b280b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/1492033270 shutdown_connections 2022-07-12T08:47:08.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.079+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/1492033270 wait complete. 2022-07-12T08:47:08.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 Processor -- start 2022-07-12T08:47:08.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 -- start start 2022-07-12T08:47:08.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 0x7f5b281449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 0x7f5b2813e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b2813e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28140000 con 0x7f5b280b3730 2022-07-12T08:47:08.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28145b10 con 0x7f5b280b05f0 2022-07-12T08:47:08.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b280b0160 con 0x7f5b280b5480 2022-07-12T08:47:08.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 0x7f5b2813e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2c9c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 0x7f5b281449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 0x7f5b2813e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57324/0 (socket says 172.21.15.61:57324) 2022-07-12T08:47:08.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2c9c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 0x7f5b281449f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:35896/0 (socket says 172.21.15.61:35896) 2022-07-12T08:47:08.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 -- 172.21.15.61:0/2561622347 learned_addr learned my addr 172.21.15.61:0/2561622347 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:08.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2d1c8700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b2813e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 -- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 msgr2=0x7f5b2813e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b2813e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 -- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 msgr2=0x7f5b281449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 0x7f5b281449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 -- 172.21.15.61:0/2561622347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20009530 con 0x7f5b280b3730 2022-07-12T08:47:08.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2c9c7700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 0x7f5b281449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:08.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b27fff700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 0x7f5b2813e220 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f5b10000ed0 tx=0x7f5b10007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b10012400 con 0x7f5b280b3730 2022-07-12T08:47:08.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b10012560 con 0x7f5b280b3730 2022-07-12T08:47:08.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b280b0390 con 0x7f5b280b3730 2022-07-12T08:47:08.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.083+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b100128c0 con 0x7f5b280b3730 2022-07-12T08:47:08.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.087+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b280af620 con 0x7f5b280b3730 2022-07-12T08:47:08.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.087+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b28057460 con 0x7f5b280b3730 2022-07-12T08:47:08.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.087+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5b10019030 con 0x7f5b280b3730 2022-07-12T08:47:08.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.087+0000 7f5b25ffb700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5b0c0539b0 0x7f5b0c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.087+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5b1006b1a0 con 0x7f5b280b3730 2022-07-12T08:47:08.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.087+0000 7f5b2c9c7700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5b0c0539b0 0x7f5b0c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.091+0000 7f5b2c9c7700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5b0c0539b0 0x7f5b0c055e70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f5b1800c000 tx=0x7f5b18008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.095+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5b100421b0 con 0x7f5b280b3730 2022-07-12T08:47:08.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.223+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5b280ba5d0 con 0x7f5b280b3730 2022-07-12T08:47:08.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.223+0000 7f5b25ffb700 1 -- 172.21.15.61:0/2561622347 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5b100457d0 con 0x7f5b280b3730 2022-07-12T08:47:08.227 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5b0c0539b0 msgr2=0x7f5b0c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5b0c0539b0 0x7f5b0c055e70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f5b1800c000 tx=0x7f5b18008040).stop 2022-07-12T08:47:08.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 msgr2=0x7f5b2813e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 0x7f5b2813e220 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f5b10000ed0 tx=0x7f5b10007ab0).stop 2022-07-12T08:47:08.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 shutdown_connections 2022-07-12T08:47:08.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5b280b5480 0x7f5b2813e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b280b3730 0x7f5b2813e220 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5b0c0539b0 0x7f5b0c055e70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 --2- 172.21.15.61:0/2561622347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5b280b05f0 0x7f5b281449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 >> 172.21.15.61:0/2561622347 conn(0x7f5b280b96b0 msgr2=0x7f5b280b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 shutdown_connections 2022-07-12T08:47:08.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.227+0000 7f5b2ec2b700 1 -- 172.21.15.61:0/2561622347 wait complete. 2022-07-12T08:47:08.240 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:47:08.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f33fff700 1 Processor -- start 2022-07-12T08:47:08.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f33fff700 1 -- start start 2022-07-12T08:47:08.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f33fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5f340b5920 0x7f5f3414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f33fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3405c3e0 con 0x7f5f340b0280 2022-07-12T08:47:08.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f33fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340b5d00 con 0x7f5f340aa830 2022-07-12T08:47:08.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f33fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3405c610 con 0x7f5f340b5920 2022-07-12T08:47:08.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f327fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5f340b0280 0x7f5f3414aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45548/0 (socket says 172.21.15.61:45548) 2022-07-12T08:47:08.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f327fc700 1 -- 172.21.15.61:0/2486097740 learned_addr learned my addr 172.21.15.61:0/2486097740 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:08.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f32ffd700 1 --2- 172.21.15.61:0/2486097740 >> v2:172.21.15.111:3300/0 conn(0x7f5f340b5920 0x7f5f3414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683419275 0 0) 0x7f5f3405c3e0 con 0x7f5f340b0280 2022-07-12T08:47:08.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f18003260 con 0x7f5f340b0280 2022-07-12T08:47:08.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978412825 0 0) 0x7f5f340b5d00 con 0x7f5f340aa830 2022-07-12T08:47:08.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3405c3e0 con 0x7f5f340aa830 2022-07-12T08:47:08.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285200056 0 0) 0x7f5f18003260 con 0x7f5f340b0280 2022-07-12T08:47:08.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 >> v2:172.21.15.111:3300/0 conn(0x7f5f340b5920 msgr2=0x7f5f3414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 --2- 172.21.15.61:0/2486097740 >> v2:172.21.15.111:3300/0 conn(0x7f5f340b5920 0x7f5f3414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 >> v1:172.21.15.111:6789/0 conn(0x7f5f340aa830 legacy=0x7f5f340aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3405c8b0 con 0x7f5f340b0280 2022-07-12T08:47:08.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f5f24001660 con 0x7f5f340b0280 2022-07-12T08:47:08.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 >> v1:172.21.15.61:6789/0 conn(0x7f5f340b0280 legacy=0x7f5f3414aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f180038e0 0x7f5f18003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f18004500 0x7f5f18004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f1800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3405c610 con 0x7f5f18004e70 2022-07-12T08:47:08.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1800bcf0 con 0x7f5f180038e0 2022-07-12T08:47:08.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1800beb0 con 0x7f5f18004500 2022-07-12T08:47:08.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f337fe700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f1800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f32ffd700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f180038e0 0x7f5f18003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.351+0000 7f5f327fc700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f18004500 0x7f5f18004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 -- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f18004500 msgr2=0x7f5f18004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f18004500 0x7f5f18004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 -- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f180038e0 msgr2=0x7f5f18003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f180038e0 0x7f5f18003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 -- 172.21.15.61:0/2486097740 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f18003260 con 0x7f5f18004e70 2022-07-12T08:47:08.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 -- 172.21.15.61:0/2486097740 reap_dead start 2022-07-12T08:47:08.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f337fe700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f1800b750 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7f5f2c001490 tx=0x7f5f2c007d80).ready entity=mon.0 client_cookie=783fa57bd4160d6f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2c006890 con 0x7f5f18004e70 2022-07-12T08:47:08.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c0069f0 con 0x7f5f18004e70 2022-07-12T08:47:08.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f31ffb700 1 -- 172.21.15.61:0/2486097740 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2c006d50 con 0x7f5f18004e70 2022-07-12T08:47:08.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 -- 172.21.15.61:0/2486097740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 msgr2=0x7f5f1800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f1800b750 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7f5f2c001490 tx=0x7f5f2c007d80).stop 2022-07-12T08:47:08.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 -- 172.21.15.61:0/2486097740 shutdown_connections 2022-07-12T08:47:08.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 --2- 172.21.15.61:0/2486097740 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f1800b750 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 -- 172.21.15.61:0/2486097740 >> 172.21.15.61:0/2486097740 conn(0x7f5f34063340 msgr2=0x7f5f340b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 -- 172.21.15.61:0/2486097740 shutdown_connections 2022-07-12T08:47:08.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.355+0000 7f5f33fff700 1 -- 172.21.15.61:0/2486097740 wait complete. 2022-07-12T08:47:08.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 Processor -- start 2022-07-12T08:47:08.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 -- start start 2022-07-12T08:47:08.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f340aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f34146030 0x7f5f340acf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f34098aa0 0x7f5f340ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34063880 con 0x7f5f18004e70 2022-07-12T08:47:08.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f340ad940 con 0x7f5f34146030 2022-07-12T08:47:08.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f33fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34148670 con 0x7f5f34098aa0 2022-07-12T08:47:08.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f340aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f340aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57338/0 (socket says 172.21.15.61:57338) 2022-07-12T08:47:08.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 -- 172.21.15.61:0/1184995849 learned_addr learned my addr 172.21.15.61:0/1184995849 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:08.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f337fe700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f34098aa0 0x7f5f340ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f327fc700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f34146030 0x7f5f340acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 -- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f34098aa0 msgr2=0x7f5f340ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f34098aa0 0x7f5f340ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 -- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f34146030 msgr2=0x7f5f340acf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f34146030 0x7f5f340acf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 -- 172.21.15.61:0/1184995849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34063ab0 con 0x7f5f18004e70 2022-07-12T08:47:08.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f337fe700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f34098aa0 0x7f5f340ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:08.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.359+0000 7f5f32ffd700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f340aefe0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f5f2800b000 tx=0x7f5f28006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f28011ed0 con 0x7f5f18004e70 2022-07-12T08:47:08.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2800f940 con 0x7f5f18004e70 2022-07-12T08:47:08.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2800fab0 con 0x7f5f18004e70 2022-07-12T08:47:08.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f34063d50 con 0x7f5f18004e70 2022-07-12T08:47:08.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f34146750 con 0x7f5f18004e70 2022-07-12T08:47:08.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5f28024070 con 0x7f5f18004e70 2022-07-12T08:47:08.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f34067690 con 0x7f5f18004e70 2022-07-12T08:47:08.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f23fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f10053ad0 0x7f5f10055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f2806c7c0 con 0x7f5f18004e70 2022-07-12T08:47:08.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.363+0000 7f5f327fc700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f10053ad0 0x7f5f10055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.367+0000 7f5f327fc700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f10053ad0 0x7f5f10055f90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f5f24005000 tx=0x7f5f340aa5c0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.371+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f34067690 con 0x7f5f18004e70 2022-07-12T08:47:08.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.503+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f34067690 con 0x7f5f18004e70 2022-07-12T08:47:08.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.503+0000 7f5f23fff700 1 -- 172.21.15.61:0/1184995849 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5f34067690 con 0x7f5f18004e70 2022-07-12T08:47:08.504 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:08.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f10053ad0 msgr2=0x7f5f10055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f10053ad0 0x7f5f10055f90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f5f24005000 tx=0x7f5f340aa5c0).stop 2022-07-12T08:47:08.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 msgr2=0x7f5f340aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f340aefe0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f5f2800b000 tx=0x7f5f28006df0).stop 2022-07-12T08:47:08.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 shutdown_connections 2022-07-12T08:47:08.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f34098aa0 0x7f5f340ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f10053ad0 0x7f5f10055f90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f34146030 0x7f5f340acf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 --2- 172.21.15.61:0/1184995849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f18004e70 0x7f5f340aefe0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 >> 172.21.15.61:0/1184995849 conn(0x7f5f34063340 msgr2=0x7f5f340b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 shutdown_connections 2022-07-12T08:47:08.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.507+0000 7f5f33fff700 1 -- 172.21.15.61:0/1184995849 wait complete. 2022-07-12T08:47:08.533 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:47:08.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.643+0000 7fec62310700 1 Processor -- start 2022-07-12T08:47:08.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 -- start start 2022-07-12T08:47:08.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec5c05fe30 0x7fec5c13e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fec5c13ea20 0x7fec5c05bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c0b55e0 0x7fec5c05c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0b1790 con 0x7fec5c05fe30 2022-07-12T08:47:08.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0b14e0 con 0x7fec5c13ea20 2022-07-12T08:47:08.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec62310700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c143170 con 0x7fec5c0b55e0 2022-07-12T08:47:08.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec5c05fe30 0x7fec5c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec5c05fe30 0x7fec5c13e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57346/0 (socket says 172.21.15.61:57346) 2022-07-12T08:47:08.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 -- 172.21.15.61:0/3831115195 learned_addr learned my addr 172.21.15.61:0/3831115195 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:08.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec60b0d700 1 --2- 172.21.15.61:0/3831115195 >> v2:172.21.15.111:3300/0 conn(0x7fec5c13ea20 0x7fec5c05bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec61b0f700 1 --2- 172.21.15.61:0/3831115195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c0b55e0 0x7fec5c05c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 -- 172.21.15.61:0/3831115195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c0b55e0 msgr2=0x7fec5c05c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 --2- 172.21.15.61:0/3831115195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c0b55e0 0x7fec5c05c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 -- 172.21.15.61:0/3831115195 >> v2:172.21.15.111:3300/0 conn(0x7fec5c13ea20 msgr2=0x7fec5c05bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 --2- 172.21.15.61:0/3831115195 >> v2:172.21.15.111:3300/0 conn(0x7fec5c13ea20 0x7fec5c05bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 -- 172.21.15.61:0/3831115195 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c060210 con 0x7fec5c05fe30 2022-07-12T08:47:08.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec6130e700 1 --2- 172.21.15.61:0/3831115195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c05fe30 0x7fec5c13e530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fec58002f70 tx=0x7fec58006eb0).ready entity=mon.0 client_cookie=2a99b3337721d662 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec4bfff700 1 -- 172.21.15.61:0/3831115195 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec5800d570 con 0x7fec5c05fe30 2022-07-12T08:47:08.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec4bfff700 1 -- 172.21.15.61:0/3831115195 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec58000e90 con 0x7fec5c05fe30 2022-07-12T08:47:08.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.647+0000 7fec4bfff700 1 -- 172.21.15.61:0/3831115195 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec5800d900 con 0x7fec5c05fe30 2022-07-12T08:47:08.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/3831115195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c05fe30 msgr2=0x7fec5c13e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- 172.21.15.61:0/3831115195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c05fe30 0x7fec5c13e530 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fec58002f70 tx=0x7fec58006eb0).stop 2022-07-12T08:47:08.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/3831115195 shutdown_connections 2022-07-12T08:47:08.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- 172.21.15.61:0/3831115195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c0b55e0 0x7fec5c05c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- 172.21.15.61:0/3831115195 >> v2:172.21.15.111:3300/0 conn(0x7fec5c13ea20 0x7fec5c05bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- 172.21.15.61:0/3831115195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c05fe30 0x7fec5c13e530 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/3831115195 >> 172.21.15.61:0/3831115195 conn(0x7fec5c063340 msgr2=0x7fec5c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/3831115195 shutdown_connections 2022-07-12T08:47:08.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/3831115195 wait complete. 2022-07-12T08:47:08.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 Processor -- start 2022-07-12T08:47:08.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- start start 2022-07-12T08:47:08.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c05fe30 0x7fec5c0ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 0x7fec5c096d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c13ea20 0x7fec5c097200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0ad420 con 0x7fec5c0b55e0 2022-07-12T08:47:08.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0ad180 con 0x7fec5c13ea20 2022-07-12T08:47:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0aeff0 con 0x7fec5c05fe30 2022-07-12T08:47:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 0x7fec5c096d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 0x7fec5c096d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57354/0 (socket says 172.21.15.61:57354) 2022-07-12T08:47:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 -- 172.21.15.61:0/2040501854 learned_addr learned my addr 172.21.15.61:0/2040501854 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec61b0f700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c13ea20 0x7fec5c097200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec6130e700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c05fe30 0x7fec5c0ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 -- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c05fe30 msgr2=0x7fec5c0ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c05fe30 0x7fec5c0ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 -- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c13ea20 msgr2=0x7fec5c097200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c13ea20 0x7fec5c097200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 -- 172.21.15.61:0/2040501854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5800b1e0 con 0x7fec5c0b55e0 2022-07-12T08:47:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec60b0d700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 0x7fec5c096d10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fec4c00c000 tx=0x7fec4c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec4c010780 con 0x7fec5c0b55e0 2022-07-12T08:47:08.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c0108e0 con 0x7fec5c0b55e0 2022-07-12T08:47:08.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c0aec80 con 0x7fec5c0b55e0 2022-07-12T08:47:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec4c010c40 con 0x7fec5c0b55e0 2022-07-12T08:47:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.651+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c0ad630 con 0x7fec5c0b55e0 2022-07-12T08:47:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.655+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5c0af1b0 con 0x7fec5c0b55e0 2022-07-12T08:47:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.655+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fec4c01c460 con 0x7fec5c0b55e0 2022-07-12T08:47:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.655+0000 7fec4a7fc700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec3c053a40 0x7fec3c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.655+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fec4c06c440 con 0x7fec5c0b55e0 2022-07-12T08:47:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.655+0000 7fec6130e700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec3c053a40 0x7fec3c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.655+0000 7fec6130e700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec3c053a40 0x7fec3c055f00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fec5801a000 tx=0x7fec58015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.659+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec4c043230 con 0x7fec5c0b55e0 2022-07-12T08:47:08.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec5c0acec0 con 0x7fec5c0b55e0 2022-07-12T08:47:08.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec4a7fc700 1 -- 172.21.15.61:0/2040501854 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+10 (secure 0 0 0) 0x7fec4c046850 con 0x7fec5c0b55e0 2022-07-12T08:47:08.789 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:08.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec3c053a40 msgr2=0x7fec3c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec3c053a40 0x7fec3c055f00 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fec5801a000 tx=0x7fec58015000).stop 2022-07-12T08:47:08.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 msgr2=0x7fec5c096d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 0x7fec5c096d10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fec4c00c000 tx=0x7fec4c006df0).stop 2022-07-12T08:47:08.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 shutdown_connections 2022-07-12T08:47:08.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fec3c053a40 0x7fec3c055f00 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fec5c13ea20 0x7fec5c097200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec5c0b55e0 0x7fec5c096d10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 --2- 172.21.15.61:0/2040501854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fec5c05fe30 0x7fec5c0ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.787+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 >> 172.21.15.61:0/2040501854 conn(0x7fec5c063340 msgr2=0x7fec5c0b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.791+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 shutdown_connections 2022-07-12T08:47:08.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.791+0000 7fec62310700 1 -- 172.21.15.61:0/2040501854 wait complete. 2022-07-12T08:47:08.802 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:47:08.803 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:08.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.907+0000 7f899ffff700 1 Processor -- start 2022-07-12T08:47:08.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.907+0000 7f899ffff700 1 -- start start 2022-07-12T08:47:08.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f89a009a930 0x7f89a00aba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0060bd0 0x7f89a009ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a0142f40 con 0x7f89a0142090 2022-07-12T08:47:08.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a00ab350 con 0x7f89a009a930 2022-07-12T08:47:08.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a00ab5f0 con 0x7f89a0060bd0 2022-07-12T08:47:08.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f89a0142090 0x7f89a0142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45580/0 (socket says 172.21.15.61:45580) 2022-07-12T08:47:08.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899f7fe700 1 -- 172.21.15.61:0/3785518968 learned_addr learned my addr 172.21.15.61:0/3785518968 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:08.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899effd700 1 --2- 172.21.15.61:0/3785518968 >> v2:172.21.15.61:3300/0 conn(0x7f89a009a930 0x7f89a00aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f8997fff700 1 --2- 172.21.15.61:0/3785518968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0060bd0 0x7f89a009ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899e7fc700 1 -- 172.21.15.61:0/3785518968 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873600421 0 0) 0x7f89a0142f40 con 0x7f89a0142090 2022-07-12T08:47:08.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899e7fc700 1 -- 172.21.15.61:0/3785518968 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8980003260 con 0x7f89a0142090 2022-07-12T08:47:08.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899effd700 1 -- 172.21.15.61:0/3785518968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0060bd0 msgr2=0x7f89a009ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:08.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899effd700 1 --2- 172.21.15.61:0/3785518968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0060bd0 0x7f89a009ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899effd700 1 -- 172.21.15.61:0/3785518968 >> v1:172.21.15.61:6789/0 conn(0x7f89a0142090 legacy=0x7f89a0142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899e7fc700 1 -- 172.21.15.61:0/3785518968 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433563445 0 0) 0x7f8980003260 con 0x7f89a0142090 2022-07-12T08:47:08.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899effd700 1 -- 172.21.15.61:0/3785518968 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a0142c50 con 0x7f89a009a930 2022-07-12T08:47:08.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899effd700 1 --2- 172.21.15.61:0/3785518968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a009a930 0x7f89a00aba90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f8998001b80 tx=0x7f8990006df0).ready entity=mon.0 client_cookie=2f953df89fdac770 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899e7fc700 1 -- 172.21.15.61:0/3785518968 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89980019e0 con 0x7f89a009a930 2022-07-12T08:47:08.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899e7fc700 1 -- 172.21.15.61:0/3785518968 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8990000e90 con 0x7f89a009a930 2022-07-12T08:47:08.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899e7fc700 1 -- 172.21.15.61:0/3785518968 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8990012460 con 0x7f89a009a930 2022-07-12T08:47:08.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- 172.21.15.61:0/3785518968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a009a930 msgr2=0x7f89a00aba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 --2- 172.21.15.61:0/3785518968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a009a930 0x7f89a00aba90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f8998001b80 tx=0x7f8990006df0).stop 2022-07-12T08:47:08.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- 172.21.15.61:0/3785518968 shutdown_connections 2022-07-12T08:47:08.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 --2- 172.21.15.61:0/3785518968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0060bd0 0x7f89a009ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 --2- 172.21.15.61:0/3785518968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a009a930 0x7f89a00aba90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- 172.21.15.61:0/3785518968 >> 172.21.15.61:0/3785518968 conn(0x7f89a00b1950 msgr2=0x7f89a00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:08.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- 172.21.15.61:0/3785518968 shutdown_connections 2022-07-12T08:47:08.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.911+0000 7f899ffff700 1 -- 172.21.15.61:0/3785518968 wait complete. 2022-07-12T08:47:08.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 Processor -- start 2022-07-12T08:47:08.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 -- start start 2022-07-12T08:47:08.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 0x7f89a01524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a009a930 0x7f89a0147990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0142090 0x7f89a0155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a01464d0 con 0x7f89a0060bd0 2022-07-12T08:47:08.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a014abc0 con 0x7f89a009a930 2022-07-12T08:47:08.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a014ae60 con 0x7f89a0142090 2022-07-12T08:47:08.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 0x7f89a01524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 0x7f89a01524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57366/0 (socket says 172.21.15.61:57366) 2022-07-12T08:47:08.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 -- 172.21.15.61:0/1959740255 learned_addr learned my addr 172.21.15.61:0/1959740255 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:08.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f8997fff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a009a930 0x7f89a0147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899f7fe700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0142090 0x7f89a0155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 -- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0142090 msgr2=0x7f89a0155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0142090 0x7f89a0155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 -- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a009a930 msgr2=0x7f89a0147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:08.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a009a930 0x7f89a0147990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:08.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 -- 172.21.15.61:0/1959740255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899000b180 con 0x7f89a0060bd0 2022-07-12T08:47:08.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899effd700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 0x7f89a01524e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f8998001b80 tx=0x7f8990014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8990016070 con 0x7f89a0060bd0 2022-07-12T08:47:08.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899000d6e0 con 0x7f89a0060bd0 2022-07-12T08:47:08.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a0147e80 con 0x7f89a0060bd0 2022-07-12T08:47:08.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f899000da40 con 0x7f89a0060bd0 2022-07-12T08:47:08.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.915+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a0144100 con 0x7f89a0060bd0 2022-07-12T08:47:08.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.919+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a0148180 con 0x7f89a0060bd0 2022-07-12T08:47:08.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.919+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f899000f030 con 0x7f89a0060bd0 2022-07-12T08:47:08.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.919+0000 7f899cff9700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8984053920 0x7f8984055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:08.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.919+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f899006cba0 con 0x7f89a0060bd0 2022-07-12T08:47:08.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.919+0000 7f8997fff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8984053920 0x7f8984055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:08.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.919+0000 7f8997fff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8984053920 0x7f8984055de0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f898800b000 tx=0x7f8988006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:08.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:08.927+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8990043970 con 0x7f89a0060bd0 2022-07-12T08:47:09.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.091+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89a014b020 con 0x7f89a0060bd0 2022-07-12T08:47:09.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.095+0000 7f899cff9700 1 -- 172.21.15.61:0/1959740255 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f8990023070 con 0x7f89a0060bd0 2022-07-12T08:47:09.095 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:09.095 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:09.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.095+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8984053920 msgr2=0x7f8984055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:09.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.095+0000 7f899ffff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8984053920 0x7f8984055de0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f898800b000 tx=0x7f8988006df0).stop 2022-07-12T08:47:09.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.095+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 msgr2=0x7f89a01524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:09.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.095+0000 7f899ffff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 0x7f89a01524e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f8998001b80 tx=0x7f8990014000).stop 2022-07-12T08:47:09.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 shutdown_connections 2022-07-12T08:47:09.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89a0142090 0x7f89a0155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:09.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8984053920 0x7f8984055de0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:09.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89a009a930 0x7f89a0147990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:09.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 --2- 172.21.15.61:0/1959740255 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89a0060bd0 0x7f89a01524e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:09.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 >> 172.21.15.61:0/1959740255 conn(0x7f89a00b1950 msgr2=0x7f89a00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:09.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 shutdown_connections 2022-07-12T08:47:09.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:09.099+0000 7f899ffff700 1 -- 172.21.15.61:0/1959740255 wait complete. 2022-07-12T08:47:09.104 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:09.327 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:47:03.397499+0000', 'modified': '2022-07-12T08:47:05.156395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6196}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6196': {'gid': 6196, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3265139715', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3265139715}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3265139715}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:47:09.328 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:47:09.329 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:47:09.448 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6196,"want_state":"up:active","state":"up:active","rank_uptime":6.0377671250000002,"mdsmap_epoch":38,"osdmap_epoch":71,"osdmap_epoch_barrier":71,"uptime":10.385821887000001} 2022-07-12T08:47:09.449 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6196, "mdsmap_epoch": 38, "osdmap_epoch": 71, "osdmap_epoch_barrier": 71, "rank_uptime": 6.037767125, "state": "up:active", "uptime": 10.385821887, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:47:09.450 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:47:09.451 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:47:09.451 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:47:09.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.453 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:09.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:47:09.483 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:47:09.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:47:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:47:09.553 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:47:09.554 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.555 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:47:09.629 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:47:09.630 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:09.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:47:09.661 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:47:09.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:47:09.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:47:09.712+0000 7f28b4286080 -1 init, newargv = 0x5635492c2780 newargc=9 2022-07-12T08:47:09.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21606]: starting ceph client 2022-07-12T08:47:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:47:09.723 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:47:09.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.724 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:47:09.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21606]: starting fuse 2022-07-12T08:47:09.736 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:47:09.736 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:47:09.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:47:09.740 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:47:09.741 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:47:09.742 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:47:09.743 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:47:09.744 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:09.744 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:47:09.745 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:47:09.746 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:47:09.750 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:09.751 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:47:09.752 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:47:09.752 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:47:09.753 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:47:09.753 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:09.753 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:47:09.754 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:47:09.755 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:47:09.755 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:47:09.756 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:47:09.756 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:09.757 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:47:09.757 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:47:09.830 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21606.asok 2022-07-12T08:47:09.835 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21606.asok 2022-07-12T08:47:09.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21606.asok status 2022-07-12T08:47:09.941 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:47:09.942 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:47:09.942 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:47:09.943 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:47:09.943 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:47:09.943 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:47:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:47:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21606", 2022-07-12T08:47:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:47:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:47:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:47:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6462, 2022-07-12T08:47:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:47:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:47:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:47:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6462 2022-07-12T08:47:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:47:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:47:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:47:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3889265026 2022-07-12T08:47:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:47:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:47:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:47:09.956 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:47:09.957 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3889265026 2022-07-12T08:47:09.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:09.957 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6462 172.21.15.121:0/3889265026", 2022-07-12T08:47:09.958 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/3889265026", 2022-07-12T08:47:09.959 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:47:09.959 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 38, 2022-07-12T08:47:09.959 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 71, 2022-07-12T08:47:09.960 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:47:09.960 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:47:09.961 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:47:09.961 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:47:09.962 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.963 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:09.964 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:09.974 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:47:09.975 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:47:09.975 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:09.976 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:09.995 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:10.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 Processor -- start 2022-07-12T08:47:10.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 -- start start 2022-07-12T08:47:10.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc3880b0150 0x7fc38809a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc38809a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc388067eb0 con 0x7fc38809ad40 2022-07-12T08:47:10.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38805f650 con 0x7fc3880b0150 2022-07-12T08:47:10.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc387fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3880b0530 con 0x7fc38805f140 2022-07-12T08:47:10.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc3877fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc38809ad40 0x7fc388144060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45596/0 (socket says 172.21.15.61:45596) 2022-07-12T08:47:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.107+0000 7fc3877fe700 1 -- 172.21.15.61:0/671021390 learned_addr learned my addr 172.21.15.61:0/671021390 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc37e7fc700 1 --2- 172.21.15.61:0/671021390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc38809a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc386ffd700 1 --2- 172.21.15.61:0/671021390 >> v2:172.21.15.111:3300/0 conn(0x7fc3880b0150 0x7fc38809a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc3867fc700 1 -- 172.21.15.61:0/671021390 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288519230 0 0) 0x7fc388067eb0 con 0x7fc38809ad40 2022-07-12T08:47:10.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc3867fc700 1 -- 172.21.15.61:0/671021390 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc368003260 con 0x7fc38809ad40 2022-07-12T08:47:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc37e7fc700 1 -- 172.21.15.61:0/671021390 >> v2:172.21.15.111:3300/0 conn(0x7fc3880b0150 msgr2=0x7fc38809a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc37e7fc700 1 --2- 172.21.15.61:0/671021390 >> v2:172.21.15.111:3300/0 conn(0x7fc3880b0150 0x7fc38809a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc37e7fc700 1 -- 172.21.15.61:0/671021390 >> v1:172.21.15.61:6789/0 conn(0x7fc38809ad40 legacy=0x7fc388144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc37e7fc700 1 -- 172.21.15.61:0/671021390 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388067c10 con 0x7fc38805f140 2022-07-12T08:47:10.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc37e7fc700 1 --2- 172.21.15.61:0/671021390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc38809a850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc370002f70 tx=0x7fc370006df0).ready entity=mon.2 client_cookie=2346f4fd01e0ef9e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc3867fc700 1 -- 172.21.15.61:0/671021390 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37000d4a0 con 0x7fc38805f140 2022-07-12T08:47:10.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc3867fc700 1 -- 172.21.15.61:0/671021390 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc370000e90 con 0x7fc38805f140 2022-07-12T08:47:10.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc3867fc700 1 -- 172.21.15.61:0/671021390 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37000d7f0 con 0x7fc38805f140 2022-07-12T08:47:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 -- 172.21.15.61:0/671021390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 msgr2=0x7fc38809a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 --2- 172.21.15.61:0/671021390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc38809a850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc370002f70 tx=0x7fc370006df0).stop 2022-07-12T08:47:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 -- 172.21.15.61:0/671021390 shutdown_connections 2022-07-12T08:47:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 --2- 172.21.15.61:0/671021390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc38809a850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 --2- 172.21.15.61:0/671021390 >> v2:172.21.15.111:3300/0 conn(0x7fc3880b0150 0x7fc38809a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 -- 172.21.15.61:0/671021390 >> 172.21.15.61:0/671021390 conn(0x7fc3880b1950 msgr2=0x7fc3880b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 -- 172.21.15.61:0/671021390 shutdown_connections 2022-07-12T08:47:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.111+0000 7fc387fff700 1 -- 172.21.15.61:0/671021390 wait complete. 2022-07-12T08:47:10.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 Processor -- start 2022-07-12T08:47:10.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 -- start start 2022-07-12T08:47:10.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc388098070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38809ad40 0x7fc388155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 0x7fc388095f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3880947b0 con 0x7fc3880b0150 2022-07-12T08:47:10.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc388155910 con 0x7fc38809ad40 2022-07-12T08:47:10.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc388155620 con 0x7fc38805f140 2022-07-12T08:47:10.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 0x7fc388095f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 0x7fc388095f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57384/0 (socket says 172.21.15.61:57384) 2022-07-12T08:47:10.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 -- 172.21.15.61:0/2221906480 learned_addr learned my addr 172.21.15.61:0/2221906480 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:10.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc37e7fc700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38809ad40 0x7fc388155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc386ffd700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc388098070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 -- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 msgr2=0x7fc388098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc388098070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 -- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38809ad40 msgr2=0x7fc388155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38809ad40 0x7fc388155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 -- 172.21.15.61:0/2221906480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37000b180 con 0x7fc3880b0150 2022-07-12T08:47:10.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc3877fe700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 0x7fc388095f60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fc380001490 tx=0x7fc380007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc380005ba0 con 0x7fc3880b0150 2022-07-12T08:47:10.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc380005d00 con 0x7fc3880b0150 2022-07-12T08:47:10.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc388099e60 con 0x7fc3880b0150 2022-07-12T08:47:10.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.115+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3800072c0 con 0x7fc3880b0150 2022-07-12T08:47:10.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.119+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3880943d0 con 0x7fc3880b0150 2022-07-12T08:47:10.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.119+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc388155ad0 con 0x7fc3880b0150 2022-07-12T08:47:10.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.119+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc380010070 con 0x7fc3880b0150 2022-07-12T08:47:10.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.119+0000 7fc384ff9700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc36c053a60 0x7fc36c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.119+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc38006b810 con 0x7fc3880b0150 2022-07-12T08:47:10.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.119+0000 7fc386ffd700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc36c053a60 0x7fc36c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.123+0000 7fc386ffd700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc36c053a60 0x7fc36c055f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fc37800b000 tx=0x7fc378006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.127+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc380042770 con 0x7fc3880b0150 2022-07-12T08:47:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.259+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc388057460 con 0x7fc3880b0150 2022-07-12T08:47:10.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.259+0000 7fc384ff9700 1 -- 172.21.15.61:0/2221906480 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16089 (secure 0 0 0) 0x7fc380045d90 con 0x7fc3880b0150 2022-07-12T08:47:10.261 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:10.261 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":71,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:04.401539+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:47:01.995190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-12T08:47:02.575389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc36c053a60 msgr2=0x7fc36c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc36c053a60 0x7fc36c055f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fc37800b000 tx=0x7fc378006df0).stop 2022-07-12T08:47:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 msgr2=0x7fc388095f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 0x7fc388095f60 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fc380001490 tx=0x7fc380007e70).stop 2022-07-12T08:47:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 shutdown_connections 2022-07-12T08:47:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3880b0150 0x7fc388095f60 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc36c053a60 0x7fc36c055f20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38809ad40 0x7fc388155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 --2- 172.21.15.61:0/2221906480 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc38805f140 0x7fc388098070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 >> 172.21.15.61:0/2221906480 conn(0x7fc3880b1950 msgr2=0x7fc3880b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 shutdown_connections 2022-07-12T08:47:10.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.263+0000 7fc387fff700 1 -- 172.21.15.61:0/2221906480 wait complete. 2022-07-12T08:47:10.275 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:47:01.995190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:47:02.575389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:47:10.276 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:47:10.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 Processor -- start 2022-07-12T08:47:10.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 -- start start 2022-07-12T08:47:10.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f386005d060 0x7f38600b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600afb50 0x7f3860143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f386005d440 con 0x7f3860142ab0 2022-07-12T08:47:10.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3860143d80 con 0x7f386005d060 2022-07-12T08:47:10.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.403+0000 7f386483f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38600674b0 con 0x7f38600afb50 2022-07-12T08:47:10.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f386005d060 0x7f38600b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385effd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3860142ab0 0x7f3860142f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59210/0 (socket says 172.21.15.61:59210) 2022-07-12T08:47:10.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385effd700 1 -- 172.21.15.61:0/142211626 learned_addr learned my addr 172.21.15.61:0/142211626 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385ffff700 1 --2- 172.21.15.61:0/142211626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600afb50 0x7f3860143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385e7fc700 1 -- 172.21.15.61:0/142211626 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848923813 0 0) 0x7f386005d440 con 0x7f3860142ab0 2022-07-12T08:47:10.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385e7fc700 1 -- 172.21.15.61:0/142211626 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384c003260 con 0x7f3860142ab0 2022-07-12T08:47:10.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385ffff700 1 -- 172.21.15.61:0/142211626 >> v2:172.21.15.111:3300/0 conn(0x7f386005d060 msgr2=0x7f38600b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:10.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385ffff700 1 --2- 172.21.15.61:0/142211626 >> v2:172.21.15.111:3300/0 conn(0x7f386005d060 0x7f38600b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385ffff700 1 -- 172.21.15.61:0/142211626 >> v1:172.21.15.111:6789/0 conn(0x7f3860142ab0 legacy=0x7f3860142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385ffff700 1 -- 172.21.15.61:0/142211626 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3860067750 con 0x7f38600afb50 2022-07-12T08:47:10.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385f7fe700 1 --2- 172.21.15.61:0/142211626 >> v2:172.21.15.111:3300/0 conn(0x7f386005d060 0x7f38600b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:10.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385ffff700 1 --2- 172.21.15.61:0/142211626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600afb50 0x7f3860143620 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3858002f70 tx=0x7f3858006df0).ready entity=mon.2 client_cookie=cbd919f4050cbe1e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385e7fc700 1 -- 172.21.15.61:0/142211626 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f385800d4a0 con 0x7f38600afb50 2022-07-12T08:47:10.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385e7fc700 1 -- 172.21.15.61:0/142211626 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3858000e90 con 0x7f38600afb50 2022-07-12T08:47:10.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f385e7fc700 1 -- 172.21.15.61:0/142211626 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f385800d7f0 con 0x7f38600afb50 2022-07-12T08:47:10.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 -- 172.21.15.61:0/142211626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600afb50 msgr2=0x7f3860143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 --2- 172.21.15.61:0/142211626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600afb50 0x7f3860143620 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f3858002f70 tx=0x7f3858006df0).stop 2022-07-12T08:47:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 -- 172.21.15.61:0/142211626 shutdown_connections 2022-07-12T08:47:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 --2- 172.21.15.61:0/142211626 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600afb50 0x7f3860143620 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 --2- 172.21.15.61:0/142211626 >> v2:172.21.15.111:3300/0 conn(0x7f386005d060 0x7f38600b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 -- 172.21.15.61:0/142211626 >> 172.21.15.61:0/142211626 conn(0x7f38600b6900 msgr2=0x7f3860066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 -- 172.21.15.61:0/142211626 shutdown_connections 2022-07-12T08:47:10.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.407+0000 7f386483f700 1 -- 172.21.15.61:0/142211626 wait complete. 2022-07-12T08:47:10.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 Processor -- start 2022-07-12T08:47:10.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 -- start start 2022-07-12T08:47:10.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f386005d060 0x7f3860148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 0x7f386009c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600b1410 0x7f38600b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600b07d0 con 0x7f38600afb50 2022-07-12T08:47:10.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f386009f020 con 0x7f386005d060 2022-07-12T08:47:10.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f386009f2a0 con 0x7f38600b1410 2022-07-12T08:47:10.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 0x7f386009c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 0x7f386009c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57394/0 (socket says 172.21.15.61:57394) 2022-07-12T08:47:10.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 -- 172.21.15.61:0/4091246380 learned_addr learned my addr 172.21.15.61:0/4091246380 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:10.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385ffff700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600b1410 0x7f38600b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385f7fe700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f386005d060 0x7f3860148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 -- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600b1410 msgr2=0x7f38600b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600b1410 0x7f38600b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 -- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f386005d060 msgr2=0x7f3860148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f386005d060 0x7f3860148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 -- 172.21.15.61:0/4091246380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385400b180 con 0x7f38600afb50 2022-07-12T08:47:10.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385effd700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 0x7f386009c910 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f3850001490 tx=0x7f3850007ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3850005ba0 con 0x7f38600afb50 2022-07-12T08:47:10.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3850005d00 con 0x7f38600afb50 2022-07-12T08:47:10.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385800b180 con 0x7f38600afb50 2022-07-12T08:47:10.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38500072c0 con 0x7f38600afb50 2022-07-12T08:47:10.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.411+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38600b0ed0 con 0x7f38600afb50 2022-07-12T08:47:10.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.415+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f385001a060 con 0x7f38600afb50 2022-07-12T08:47:10.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.415+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860145f50 con 0x7f38600afb50 2022-07-12T08:47:10.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.415+0000 7f385cff9700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3840053920 0x7f3840055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.415+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f385006ba70 con 0x7f38600afb50 2022-07-12T08:47:10.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.415+0000 7f385f7fe700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3840053920 0x7f3840055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.415+0000 7f385f7fe700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3840053920 0x7f3840055de0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f385400b000 tx=0x7f3854006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.423+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38500428c0 con 0x7f38600afb50 2022-07-12T08:47:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.547+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3860057460 con 0x7f38600afb50 2022-07-12T08:47:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.547+0000 7f385cff9700 1 -- 172.21.15.61:0/4091246380 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v71) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3850018030 con 0x7f38600afb50 2022-07-12T08:47:10.551 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:47:10.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3840053920 msgr2=0x7f3840055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3840053920 0x7f3840055de0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f385400b000 tx=0x7f3854006df0).stop 2022-07-12T08:47:10.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 msgr2=0x7f386009c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 0x7f386009c910 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f3850001490 tx=0x7f3850007ea0).stop 2022-07-12T08:47:10.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 shutdown_connections 2022-07-12T08:47:10.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f38600b1410 0x7f38600b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38600afb50 0x7f386009c910 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3840053920 0x7f3840055de0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 --2- 172.21.15.61:0/4091246380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f386005d060 0x7f3860148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 >> 172.21.15.61:0/4091246380 conn(0x7f38600b6900 msgr2=0x7f386005fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 shutdown_connections 2022-07-12T08:47:10.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.551+0000 7f386483f700 1 -- 172.21.15.61:0/4091246380 wait complete. 2022-07-12T08:47:10.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:47:10.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 Processor -- start 2022-07-12T08:47:10.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 -- start start 2022-07-12T08:47:10.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 0x7fd590140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd590099e60 0x7fd5901418a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5900b1ea0 con 0x7fd5900a9cd0 2022-07-12T08:47:10.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59005d2a0 con 0x7fd590099e60 2022-07-12T08:47:10.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd597fc0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59005d000 con 0x7fd5900b36c0 2022-07-12T08:47:10.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd59655d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd5900a9cd0 0x7fd5900b2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45622/0 (socket says 172.21.15.61:45622) 2022-07-12T08:47:10.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd59655d700 1 -- 172.21.15.61:0/2660606186 learned_addr learned my addr 172.21.15.61:0/2660606186 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:10.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd59555b700 1 --2- 172.21.15.61:0/2660606186 >> v2:172.21.15.61:3300/0 conn(0x7fd590099e60 0x7fd5901418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd595d5c700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 0x7fd590140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.671+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555138706 0 0) 0x7fd5900b1ea0 con 0x7fd5900a9cd0 2022-07-12T08:47:10.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd578003260 con 0x7fd5900a9cd0 2022-07-12T08:47:10.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279204042 0 0) 0x7fd578003260 con 0x7fd5900a9cd0 2022-07-12T08:47:10.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 msgr2=0x7fd590140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:10.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 0x7fd590140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 >> v2:172.21.15.61:3300/0 conn(0x7fd590099e60 msgr2=0x7fd5901418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:10.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 --2- 172.21.15.61:0/2660606186 >> v2:172.21.15.61:3300/0 conn(0x7fd590099e60 0x7fd5901418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5901413b0 con 0x7fd5900a9cd0 2022-07-12T08:47:10.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fd588001a00 con 0x7fd5900a9cd0 2022-07-12T08:47:10.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 >> v1:172.21.15.61:6789/0 conn(0x7fd5900a9cd0 legacy=0x7fd5900b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd578003750 0x7fd578003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 0x7fd578004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd578004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59005d2a0 con 0x7fd5780049f0 2022-07-12T08:47:10.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59005d000 con 0x7fd578003750 2022-07-12T08:47:10.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd57800b750 con 0x7fd5900b36c0 2022-07-12T08:47:10.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd578004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59555b700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 0x7fd578004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd595d5c700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd578003750 0x7fd578003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 -- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 msgr2=0x7fd578004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:10.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900b36c0 0x7fd578004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 -- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd578003750 msgr2=0x7fd578003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:10.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd578003750 0x7fd578003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 -- 172.21.15.61:0/2660606186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd578003260 con 0x7fd5780049f0 2022-07-12T08:47:10.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd578004e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fd588001710 tx=0x7fd5880021f0).ready entity=mon.0 client_cookie=2e9fdf394974787 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58800cc40 con 0x7fd5780049f0 2022-07-12T08:47:10.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58800cda0 con 0x7fd5780049f0 2022-07-12T08:47:10.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd594d5a700 1 -- 172.21.15.61:0/2660606186 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd588004b50 con 0x7fd5780049f0 2022-07-12T08:47:10.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd597fc0700 1 -- 172.21.15.61:0/2660606186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 msgr2=0x7fd578004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd597fc0700 1 --2- 172.21.15.61:0/2660606186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd578004e70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7fd588001710 tx=0x7fd5880021f0).stop 2022-07-12T08:47:10.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd59655d700 1 -- 172.21.15.61:0/2660606186 reap_dead start 2022-07-12T08:47:10.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd597fc0700 1 -- 172.21.15.61:0/2660606186 shutdown_connections 2022-07-12T08:47:10.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd597fc0700 1 -- 172.21.15.61:0/2660606186 >> 172.21.15.61:0/2660606186 conn(0x7fd5900b6e40 msgr2=0x7fd5900b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd597fc0700 1 -- 172.21.15.61:0/2660606186 shutdown_connections 2022-07-12T08:47:10.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.675+0000 7fd597fc0700 1 -- 172.21.15.61:0/2660606186 wait complete. 2022-07-12T08:47:10.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 Processor -- start 2022-07-12T08:47:10.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 -- start start 2022-07-12T08:47:10.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd5900afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5900acde0 0x7fd5900669e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900abf20 0x7fd590063d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900ac3d0 con 0x7fd5780049f0 2022-07-12T08:47:10.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590066020 con 0x7fd5900acde0 2022-07-12T08:47:10.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5900662a0 con 0x7fd5900abf20 2022-07-12T08:47:10.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd5900afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd59555b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5900acde0 0x7fd5900669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd5900afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57414/0 (socket says 172.21.15.61:57414) 2022-07-12T08:47:10.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 -- 172.21.15.61:0/4070378862 learned_addr learned my addr 172.21.15.61:0/4070378862 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:10.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd59655d700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900abf20 0x7fd590063d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 -- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900abf20 msgr2=0x7fd590063d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900abf20 0x7fd590063d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 -- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5900acde0 msgr2=0x7fd5900669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5900acde0 0x7fd5900669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 -- 172.21.15.61:0/4070378862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590066540 con 0x7fd5780049f0 2022-07-12T08:47:10.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd595d5c700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd5900afe00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fd58c00b000 tx=0x7fd58c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c012610 con 0x7fd5780049f0 2022-07-12T08:47:10.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c012770 con 0x7fd5780049f0 2022-07-12T08:47:10.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd590064270 con 0x7fd5780049f0 2022-07-12T08:47:10.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c012ad0 con 0x7fd5780049f0 2022-07-12T08:47:10.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.679+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5900aa250 con 0x7fd5780049f0 2022-07-12T08:47:10.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.683+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5900666d0 con 0x7fd5780049f0 2022-07-12T08:47:10.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.683+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd58c012c30 con 0x7fd5780049f0 2022-07-12T08:47:10.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.683+0000 7fd57effd700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd5700539b0 0x7fd570055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.683+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd58c06c820 con 0x7fd5780049f0 2022-07-12T08:47:10.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.683+0000 7fd59555b700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd5700539b0 0x7fd570055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.683+0000 7fd59555b700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd5700539b0 0x7fd570055e70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fd58000b000 tx=0x7fd580006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.687+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd58c043670 con 0x7fd5780049f0 2022-07-12T08:47:10.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.831+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd590057460 con 0x7fd5780049f0 2022-07-12T08:47:10.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.831+0000 7fd57effd700 1 -- 172.21.15.61:0/4070378862 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd58c046c90 con 0x7fd5780049f0 2022-07-12T08:47:10.835 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:10.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd5700539b0 msgr2=0x7fd570055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd5700539b0 0x7fd570055e70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fd58000b000 tx=0x7fd580006df0).stop 2022-07-12T08:47:10.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 msgr2=0x7fd5900afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd5900afe00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7fd58c00b000 tx=0x7fd58c008040).stop 2022-07-12T08:47:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 shutdown_connections 2022-07-12T08:47:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd5900abf20 0x7fd590063d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd5700539b0 0x7fd570055e70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd5900acde0 0x7fd5900669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 --2- 172.21.15.61:0/4070378862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5780049f0 0x7fd5900afe00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 >> 172.21.15.61:0/4070378862 conn(0x7fd5900b6e40 msgr2=0x7fd5900b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 shutdown_connections 2022-07-12T08:47:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.835+0000 7fd597fc0700 1 -- 172.21.15.61:0/4070378862 wait complete. 2022-07-12T08:47:10.845 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:47:10.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.963+0000 7f4dc1bd2700 1 Processor -- start 2022-07-12T08:47:10.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.963+0000 7f4dc1bd2700 1 -- start start 2022-07-12T08:47:10.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.963+0000 7f4dc1bd2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4dbc142eb0 0x7f4dbc05c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.963+0000 7f4dc1bd2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc143500 con 0x7f4dbc0b6130 2022-07-12T08:47:10.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dbc143290 con 0x7f4dbc05d130 2022-07-12T08:47:10.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dbc0b65c0 con 0x7f4dbc142eb0 2022-07-12T08:47:10.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc0bd0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4dbc142eb0 0x7f4dbc05c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc13d1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4dbc0b6130 0x7f4dbc0b1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45644/0 (socket says 172.21.15.61:45644) 2022-07-12T08:47:10.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc13d1700 1 -- 172.21.15.61:0/2901770111 learned_addr learned my addr 172.21.15.61:0/2901770111 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:10.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965964324 0 0) 0x7f4dbc143500 con 0x7f4dbc0b6130 2022-07-12T08:47:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d9c003260 con 0x7f4dbc0b6130 2022-07-12T08:47:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225596942 0 0) 0x7f4dbc143290 con 0x7f4dbc05d130 2022-07-12T08:47:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dbc143500 con 0x7f4dbc05d130 2022-07-12T08:47:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4238763736 0 0) 0x7f4d9c003260 con 0x7f4dbc0b6130 2022-07-12T08:47:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc0bd0700 1 -- 172.21.15.61:0/2901770111 >> v1:172.21.15.111:6789/0 conn(0x7f4dbc05d130 legacy=0x7f4dbc0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc0bd0700 1 -- 172.21.15.61:0/2901770111 >> v1:172.21.15.61:6789/0 conn(0x7f4dbc0b6130 legacy=0x7f4dbc0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc0bd0700 1 -- 172.21.15.61:0/2901770111 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc143760 con 0x7f4dbc142eb0 2022-07-12T08:47:10.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc0bd0700 1 --2- 172.21.15.61:0/2901770111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc05c6d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f4dbc143c30 tx=0x7f4dac006df0).ready entity=mon.0 client_cookie=7cf4c3cd5ef48a52 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4db40019e0 con 0x7f4dbc142eb0 2022-07-12T08:47:10.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dac000e90 con 0x7f4dbc142eb0 2022-07-12T08:47:10.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dbb7fe700 1 -- 172.21.15.61:0/2901770111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4dac012460 con 0x7f4dbc142eb0 2022-07-12T08:47:10.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/2901770111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 msgr2=0x7f4dbc05c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/2901770111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc05c6d0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 rx=0x7f4dbc143c30 tx=0x7f4dac006df0).stop 2022-07-12T08:47:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/2901770111 shutdown_connections 2022-07-12T08:47:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/2901770111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc05c6d0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/2901770111 >> 172.21.15.61:0/2901770111 conn(0x7f4dbc066d70 msgr2=0x7f4dbc067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/2901770111 shutdown_connections 2022-07-12T08:47:10.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.967+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/2901770111 wait complete. 2022-07-12T08:47:10.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 Processor -- start 2022-07-12T08:47:10.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 -- start start 2022-07-12T08:47:10.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4dbc05d130 0x7f4dbc0af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dbc0b6130 0x7f4dbc154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc0b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc1454a0 con 0x7f4dbc142eb0 2022-07-12T08:47:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0aed90 con 0x7f4dbc0b6130 2022-07-12T08:47:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc1bd2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dbc0ae850 con 0x7f4dbc05d130 2022-07-12T08:47:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc0b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc0bd0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4dbc05d130 0x7f4dbc0af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dbbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dbc0b6130 0x7f4dbc154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc0b2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57432/0 (socket says 172.21.15.61:57432) 2022-07-12T08:47:10.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc0bd0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4dbc05d130 0x7f4dbc0af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39624/0 (socket says 172.21.15.61:39624) 2022-07-12T08:47:10.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 -- 172.21.15.61:0/840216658 learned_addr learned my addr 172.21.15.61:0/840216658 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:10.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 -- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4dbc05d130 msgr2=0x7f4dbc0af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4dbc05d130 0x7f4dbc0af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 -- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dbc0b6130 msgr2=0x7f4dbc154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:10.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dbc0b6130 0x7f4dbc154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:10.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 -- 172.21.15.61:0/840216658 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac00b180 con 0x7f4dbc142eb0 2022-07-12T08:47:10.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dbbfff700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dbc0b6130 0x7f4dbc154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:10.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4dc13d1700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc0b2c80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f4db4001bc0 tx=0x7f4db40019e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:10.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4db4004ab0 con 0x7f4dbc142eb0 2022-07-12T08:47:10.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.971+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db4004c10 con 0x7f4dbc142eb0 2022-07-12T08:47:10.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dbc0aeaf0 con 0x7f4dbc142eb0 2022-07-12T08:47:10.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4db4007170 con 0x7f4dbc142eb0 2022-07-12T08:47:10.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dbc1449a0 con 0x7f4dbc142eb0 2022-07-12T08:47:10.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dbc146270 con 0x7f4dbc142eb0 2022-07-12T08:47:10.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4db401b070 con 0x7f4dbc142eb0 2022-07-12T08:47:10.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4db9ffb700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4da00539b0 0x7f4da0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4db406b770 con 0x7f4dbc142eb0 2022-07-12T08:47:11.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.975+0000 7f4dc0bd0700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4da00539b0 0x7f4da0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.979+0000 7f4dc0bd0700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4da00539b0 0x7f4da0055e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f4dac01a000 tx=0x7f4dac014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:10.983+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4db4042630 con 0x7f4dbc142eb0 2022-07-12T08:47:11.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.111+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4dbc06b2b0 con 0x7f4dbc142eb0 2022-07-12T08:47:11.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.111+0000 7f4db9ffb700 1 -- 172.21.15.61:0/840216658 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4db4045c50 con 0x7f4dbc142eb0 2022-07-12T08:47:11.115 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:11.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4da00539b0 msgr2=0x7f4da0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4da00539b0 0x7f4da0055e70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f4dac01a000 tx=0x7f4dac014000).stop 2022-07-12T08:47:11.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 msgr2=0x7f4dbc0b2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc0b2c80 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f4db4001bc0 tx=0x7f4db40019e0).stop 2022-07-12T08:47:11.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 shutdown_connections 2022-07-12T08:47:11.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4dbc142eb0 0x7f4dbc0b2c80 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4da00539b0 0x7f4da0055e70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4dbc0b6130 0x7f4dbc154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 --2- 172.21.15.61:0/840216658 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4dbc05d130 0x7f4dbc0af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 >> 172.21.15.61:0/840216658 conn(0x7f4dbc066d70 msgr2=0x7f4dbc0b8140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:11.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 shutdown_connections 2022-07-12T08:47:11.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.115+0000 7f4dc1bd2700 1 -- 172.21.15.61:0/840216658 wait complete. 2022-07-12T08:47:11.129 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:47:11.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5ffff700 1 Processor -- start 2022-07-12T08:47:11.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5ffff700 1 -- start start 2022-07-12T08:47:11.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5ffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcb60061840 0x7fcb60142f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb600ac340 con 0x7fcb600af7e0 2022-07-12T08:47:11.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5ffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb60061c20 con 0x7fcb600b1650 2022-07-12T08:47:11.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5ffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6013f900 con 0x7fcb60061840 2022-07-12T08:47:11.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcb600af7e0 0x7fcb6014a290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45658/0 (socket says 172.21.15.61:45658) 2022-07-12T08:47:11.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5f7fe700 1 -- 172.21.15.61:0/3488269203 learned_addr learned my addr 172.21.15.61:0/3488269203 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:11.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5effd700 1 --2- 172.21.15.61:0/3488269203 >> v2:172.21.15.111:3300/0 conn(0x7fcb60061840 0x7fcb60142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644606222 0 0) 0x7fcb600ac340 con 0x7fcb600af7e0 2022-07-12T08:47:11.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb40003260 con 0x7fcb600af7e0 2022-07-12T08:47:11.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152868046 0 0) 0x7fcb60061c20 con 0x7fcb600b1650 2022-07-12T08:47:11.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb600ac340 con 0x7fcb600b1650 2022-07-12T08:47:11.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5effd700 1 -- 172.21.15.61:0/3488269203 >> v1:172.21.15.111:6789/0 conn(0x7fcb600b1650 legacy=0x7fcb600af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5effd700 1 -- 172.21.15.61:0/3488269203 >> v1:172.21.15.61:6789/0 conn(0x7fcb600af7e0 legacy=0x7fcb6014a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5effd700 1 -- 172.21.15.61:0/3488269203 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb600aabd0 con 0x7fcb60061840 2022-07-12T08:47:11.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5effd700 1 --2- 172.21.15.61:0/3488269203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb60061840 0x7fcb60142f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fcb500032f0 tx=0x7fcb500091d0).ready entity=mon.1 client_cookie=a630ad6672d04aca server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.239+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb5000b600 con 0x7fcb60061840 2022-07-12T08:47:11.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5000b760 con 0x7fcb60061840 2022-07-12T08:47:11.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5dffb700 1 -- 172.21.15.61:0/3488269203 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb5000b8d0 con 0x7fcb60061840 2022-07-12T08:47:11.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 -- 172.21.15.61:0/3488269203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb60061840 msgr2=0x7fcb60142f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/3488269203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb60061840 0x7fcb60142f50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fcb500032f0 tx=0x7fcb500091d0).stop 2022-07-12T08:47:11.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 -- 172.21.15.61:0/3488269203 shutdown_connections 2022-07-12T08:47:11.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/3488269203 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb60061840 0x7fcb60142f50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 -- 172.21.15.61:0/3488269203 >> 172.21.15.61:0/3488269203 conn(0x7fcb600b57c0 msgr2=0x7fcb600b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:11.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 -- 172.21.15.61:0/3488269203 shutdown_connections 2022-07-12T08:47:11.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 -- 172.21.15.61:0/3488269203 wait complete. 2022-07-12T08:47:11.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 Processor -- start 2022-07-12T08:47:11.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.243+0000 7fcb5ffff700 1 -- start start 2022-07-12T08:47:11.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 0x7fcb60151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb600af7e0 0x7fcb600b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcb600b1650 0x7fcb60092ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb600928f0 con 0x7fcb60061840 2022-07-12T08:47:11.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb600906f0 con 0x7fcb600af7e0 2022-07-12T08:47:11.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb600901f0 con 0x7fcb600b1650 2022-07-12T08:47:11.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 0x7fcb60151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 0x7fcb60151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57442/0 (socket says 172.21.15.61:57442) 2022-07-12T08:47:11.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 -- 172.21.15.61:0/1076298111 learned_addr learned my addr 172.21.15.61:0/1076298111 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:11.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5e7fc700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb600af7e0 0x7fcb600b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5f7fe700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcb600b1650 0x7fcb60092ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 -- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcb600b1650 msgr2=0x7fcb60092ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcb600b1650 0x7fcb60092ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 -- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb600af7e0 msgr2=0x7fcb600b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb600af7e0 0x7fcb600b2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 -- 172.21.15.61:0/1076298111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb50009470 con 0x7fcb60061840 2022-07-12T08:47:11.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5effd700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 0x7fcb60151fd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fcb500032f0 tx=0x7fcb50016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb5000b600 con 0x7fcb60061840 2022-07-12T08:47:11.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5000beb0 con 0x7fcb60061840 2022-07-12T08:47:11.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb5001f5e0 con 0x7fcb60061840 2022-07-12T08:47:11.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6008f690 con 0x7fcb60061840 2022-07-12T08:47:11.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.247+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb600912b0 con 0x7fcb60061840 2022-07-12T08:47:11.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.251+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcb5000f070 con 0x7fcb60061840 2022-07-12T08:47:11.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.251+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb60090030 con 0x7fcb60061840 2022-07-12T08:47:11.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.251+0000 7fcb57fff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcb44053a60 0x7fcb44055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.251+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcb5006c8e0 con 0x7fcb60061840 2022-07-12T08:47:11.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.251+0000 7fcb5e7fc700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcb44053a60 0x7fcb44055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.251+0000 7fcb5e7fc700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcb44053a60 0x7fcb44055f20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fcb4800b000 tx=0x7fcb48009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.259+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb500436b0 con 0x7fcb60061840 2022-07-12T08:47:11.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.383+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb60057460 con 0x7fcb60061840 2022-07-12T08:47:11.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.383+0000 7fcb57fff700 1 -- 172.21.15.61:0/1076298111 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+10 (secure 0 0 0) 0x7fcb500240a0 con 0x7fcb60061840 2022-07-12T08:47:11.387 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:11.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcb44053a60 msgr2=0x7fcb44055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcb44053a60 0x7fcb44055f20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fcb4800b000 tx=0x7fcb48009040).stop 2022-07-12T08:47:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 msgr2=0x7fcb60151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 0x7fcb60151fd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7fcb500032f0 tx=0x7fcb50016000).stop 2022-07-12T08:47:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 shutdown_connections 2022-07-12T08:47:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcb600b1650 0x7fcb60092ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcb44053a60 0x7fcb44055f20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb600af7e0 0x7fcb600b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 --2- 172.21.15.61:0/1076298111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb60061840 0x7fcb60151fd0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 >> 172.21.15.61:0/1076298111 conn(0x7fcb600b57c0 msgr2=0x7fcb6013ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 shutdown_connections 2022-07-12T08:47:11.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.387+0000 7fcb5ffff700 1 -- 172.21.15.61:0/1076298111 wait complete. 2022-07-12T08:47:11.413 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:47:11.413 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:11.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 Processor -- start 2022-07-12T08:47:11.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 -- start start 2022-07-12T08:47:11.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f640805c050 0x7f64080ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64080b01f0 0x7f640814a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64080ad2f0 con 0x7f640805cb00 2022-07-12T08:47:11.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64080b0a40 con 0x7f640805c050 2022-07-12T08:47:11.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640e943700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640805c430 con 0x7f64080b01f0 2022-07-12T08:47:11.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6407fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f640805c050 0x7f64080ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640cee0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f640805cb00 0x7f64081443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45672/0 (socket says 172.21.15.61:45672) 2022-07-12T08:47:11.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f640cee0700 1 -- 172.21.15.61:0/2103936952 learned_addr learned my addr 172.21.15.61:0/2103936952 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:11.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f64077fe700 1 --2- 172.21.15.61:0/2103936952 >> v2:172.21.15.111:3300/0 conn(0x7f64080b01f0 0x7f640814a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6407fff700 1 -- 172.21.15.61:0/2103936952 >> v2:172.21.15.111:3300/0 conn(0x7f64080b01f0 msgr2=0x7f640814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:11.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6407fff700 1 --2- 172.21.15.61:0/2103936952 >> v2:172.21.15.111:3300/0 conn(0x7f64080b01f0 0x7f640814a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6407fff700 1 -- 172.21.15.61:0/2103936952 >> v1:172.21.15.61:6789/0 conn(0x7f640805cb00 legacy=0x7f64081443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6406ffd700 1 -- 172.21.15.61:0/2103936952 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224239830 0 0) 0x7f64080ad2f0 con 0x7f640805cb00 2022-07-12T08:47:11.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6407fff700 1 -- 172.21.15.61:0/2103936952 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64080b2420 con 0x7f640805c050 2022-07-12T08:47:11.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6407fff700 1 --2- 172.21.15.61:0/2103936952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640805c050 0x7f64080ac710 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f63f0002f70 tx=0x7f63f0006df0).ready entity=mon.0 client_cookie=5f27943e424e3dc5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6406ffd700 1 -- 172.21.15.61:0/2103936952 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63f000d570 con 0x7f640805c050 2022-07-12T08:47:11.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6406ffd700 1 -- 172.21.15.61:0/2103936952 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0000e90 con 0x7f640805c050 2022-07-12T08:47:11.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.519+0000 7f6406ffd700 1 -- 172.21.15.61:0/2103936952 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63f000d8c0 con 0x7f640805c050 2022-07-12T08:47:11.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 -- 172.21.15.61:0/2103936952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640805c050 msgr2=0x7f64080ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 --2- 172.21.15.61:0/2103936952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640805c050 0x7f64080ac710 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f63f0002f70 tx=0x7f63f0006df0).stop 2022-07-12T08:47:11.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 -- 172.21.15.61:0/2103936952 shutdown_connections 2022-07-12T08:47:11.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 --2- 172.21.15.61:0/2103936952 >> v2:172.21.15.111:3300/0 conn(0x7f64080b01f0 0x7f640814a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 --2- 172.21.15.61:0/2103936952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f640805c050 0x7f64080ac710 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 -- 172.21.15.61:0/2103936952 >> 172.21.15.61:0/2103936952 conn(0x7f6408062c00 msgr2=0x7f6408062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 -- 172.21.15.61:0/2103936952 shutdown_connections 2022-07-12T08:47:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 -- 172.21.15.61:0/2103936952 wait complete. 2022-07-12T08:47:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 Processor -- start 2022-07-12T08:47:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.523+0000 7f640e943700 1 -- start start 2022-07-12T08:47:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f640805c050 0x7f6408156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640805cb00 0x7f64080ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 0x7f6408143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408159a60 con 0x7f64080b01f0 2022-07-12T08:47:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640814a4e0 con 0x7f640805cb00 2022-07-12T08:47:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6408159c90 con 0x7f640805c050 2022-07-12T08:47:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 0x7f6408143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 0x7f6408143180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57460/0 (socket says 172.21.15.61:57460) 2022-07-12T08:47:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 -- 172.21.15.61:0/2014217119 learned_addr learned my addr 172.21.15.61:0/2014217119 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f6407fff700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f640805c050 0x7f6408156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f64077fe700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640805cb00 0x7f64080ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 -- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f640805c050 msgr2=0x7f6408156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f640805c050 0x7f6408156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 -- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640805cb00 msgr2=0x7f64080ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640805cb00 0x7f64080ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 -- 172.21.15.61:0/2014217119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f000b180 con 0x7f64080b01f0 2022-07-12T08:47:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f64077fe700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640805cb00 0x7f64080ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640cee0700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 0x7f6408143180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7f6400000bc0 tx=0x7f6400007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64000070d0 con 0x7f64080b01f0 2022-07-12T08:47:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400007230 con 0x7f64080b01f0 2022-07-12T08:47:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640814a130 con 0x7f64080b01f0 2022-07-12T08:47:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6400005610 con 0x7f64080b01f0 2022-07-12T08:47:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64080adef0 con 0x7f64080b01f0 2022-07-12T08:47:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.527+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6408159e50 con 0x7f64080b01f0 2022-07-12T08:47:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.531+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6400010070 con 0x7f64080b01f0 2022-07-12T08:47:11.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.531+0000 7f64057fa700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63ec053af0 0x7f63ec055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:11.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.531+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f640006b520 con 0x7f64080b01f0 2022-07-12T08:47:11.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.531+0000 7f6407fff700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63ec053af0 0x7f63ec055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:11.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.531+0000 7f6407fff700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63ec053af0 0x7f63ec055fb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f63f001a000 tx=0x7f63f0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:11.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.535+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6400042480 con 0x7f64080b01f0 2022-07-12T08:47:11.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.699+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6408057460 con 0x7f64080b01f0 2022-07-12T08:47:11.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f64057fa700 1 -- 172.21.15.61:0/2014217119 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f6400007ba0 con 0x7f64080b01f0 2022-07-12T08:47:11.702 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:11.703 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:11.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63ec053af0 msgr2=0x7f63ec055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63ec053af0 0x7f63ec055fb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f63f001a000 tx=0x7f63f0014000).stop 2022-07-12T08:47:11.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 msgr2=0x7f6408143180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:11.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 0x7f6408143180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 rx=0x7f6400000bc0 tx=0x7f6400007a80).stop 2022-07-12T08:47:11.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 shutdown_connections 2022-07-12T08:47:11.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64080b01f0 0x7f6408143180 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63ec053af0 0x7f63ec055fb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f640805cb00 0x7f64080ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 --2- 172.21.15.61:0/2014217119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f640805c050 0x7f6408156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:11.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 >> 172.21.15.61:0/2014217119 conn(0x7f6408062c00 msgr2=0x7f6408062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:11.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 shutdown_connections 2022-07-12T08:47:11.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:11.703+0000 7f640e943700 1 -- 172.21.15.61:0/2014217119 wait complete. 2022-07-12T08:47:11.708 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:11.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:47:03.397499+0000', 'modified': '2022-07-12T08:47:05.156395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6196}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6196': {'gid': 6196, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3265139715', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3265139715}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3265139715}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:47:11.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:47:11.934 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:47:12.054 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6196,"want_state":"up:active","state":"up:active","rank_uptime":8.6426539200000008,"mdsmap_epoch":38,"osdmap_epoch":71,"osdmap_epoch_barrier":71,"uptime":12.990719627000001} 2022-07-12T08:47:12.055 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6196, "mdsmap_epoch": 38, "osdmap_epoch": 71, "osdmap_epoch_barrier": 71, "rank_uptime": 8.64265392, "state": "up:active", "uptime": 12.990719627, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:47:12.056 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:47:12.056 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:47:12.057 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:47:12.060 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.061 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:12.077 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:47:12.145 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:47:12.147 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:47:12.160 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:47:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:47:12.163 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.164 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:47:12.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:47:12.215 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:12.227 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:47:12.247 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:47:12.248 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:47:12.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:47:12.299+0000 7fac6258a080 -1 init, newargv = 0x55f6ec0c65f0 newargc=9 2022-07-12T08:47:12.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21144]: starting ceph client 2022-07-12T08:47:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:47:12.314 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:47:12.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.316 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:47:12.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21144]: starting fuse 2022-07-12T08:47:12.326 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:47:12.327 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:47:12.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.328 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:47:12.328 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:47:12.329 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:47:12.329 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:47:12.330 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:47:12.330 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:47:12.331 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:47:12.337 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:47:12.338 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:47:12.339 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:47:12.340 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:47:12.341 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:47:12.341 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:47:12.342 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:47:12.343 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:47:12.344 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:47:12.345 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:47:12.345 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:47:12.347 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:47:12.348 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:47:12.348 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:47:12.349 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:47:12.350 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:47:12.412 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21144.asok 2022-07-12T08:47:12.419 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21144.asok 2022-07-12T08:47:12.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.421 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21144.asok status 2022-07-12T08:47:12.517 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:47:12.517 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:47:12.518 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:47:12.518 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:47:12.518 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:47:12.518 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:47:12.519 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:47:12.519 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21144", 2022-07-12T08:47:12.519 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:47:12.519 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:47:12.520 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:47:12.520 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:47:12.520 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 6510, 2022-07-12T08:47:12.521 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:47:12.521 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:47:12.521 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:47:12.521 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 6510 2022-07-12T08:47:12.521 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:47:12.522 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:47:12.522 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:47:12.522 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:47:12.522 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 14387408 2022-07-12T08:47:12.523 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:47:12.523 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:47:12.523 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:47:12.523 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:47:12.524 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:47:12.524 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 14387408 2022-07-12T08:47:12.526 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:47:12.526 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.6510 172.21.15.130:0/14387408", 2022-07-12T08:47:12.526 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/14387408", 2022-07-12T08:47:12.526 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:47:12.527 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 38, 2022-07-12T08:47:12.527 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 71, 2022-07-12T08:47:12.527 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:47:12.527 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:47:12.527 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:47:12.528 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:47:12.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.530 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:12.530 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:12.561 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:47:12.562 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:47:12.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:12.563 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:12.629 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 Processor -- start 2022-07-12T08:47:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 -- start start 2022-07-12T08:47:12.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fabf405c8e0 0x7fabf40ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:12.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40aed50 0x7fabf40af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:12.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf413f350 con 0x7fabf40ad8b0 2022-07-12T08:47:12.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf40b1c60 con 0x7fabf405c8e0 2022-07-12T08:47:12.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf405c420 con 0x7fabf40aed50 2022-07-12T08:47:12.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf1d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fabf40ad8b0 0x7fabf405bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59284/0 (socket says 172.21.15.61:59284) 2022-07-12T08:47:12.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf1d9b700 1 -- 172.21.15.61:0/3625769978 learned_addr learned my addr 172.21.15.61:0/3625769978 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:12.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf259c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fabf405c8e0 0x7fabf40ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:12.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.743+0000 7fabf2d9d700 1 --2- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40aed50 0x7fabf40af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:12.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf259c700 1 -- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40aed50 msgr2=0x7fabf40af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:12.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf259c700 1 --2- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40aed50 0x7fabf40af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf259c700 1 -- 172.21.15.61:0/3625769978 >> v1:172.21.15.111:6789/0 conn(0x7fabf40ad8b0 legacy=0x7fabf405bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf259c700 1 -- 172.21.15.61:0/3625769978 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf413f5f0 con 0x7fabf405c8e0 2022-07-12T08:47:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf259c700 1 --2- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf405c8e0 0x7fabf40ae860 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fabf405c560 tx=0x7fabe4006df0).ready entity=mon.1 client_cookie=28cf012bb970463b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf159a700 1 -- 172.21.15.61:0/3625769978 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe400d570 con 0x7fabf405c8e0 2022-07-12T08:47:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf159a700 1 -- 172.21.15.61:0/3625769978 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe4000e90 con 0x7fabf405c8e0 2022-07-12T08:47:12.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf159a700 1 -- 172.21.15.61:0/3625769978 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe400d8c0 con 0x7fabf405c8e0 2022-07-12T08:47:12.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 -- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf405c8e0 msgr2=0x7fabf40ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:12.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 --2- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf405c8e0 0x7fabf40ae860 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fabf405c560 tx=0x7fabe4006df0).stop 2022-07-12T08:47:12.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 -- 172.21.15.61:0/3625769978 shutdown_connections 2022-07-12T08:47:12.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 --2- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40aed50 0x7fabf40af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 --2- 172.21.15.61:0/3625769978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf405c8e0 0x7fabf40ae860 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 -- 172.21.15.61:0/3625769978 >> 172.21.15.61:0/3625769978 conn(0x7fabf40b5780 msgr2=0x7fabf40b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:12.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 -- 172.21.15.61:0/3625769978 shutdown_connections 2022-07-12T08:47:12.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.747+0000 7fabf359e700 1 -- 172.21.15.61:0/3625769978 wait complete. 2022-07-12T08:47:12.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 Processor -- start 2022-07-12T08:47:12.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 -- start start 2022-07-12T08:47:12.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 0x7fabf413e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:12.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40ad8b0 0x7fabf41533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:12.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf40aed50 0x7fabf41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:12.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf4143130 con 0x7fabf405c8e0 2022-07-12T08:47:12.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf4152fc0 con 0x7fabf40aed50 2022-07-12T08:47:12.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf413b9c0 con 0x7fabf40ad8b0 2022-07-12T08:47:12.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 0x7fabf413e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:12.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 0x7fabf413e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57470/0 (socket says 172.21.15.61:57470) 2022-07-12T08:47:12.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 -- 172.21.15.61:0/2987893991 learned_addr learned my addr 172.21.15.61:0/2987893991 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:12.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf1d9b700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40ad8b0 0x7fabf41533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf2d9d700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf40aed50 0x7fabf41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 -- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40ad8b0 msgr2=0x7fabf41533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40ad8b0 0x7fabf41533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 -- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf40aed50 msgr2=0x7fabf41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf40aed50 0x7fabf41538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.751+0000 7fabf259c700 1 -- 172.21.15.61:0/2987893991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe400b180 con 0x7fabf405c8e0 2022-07-12T08:47:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabf259c700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 0x7fabf413e880 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7fabf405c560 tx=0x7fabe4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe4016070 con 0x7fabf405c8e0 2022-07-12T08:47:12.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe4002f70 con 0x7fabf405c8e0 2022-07-12T08:47:12.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf40652b0 con 0x7fabf405c8e0 2022-07-12T08:47:12.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabe4013de0 con 0x7fabf405c8e0 2022-07-12T08:47:12.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf4153dd0 con 0x7fabf405c8e0 2022-07-12T08:47:12.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf4153130 con 0x7fabf405c8e0 2022-07-12T08:47:12.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fabe4025030 con 0x7fabf405c8e0 2022-07-12T08:47:12.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.755+0000 7fabeb7fe700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabd80539b0 0x7fabd8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:12.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.759+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fabe406ce40 con 0x7fabf405c8e0 2022-07-12T08:47:12.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.759+0000 7fabf1d9b700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabd80539b0 0x7fabd8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:12.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.759+0000 7fabf1d9b700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabd80539b0 0x7fabd8055e70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fabdc00d000 tx=0x7fabdc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:12.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.763+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabe4043c90 con 0x7fabf405c8e0 2022-07-12T08:47:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabf4140650 con 0x7fabf405c8e0 2022-07-12T08:47:12.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabeb7fe700 1 -- 172.21.15.61:0/2987893991 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fabe4023070 con 0x7fabf405c8e0 2022-07-12T08:47:12.935 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:12.936 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:12.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabd80539b0 msgr2=0x7fabd8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabd80539b0 0x7fabd8055e70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fabdc00d000 tx=0x7fabdc009040).stop 2022-07-12T08:47:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 msgr2=0x7fabf413e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 0x7fabf413e880 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 rx=0x7fabf405c560 tx=0x7fabe4014000).stop 2022-07-12T08:47:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 shutdown_connections 2022-07-12T08:47:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabd80539b0 0x7fabd8055e70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabf40aed50 0x7fabf41538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabf40ad8b0 0x7fabf41533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 --2- 172.21.15.61:0/2987893991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabf405c8e0 0x7fabf413e880 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 >> 172.21.15.61:0/2987893991 conn(0x7fabf40b5780 msgr2=0x7fabf40b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 shutdown_connections 2022-07-12T08:47:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:12.935+0000 7fabf359e700 1 -- 172.21.15.61:0/2987893991 wait complete. 2022-07-12T08:47:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:12.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:13.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 Processor -- start 2022-07-12T08:47:13.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 -- start start 2022-07-12T08:47:13.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f6f4c060070 0x7f6f4c0606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:13.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f4c144a20 0x7f6f4c0a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:13.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0b29b0 con 0x7f6f4c0afb00 2022-07-12T08:47:13.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c0afee0 con 0x7f6f4c144a20 2022-07-12T08:47:13.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f53243700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c05cc50 con 0x7f6f4c060070 2022-07-12T08:47:13.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f517e0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f4c144a20 0x7f6f4c0a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:13.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f517e0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f4c144a20 0x7f6f4c0a91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57480/0 (socket says 172.21.15.61:57480) 2022-07-12T08:47:13.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.055+0000 7f6f517e0700 1 -- 172.21.15.61:0/3794449893 learned_addr learned my addr 172.21.15.61:0/3794449893 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:13.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f4bfff700 1 --2- 172.21.15.61:0/3794449893 >> v2:172.21.15.111:3300/0 conn(0x7f6f4c060070 0x7f6f4c0606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:13.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f517e0700 1 -- 172.21.15.61:0/3794449893 >> v2:172.21.15.111:3300/0 conn(0x7f6f4c060070 msgr2=0x7f6f4c0606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:13.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f517e0700 1 --2- 172.21.15.61:0/3794449893 >> v2:172.21.15.111:3300/0 conn(0x7f6f4c060070 0x7f6f4c0606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f517e0700 1 -- 172.21.15.61:0/3794449893 >> v1:172.21.15.111:6789/0 conn(0x7f6f4c0afb00 legacy=0x7f6f4c144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:13.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f517e0700 1 -- 172.21.15.61:0/3794449893 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c0b26c0 con 0x7f6f4c144a20 2022-07-12T08:47:13.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f517e0700 1 --2- 172.21.15.61:0/3794449893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c144a20 0x7f6f4c0a91b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f6f440032f0 tx=0x7f6f44009290).ready entity=mon.0 client_cookie=fc8789955a991ae2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:13.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f4b7fe700 1 -- 172.21.15.61:0/3794449893 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f4400b870 con 0x7f6f4c144a20 2022-07-12T08:47:13.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f4b7fe700 1 -- 172.21.15.61:0/3794449893 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4400b9d0 con 0x7f6f4c144a20 2022-07-12T08:47:13.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f4b7fe700 1 -- 172.21.15.61:0/3794449893 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f4400bb40 con 0x7f6f4c144a20 2022-07-12T08:47:13.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 -- 172.21.15.61:0/3794449893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c144a20 msgr2=0x7f6f4c0a91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:13.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 --2- 172.21.15.61:0/3794449893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c144a20 0x7f6f4c0a91b0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f6f440032f0 tx=0x7f6f44009290).stop 2022-07-12T08:47:13.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 -- 172.21.15.61:0/3794449893 shutdown_connections 2022-07-12T08:47:13.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 --2- 172.21.15.61:0/3794449893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c144a20 0x7f6f4c0a91b0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 --2- 172.21.15.61:0/3794449893 >> v2:172.21.15.111:3300/0 conn(0x7f6f4c060070 0x7f6f4c0606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 -- 172.21.15.61:0/3794449893 >> 172.21.15.61:0/3794449893 conn(0x7f6f4c063ab0 msgr2=0x7f6f4c063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:13.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 -- 172.21.15.61:0/3794449893 shutdown_connections 2022-07-12T08:47:13.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 -- 172.21.15.61:0/3794449893 wait complete. 2022-07-12T08:47:13.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 Processor -- start 2022-07-12T08:47:13.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.059+0000 7f6f53243700 1 -- start start 2022-07-12T08:47:13.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f4c060070 0x7f6f4c09ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:13.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 0x7f6f4c0ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:13.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f4c144a20 0x7f6f4c143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:13.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c09e1a0 con 0x7f6f4c0afb00 2022-07-12T08:47:13.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c0669b0 con 0x7f6f4c144a20 2022-07-12T08:47:13.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c152f60 con 0x7f6f4c060070 2022-07-12T08:47:13.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 0x7f6f4c0ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:13.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 0x7f6f4c0ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57486/0 (socket says 172.21.15.61:57486) 2022-07-12T08:47:13.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 -- 172.21.15.61:0/3621427076 learned_addr learned my addr 172.21.15.61:0/3621427076 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:13.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f50fdf700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f4c060070 0x7f6f4c09ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:13.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f517e0700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f4c144a20 0x7f6f4c143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:13.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 -- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f4c060070 msgr2=0x7f6f4c09ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:13.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f4c060070 0x7f6f4c09ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 -- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f4c144a20 msgr2=0x7f6f4c143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:13.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f4c144a20 0x7f6f4c143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 -- 172.21.15.61:0/3621427076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44009530 con 0x7f6f4c0afb00 2022-07-12T08:47:13.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f4bfff700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 0x7f6f4c0ab7e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f6f3800c000 tx=0x7f6f38006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:13.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f38010780 con 0x7f6f4c0afb00 2022-07-12T08:47:13.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f380108e0 con 0x7f6f4c0afb00 2022-07-12T08:47:13.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c067490 con 0x7f6f4c0afb00 2022-07-12T08:47:13.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f38010c40 con 0x7f6f4c0afb00 2022-07-12T08:47:13.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.063+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c143a90 con 0x7f6f4c0afb00 2022-07-12T08:47:13.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.067+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c09ef80 con 0x7f6f4c0afb00 2022-07-12T08:47:13.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.067+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6f3801c460 con 0x7f6f4c0afb00 2022-07-12T08:47:13.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.067+0000 7f6f49ffb700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f2c053920 0x7f6f2c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:13.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.067+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6f3806c3b0 con 0x7f6f4c0afb00 2022-07-12T08:47:13.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.067+0000 7f6f50fdf700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f2c053920 0x7f6f2c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:13.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.067+0000 7f6f50fdf700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f2c053920 0x7f6f2c055de0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6f4000d000 tx=0x7f6f40007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:13.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.071+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f38043120 con 0x7f6f4c0afb00 2022-07-12T08:47:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.231+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f4c057460 con 0x7f6f4c0afb00 2022-07-12T08:47:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.231+0000 7f6f49ffb700 1 -- 172.21.15.61:0/3621427076 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f6f38046740 con 0x7f6f4c0afb00 2022-07-12T08:47:13.235 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:13.236 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f2c053920 msgr2=0x7f6f2c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f2c053920 0x7f6f2c055de0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f6f4000d000 tx=0x7f6f40007b90).stop 2022-07-12T08:47:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 msgr2=0x7f6f4c0ab7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:13.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 0x7f6f4c0ab7e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 rx=0x7f6f3800c000 tx=0x7f6f38006df0).stop 2022-07-12T08:47:13.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 shutdown_connections 2022-07-12T08:47:13.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6f2c053920 0x7f6f2c055de0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f4c144a20 0x7f6f4c143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f4c0afb00 0x7f6f4c0ab7e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 --2- 172.21.15.61:0/3621427076 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f4c060070 0x7f6f4c09ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:13.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 >> 172.21.15.61:0/3621427076 conn(0x7f6f4c063ab0 msgr2=0x7f6f4c0aa490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:13.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 shutdown_connections 2022-07-12T08:47:13.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:13.235+0000 7f6f53243700 1 -- 172.21.15.61:0/3621427076 wait complete. 2022-07-12T08:47:13.244 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:13.253 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_cap_revoke_eviction_timeout 30.0 2022-07-12T08:47:13.360 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_cap_revoke_eviction_timeout = '30.000000' "} 2022-07-12T08:47:13.361 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_cap_revoke_eviction_timeout = '30.000000' " } 2022-07-12T08:47:13.362 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:13.362 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:13.362 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:13.371 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:47:13.375 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:47:13.377 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:47:13.378 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-07-12T08:47:13.378 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:13.379 DEBUG:teuthology.orchestra.run.smithi121:> with open("/home/ubuntu/cephtest/mnt.0/background_file", '"'"'w'"'"') as f: 2022-07-12T08:47:13.379 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-07-12T08:47:13.380 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-07-12T08:47:13.381 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content2'"'"') 2022-07-12T08:47:13.382 DEBUG:teuthology.orchestra.run.smithi121:> while True: 2022-07-12T08:47:13.384 DEBUG:teuthology.orchestra.run.smithi121:> time.sleep(1) 2022-07-12T08:47:13.384 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:47:13.391 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-12T08:47:13.423 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-12T08:47:13.425 INFO:teuthology.orchestra.run.smithi121.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/background_file': No such file or directory 2022-07-12T08:47:14.428 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-12T08:47:14.451 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.0/background_file 2022-07-12T08:47:14.453 DEBUG:tasks.cephfs.mount:File background_file became visible from 0 after 1s 2022-07-12T08:47:14.454 DEBUG:teuthology.orchestra.run.smithi130:> sudo ls /home/ubuntu/cephtest/mnt.1/background_file 2022-07-12T08:47:14.495 INFO:teuthology.orchestra.run.smithi130.stdout:/home/ubuntu/cephtest/mnt.1/background_file 2022-07-12T08:47:14.497 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s 2022-07-12T08:47:14.498 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:47:14.519 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:14.519 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:14.520 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:14.553 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:47:14.554 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:47:14.554 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:47:14.555 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:47:14.555 DEBUG:teuthology.orchestra.run.smithi130:> import time 2022-07-12T08:47:14.556 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:47:14.556 DEBUG:teuthology.orchestra.run.smithi130:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2022-07-12T08:47:14.556 DEBUG:teuthology.orchestra.run.smithi130:> try: 2022-07-12T08:47:14.557 DEBUG:teuthology.orchestra.run.smithi130:> while True: 2022-07-12T08:47:14.557 DEBUG:teuthology.orchestra.run.smithi130:> os.write(fd, b'"'"'content'"'"') 2022-07-12T08:47:14.558 DEBUG:teuthology.orchestra.run.smithi130:> time.sleep(1) 2022-07-12T08:47:14.558 DEBUG:teuthology.orchestra.run.smithi130:> if not False: 2022-07-12T08:47:14.559 DEBUG:teuthology.orchestra.run.smithi130:> break 2022-07-12T08:47:14.559 DEBUG:teuthology.orchestra.run.smithi130:> except IOError as e: 2022-07-12T08:47:14.560 DEBUG:teuthology.orchestra.run.smithi130:> pass 2022-07-12T08:47:14.560 DEBUG:teuthology.orchestra.run.smithi130:> os.close(fd) 2022-07-12T08:47:14.560 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:47:33.049 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:33.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:33.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:33.062 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:33.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:47:55.292 INFO:tasks.cephfs.test_misc:cap_waiter waited 40.72899651527405s 2022-07-12T08:47:55.294 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist ls --format=json 2022-07-12T08:47:55.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 Processor -- start 2022-07-12T08:47:55.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 -- start start 2022-07-12T08:47:55.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe910065490 0x7fe9100b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe9100b6550 0x7fe91013fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91005f1e0 con 0x7fe9100b0df0 2022-07-12T08:47:55.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe910146850 con 0x7fe9100b6550 2022-07-12T08:47:55.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91729f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100b11d0 con 0x7fe910065490 2022-07-12T08:47:55.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe916a9e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe9100b6550 0x7fe91013fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe915a9c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe910065490 0x7fe9100b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91629d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fe9100b0df0 0x7fe910146140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59314/0 (socket says 172.21.15.61:59314) 2022-07-12T08:47:55.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe91629d700 1 -- 172.21.15.61:0/2935768596 learned_addr learned my addr 172.21.15.61:0/2935768596 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:55.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe916a9e700 1 -- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe910065490 msgr2=0x7fe9100b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe916a9e700 1 --2- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe910065490 0x7fe9100b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe916a9e700 1 -- 172.21.15.61:0/2935768596 >> v1:172.21.15.111:6789/0 conn(0x7fe9100b0df0 legacy=0x7fe910146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe916a9e700 1 -- 172.21.15.61:0/2935768596 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe910060170 con 0x7fe9100b6550 2022-07-12T08:47:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.408+0000 7fe915a9c700 1 --2- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe910065490 0x7fe9100b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe916a9e700 1 --2- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b6550 0x7fe91013fcb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fe9080032f0 tx=0x7fe9080091d0).ready entity=mon.1 client_cookie=7c93c92fa2ad7fe8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:55.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91529b700 1 -- 172.21.15.61:0/2935768596 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe90800b6c0 con 0x7fe9100b6550 2022-07-12T08:47:55.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91529b700 1 -- 172.21.15.61:0/2935768596 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90800b820 con 0x7fe9100b6550 2022-07-12T08:47:55.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91529b700 1 -- 172.21.15.61:0/2935768596 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe90800b990 con 0x7fe9100b6550 2022-07-12T08:47:55.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 -- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b6550 msgr2=0x7fe91013fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 --2- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b6550 0x7fe91013fcb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fe9080032f0 tx=0x7fe9080091d0).stop 2022-07-12T08:47:55.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 -- 172.21.15.61:0/2935768596 shutdown_connections 2022-07-12T08:47:55.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 --2- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b6550 0x7fe91013fcb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 --2- 172.21.15.61:0/2935768596 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe910065490 0x7fe9100b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 -- 172.21.15.61:0/2935768596 >> 172.21.15.61:0/2935768596 conn(0x7fe910063ad0 msgr2=0x7fe910063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:55.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 -- 172.21.15.61:0/2935768596 shutdown_connections 2022-07-12T08:47:55.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 -- 172.21.15.61:0/2935768596 wait complete. 2022-07-12T08:47:55.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 Processor -- start 2022-07-12T08:47:55.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.412+0000 7fe91729f700 1 -- start start 2022-07-12T08:47:55.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b0df0 0x7fe9100b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 0x7fe9100add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe9100b0200 0x7fe910151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100b58d0 con 0x7fe9100b6550 2022-07-12T08:47:55.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9100ad5d0 con 0x7fe9100b0df0 2022-07-12T08:47:55.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe910146080 con 0x7fe9100b0200 2022-07-12T08:47:55.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 0x7fe9100add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 0x7fe9100add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57500/0 (socket says 172.21.15.61:57500) 2022-07-12T08:47:55.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 -- 172.21.15.61:0/1296702960 learned_addr learned my addr 172.21.15.61:0/1296702960 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:55.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91629d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b0df0 0x7fe9100b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe916a9e700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe9100b0200 0x7fe910151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 -- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe9100b0200 msgr2=0x7fe910151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe9100b0200 0x7fe910151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 -- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b0df0 msgr2=0x7fe9100b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b0df0 0x7fe9100b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 -- 172.21.15.61:0/1296702960 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90000b180 con 0x7fe9100b6550 2022-07-12T08:47:55.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe915a9c700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 0x7fe9100add90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fe90000c000 tx=0x7fe900006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:55.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe900010780 con 0x7fe9100b6550 2022-07-12T08:47:55.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9000108e0 con 0x7fe9100b6550 2022-07-12T08:47:55.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908009470 con 0x7fe9100b6550 2022-07-12T08:47:55.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe900010c40 con 0x7fe9100b6550 2022-07-12T08:47:55.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9100b5b00 con 0x7fe9100b6550 2022-07-12T08:47:55.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe90001c460 con 0x7fe9100b6550 2022-07-12T08:47:55.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9100ad7d0 con 0x7fe9100b6550 2022-07-12T08:47:55.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe8ff7fe700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8f0053a40 0x7fe8f0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.416+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fe90006c360 con 0x7fe9100b6550 2022-07-12T08:47:55.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.420+0000 7fe91629d700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8f0053a40 0x7fe8f0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.420+0000 7fe91629d700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8f0053a40 0x7fe8f0055f00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fe90c00d000 tx=0x7fe90c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:55.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.420+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe900043230 con 0x7fe9100b6550 2022-07-12T08:47:55.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.560+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist ls", "format": "json"} v 0) v1 -- 0x7fe910146240 con 0x7fe9100b6550 2022-07-12T08:47:55.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.560+0000 7fe8ff7fe700 1 -- 172.21.15.61:0/1296702960 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist ls", "format": "json"}]=0 listed 1 entries v72) v1 ==== 98+0+81 (secure 0 0 0) 0x7fe900046850 con 0x7fe9100b6550 2022-07-12T08:47:55.569 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:55.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8f0053a40 msgr2=0x7fe8f0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8f0053a40 0x7fe8f0055f00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fe90c00d000 tx=0x7fe90c009040).stop 2022-07-12T08:47:55.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 msgr2=0x7fe9100add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 0x7fe9100add90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fe90000c000 tx=0x7fe900006df0).stop 2022-07-12T08:47:55.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 shutdown_connections 2022-07-12T08:47:55.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe9100b0200 0x7fe910151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9100b6550 0x7fe9100add90 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8f0053a40 0x7fe8f0055f00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 --2- 172.21.15.61:0/1296702960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9100b0df0 0x7fe9100b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 >> 172.21.15.61:0/1296702960 conn(0x7fe910063ad0 msgr2=0x7fe91013f150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:55.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 shutdown_connections 2022-07-12T08:47:55.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.564+0000 7fe91729f700 1 -- 172.21.15.61:0/1296702960 wait complete. 2022-07-12T08:47:55.578 INFO:teuthology.orchestra.run.smithi061.stderr:listed 1 entries 2022-07-12T08:47:55.579 INFO:teuthology.orchestra.run.smithi061.stdout:[{"addr":"172.21.15.121:0/3889265026","until":"2022-07-12T09:47:49.061933+0000"}] 2022-07-12T08:47:55.597 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-12T08:47:55.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:55.599 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:55.600 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:55.617 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-12T08:47:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:47:55.621 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-12T08:47:55.622 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:47:55.623 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:47:55.624 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:55.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:55.727 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:55.739 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:55.775 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:47:55.778 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:47:55.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:47:55.779 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:55.801 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:47:55.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 Processor -- start 2022-07-12T08:47:55.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 -- start start 2022-07-12T08:47:55.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8af80ab290 0x7f8af80ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80afa60 0x7f8af80aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af80abc70 con 0x7f8af805f140 2022-07-12T08:47:55.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af805f580 con 0x7f8af80ab290 2022-07-12T08:47:55.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afd524700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af805c720 con 0x7f8af80afa60 2022-07-12T08:47:55.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8af80ab290 0x7f8af80ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8af805f140 0x7f8af805ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45724/0 (socket says 172.21.15.61:45724) 2022-07-12T08:47:55.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af77fe700 1 -- 172.21.15.61:0/1189098258 learned_addr learned my addr 172.21.15.61:0/1189098258 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:55.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afcd23700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80afa60 0x7f8af80aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af7fff700 1 -- 172.21.15.61:0/1189098258 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80afa60 msgr2=0x7f8af80aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:55.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af7fff700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80afa60 0x7f8af80aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af7fff700 1 -- 172.21.15.61:0/1189098258 >> v1:172.21.15.61:6789/0 conn(0x7f8af805f140 legacy=0x7f8af805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af6ffd700 1 -- 172.21.15.61:0/1189098258 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416416392 0 0) 0x7f8af80abc70 con 0x7f8af805f140 2022-07-12T08:47:55.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8af7fff700 1 -- 172.21.15.61:0/1189098258 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af805c9c0 con 0x7f8af80ab290 2022-07-12T08:47:55.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.912+0000 7f8afcd23700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80afa60 0x7f8af80aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:55.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8af7fff700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80ab290 0x7f8af80ab670 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f8aec002f70 tx=0x7f8aec006df0).ready entity=mon.0 client_cookie=395ea678848c867a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:55.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8af6ffd700 1 -- 172.21.15.61:0/1189098258 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aec00d570 con 0x7f8af80ab290 2022-07-12T08:47:55.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8af6ffd700 1 -- 172.21.15.61:0/1189098258 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec000e90 con 0x7f8af80ab290 2022-07-12T08:47:55.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8af6ffd700 1 -- 172.21.15.61:0/1189098258 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8aec00d8c0 con 0x7f8af80ab290 2022-07-12T08:47:55.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 -- 172.21.15.61:0/1189098258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80ab290 msgr2=0x7f8af80ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80ab290 0x7f8af80ab670 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f8aec002f70 tx=0x7f8aec006df0).stop 2022-07-12T08:47:55.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 -- 172.21.15.61:0/1189098258 shutdown_connections 2022-07-12T08:47:55.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80afa60 0x7f8af80aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 --2- 172.21.15.61:0/1189098258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80ab290 0x7f8af80ab670 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 -- 172.21.15.61:0/1189098258 >> 172.21.15.61:0/1189098258 conn(0x7f8af80b1950 msgr2=0x7f8af80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:55.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 -- 172.21.15.61:0/1189098258 shutdown_connections 2022-07-12T08:47:55.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.916+0000 7f8afd524700 1 -- 172.21.15.61:0/1189098258 wait complete. 2022-07-12T08:47:55.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 Processor -- start 2022-07-12T08:47:55.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 -- start start 2022-07-12T08:47:55.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af805f140 0x7f8af8063c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80ab290 0x7f8af81597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 0x7f8af8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af814a280 con 0x7f8af80afa60 2022-07-12T08:47:55.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8159430 con 0x7f8af805f140 2022-07-12T08:47:55.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afd524700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af8148610 con 0x7f8af80ab290 2022-07-12T08:47:55.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 0x7f8af8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 0x7f8af8159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57516/0 (socket says 172.21.15.61:57516) 2022-07-12T08:47:55.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 -- 172.21.15.61:0/1769549749 learned_addr learned my addr 172.21.15.61:0/1769549749 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:55.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8af77fe700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80ab290 0x7f8af81597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8af7fff700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af805f140 0x7f8af8063c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 -- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80ab290 msgr2=0x7f8af81597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80ab290 0x7f8af81597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 -- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af805f140 msgr2=0x7f8af8063c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:55.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af805f140 0x7f8af8063c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:55.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 -- 172.21.15.61:0/1769549749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec00b180 con 0x7f8af80afa60 2022-07-12T08:47:55.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8af77fe700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80ab290 0x7f8af81597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:47:55.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8afcd23700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 0x7f8af8159ce0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f8af8065e40 tx=0x7f8af0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:55.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.920+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8af0012610 con 0x7f8af80afa60 2022-07-12T08:47:55.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8af0012770 con 0x7f8af80afa60 2022-07-12T08:47:55.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af8067050 con 0x7f8af80afa60 2022-07-12T08:47:55.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8af0012ad0 con 0x7f8af80afa60 2022-07-12T08:47:55.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af8149640 con 0x7f8af80afa60 2022-07-12T08:47:55.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af815a370 con 0x7f8af80afa60 2022-07-12T08:47:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8af0012c30 con 0x7f8af80afa60 2022-07-12T08:47:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8af57fa700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ad80539b0 0x7f8ad8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f8af006c1a0 con 0x7f8af80afa60 2022-07-12T08:47:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.924+0000 7f8af7fff700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ad80539b0 0x7f8ad8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.928+0000 7f8af7fff700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ad80539b0 0x7f8ad8055e70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f8aec01c000 tx=0x7f8aec015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:55.932+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8af00430d0 con 0x7f8af80afa60 2022-07-12T08:47:56.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.060+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8af8057460 con 0x7f8af80afa60 2022-07-12T08:47:56.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.060+0000 7f8af57fa700 1 -- 172.21.15.61:0/1769549749 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16151 (secure 0 0 0) 0x7f8af00466f0 con 0x7f8af80afa60 2022-07-12T08:47:56.064 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:56.065 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:47:49.277545+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:47:01.995190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-12T08:47:02.575389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.121:0/3889265026":"2022-07-12T09:47:49.061933+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:47:56.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ad80539b0 msgr2=0x7f8ad8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ad80539b0 0x7f8ad8055e70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f8aec01c000 tx=0x7f8aec015000).stop 2022-07-12T08:47:56.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 msgr2=0x7f8af8159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 0x7f8af8159ce0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f8af8065e40 tx=0x7f8af0008040).stop 2022-07-12T08:47:56.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 shutdown_connections 2022-07-12T08:47:56.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8af80afa60 0x7f8af8159ce0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8af80ab290 0x7f8af81597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ad80539b0 0x7f8ad8055e70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 --2- 172.21.15.61:0/1769549749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8af805f140 0x7f8af8063c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 >> 172.21.15.61:0/1769549749 conn(0x7f8af80b1950 msgr2=0x7f8af805bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 shutdown_connections 2022-07-12T08:47:56.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.064+0000 7f8afd524700 1 -- 172.21.15.61:0/1769549749 wait complete. 2022-07-12T08:47:56.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:47:01.995190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:47:02.575389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:47:56.101 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:47:56.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 Processor -- start 2022-07-12T08:47:56.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 -- start start 2022-07-12T08:47:56.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea014b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feea00b1000 0x7feea00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea0099010 con 0x7feea00afdc0 2022-07-12T08:47:56.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea0060ff0 con 0x7feea00b1000 2022-07-12T08:47:56.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7feea53a0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea014a930 con 0x7feea0060c10 2022-07-12T08:47:56.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea014b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feea00b1000 0x7feea00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea014b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39716/0 (socket says 172.21.15.61:39716) 2022-07-12T08:47:56.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9ffff700 1 -- 172.21.15.61:0/1622907249 learned_addr learned my addr 172.21.15.61:0/1622907249 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:56.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9f7fe700 1 -- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 msgr2=0x7feea014b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:56.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9f7fe700 1 --2- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea014b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.212+0000 7fee9f7fe700 1 -- 172.21.15.61:0/1622907249 >> v1:172.21.15.111:6789/0 conn(0x7feea00afdc0 legacy=0x7feea009a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7fee9f7fe700 1 -- 172.21.15.61:0/1622907249 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea014abd0 con 0x7feea00b1000 2022-07-12T08:47:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7fee9f7fe700 1 --2- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00af8d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fee900032f0 tx=0x7fee900091d0).ready entity=mon.1 client_cookie=7857669c02c740c7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7fee9effd700 1 -- 172.21.15.61:0/1622907249 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee9000b6c0 con 0x7feea00b1000 2022-07-12T08:47:56.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7fee9effd700 1 -- 172.21.15.61:0/1622907249 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9000b820 con 0x7feea00b1000 2022-07-12T08:47:56.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7fee9effd700 1 -- 172.21.15.61:0/1622907249 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee9000b990 con 0x7feea00b1000 2022-07-12T08:47:56.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 -- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 msgr2=0x7feea00af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 --2- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00af8d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fee900032f0 tx=0x7fee900091d0).stop 2022-07-12T08:47:56.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 -- 172.21.15.61:0/1622907249 shutdown_connections 2022-07-12T08:47:56.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 --2- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00af8d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 --2- 172.21.15.61:0/1622907249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea014b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 -- 172.21.15.61:0/1622907249 >> 172.21.15.61:0/1622907249 conn(0x7feea005bd10 msgr2=0x7feea005c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 -- 172.21.15.61:0/1622907249 shutdown_connections 2022-07-12T08:47:56.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.216+0000 7feea53a0700 1 -- 172.21.15.61:0/1622907249 wait complete. 2022-07-12T08:47:56.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 Processor -- start 2022-07-12T08:47:56.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 -- start start 2022-07-12T08:47:56.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea009a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 0x7feea0065f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00663f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea00ae8a0 con 0x7feea00afdc0 2022-07-12T08:47:56.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea00648d0 con 0x7feea00b1000 2022-07-12T08:47:56.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea53a0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea00627c0 con 0x7feea0060c10 2022-07-12T08:47:56.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 0x7feea0065f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 0x7feea0065f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57528/0 (socket says 172.21.15.61:57528) 2022-07-12T08:47:56.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 -- 172.21.15.61:0/3090701823 learned_addr learned my addr 172.21.15.61:0/3090701823 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:56.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9ffff700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea009a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7feea4b9f700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 -- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 msgr2=0x7feea009a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea009a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 -- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 msgr2=0x7feea00663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 -- 172.21.15.61:0/3090701823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90009470 con 0x7feea00afdc0 2022-07-12T08:47:56.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9f7fe700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 0x7feea0065f00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7fee900032f0 tx=0x7fee90016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee90011e20 con 0x7feea00afdc0 2022-07-12T08:47:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.220+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee90020400 con 0x7feea00afdc0 2022-07-12T08:47:56.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea00668e0 con 0x7feea00afdc0 2022-07-12T08:47:56.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee90020760 con 0x7feea00afdc0 2022-07-12T08:47:56.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea00af2a0 con 0x7feea00afdc0 2022-07-12T08:47:56.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fee90027030 con 0x7feea00afdc0 2022-07-12T08:47:56.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea00af4c0 con 0x7feea00afdc0 2022-07-12T08:47:56.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7fee9d7fa700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fee80053a40 0x7fee80055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fee9006cf60 con 0x7feea00afdc0 2022-07-12T08:47:56.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7fee9ffff700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fee80053a40 0x7fee80055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.224+0000 7fee9ffff700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fee80053a40 0x7fee80055f00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fee94008000 tx=0x7fee94008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.232+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee90043df0 con 0x7feea00afdc0 2022-07-12T08:47:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.360+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7feea0064720 con 0x7feea00afdc0 2022-07-12T08:47:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.360+0000 7fee9d7fa700 1 -- 172.21.15.61:0/3090701823 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7fee900250b0 con 0x7feea00afdc0 2022-07-12T08:47:56.366 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:47:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fee80053a40 msgr2=0x7fee80055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fee80053a40 0x7fee80055f00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fee94008000 tx=0x7fee94008040).stop 2022-07-12T08:47:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 msgr2=0x7feea0065f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 0x7feea0065f00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7fee900032f0 tx=0x7fee90016000).stop 2022-07-12T08:47:56.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 shutdown_connections 2022-07-12T08:47:56.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fee80053a40 0x7fee80055f00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feea00b1000 0x7feea00663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feea00afdc0 0x7feea0065f00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 --2- 172.21.15.61:0/3090701823 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feea0060c10 0x7feea009a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 >> 172.21.15.61:0/3090701823 conn(0x7feea005bd10 msgr2=0x7feea01486b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 shutdown_connections 2022-07-12T08:47:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.364+0000 7feea53a0700 1 -- 172.21.15.61:0/3090701823 wait complete. 2022-07-12T08:47:56.376 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:47:56.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 Processor -- start 2022-07-12T08:47:56.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 -- start start 2022-07-12T08:47:56.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85013fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8501388d0 0x7ff850138d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8500b04e0 con 0x7ff850061ac0 2022-07-12T08:47:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85013f2c0 con 0x7ff8501388d0 2022-07-12T08:47:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.484+0000 7ff8551fc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500616e0 con 0x7ff850061300 2022-07-12T08:47:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8501388d0 0x7ff850138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85013fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8501388d0 0x7ff850138d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36112/0 (socket says 172.21.15.61:36112) 2022-07-12T08:47:56.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 -- 172.21.15.61:0/2051197852 learned_addr learned my addr 172.21.15.61:0/2051197852 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:56.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84e59c700 1 --1- 172.21.15.61:0/2051197852 >> v1:172.21.15.111:6789/0 conn(0x7ff850061ac0 0x7ff8500b08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59354/0 (socket says 172.21.15.61:59354) 2022-07-12T08:47:56.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 -- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 msgr2=0x7ff85013fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:56.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 --2- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85013fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 -- 172.21.15.61:0/2051197852 >> v1:172.21.15.111:6789/0 conn(0x7ff850061ac0 legacy=0x7ff8500b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 -- 172.21.15.61:0/2051197852 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85013f4b0 con 0x7ff8501388d0 2022-07-12T08:47:56.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84f59e700 1 --2- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff850138d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff8480032f0 tx=0x7ff848009290).ready entity=mon.1 client_cookie=e655d3b9002a9ce1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84dd9b700 1 -- 172.21.15.61:0/2051197852 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff84800b8a0 con 0x7ff8501388d0 2022-07-12T08:47:56.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84dd9b700 1 -- 172.21.15.61:0/2051197852 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84800ba00 con 0x7ff8501388d0 2022-07-12T08:47:56.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff84dd9b700 1 -- 172.21.15.61:0/2051197852 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff84800bb70 con 0x7ff8501388d0 2022-07-12T08:47:56.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 -- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 msgr2=0x7ff850138d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 --2- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff850138d40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff8480032f0 tx=0x7ff848009290).stop 2022-07-12T08:47:56.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 -- 172.21.15.61:0/2051197852 shutdown_connections 2022-07-12T08:47:56.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 --2- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff850138d40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 --2- 172.21.15.61:0/2051197852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85013fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 -- 172.21.15.61:0/2051197852 >> 172.21.15.61:0/2051197852 conn(0x7ff8500b5a40 msgr2=0x7ff8500b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 -- 172.21.15.61:0/2051197852 shutdown_connections 2022-07-12T08:47:56.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.488+0000 7ff8551fc700 1 -- 172.21.15.61:0/2051197852 wait complete. 2022-07-12T08:47:56.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 Processor -- start 2022-07-12T08:47:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 -- start start 2022-07-12T08:47:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85012e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff850061ac0 0x7ff8500a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff8500ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850141b20 con 0x7ff850061ac0 2022-07-12T08:47:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850141dc0 con 0x7ff8501388d0 2022-07-12T08:47:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85013d2b0 con 0x7ff850061300 2022-07-12T08:47:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85012e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff850061ac0 0x7ff8500a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85012e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39736/0 (socket says 172.21.15.61:39736) 2022-07-12T08:47:56.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 -- 172.21.15.61:0/671277945 learned_addr learned my addr 172.21.15.61:0/671277945 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:56.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff8500ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 -- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 msgr2=0x7ff8500ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff8500ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 -- 172.21.15.61:0/671277945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff850061ac0 msgr2=0x7ff8500a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff850061ac0 0x7ff8500a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 -- 172.21.15.61:0/671277945 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848009530 con 0x7ff850061300 2022-07-12T08:47:56.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff84ed9d700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85012e040 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff84400c000 tx=0x7ff844006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff844010780 con 0x7ff850061300 2022-07-12T08:47:56.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8440108e0 con 0x7ff850061300 2022-07-12T08:47:56.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8500a9ef0 con 0x7ff850061300 2022-07-12T08:47:56.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff844010c40 con 0x7ff850061300 2022-07-12T08:47:56.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.492+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8501416f0 con 0x7ff850061300 2022-07-12T08:47:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.496+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85013d470 con 0x7ff850061300 2022-07-12T08:47:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.496+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff84401c460 con 0x7ff850061300 2022-07-12T08:47:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.496+0000 7ff83ffff700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff82c053920 0x7ff82c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.496+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7ff84406c330 con 0x7ff850061300 2022-07-12T08:47:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.496+0000 7ff84e59c700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff82c053920 0x7ff82c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.496+0000 7ff84e59c700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff82c053920 0x7ff82c055de0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7ff84000d000 tx=0x7ff840007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.508+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff844043180 con 0x7ff850061300 2022-07-12T08:47:56.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.632+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff850057460 con 0x7ff850061300 2022-07-12T08:47:56.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.632+0000 7ff83ffff700 1 -- 172.21.15.61:0/671277945 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff8440467a0 con 0x7ff850061300 2022-07-12T08:47:56.638 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:56.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff82c053920 msgr2=0x7ff82c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff82c053920 0x7ff82c055de0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7ff84000d000 tx=0x7ff840007b80).stop 2022-07-12T08:47:56.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 msgr2=0x7ff85012e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85012e040 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ff84400c000 tx=0x7ff844006df0).stop 2022-07-12T08:47:56.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 shutdown_connections 2022-07-12T08:47:56.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff82c053920 0x7ff82c055de0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8501388d0 0x7ff8500ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff850061ac0 0x7ff8500a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 --2- 172.21.15.61:0/671277945 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff850061300 0x7ff85012e040 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 >> 172.21.15.61:0/671277945 conn(0x7ff8500b5a40 msgr2=0x7ff8500b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 shutdown_connections 2022-07-12T08:47:56.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.636+0000 7ff8551fc700 1 -- 172.21.15.61:0/671277945 wait complete. 2022-07-12T08:47:56.651 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:47:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.768+0000 7fe565e1f700 1 Processor -- start 2022-07-12T08:47:56.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.768+0000 7fe565e1f700 1 -- start start 2022-07-12T08:47:56.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5600b0280 0x7fe56014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600aa830 0x7fe5600aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe56005c3e0 con 0x7fe5600b5920 2022-07-12T08:47:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5600b5d00 con 0x7fe5600b0280 2022-07-12T08:47:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56005c610 con 0x7fe5600aa830 2022-07-12T08:47:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5600b0280 0x7fe56014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5600b0280 0x7fe56014b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57550/0 (socket says 172.21.15.61:57550) 2022-07-12T08:47:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 -- 172.21.15.61:0/1051904385 learned_addr learned my addr 172.21.15.61:0/1051904385 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe56561e700 1 --2- 172.21.15.61:0/1051904385 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600aa830 0x7fe5600aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 -- 172.21.15.61:0/1051904385 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600aa830 msgr2=0x7fe5600aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 --2- 172.21.15.61:0/1051904385 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600aa830 0x7fe5600aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 -- 172.21.15.61:0/1051904385 >> v1:172.21.15.61:6789/0 conn(0x7fe5600b5920 legacy=0x7fe56014a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 -- 172.21.15.61:0/1051904385 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56005c8b0 con 0x7fe5600b0280 2022-07-12T08:47:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55ffff700 1 --2- 172.21.15.61:0/1051904385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600b0280 0x7fe56014b0f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fe5500032f0 tx=0x7fe550009290).ready entity=mon.0 client_cookie=202d9d12a4908327 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55f7fe700 1 -- 172.21.15.61:0/1051904385 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe55000b8a0 con 0x7fe5600b0280 2022-07-12T08:47:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55f7fe700 1 -- 172.21.15.61:0/1051904385 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55000ba00 con 0x7fe5600b0280 2022-07-12T08:47:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe55f7fe700 1 -- 172.21.15.61:0/1051904385 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe55000bb70 con 0x7fe5600b0280 2022-07-12T08:47:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- 172.21.15.61:0/1051904385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600b0280 msgr2=0x7fe56014b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 --2- 172.21.15.61:0/1051904385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600b0280 0x7fe56014b0f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fe5500032f0 tx=0x7fe550009290).stop 2022-07-12T08:47:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- 172.21.15.61:0/1051904385 shutdown_connections 2022-07-12T08:47:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 --2- 172.21.15.61:0/1051904385 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600aa830 0x7fe5600aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 --2- 172.21.15.61:0/1051904385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600b0280 0x7fe56014b0f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- 172.21.15.61:0/1051904385 >> 172.21.15.61:0/1051904385 conn(0x7fe560063340 msgr2=0x7fe5600b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- 172.21.15.61:0/1051904385 shutdown_connections 2022-07-12T08:47:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.772+0000 7fe565e1f700 1 -- 172.21.15.61:0/1051904385 wait complete. 2022-07-12T08:47:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 Processor -- start 2022-07-12T08:47:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 -- start start 2022-07-12T08:47:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 0x7fe560098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5600b0280 0x7fe560148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600b5920 0x7fe5601488f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5600af060 con 0x7fe5600aa830 2022-07-12T08:47:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe560149630 con 0x7fe5600b0280 2022-07-12T08:47:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe560146b40 con 0x7fe5600b5920 2022-07-12T08:47:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 0x7fe560098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 0x7fe560098a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57554/0 (socket says 172.21.15.61:57554) 2022-07-12T08:47:56.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 -- 172.21.15.61:0/3778401990 learned_addr learned my addr 172.21.15.61:0/3778401990 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:56.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe56561e700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600b5920 0x7fe5601488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe55ffff700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5600b0280 0x7fe560148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 -- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600b5920 msgr2=0x7fe5601488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600b5920 0x7fe5601488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 -- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5600b0280 msgr2=0x7fe560148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5600b0280 0x7fe560148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 -- 172.21.15.61:0/3778401990 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe550009530 con 0x7fe5600aa830 2022-07-12T08:47:56.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe56561e700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600b5920 0x7fe5601488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:56.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe564e1d700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 0x7fe560098a40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7fe5600b5f30 tx=0x7fe554007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe55400ce30 con 0x7fe5600aa830 2022-07-12T08:47:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe554007240 con 0x7fe5600aa830 2022-07-12T08:47:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5601498c0 con 0x7fe5600aa830 2022-07-12T08:47:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe554006700 con 0x7fe5600aa830 2022-07-12T08:47:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.776+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe560148f80 con 0x7fe5600aa830 2022-07-12T08:47:56.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.780+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5600af1d0 con 0x7fe5600aa830 2022-07-12T08:47:56.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.780+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe554006860 con 0x7fe5600aa830 2022-07-12T08:47:56.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.780+0000 7fe55dffb700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe540053a60 0x7fe540055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:56.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.780+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fe55406b7c0 con 0x7fe5600aa830 2022-07-12T08:47:56.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.780+0000 7fe55ffff700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe540053a60 0x7fe540055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:56.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.780+0000 7fe55ffff700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe540053a60 0x7fe540055f20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fe550019000 tx=0x7fe550016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:56.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.784+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe5600af1d0 con 0x7fe5600aa830 2022-07-12T08:47:56.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.912+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe5600aeeb0 con 0x7fe5600aa830 2022-07-12T08:47:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.912+0000 7fe55dffb700 1 -- 172.21.15.61:0/3778401990 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+10 (secure 0 0 0) 0x7fe554069e60 con 0x7fe5600aa830 2022-07-12T08:47:56.915 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:56.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe540053a60 msgr2=0x7fe540055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe540053a60 0x7fe540055f20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fe550019000 tx=0x7fe550016000).stop 2022-07-12T08:47:56.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 msgr2=0x7fe560098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:56.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 0x7fe560098a40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 rx=0x7fe5600b5f30 tx=0x7fe554007bd0).stop 2022-07-12T08:47:56.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 shutdown_connections 2022-07-12T08:47:56.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe5600b5920 0x7fe5601488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe540053a60 0x7fe540055f20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe5600b0280 0x7fe560148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 --2- 172.21.15.61:0/3778401990 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5600aa830 0x7fe560098a40 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:56.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 >> 172.21.15.61:0/3778401990 conn(0x7fe560063340 msgr2=0x7fe5600b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:56.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 shutdown_connections 2022-07-12T08:47:56.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:56.916+0000 7fe565e1f700 1 -- 172.21.15.61:0/3778401990 wait complete. 2022-07-12T08:47:56.928 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:47:57.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.036+0000 7fd85deb7700 1 Processor -- start 2022-07-12T08:47:57.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.036+0000 7fd85deb7700 1 -- start start 2022-07-12T08:47:57.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd8580af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd85805bdf0 0x7fd85805c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd858143610 con 0x7fd8580af4b0 2022-07-12T08:47:57.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd858143840 con 0x7fd85805bdf0 2022-07-12T08:47:57.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8580afef0 con 0x7fd858060b50 2022-07-12T08:47:57.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85ceb5700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd8580af4b0 0x7fd858142120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45780/0 (socket says 172.21.15.61:45780) 2022-07-12T08:47:57.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85d6b6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd85805bdf0 0x7fd85805c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85ceb5700 1 -- 172.21.15.61:0/1008246478 learned_addr learned my addr 172.21.15.61:0/1008246478 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd857fff700 1 --2- 172.21.15.61:0/1008246478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd8580af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd8577fe700 1 -- 172.21.15.61:0/1008246478 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396502720 0 0) 0x7fd858143610 con 0x7fd8580af4b0 2022-07-12T08:47:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd8577fe700 1 -- 172.21.15.61:0/1008246478 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd844003260 con 0x7fd8580af4b0 2022-07-12T08:47:57.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85d6b6700 1 -- 172.21.15.61:0/1008246478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 msgr2=0x7fd8580af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:57.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85d6b6700 1 --2- 172.21.15.61:0/1008246478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd8580af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85d6b6700 1 -- 172.21.15.61:0/1008246478 >> v1:172.21.15.61:6789/0 conn(0x7fd8580af4b0 legacy=0x7fd858142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85d6b6700 1 -- 172.21.15.61:0/1008246478 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd858142830 con 0x7fd85805bdf0 2022-07-12T08:47:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85d6b6700 1 --2- 172.21.15.61:0/1008246478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85805bdf0 0x7fd85805c230 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fd8500032f0 tx=0x7fd8500091d0).ready entity=mon.0 client_cookie=d23ec99c6754c4e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd8577fe700 1 -- 172.21.15.61:0/1008246478 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd85000b600 con 0x7fd85805bdf0 2022-07-12T08:47:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd8577fe700 1 -- 172.21.15.61:0/1008246478 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85000b760 con 0x7fd85805bdf0 2022-07-12T08:47:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd8577fe700 1 -- 172.21.15.61:0/1008246478 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd85000b8d0 con 0x7fd85805bdf0 2022-07-12T08:47:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- 172.21.15.61:0/1008246478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85805bdf0 msgr2=0x7fd85805c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 --2- 172.21.15.61:0/1008246478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85805bdf0 0x7fd85805c230 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7fd8500032f0 tx=0x7fd8500091d0).stop 2022-07-12T08:47:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- 172.21.15.61:0/1008246478 shutdown_connections 2022-07-12T08:47:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 --2- 172.21.15.61:0/1008246478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd85805bdf0 0x7fd85805c230 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 --2- 172.21.15.61:0/1008246478 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd8580af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- 172.21.15.61:0/1008246478 >> 172.21.15.61:0/1008246478 conn(0x7fd8580b92c0 msgr2=0x7fd8580b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- 172.21.15.61:0/1008246478 shutdown_connections 2022-07-12T08:47:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.040+0000 7fd85deb7700 1 -- 172.21.15.61:0/1008246478 wait complete. 2022-07-12T08:47:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 Processor -- start 2022-07-12T08:47:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 -- start start 2022-07-12T08:47:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd85805bdf0 0x7fd858149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd85813f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 0x7fd85813f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85809a480 con 0x7fd8580af4b0 2022-07-12T08:47:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd858144190 con 0x7fd85805bdf0 2022-07-12T08:47:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85deb7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd858148e70 con 0x7fd858060b50 2022-07-12T08:47:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 0x7fd85813f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 0x7fd85813f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57570/0 (socket says 172.21.15.61:57570) 2022-07-12T08:47:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 -- 172.21.15.61:0/2850925317 learned_addr learned my addr 172.21.15.61:0/2850925317 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85ceb5700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd85805bdf0 0x7fd858149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd857fff700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd85813f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 -- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 msgr2=0x7fd85813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd85813f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 -- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd85805bdf0 msgr2=0x7fd858149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd85805bdf0 0x7fd858149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 -- 172.21.15.61:0/2850925317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd850009470 con 0x7fd8580af4b0 2022-07-12T08:47:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85ceb5700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd85805bdf0 0x7fd858149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd85d6b6700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 0x7fd85813f940 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7fd8500032f0 tx=0x7fd850016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.044+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd85000b600 con 0x7fd8580af4b0 2022-07-12T08:47:57.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd850011cc0 con 0x7fd8580af4b0 2022-07-12T08:47:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85809b2a0 con 0x7fd8580af4b0 2022-07-12T08:47:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd85001f5e0 con 0x7fd8580af4b0 2022-07-12T08:47:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85809b9e0 con 0x7fd8580af4b0 2022-07-12T08:47:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd858057460 con 0x7fd8580af4b0 2022-07-12T08:47:57.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd850026030 con 0x7fd8580af4b0 2022-07-12T08:47:57.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd855ffb700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd838053920 0x7fd838055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fd85006cae0 con 0x7fd8580af4b0 2022-07-12T08:47:57.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.048+0000 7fd85ceb5700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd838053920 0x7fd838055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.052+0000 7fd85ceb5700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd838053920 0x7fd838055de0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd84c00b000 tx=0x7fd84c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:57.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.056+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd850043900 con 0x7fd8580af4b0 2022-07-12T08:47:57.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd85809bf80 con 0x7fd8580af4b0 2022-07-12T08:47:57.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd855ffb700 1 -- 172.21.15.61:0/2850925317 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd8500240a0 con 0x7fd8580af4b0 2022-07-12T08:47:57.183 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:47:57.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd838053920 msgr2=0x7fd838055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd838053920 0x7fd838055de0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd84c00b000 tx=0x7fd84c009040).stop 2022-07-12T08:47:57.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 msgr2=0x7fd85813f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 0x7fd85813f940 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7fd8500032f0 tx=0x7fd850016000).stop 2022-07-12T08:47:57.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 shutdown_connections 2022-07-12T08:47:57.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8580af4b0 0x7fd85813f940 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd858060b50 0x7fd85813f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.180+0000 7fd85deb7700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd838053920 0x7fd838055de0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.184+0000 7fd85deb7700 1 --2- 172.21.15.61:0/2850925317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd85805bdf0 0x7fd858149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.184+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 >> 172.21.15.61:0/2850925317 conn(0x7fd8580b92c0 msgr2=0x7fd8580b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:57.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.184+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 shutdown_connections 2022-07-12T08:47:57.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.184+0000 7fd85deb7700 1 -- 172.21.15.61:0/2850925317 wait complete. 2022-07-12T08:47:57.196 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:47:57.197 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:57.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 Processor -- start 2022-07-12T08:47:57.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 -- start start 2022-07-12T08:47:57.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53240614a0 0x7f53240b0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5324146420 0x7f53240af510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53240acab0 con 0x7f53240b02a0 2022-07-12T08:47:57.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532405c150 con 0x7f5324146420 2022-07-12T08:47:57.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532ba64700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53240b0ff0 con 0x7f53240614a0 2022-07-12T08:47:57.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5324146420 0x7f53240af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5324146420 0x7f53240af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57578/0 (socket says 172.21.15.61:57578) 2022-07-12T08:47:57.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f5328fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53240614a0 0x7f53240b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 -- 172.21.15.61:0/125364580 learned_addr learned my addr 172.21.15.61:0/125364580 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f5329800700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f53240b02a0 0x7f53240b1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59394/0 (socket says 172.21.15.61:59394) 2022-07-12T08:47:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 -- 172.21.15.61:0/125364580 >> v2:172.21.15.111:3300/0 conn(0x7f53240614a0 msgr2=0x7f53240b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:47:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 --2- 172.21.15.61:0/125364580 >> v2:172.21.15.111:3300/0 conn(0x7f53240614a0 0x7f53240b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 -- 172.21.15.61:0/125364580 >> v1:172.21.15.111:6789/0 conn(0x7f53240b02a0 legacy=0x7f53240b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.316+0000 7f532a001700 1 -- 172.21.15.61:0/125364580 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240b2170 con 0x7f5324146420 2022-07-12T08:47:57.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532a001700 1 --2- 172.21.15.61:0/125364580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f53240af510 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f531c0032f0 tx=0x7f531c009290).ready entity=mon.0 client_cookie=c98991e2a76e7230 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:57.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f5313fff700 1 -- 172.21.15.61:0/125364580 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f531c00b870 con 0x7f5324146420 2022-07-12T08:47:57.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f5313fff700 1 -- 172.21.15.61:0/125364580 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c00b9d0 con 0x7f5324146420 2022-07-12T08:47:57.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f5313fff700 1 -- 172.21.15.61:0/125364580 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f531c00bb40 con 0x7f5324146420 2022-07-12T08:47:57.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 -- 172.21.15.61:0/125364580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 msgr2=0x7f53240af510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 --2- 172.21.15.61:0/125364580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f53240af510 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7f531c0032f0 tx=0x7f531c009290).stop 2022-07-12T08:47:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 -- 172.21.15.61:0/125364580 shutdown_connections 2022-07-12T08:47:57.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 --2- 172.21.15.61:0/125364580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f53240af510 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 --2- 172.21.15.61:0/125364580 >> v2:172.21.15.111:3300/0 conn(0x7f53240614a0 0x7f53240b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 -- 172.21.15.61:0/125364580 >> 172.21.15.61:0/125364580 conn(0x7f532405f140 msgr2=0x7f532405f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:57.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 -- 172.21.15.61:0/125364580 shutdown_connections 2022-07-12T08:47:57.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.320+0000 7f532ba64700 1 -- 172.21.15.61:0/125364580 wait complete. 2022-07-12T08:47:57.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 Processor -- start 2022-07-12T08:47:57.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 -- start start 2022-07-12T08:47:57.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53240614a0 0x7f5324140c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53240b02a0 0x7f5324152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f532408f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5324094380 con 0x7f5324146420 2022-07-12T08:47:57.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5324152010 con 0x7f53240b02a0 2022-07-12T08:47:57.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532ba64700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53241522b0 con 0x7f53240614a0 2022-07-12T08:47:57.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f532408f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f532408f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57584/0 (socket says 172.21.15.61:57584) 2022-07-12T08:47:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 -- 172.21.15.61:0/181931405 learned_addr learned my addr 172.21.15.61:0/181931405 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f5328fff700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53240b02a0 0x7f5324152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f5329800700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53240614a0 0x7f5324140c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 -- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53240614a0 msgr2=0x7f5324140c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53240614a0 0x7f5324140c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 -- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53240b02a0 msgr2=0x7f5324152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53240b02a0 0x7f5324152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 -- 172.21.15.61:0/181931405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c009530 con 0x7f5324146420 2022-07-12T08:47:57.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f5329800700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53240614a0 0x7f5324140c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:57.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.324+0000 7f532a001700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f532408f690 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f531c0032f0 tx=0x7f531c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:57.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f531c00b870 con 0x7f5324146420 2022-07-12T08:47:57.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531c021480 con 0x7f5324146420 2022-07-12T08:47:57.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53241410f0 con 0x7f5324146420 2022-07-12T08:47:57.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f531c0217e0 con 0x7f5324146420 2022-07-12T08:47:57.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532409b7d0 con 0x7f5324146420 2022-07-12T08:47:57.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53240abd10 con 0x7f5324146420 2022-07-12T08:47:57.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f531c00f070 con 0x7f5324146420 2022-07-12T08:47:57.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f53127fc700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5304053a40 0x7f5304055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f531c06d330 con 0x7f5324146420 2022-07-12T08:47:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f5329800700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5304053a40 0x7f5304055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:57.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.328+0000 7f5329800700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5304053a40 0x7f5304055f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f532000b000 tx=0x7f5320000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:57.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.332+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f531c044310 con 0x7f5324146420 2022-07-12T08:47:57.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.496+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5324152470 con 0x7f5324146420 2022-07-12T08:47:57.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.496+0000 7f53127fc700 1 -- 172.21.15.61:0/181931405 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f531c015070 con 0x7f5324146420 2022-07-12T08:47:57.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:57.500 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:57.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.496+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5304053a40 msgr2=0x7f5304055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5304053a40 0x7f5304055f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f532000b000 tx=0x7f5320000bc0).stop 2022-07-12T08:47:57.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 msgr2=0x7f532408f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:57.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f532408f690 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7f531c0032f0 tx=0x7f531c016000).stop 2022-07-12T08:47:57.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 shutdown_connections 2022-07-12T08:47:57.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5324146420 0x7f532408f690 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5304053a40 0x7f5304055f00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53240b02a0 0x7f5324152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 --2- 172.21.15.61:0/181931405 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53240614a0 0x7f5324140c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 >> 172.21.15.61:0/181931405 conn(0x7f532405f140 msgr2=0x7f5324144680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 shutdown_connections 2022-07-12T08:47:57.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:57.500+0000 7f532ba64700 1 -- 172.21.15.61:0/181931405 wait complete. 2022-07-12T08:47:57.505 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:57.717 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:47:03.397499+0000', 'modified': '2022-07-12T08:47:05.156395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6196}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6196': {'gid': 6196, 'name': 'a', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3265139715', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3265139715}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3265139715}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:47:57.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:47:57.719 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:47:57.843 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6196,"want_state":"up:active","state":"up:active","rank_uptime":54.429881332999997,"mdsmap_epoch":38,"osdmap_epoch":72,"osdmap_epoch_barrier":72,"uptime":58.777941169000002} 2022-07-12T08:47:57.844 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6196, "mdsmap_epoch": 38, "osdmap_epoch": 72, "osdmap_epoch_barrier": 72, "rank_uptime": 54.429881333, "state": "up:active", "uptime": 58.777941169, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:47:57.847 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:47:57.847 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:47:57.849 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:47:57.849 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:57.850 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:57.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:47:57.895 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:47:57.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:47:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:47:57.957 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:47:57.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:57.959 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:47:57.977 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:47:57.978 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:47:58.055 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:47:58.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:47:58.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:47:58.099+0000 7f2b07b12080 -1 init, newargv = 0x55c12aab65d0 newargc=9 2022-07-12T08:47:58.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21685]: starting ceph client 2022-07-12T08:47:58.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21685]: starting fuse 2022-07-12T08:47:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:47:58.119 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:47:58.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.120 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:47:58.125 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:47:58.125 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:47:58.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:47:58.127 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:47:58.127 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:47:58.129 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:47:58.130 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:47:58.131 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:58.133 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:47:58.134 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:47:58.134 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:47:58.135 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:58.136 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:47:58.137 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:47:58.138 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:47:58.139 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:47:58.140 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:58.140 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:47:58.141 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:47:58.143 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:47:58.143 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:47:58.144 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:47:58.144 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:47:58.144 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:47:58.145 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:47:58.210 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21685.asok 2022-07-12T08:47:58.215 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21685.asok 2022-07-12T08:47:58.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21685.asok status 2022-07-12T08:47:58.305 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:47:58.305 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:47:58.306 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:47:58.306 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:47:58.306 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:47:58.307 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:47:58.307 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:47:58.308 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21685", 2022-07-12T08:47:58.308 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:47:58.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:58.308 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:47:58.309 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:47:58.309 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6570, 2022-07-12T08:47:58.309 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:47:58.310 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:47:58.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:47:58.311 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6570 2022-07-12T08:47:58.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:58.311 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:47:58.312 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:47:58.312 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:47:58.312 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2228558543 2022-07-12T08:47:58.312 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:47:58.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:58.313 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:47:58.313 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:47:58.313 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:47:58.314 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2228558543 2022-07-12T08:47:58.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:47:58.315 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6570 172.21.15.121:0/2228558543", 2022-07-12T08:47:58.315 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2228558543", 2022-07-12T08:47:58.315 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:47:58.316 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 38, 2022-07-12T08:47:58.316 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 72, 2022-07-12T08:47:58.316 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:47:58.317 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:47:58.317 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:47:58.318 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:47:58.319 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.320 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:58.320 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.331 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:47:58.332 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:47:58.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.333 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.353 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:47:58.354 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:47:58.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:47:58.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:47:58.424 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:47:58.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:47:58.472 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:47:58.474 INFO:tasks.cephfs.mount:Terminating background process 2022-07-12T08:47:58.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.475 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:58.476 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.486 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:47:58.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:47:58.487 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:47:58.488 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:47:58.489 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21685]: fuse finished with error 0 and tester_r 0 2022-07-12T08:47:58.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.510 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:58.511 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.559 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:47:58.560 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:47:58.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:47:58.677 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:47:58.687 INFO:tasks.cephfs.mount:Terminating background process 2022-07-12T08:47:58.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.689 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:58.690 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:58.701 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:47:58.703 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:47:58.705 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:47:58.706 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:47:58.707 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:58.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21144]: fuse finished with error 0 and tester_r 0 2022-07-12T08:47:58.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.768 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:58.768 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:58.776 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:47:58.777 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:47:58.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:47:58.904 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:47:58.945 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:47:58.946 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:47:58.946 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:47:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:47:58.965 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:47:58.965 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:47:58.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:47:58.967 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:47:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:47:58.979 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:47:58.979 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:47:59.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b7d880700 1 Processor -- start 2022-07-12T08:47:59.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b7d880700 1 -- start start 2022-07-12T08:47:59.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b7d880700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1b7813f670 0x7f1b780ae350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b7d880700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b780b30b0 con 0x7f1b78060250 2022-07-12T08:47:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b7d880700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b780b2ba0 con 0x7f1b780b27c0 2022-07-12T08:47:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b7d880700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b7813fad0 con 0x7f1b7813f670 2022-07-12T08:47:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b6e7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1b78060250 0x7f1b7813ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45808/0 (socket says 172.21.15.61:45808) 2022-07-12T08:47:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b777fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1b7813f670 0x7f1b780ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b6e7fc700 1 -- 172.21.15.61:0/755293967 learned_addr learned my addr 172.21.15.61:0/755293967 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:59.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.096+0000 7f1b767fc700 1 -- 172.21.15.61:0/755293967 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257113795 0 0) 0x7f1b780b30b0 con 0x7f1b78060250 2022-07-12T08:47:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b767fc700 1 -- 172.21.15.61:0/755293967 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58003260 con 0x7f1b78060250 2022-07-12T08:47:59.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b777fe700 1 -- 172.21.15.61:0/755293967 >> v1:172.21.15.111:6789/0 conn(0x7f1b780b27c0 legacy=0x7f1b780af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b777fe700 1 -- 172.21.15.61:0/755293967 >> v1:172.21.15.61:6789/0 conn(0x7f1b78060250 legacy=0x7f1b7813ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b777fe700 1 -- 172.21.15.61:0/755293967 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b780b32e0 con 0x7f1b7813f670 2022-07-12T08:47:59.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b777fe700 1 --2- 172.21.15.61:0/755293967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b7813f670 0x7f1b780ae350 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f1b700032f0 tx=0x7f1b700091d0).ready entity=mon.0 client_cookie=7bc16662bd4e08ce server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:59.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b767fc700 1 -- 172.21.15.61:0/755293967 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b7000b6c0 con 0x7f1b7813f670 2022-07-12T08:47:59.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b767fc700 1 -- 172.21.15.61:0/755293967 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7000b820 con 0x7f1b7813f670 2022-07-12T08:47:59.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b767fc700 1 -- 172.21.15.61:0/755293967 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b7000b990 con 0x7f1b7813f670 2022-07-12T08:47:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- 172.21.15.61:0/755293967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b7813f670 msgr2=0x7f1b780ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 --2- 172.21.15.61:0/755293967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b7813f670 0x7f1b780ae350 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f1b700032f0 tx=0x7f1b700091d0).stop 2022-07-12T08:47:59.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- 172.21.15.61:0/755293967 shutdown_connections 2022-07-12T08:47:59.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 --2- 172.21.15.61:0/755293967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b7813f670 0x7f1b780ae350 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- 172.21.15.61:0/755293967 >> 172.21.15.61:0/755293967 conn(0x7f1b780633a0 msgr2=0x7f1b780637b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- 172.21.15.61:0/755293967 shutdown_connections 2022-07-12T08:47:59.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- 172.21.15.61:0/755293967 wait complete. 2022-07-12T08:47:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 Processor -- start 2022-07-12T08:47:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- start start 2022-07-12T08:47:59.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 0x7f1b78064a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b780b27c0 0x7f1b780659d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7813f670 0x7f1b780b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781508b0 con 0x7f1b78060250 2022-07-12T08:47:59.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781531e0 con 0x7f1b7813f670 2022-07-12T08:47:59.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b7d880700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b781504f0 con 0x7f1b780b27c0 2022-07-12T08:47:59.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b76ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 0x7f1b78064a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b76ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 0x7f1b78064a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57594/0 (socket says 172.21.15.61:57594) 2022-07-12T08:47:59.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.100+0000 7f1b76ffd700 1 -- 172.21.15.61:0/3765328820 learned_addr learned my addr 172.21.15.61:0/3765328820 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:59.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b6e7fc700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b780b27c0 0x7f1b780659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b777fe700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7813f670 0x7f1b780b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b76ffd700 1 -- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b780b27c0 msgr2=0x7f1b780659d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b76ffd700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b780b27c0 0x7f1b780659d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b76ffd700 1 -- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7813f670 msgr2=0x7f1b780b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b76ffd700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7813f670 0x7f1b780b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b76ffd700 1 -- 172.21.15.61:0/3765328820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70009470 con 0x7f1b78060250 2022-07-12T08:47:59.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b6e7fc700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b780b27c0 0x7f1b780659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:59.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b777fe700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7813f670 0x7f1b780b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:47:59.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b76ffd700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 0x7f1b78064a70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f1b78062f60 tx=0x7f1b68007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:59.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b6800ce30 con 0x7f1b78060250 2022-07-12T08:47:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6800f470 con 0x7f1b78060250 2022-07-12T08:47:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b780b5fa0 con 0x7f1b78060250 2022-07-12T08:47:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b6800f7d0 con 0x7f1b78060250 2022-07-12T08:47:59.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b78065720 con 0x7f1b78060250 2022-07-12T08:47:59.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1b68019030 con 0x7f1b78060250 2022-07-12T08:47:59.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b78150710 con 0x7f1b78060250 2022-07-12T08:47:59.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b74ff9700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b5c0539d0 0x7f1b5c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f1b6806b2d0 con 0x7f1b78060250 2022-07-12T08:47:59.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.104+0000 7f1b6e7fc700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b5c0539d0 0x7f1b5c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.108+0000 7f1b6e7fc700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b5c0539d0 0x7f1b5c055e90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1b6000b000 tx=0x7f1b60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:59.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.112+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b78150710 con 0x7f1b78060250 2022-07-12T08:47:59.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.240+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f1b78150710 con 0x7f1b78060250 2022-07-12T08:47:59.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.244+0000 7f1b74ff9700 1 -- 172.21.15.61:0/3765328820 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v72) v1 ==== 81+0+33773 (secure 0 0 0) 0x7f1b78150710 con 0x7f1b78060250 2022-07-12T08:47:59.248 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:59.249 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:47:59.249 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 72, 2022-07-12T08:47:59.250 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:47:59.250 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:47:59.251 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:47:49.277545+0000", 2022-07-12T08:47:59.251 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:47:59.252 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:47:59.252 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:47:59.252 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:47:59.253 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:47:59.253 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:47:59.254 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:47:59.255 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:47:59.255 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:47:59.256 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:47:59.256 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:47:59.257 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:47:59.257 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:47:59.258 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:47:59.258 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:47:59.259 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 14, 2022-07-12T08:47:59.260 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:47:59.260 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:47:59.261 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:47:59.261 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:47:59.262 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:47:59.262 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.263 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:47:59.263 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:47:59.264 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:47:59.264 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:47:59.265 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:47:59.265 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:47:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:47:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:47:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:47:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:47:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:47:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:47:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:47:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:47:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:47:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:47:59.271 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:47:59.271 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:47:59.272 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:47:59.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:47:59.273 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:47:59.273 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:47:59.274 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:47:59.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:47:59.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:47:59.276 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:47:59.276 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:47:59.277 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:47:59.277 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:47:59.278 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:47:59.278 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:47:59.279 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:47:59.279 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:47:59.280 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:47:59.280 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:47:59.280 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:47:59.281 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:47:59.281 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:47:59.282 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:47:59.283 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:47:59.283 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:47:59.284 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:47:59.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:47:59.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:47:59.285 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:47:59.285 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:47:59.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:47:59.286 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:47:59.287 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:47:59.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:47:59.288 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.288 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:47:59.289 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:47:59.289 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:47:59.290 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:47:59.290 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:47:59.291 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:47:59.291 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:47:59.291 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:47:59.292 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:47:59.292 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:47:59.293 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:47:59.293 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:47:59.294 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:47:59.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.295 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:47:59.295 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:47:59.296 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.296 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.297 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.297 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:47:59.298 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:47:59.298 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:47:59.299 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:47:59.299 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:47:59.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:47:59.300 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:47:59.301 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:47:59.301 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:47:59.302 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:47:59.302 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:47:59.303 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:47:59.303 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:47:59.304 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:47:59.304 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:47:59.305 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:47:59.305 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:47:59.306 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:47:59.306 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:47:59.307 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:47:59.307 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:47:59.308 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:47:59.308 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:47:59.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.309 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:47:59.309 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:47:59.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:47:59.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:47:59.311 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:47:59.311 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:47:59.312 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:47:59.312 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:47:59.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:47:59.313 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:47:59.314 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:47:59.314 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:47:59.315 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:47:59.315 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:47:59.316 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:47:59.316 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:47:59.317 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:47:59.317 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:47:59.318 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:47:59.318 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:47:59.319 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:47:59.319 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:47:59.320 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:47:59.320 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:47:59.320 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:47:59.321 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:47:59.321 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:47:59.322 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.322 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:47:59.323 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:47:59.323 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:47:59.324 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:47:59.324 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:47:59.325 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:47:59.325 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:47:59.326 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:47:59.326 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:47:59.327 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:47:59.327 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:47:59.328 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:47:59.328 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:47:59.328 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:47:59.329 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.329 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.330 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.330 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 13, 2022-07-12T08:47:59.330 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:47:59.331 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:47:01.995190+0000", 2022-07-12T08:47:59.331 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:47:59.332 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:47:59.332 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:47:59.333 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:47:59.333 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:47:59.333 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:47:59.334 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:47:59.334 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:47:59.335 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:47:59.335 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:47:59.336 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:47:59.336 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:47:59.337 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:47:59.337 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:47:59.338 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:47:59.340 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:47:59.340 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:47:59.340 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:47:59.341 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:47:59.341 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:47:59.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.342 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "70", 2022-07-12T08:47:59.342 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:47:59.343 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:47:59.343 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:47:59.343 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:47:59.344 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:47:59.345 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:47:59.345 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:47:59.346 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:47:59.346 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:47:59.347 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:47:59.347 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:47:59.348 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:47:59.348 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:47:59.348 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:47:59.349 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:47:59.349 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:47:59.350 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:47:59.350 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:47:59.351 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:47:59.351 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:47:59.351 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:47:59.352 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:47:59.352 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:47:59.353 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:47:59.353 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:47:59.354 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:47:59.354 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.355 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:47:59.355 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:47:59.356 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:47:59.356 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:47:59.357 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:47:59.357 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:47:59.358 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:47:59.358 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:47:59.358 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:47:59.359 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:47:59.359 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:47:59.360 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:47:59.360 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:47:59.360 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:47:59.361 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:47:59.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.361 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:47:59.362 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:47:59.362 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:47:59.363 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.363 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.364 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.364 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 14, 2022-07-12T08:47:59.365 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:47:59.365 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:47:02.575389+0000", 2022-07-12T08:47:59.365 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:47:59.366 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:47:59.366 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:47:59.367 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:47:59.367 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:47:59.367 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:47:59.368 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:47:59.368 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:47:59.368 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:47:59.369 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:47:59.369 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:47:59.370 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:47:59.370 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:47:59.370 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:47:59.371 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:47:59.371 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:47:59.371 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:47:59.371 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:47:59.372 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:47:59.372 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:47:59.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.372 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "71", 2022-07-12T08:47:59.373 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:47:59.373 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:47:59.373 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:47:59.373 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:47:59.374 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:47:59.374 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:47:59.374 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:47:59.374 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:47:59.375 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:47:59.375 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:47:59.375 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:47:59.375 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:47:59.375 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:47:59.376 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:47:59.376 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:47:59.376 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:47:59.377 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:47:59.377 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:47:59.378 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:47:59.378 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:47:59.378 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:47:59.379 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:47:59.379 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:47:59.380 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:47:59.380 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:47:59.381 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:47:59.381 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.381 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:47:59.382 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:47:59.382 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:47:59.383 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:47:59.383 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:47:59.384 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:47:59.384 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:47:59.384 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:47:59.385 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:47:59.385 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:47:59.386 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:47:59.386 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:47:59.386 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:47:59.387 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:47:59.387 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:47:59.388 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.388 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.388 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.389 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:47:59.389 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:47:59.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.389 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:47:59.390 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:47:59.391 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.391 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.391 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.391 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.392 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.392 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.392 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.392 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:47:59.393 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.393 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.393 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.394 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.394 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:47:59.394 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.395 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.395 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.395 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:47:59.395 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.395 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.396 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.396 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.396 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.396 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.397 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:47:59.397 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.397 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.398 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.398 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:47:59.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.399 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.399 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.399 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.400 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:47:59.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.401 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.401 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.402 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:47:59.402 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.402 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.402 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.403 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.403 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.403 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.403 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:47:59.404 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.404 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.404 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.404 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:47:59.405 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:47:59.405 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.405 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.405 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.406 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:47:59.406 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:47:59.406 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:47:59.406 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:47:59.407 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.407 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.407 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.407 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.408 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.408 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:47:59.408 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:47:59.409 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.409 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.409 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.409 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.409 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.410 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.410 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.410 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:47:59.410 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.411 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.411 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.411 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.411 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:47:59.412 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.412 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.413 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:47:59.413 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.413 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.413 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.414 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.414 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.414 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.415 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:47:59.415 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.415 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.416 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.416 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:47:59.416 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.416 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.416 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.417 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.417 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.418 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:47:59.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.418 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.419 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:47:59.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.419 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.419 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.420 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.420 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.420 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.420 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.421 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.421 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:47:59.421 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.421 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.422 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.422 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:47:59.422 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:47:59.422 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.423 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.423 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.423 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:47:59.423 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:47:59.423 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:47:59.424 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:47:59.424 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.424 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.424 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.425 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.425 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.425 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:47:59.426 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:47:59.426 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.426 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.426 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.427 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.427 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.427 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.427 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.428 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:47:59.428 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.428 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.428 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.429 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.429 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.429 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:47:59.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.430 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.430 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:47:59.431 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.431 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.431 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.431 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.432 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.432 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.433 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:47:59.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.434 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:47:59.434 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.434 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.435 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.435 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.435 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.435 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.436 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:47:59.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.437 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:47:59.437 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.438 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.438 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.438 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.439 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.439 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.439 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.439 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.440 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:47:59.440 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.440 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.440 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.440 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.441 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:47:59.441 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:47:59.441 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.441 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.442 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.442 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:47:59.442 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:47:59.442 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:47:59.442 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:47:59.443 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.443 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.443 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.443 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.444 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.444 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.444 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:47:59.444 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:47:59.444 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.445 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.445 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.445 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.445 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.446 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.446 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.446 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:47:59.446 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.447 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.447 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.447 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.448 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.448 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:47:59.448 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.448 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.448 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.449 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.449 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:47:59.449 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.449 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.450 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.450 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.450 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.450 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.451 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.451 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.452 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:47:59.452 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.453 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.453 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:47:59.454 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.454 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.455 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.455 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.455 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.456 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.457 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:47:59.457 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.458 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.458 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.459 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.459 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:47:59.460 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.460 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.460 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.461 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.462 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.462 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.462 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.463 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:47:59.463 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.464 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.464 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.465 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.465 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:47:59.466 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:47:59.466 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.466 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.467 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.467 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:47:59.467 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:47:59.468 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:47:59.468 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:47:59.468 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.468 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.468 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.469 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.469 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.469 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.469 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:47:59.469 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:47:59.470 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.470 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.471 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.471 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.471 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.472 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.472 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.473 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-12T08:47:59.473 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.474 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.474 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.474 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.475 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.475 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.476 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:47:59.476 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.476 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.477 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.477 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.478 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:47:59.478 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.479 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.479 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.480 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.480 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.480 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.480 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.481 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.481 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:47:59.481 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.481 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.482 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.482 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.482 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:47:59.483 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.483 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.483 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.483 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.484 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.484 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.484 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.484 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:47:59.485 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.486 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.486 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.486 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.487 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:47:59.487 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.487 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.488 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.488 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.488 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.489 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.490 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.490 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:47:59.490 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.490 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.491 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.491 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.491 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:47:59.492 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:47:59.492 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.492 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.492 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.493 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:47:59.493 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:47:59.493 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:47:59.493 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:47:59.493 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.494 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.494 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.494 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.494 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.495 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.495 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:47:59.495 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:47:59.495 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.496 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.496 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.496 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.496 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.497 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.497 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.497 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:47:59.497 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.502 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.503 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.503 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.504 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.504 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.505 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:47:59.505 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.506 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.507 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.507 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:47:59.508 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.509 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.509 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.509 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.510 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.510 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.511 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.511 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.512 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:47:59.512 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.513 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.514 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.514 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.515 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:47:59.515 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.516 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.516 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.516 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.517 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.517 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.519 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.519 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.520 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:47:59.520 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.521 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.521 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.522 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.523 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:47:59.523 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.524 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.524 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.525 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.525 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.525 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.526 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.526 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.527 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:47:59.527 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.528 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.528 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.529 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.529 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:47:59.530 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:47:59.531 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.531 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.532 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.532 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:47:59.532 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:47:59.533 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:47:59.533 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:47:59.534 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.534 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.535 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.535 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.536 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.536 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.536 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:47:59.537 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:47:59.537 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.538 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.538 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.539 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.539 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.540 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.542 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.542 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:47:59.543 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.543 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.543 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.544 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.544 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.545 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.545 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:47:59.545 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.546 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.546 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.547 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.547 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:47:59.549 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.551 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.552 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.553 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.553 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.554 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.554 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:47:59.555 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.555 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.556 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.556 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.556 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:47:59.557 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.557 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.565 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.568 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.569 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.569 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.570 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.570 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:47:59.570 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.571 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.572 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.572 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.572 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:47:59.573 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.573 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.574 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.574 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.574 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.575 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.575 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.575 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.576 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:47:59.576 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.576 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.576 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.576 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.577 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:47:59.577 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:47:59.577 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.577 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.578 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.578 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:47:59.578 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:47:59.579 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:47:59.579 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:47:59.579 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.579 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.580 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.580 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.580 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.580 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.581 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:47:59.581 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:47:59.581 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:47:59.582 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:47:59.582 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:47:59.583 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:47:59.583 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:47:59.583 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:47:59.584 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:47:59.584 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-12T08:47:59.584 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:47:59.584 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:47:59.585 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:47:59.585 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.585 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.586 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.586 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:47:59.586 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.587 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.587 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.588 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.588 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:47:59.588 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.589 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.589 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.590 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.590 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:47:59.591 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.592 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.593 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.593 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:47:59.594 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.594 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.594 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.594 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.595 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:47:59.595 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.595 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.595 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:47:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.597 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.597 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:47:59.597 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.597 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.597 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.598 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.598 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:47:59.598 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.598 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.599 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.599 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.599 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:47:59.599 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:47:59.599 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.600 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:47:59.600 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:47:59.600 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.601 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.601 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.601 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:47:59.601 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:47:59.602 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:47:59.602 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.602 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.603 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.603 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:47:59.603 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:47:59.603 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:47:59.604 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:47:59.604 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:47:59.604 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:47:59.604 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:47:59.604 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:47:59.605 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.605 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:47:59.605 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:47:59.605 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.606 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:47:59.606 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.606 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.606 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.606 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.607 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.607 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:47:59.607 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.608 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.608 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.608 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:47:59.609 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.609 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.609 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.610 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.610 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.610 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:47:59.610 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.611 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.611 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.611 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:47:59.612 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.612 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.612 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.612 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.613 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.613 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:47:59.613 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.613 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.615 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.615 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:47:59.615 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.615 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.615 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.616 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.616 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.616 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:47:59.616 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.617 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.617 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.617 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:47:59.617 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.618 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.618 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.618 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.618 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.619 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:47:59.619 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.619 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.620 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.620 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:47:59.620 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.620 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.620 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.621 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.621 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.621 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:47:59.621 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.622 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.622 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.622 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:47:59.622 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.622 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.623 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.623 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.623 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.624 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:47:59.625 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.625 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.625 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:47:59.625 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:47:59.626 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:47:59.626 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:47:59.626 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:47:59.626 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:47:59.627 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:47:59.627 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:47:59.627 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:47:59.627 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.627 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:47:59.628 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:47:59.628 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:47:59.628 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:47:59.628 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:47:59.629 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:47:59.629 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.121:0/3889265026": "2022-07-12T09:47:49.061933+0000" 2022-07-12T08:47:59.629 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.629 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:47:59.630 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:47:59.630 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:47:59.630 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:47:59.630 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:47:59.631 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:47:59.631 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:47:59.631 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:47:59.631 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:47:59.632 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:47:59.632 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:47:59.632 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:47:59.632 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:47:59.632 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:47:59.633 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:47:59.633 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:47:59.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b5c0539d0 msgr2=0x7f1b5c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b5c0539d0 0x7f1b5c055e90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f1b6000b000 tx=0x7f1b60009040).stop 2022-07-12T08:47:59.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 msgr2=0x7f1b78064a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 0x7f1b78064a70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f1b78062f60 tx=0x7f1b68007c50).stop 2022-07-12T08:47:59.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 shutdown_connections 2022-07-12T08:47:59.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b5c0539d0 0x7f1b5c055e90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b7813f670 0x7f1b780b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b780b27c0 0x7f1b780659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 --2- 172.21.15.61:0/3765328820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b78060250 0x7f1b78064a70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 >> 172.21.15.61:0/3765328820 conn(0x7f1b780633a0 msgr2=0x7f1b780a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:59.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 shutdown_connections 2022-07-12T08:47:59.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.248+0000 7f1b7d880700 1 -- 172.21.15.61:0/3765328820 wait complete. 2022-07-12T08:47:59.639 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:47:59.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.748+0000 7fb368dfb700 1 Processor -- start 2022-07-12T08:47:59.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.748+0000 7fb368dfb700 1 -- start start 2022-07-12T08:47:59.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.748+0000 7fb368dfb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb36409bce0 0x7fb3640af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.748+0000 7fb368dfb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3640aecb0 con 0x7fb3640b3f60 2022-07-12T08:47:59.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.748+0000 7fb368dfb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3641435c0 con 0x7fb3640af280 2022-07-12T08:47:59.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.748+0000 7fb368dfb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb364143b00 con 0x7fb36409bce0 2022-07-12T08:47:59.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb363fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb3640b3f60 0x7fb3640b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45824/0 (socket says 172.21.15.61:45824) 2022-07-12T08:47:59.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb363fff700 1 -- 172.21.15.61:0/50998717 learned_addr learned my addr 172.21.15.61:0/50998717 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:47:59.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb362ffd700 1 --2- 172.21.15.61:0/50998717 >> v2:172.21.15.111:3300/0 conn(0x7fb36409bce0 0x7fb3640af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb3627fc700 1 -- 172.21.15.61:0/50998717 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307475576 0 0) 0x7fb3640aecb0 con 0x7fb3640b3f60 2022-07-12T08:47:59.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb3627fc700 1 -- 172.21.15.61:0/50998717 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb350003260 con 0x7fb3640b3f60 2022-07-12T08:47:59.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb362ffd700 1 -- 172.21.15.61:0/50998717 >> v1:172.21.15.111:6789/0 conn(0x7fb3640af280 legacy=0x7fb364060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb362ffd700 1 -- 172.21.15.61:0/50998717 >> v1:172.21.15.61:6789/0 conn(0x7fb3640b3f60 legacy=0x7fb3640b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb362ffd700 1 -- 172.21.15.61:0/50998717 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364143860 con 0x7fb36409bce0 2022-07-12T08:47:59.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb362ffd700 1 --2- 172.21.15.61:0/50998717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb3640af660 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fb354002f70 tx=0x7fb354006df0).ready entity=mon.1 client_cookie=80871d7acb26844a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:59.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb3627fc700 1 -- 172.21.15.61:0/50998717 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35400d610 con 0x7fb36409bce0 2022-07-12T08:47:59.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb3627fc700 1 -- 172.21.15.61:0/50998717 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb354000e90 con 0x7fb36409bce0 2022-07-12T08:47:59.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb3627fc700 1 -- 172.21.15.61:0/50998717 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35400d960 con 0x7fb36409bce0 2022-07-12T08:47:59.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 -- 172.21.15.61:0/50998717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 msgr2=0x7fb3640af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 --2- 172.21.15.61:0/50998717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb3640af660 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fb354002f70 tx=0x7fb354006df0).stop 2022-07-12T08:47:59.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 -- 172.21.15.61:0/50998717 shutdown_connections 2022-07-12T08:47:59.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 --2- 172.21.15.61:0/50998717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb3640af660 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 -- 172.21.15.61:0/50998717 >> 172.21.15.61:0/50998717 conn(0x7fb3640b92c0 msgr2=0x7fb3640b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:59.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 -- 172.21.15.61:0/50998717 shutdown_connections 2022-07-12T08:47:59.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.752+0000 7fb368dfb700 1 -- 172.21.15.61:0/50998717 wait complete. 2022-07-12T08:47:59.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 Processor -- start 2022-07-12T08:47:59.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 -- start start 2022-07-12T08:47:59.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb364155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3640af280 0x7fb364141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3640b3f60 0x7fb3640b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364142d00 con 0x7fb3640b3f60 2022-07-12T08:47:59.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36409dca0 con 0x7fb36409bce0 2022-07-12T08:47:59.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36409ded0 con 0x7fb3640af280 2022-07-12T08:47:59.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb363fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3640b3f60 0x7fb3640b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb363fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3640b3f60 0x7fb3640b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57612/0 (socket says 172.21.15.61:57612) 2022-07-12T08:47:59.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb363fff700 1 -- 172.21.15.61:0/1659467762 learned_addr learned my addr 172.21.15.61:0/1659467762 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:47:59.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3640af280 0x7fb364141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb3637fe700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb364155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 -- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 msgr2=0x7fb364155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb364155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 -- 172.21.15.61:0/1659467762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3640b3f60 msgr2=0x7fb3640b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3640b3f60 0x7fb3640b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 -- 172.21.15.61:0/1659467762 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35400b180 con 0x7fb3640af280 2022-07-12T08:47:59.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb362ffd700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3640af280 0x7fb364141f50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb3640b1b30 tx=0x7fb354014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:59.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb354013d20 con 0x7fb3640af280 2022-07-12T08:47:59.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35400d290 con 0x7fb3640af280 2022-07-12T08:47:59.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb35401e5f0 con 0x7fb3640af280 2022-07-12T08:47:59.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36409e170 con 0x7fb3640af280 2022-07-12T08:47:59.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.756+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3640b3640 con 0x7fb3640af280 2022-07-12T08:47:59.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.760+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36409f120 con 0x7fb3640af280 2022-07-12T08:47:59.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.760+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb35401c070 con 0x7fb3640af280 2022-07-12T08:47:59.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.760+0000 7fb360ff9700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3440539b0 0x7fb344055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:47:59.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.760+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fb35406c8e0 con 0x7fb3640af280 2022-07-12T08:47:59.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.760+0000 7fb3637fe700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3440539b0 0x7fb344055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:47:59.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.760+0000 7fb3637fe700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3440539b0 0x7fb344055e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb35800d000 tx=0x7fb358007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:47:59.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.768+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb36409f120 con 0x7fb3640af280 2022-07-12T08:47:59.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.936+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb36409f120 con 0x7fb3640af280 2022-07-12T08:47:59.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.936+0000 7fb360ff9700 1 -- 172.21.15.61:0/1659467762 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fb36409f120 con 0x7fb3640af280 2022-07-12T08:47:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:47:59.940 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":38,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:47:05.156395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6196":{"gid":6196,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3265139715","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3265139715},{"type":"v1","addr":"172.21.15.61:6833","nonce":3265139715}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:47:59.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3440539b0 msgr2=0x7fb344055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3440539b0 0x7fb344055e70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb35800d000 tx=0x7fb358007b80).stop 2022-07-12T08:47:59.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3640af280 msgr2=0x7fb364141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:47:59.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3640af280 0x7fb364141f50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb3640b1b30 tx=0x7fb354014000).stop 2022-07-12T08:47:59.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 shutdown_connections 2022-07-12T08:47:59.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3640b3f60 0x7fb3640b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3640af280 0x7fb364141f50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3440539b0 0x7fb344055e70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 --2- 172.21.15.61:0/1659467762 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36409bce0 0x7fb364155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:47:59.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 >> 172.21.15.61:0/1659467762 conn(0x7fb3640b92c0 msgr2=0x7fb3640602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:47:59.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 shutdown_connections 2022-07-12T08:47:59.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:47:59.940+0000 7fb368dfb700 1 -- 172.21.15.61:0/1659467762 wait complete. 2022-07-12T08:47:59.950 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 38 2022-07-12T08:47:59.953 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:48:00.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 Processor -- start 2022-07-12T08:48:00.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 -- start start 2022-07-12T08:48:00.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300b4250 0x7f4e30066440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:00.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4e300ad1c0 0x7f4e30140050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:00.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e300b4630 con 0x7f4e300b2480 2022-07-12T08:48:00.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e30066930 con 0x7f4e300ad1c0 2022-07-12T08:48:00.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e3806a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e300acce0 con 0x7f4e300b4250 2022-07-12T08:48:00.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e35605700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f4e300b2480 0x7f4e30144350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59436/0 (socket says 172.21.15.61:59436) 2022-07-12T08:48:00.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e35605700 1 -- 172.21.15.61:0/4268951851 learned_addr learned my addr 172.21.15.61:0/4268951851 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:00.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e36607700 1 --2- 172.21.15.61:0/4268951851 >> v2:172.21.15.111:3300/0 conn(0x7f4e300ad1c0 0x7f4e30140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:00.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.060+0000 7f4e35e06700 1 --2- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300b4250 0x7f4e30066440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:00.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e34e04700 1 -- 172.21.15.61:0/4268951851 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655081250 0 0) 0x7f4e300b4630 con 0x7f4e300b2480 2022-07-12T08:48:00.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e34e04700 1 -- 172.21.15.61:0/4268951851 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c003260 con 0x7f4e300b2480 2022-07-12T08:48:00.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e36607700 1 -- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300b4250 msgr2=0x7f4e30066440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:00.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e36607700 1 --2- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300b4250 0x7f4e30066440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:00.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e36607700 1 -- 172.21.15.61:0/4268951851 >> v1:172.21.15.111:6789/0 conn(0x7f4e300b2480 legacy=0x7f4e30144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:00.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e36607700 1 -- 172.21.15.61:0/4268951851 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e30065fa0 con 0x7f4e300ad1c0 2022-07-12T08:48:00.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e36607700 1 --2- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300ad1c0 0x7f4e30140050 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f4e20002f70 tx=0x7f4e20006df0).ready entity=mon.1 client_cookie=ccd2b068852fda4b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:00.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e34e04700 1 -- 172.21.15.61:0/4268951851 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2000d4a0 con 0x7f4e300ad1c0 2022-07-12T08:48:00.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e34e04700 1 -- 172.21.15.61:0/4268951851 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e20000e90 con 0x7f4e300ad1c0 2022-07-12T08:48:00.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e34e04700 1 -- 172.21.15.61:0/4268951851 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2000d7f0 con 0x7f4e300ad1c0 2022-07-12T08:48:00.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 -- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300ad1c0 msgr2=0x7f4e30140050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 --2- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300ad1c0 0x7f4e30140050 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f4e20002f70 tx=0x7f4e20006df0).stop 2022-07-12T08:48:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 -- 172.21.15.61:0/4268951851 shutdown_connections 2022-07-12T08:48:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 --2- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300ad1c0 0x7f4e30140050 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 --2- 172.21.15.61:0/4268951851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300b4250 0x7f4e30066440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 -- 172.21.15.61:0/4268951851 >> 172.21.15.61:0/4268951851 conn(0x7f4e30064e10 msgr2=0x7f4e30065220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:00.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 -- 172.21.15.61:0/4268951851 shutdown_connections 2022-07-12T08:48:00.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.064+0000 7f4e3806a700 1 -- 172.21.15.61:0/4268951851 wait complete. 2022-07-12T08:48:00.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 Processor -- start 2022-07-12T08:48:00.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 -- start start 2022-07-12T08:48:00.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300ad1c0 0x7f4e30142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:00.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 0x7f4e300b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:00.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300b4250 0x7f4e30145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:00.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3013bf10 con 0x7f4e300b2480 2022-07-12T08:48:00.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30060f70 con 0x7f4e300b4250 2022-07-12T08:48:00.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e3806a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30153190 con 0x7f4e300ad1c0 2022-07-12T08:48:00.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e35605700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 0x7f4e300b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:00.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e35605700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 0x7f4e300b1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57624/0 (socket says 172.21.15.61:57624) 2022-07-12T08:48:00.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e35605700 1 -- 172.21.15.61:0/1231507975 learned_addr learned my addr 172.21.15.61:0/1231507975 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:00.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e35e06700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300ad1c0 0x7f4e30142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:00.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.068+0000 7f4e36607700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300b4250 0x7f4e30145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:00.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35605700 1 -- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300ad1c0 msgr2=0x7f4e30142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:00.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35605700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300ad1c0 0x7f4e30142330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:00.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35605700 1 -- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300b4250 msgr2=0x7f4e30145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:00.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35605700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300b4250 0x7f4e30145850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:00.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35605700 1 -- 172.21.15.61:0/1231507975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2000b180 con 0x7f4e300b2480 2022-07-12T08:48:00.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35e06700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300ad1c0 0x7f4e30142330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:00.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35605700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 0x7f4e300b1220 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f4e2c001490 tx=0x7f4e2c007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:00.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2c005ba0 con 0x7f4e300b2480 2022-07-12T08:48:00.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c005d00 con 0x7f4e300b2480 2022-07-12T08:48:00.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3005f300 con 0x7f4e300b2480 2022-07-12T08:48:00.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e2c0072c0 con 0x7f4e300b2480 2022-07-12T08:48:00.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e300af570 con 0x7f4e300b2480 2022-07-12T08:48:00.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4e2c010070 con 0x7f4e300b2480 2022-07-12T08:48:00.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e30061130 con 0x7f4e300b2480 2022-07-12T08:48:00.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e26ffd700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4e10053a40 0x7f4e10055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:00.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f4e2c06ba40 con 0x7f4e300b2480 2022-07-12T08:48:00.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.072+0000 7f4e35e06700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4e10053a40 0x7f4e10055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:00.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.076+0000 7f4e35e06700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4e10053a40 0x7f4e10055f00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f4e2800b000 tx=0x7f4e28006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:00.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.080+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e2c0427e0 con 0x7f4e300b2480 2022-07-12T08:48:00.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:00.248+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4e300af880 con 0x7f4e300b2480 2022-07-12T08:48:01.178 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:48:01.176+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:48:01.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.188+0000 7f4e26ffd700 1 -- 172.21.15.61:0/1231507975 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v39) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4e2c045e00 con 0x7f4e300b2480 2022-07-12T08:48:01.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4e10053a40 msgr2=0x7f4e10055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4e10053a40 0x7f4e10055f00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f4e2800b000 tx=0x7f4e28006df0).stop 2022-07-12T08:48:01.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 msgr2=0x7f4e300b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 0x7f4e300b1220 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 rx=0x7f4e2c001490 tx=0x7f4e2c007ed0).stop 2022-07-12T08:48:01.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 shutdown_connections 2022-07-12T08:48:01.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4e10053a40 0x7f4e10055f00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e300b4250 0x7f4e30145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e300b2480 0x7f4e300b1220 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 --2- 172.21.15.61:0/1231507975 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4e300ad1c0 0x7f4e30142330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 >> 172.21.15.61:0/1231507975 conn(0x7f4e30064e10 msgr2=0x7f4e301467d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:01.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 shutdown_connections 2022-07-12T08:48:01.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.192+0000 7f4e3806a700 1 -- 172.21.15.61:0/1231507975 wait complete. 2022-07-12T08:48:01.200 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:48:01.210 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:01.234 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:48:01.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.312+0000 7f9b1accc700 1 Processor -- start 2022-07-12T08:48:01.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.312+0000 7f9b1accc700 1 -- start start 2022-07-12T08:48:01.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9b140b6780 0x7f9b140b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9b141436b0 0x7f9b14065a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b14140010 con 0x7f9b140b25c0 2022-07-12T08:48:01.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b140668f0 con 0x7f9b141436b0 2022-07-12T08:48:01.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b14142f70 con 0x7f9b140b6780 2022-07-12T08:48:01.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1a4cb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9b141436b0 0x7f9b14065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b194c9700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9b140b25c0 0x7f9b1413f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45862/0 (socket says 172.21.15.61:45862) 2022-07-12T08:48:01.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b194c9700 1 -- 172.21.15.61:0/46975689 learned_addr learned my addr 172.21.15.61:0/46975689 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:01.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b19cca700 1 --2- 172.21.15.61:0/46975689 >> v2:172.21.15.111:3300/0 conn(0x7f9b140b6780 0x7f9b140b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b18cc8700 1 -- 172.21.15.61:0/46975689 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144074654 0 0) 0x7f9b14140010 con 0x7f9b140b25c0 2022-07-12T08:48:01.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b18cc8700 1 -- 172.21.15.61:0/46975689 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9afc003260 con 0x7f9b140b25c0 2022-07-12T08:48:01.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1a4cb700 1 -- 172.21.15.61:0/46975689 >> v2:172.21.15.111:3300/0 conn(0x7f9b140b6780 msgr2=0x7f9b140b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:01.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1a4cb700 1 --2- 172.21.15.61:0/46975689 >> v2:172.21.15.111:3300/0 conn(0x7f9b140b6780 0x7f9b140b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1a4cb700 1 -- 172.21.15.61:0/46975689 >> v1:172.21.15.61:6789/0 conn(0x7f9b140b25c0 legacy=0x7f9b1413f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1a4cb700 1 -- 172.21.15.61:0/46975689 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b14143210 con 0x7f9b141436b0 2022-07-12T08:48:01.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1a4cb700 1 --2- 172.21.15.61:0/46975689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b141436b0 0x7f9b14065a80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f9b14143590 tx=0x7f9b0c0091d0).ready entity=mon.0 client_cookie=f3d519ca07cf9ea3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:01.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b18cc8700 1 -- 172.21.15.61:0/46975689 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b0c00b5a0 con 0x7f9b141436b0 2022-07-12T08:48:01.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b18cc8700 1 -- 172.21.15.61:0/46975689 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c00b700 con 0x7f9b141436b0 2022-07-12T08:48:01.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b18cc8700 1 -- 172.21.15.61:0/46975689 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b0c00b870 con 0x7f9b141436b0 2022-07-12T08:48:01.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- 172.21.15.61:0/46975689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b141436b0 msgr2=0x7f9b14065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 --2- 172.21.15.61:0/46975689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b141436b0 0x7f9b14065a80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f9b14143590 tx=0x7f9b0c0091d0).stop 2022-07-12T08:48:01.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- 172.21.15.61:0/46975689 shutdown_connections 2022-07-12T08:48:01.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 --2- 172.21.15.61:0/46975689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b141436b0 0x7f9b14065a80 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 --2- 172.21.15.61:0/46975689 >> v2:172.21.15.111:3300/0 conn(0x7f9b140b6780 0x7f9b140b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- 172.21.15.61:0/46975689 >> 172.21.15.61:0/46975689 conn(0x7f9b14064570 msgr2=0x7f9b140b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:01.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- 172.21.15.61:0/46975689 shutdown_connections 2022-07-12T08:48:01.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.316+0000 7f9b1accc700 1 -- 172.21.15.61:0/46975689 wait complete. 2022-07-12T08:48:01.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 Processor -- start 2022-07-12T08:48:01.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 -- start start 2022-07-12T08:48:01.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b140b25c0 0x7f9b140b6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 0x7f9b1413e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9b141436b0 0x7f9b14155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b5a30 con 0x7f9b140b6780 2022-07-12T08:48:01.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b5100 con 0x7f9b140b25c0 2022-07-12T08:48:01.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b140b53a0 con 0x7f9b141436b0 2022-07-12T08:48:01.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 0x7f9b1413e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 0x7f9b1413e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57650/0 (socket says 172.21.15.61:57650) 2022-07-12T08:48:01.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 -- 172.21.15.61:0/556537769 learned_addr learned my addr 172.21.15.61:0/556537769 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:01.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b19cca700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b140b25c0 0x7f9b140b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1a4cb700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9b141436b0 0x7f9b14155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 -- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9b141436b0 msgr2=0x7f9b14155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9b141436b0 0x7f9b14155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 -- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b140b25c0 msgr2=0x7f9b140b6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b140b25c0 0x7f9b140b6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 -- 172.21.15.61:0/556537769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c009470 con 0x7f9b140b6780 2022-07-12T08:48:01.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b194c9700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 0x7f9b1413e620 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f9b04001110 tx=0x7f9b04007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:01.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b040070d0 con 0x7f9b140b6780 2022-07-12T08:48:01.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b04007230 con 0x7f9b140b6780 2022-07-12T08:48:01.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b04005610 con 0x7f9b140b6780 2022-07-12T08:48:01.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1413eb10 con 0x7f9b140b6780 2022-07-12T08:48:01.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.320+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b140b3ea0 con 0x7f9b140b6780 2022-07-12T08:48:01.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.324+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b14057460 con 0x7f9b140b6780 2022-07-12T08:48:01.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.324+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9b0401a070 con 0x7f9b140b6780 2022-07-12T08:48:01.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.324+0000 7f9b02ffd700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9af4053920 0x7f9af4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.324+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f9b0406b280 con 0x7f9b140b6780 2022-07-12T08:48:01.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.324+0000 7f9b19cca700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9af4053920 0x7f9af4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.324+0000 7f9b19cca700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9af4053920 0x7f9af4055de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f9b1000b000 tx=0x7f9b10006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:01.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.332+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9b04042050 con 0x7f9b140b6780 2022-07-12T08:48:01.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.496+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b140b6e30 con 0x7f9b140b6780 2022-07-12T08:48:01.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.496+0000 7f9b02ffd700 1 -- 172.21.15.61:0/556537769 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f9b04045670 con 0x7f9b140b6780 2022-07-12T08:48:01.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:01.500 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":39,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:47:03.397499+0000","modified":"2022-07-12T08:48:01.177774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-12T08:48:01.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9af4053920 msgr2=0x7f9af4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9af4053920 0x7f9af4055de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f9b1000b000 tx=0x7f9b10006df0).stop 2022-07-12T08:48:01.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 msgr2=0x7f9b1413e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 0x7f9b1413e620 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f9b04001110 tx=0x7f9b04007b20).stop 2022-07-12T08:48:01.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 shutdown_connections 2022-07-12T08:48:01.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9b141436b0 0x7f9b14155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9b140b6780 0x7f9b1413e620 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9af4053920 0x7f9af4055de0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 --2- 172.21.15.61:0/556537769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9b140b25c0 0x7f9b140b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 >> 172.21.15.61:0/556537769 conn(0x7f9b14064570 msgr2=0x7f9b140b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:01.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 shutdown_connections 2022-07-12T08:48:01.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.500+0000 7f9b1accc700 1 -- 172.21.15.61:0/556537769 wait complete. 2022-07-12T08:48:01.505 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 39 2022-07-12T08:48:01.519 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:48:01.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf43fff700 1 Processor -- start 2022-07-12T08:48:01.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf43fff700 1 -- start start 2022-07-12T08:48:01.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf43fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf441404c0 0x7fcf440b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf43fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4405d440 con 0x7fcf440af950 2022-07-12T08:48:01.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf43fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf440afed0 con 0x7fcf4405d060 2022-07-12T08:48:01.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf43fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4409b140 con 0x7fcf441404c0 2022-07-12T08:48:01.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf437fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcf440af950 0x7fcf44099280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45878/0 (socket says 172.21.15.61:45878) 2022-07-12T08:48:01.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf437fe700 1 -- 172.21.15.61:0/247021882 learned_addr learned my addr 172.21.15.61:0/247021882 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:01.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.624+0000 7fcf427fc700 1 --2- 172.21.15.61:0/247021882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf441404c0 0x7fcf440b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074875106 0 0) 0x7fcf440afed0 con 0x7fcf4405d060 2022-07-12T08:48:01.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf24003260 con 0x7fcf4405d060 2022-07-12T08:48:01.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787615411 0 0) 0x7fcf4405d440 con 0x7fcf440af950 2022-07-12T08:48:01.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf440afed0 con 0x7fcf440af950 2022-07-12T08:48:01.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf427fc700 1 -- 172.21.15.61:0/247021882 >> v1:172.21.15.111:6789/0 conn(0x7fcf4405d060 legacy=0x7fcf4413fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf427fc700 1 -- 172.21.15.61:0/247021882 >> v1:172.21.15.61:6789/0 conn(0x7fcf440af950 legacy=0x7fcf44099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf427fc700 1 -- 172.21.15.61:0/247021882 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf440b0170 con 0x7fcf441404c0 2022-07-12T08:48:01.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf427fc700 1 --2- 172.21.15.61:0/247021882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf441404c0 0x7fcf440b10b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fcf2c002f70 tx=0x7fcf2c006df0).ready entity=mon.2 client_cookie=53a4e4eb0ecfc953 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:01.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf2c00d5c0 con 0x7fcf441404c0 2022-07-12T08:48:01.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c000e90 con 0x7fcf441404c0 2022-07-12T08:48:01.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf41ffb700 1 -- 172.21.15.61:0/247021882 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf2c00d910 con 0x7fcf441404c0 2022-07-12T08:48:01.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 -- 172.21.15.61:0/247021882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf441404c0 msgr2=0x7fcf440b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 --2- 172.21.15.61:0/247021882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf441404c0 0x7fcf440b10b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fcf2c002f70 tx=0x7fcf2c006df0).stop 2022-07-12T08:48:01.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 -- 172.21.15.61:0/247021882 shutdown_connections 2022-07-12T08:48:01.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 --2- 172.21.15.61:0/247021882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf441404c0 0x7fcf440b10b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 -- 172.21.15.61:0/247021882 >> 172.21.15.61:0/247021882 conn(0x7fcf440b1a50 msgr2=0x7fcf440b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:01.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 -- 172.21.15.61:0/247021882 shutdown_connections 2022-07-12T08:48:01.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.628+0000 7fcf43fff700 1 -- 172.21.15.61:0/247021882 wait complete. 2022-07-12T08:48:01.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 Processor -- start 2022-07-12T08:48:01.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 -- start start 2022-07-12T08:48:01.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf4405d060 0x7fcf440aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf440af950 0x7fcf441461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 0x7fcf44142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440af4d0 con 0x7fcf440af950 2022-07-12T08:48:01.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440af280 con 0x7fcf441404c0 2022-07-12T08:48:01.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf441508c0 con 0x7fcf4405d060 2022-07-12T08:48:01.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf427fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf440af950 0x7fcf441461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 0x7fcf44142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf427fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf440af950 0x7fcf441461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57664/0 (socket says 172.21.15.61:57664) 2022-07-12T08:48:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf42ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf4405d060 0x7fcf440aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 0x7fcf44142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36240/0 (socket says 172.21.15.61:36240) 2022-07-12T08:48:01.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf427fc700 1 -- 172.21.15.61:0/1964870858 learned_addr learned my addr 172.21.15.61:0/1964870858 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:01.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 -- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf4405d060 msgr2=0x7fcf440aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf4405d060 0x7fcf440aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 -- 172.21.15.61:0/1964870858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf440af950 msgr2=0x7fcf441461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:01.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf440af950 0x7fcf441461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:01.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 -- 172.21.15.61:0/1964870858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c00b180 con 0x7fcf441404c0 2022-07-12T08:48:01.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf437fe700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 0x7fcf44142da0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fcf3c001110 tx=0x7fcf3c007a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:01.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf3c00ce80 con 0x7fcf441404c0 2022-07-12T08:48:01.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c0070d0 con 0x7fcf441404c0 2022-07-12T08:48:01.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf44150b60 con 0x7fcf441404c0 2022-07-12T08:48:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf3c007470 con 0x7fcf441404c0 2022-07-12T08:48:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.632+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf441468b0 con 0x7fcf441404c0 2022-07-12T08:48:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.636+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf44057460 con 0x7fcf441404c0 2022-07-12T08:48:01.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.636+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcf3c010070 con 0x7fcf441404c0 2022-07-12T08:48:01.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.636+0000 7fcf3bfff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf28053a40 0x7fcf28055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:01.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.636+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fcf3c06b4d0 con 0x7fcf441404c0 2022-07-12T08:48:01.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.636+0000 7fcf42ffd700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf28053a40 0x7fcf28055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:01.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.636+0000 7fcf42ffd700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf28053a40 0x7fcf28055f00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcf3400d000 tx=0x7fcf34000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:01.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.644+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf3c0423b0 con 0x7fcf441404c0 2022-07-12T08:48:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:01.808+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcf440625a0 con 0x7fcf441404c0 2022-07-12T08:48:02.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.200+0000 7fcf3bfff700 1 -- 172.21.15.61:0/1964870858 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v40) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcf3c0459d0 con 0x7fcf441404c0 2022-07-12T08:48:02.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf28053a40 msgr2=0x7fcf28055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:02.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf28053a40 0x7fcf28055f00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcf3400d000 tx=0x7fcf34000f40).stop 2022-07-12T08:48:02.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 msgr2=0x7fcf44142da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:02.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 0x7fcf44142da0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fcf3c001110 tx=0x7fcf3c007a40).stop 2022-07-12T08:48:02.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 shutdown_connections 2022-07-12T08:48:02.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf28053a40 0x7fcf28055f00 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf441404c0 0x7fcf44142da0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf440af950 0x7fcf441461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 --2- 172.21.15.61:0/1964870858 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf4405d060 0x7fcf440aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 >> 172.21.15.61:0/1964870858 conn(0x7fcf440b1a50 msgr2=0x7fcf4413f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 shutdown_connections 2022-07-12T08:48:02.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.204+0000 7fcf43fff700 1 -- 172.21.15.61:0/1964870858 wait complete. 2022-07-12T08:48:02.229 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:48:02.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.344+0000 7fd63d412700 1 Processor -- start 2022-07-12T08:48:02.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.344+0000 7fd63d412700 1 -- start start 2022-07-12T08:48:02.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6380b1080 0x7fd6380b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd6380b4090 0x7fd63805c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd638142f30 0x7fd6380afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6380b5740 con 0x7fd638142f30 2022-07-12T08:48:02.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6380b1460 con 0x7fd6380b4090 2022-07-12T08:48:02.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380b6880 con 0x7fd6380b1080 2022-07-12T08:48:02.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd638142f30 0x7fd6380afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd638142f30 0x7fd6380afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57676/0 (socket says 172.21.15.61:57676) 2022-07-12T08:48:02.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 -- 172.21.15.61:0/1694190518 learned_addr learned my addr 172.21.15.61:0/1694190518 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:02.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6367fc700 1 --2- 172.21.15.61:0/1694190518 >> v2:172.21.15.111:3300/0 conn(0x7fd6380b4090 0x7fd63805c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd636ffd700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6380b1080 0x7fd6380b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 -- 172.21.15.61:0/1694190518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6380b1080 msgr2=0x7fd6380b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:02.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6380b1080 0x7fd6380b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 -- 172.21.15.61:0/1694190518 >> v2:172.21.15.111:3300/0 conn(0x7fd6380b4090 msgr2=0x7fd63805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:02.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 --2- 172.21.15.61:0/1694190518 >> v2:172.21.15.111:3300/0 conn(0x7fd6380b4090 0x7fd63805c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 -- 172.21.15.61:0/1694190518 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63805bdb0 con 0x7fd638142f30 2022-07-12T08:48:02.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd636ffd700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6380b1080 0x7fd6380b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:02.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd6377fe700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd638142f30 0x7fd6380afb90 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7fd6300032f0 tx=0x7fd630009290).ready entity=mon.0 client_cookie=36c1cf6c812b3654 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:02.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd635ffb700 1 -- 172.21.15.61:0/1694190518 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd63000b870 con 0x7fd638142f30 2022-07-12T08:48:02.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd635ffb700 1 -- 172.21.15.61:0/1694190518 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63000b9d0 con 0x7fd638142f30 2022-07-12T08:48:02.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd635ffb700 1 -- 172.21.15.61:0/1694190518 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd63000bb40 con 0x7fd638142f30 2022-07-12T08:48:02.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- 172.21.15.61:0/1694190518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd638142f30 msgr2=0x7fd6380afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:02.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd638142f30 0x7fd6380afb90 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 rx=0x7fd6300032f0 tx=0x7fd630009290).stop 2022-07-12T08:48:02.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- 172.21.15.61:0/1694190518 shutdown_connections 2022-07-12T08:48:02.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd638142f30 0x7fd6380afb90 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- 172.21.15.61:0/1694190518 >> v2:172.21.15.111:3300/0 conn(0x7fd6380b4090 0x7fd63805c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 --2- 172.21.15.61:0/1694190518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6380b1080 0x7fd6380b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- 172.21.15.61:0/1694190518 >> 172.21.15.61:0/1694190518 conn(0x7fd638067020 msgr2=0x7fd638067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:02.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- 172.21.15.61:0/1694190518 shutdown_connections 2022-07-12T08:48:02.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.348+0000 7fd63d412700 1 -- 172.21.15.61:0/1694190518 wait complete. 2022-07-12T08:48:02.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 Processor -- start 2022-07-12T08:48:02.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 -- start start 2022-07-12T08:48:02.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 0x7fd638152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380b4090 0x7fd638143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd638142f30 0x7fd638152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638155920 con 0x7fd6380b1080 2022-07-12T08:48:02.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638116e50 con 0x7fd6380b4090 2022-07-12T08:48:02.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd63d412700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381170f0 con 0x7fd638142f30 2022-07-12T08:48:02.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 0x7fd638152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 0x7fd638152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57678/0 (socket says 172.21.15.61:57678) 2022-07-12T08:48:02.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd6377fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd638142f30 0x7fd638152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd6367fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380b4090 0x7fd638143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 -- 172.21.15.61:0/1984109504 learned_addr learned my addr 172.21.15.61:0/1984109504 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:02.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 -- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd638142f30 msgr2=0x7fd638152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:02.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd638142f30 0x7fd638152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 -- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380b4090 msgr2=0x7fd638143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:02.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380b4090 0x7fd638143340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:02.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.352+0000 7fd636ffd700 1 -- 172.21.15.61:0/1984109504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd630009530 con 0x7fd6380b1080 2022-07-12T08:48:02.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd636ffd700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 0x7fd638152310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fd628006940 tx=0x7fd628008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:02.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd628011850 con 0x7fd6380b1080 2022-07-12T08:48:02.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6280119b0 con 0x7fd6380b1080 2022-07-12T08:48:02.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd638112ae0 con 0x7fd6380b1080 2022-07-12T08:48:02.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd628011d10 con 0x7fd6380b1080 2022-07-12T08:48:02.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd638143a50 con 0x7fd6380b1080 2022-07-12T08:48:02.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd638057460 con 0x7fd6380b1080 2022-07-12T08:48:02.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd628006030 con 0x7fd6380b1080 2022-07-12T08:48:02.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd62ffff700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd61c0539b0 0x7fd61c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:02.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fd62806bd60 con 0x7fd6380b1080 2022-07-12T08:48:02.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.356+0000 7fd6367fc700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd61c0539b0 0x7fd61c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:02.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.360+0000 7fd6367fc700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd61c0539b0 0x7fd61c055e70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd62000b000 tx=0x7fd620006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:02.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.364+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd628042be0 con 0x7fd6380b1080 2022-07-12T08:48:02.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:02.488+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd638142d10 con 0x7fd6380b1080 2022-07-12T08:48:03.148 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:03.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:03.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:03.159 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:03.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:03.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.204+0000 7fd62ffff700 1 -- 172.21.15.61:0/1984109504 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd628046200 con 0x7fd6380b1080 2022-07-12T08:48:03.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd61c0539b0 msgr2=0x7fd61c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:03.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd61c0539b0 0x7fd61c055e70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fd62000b000 tx=0x7fd620006df0).stop 2022-07-12T08:48:03.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 msgr2=0x7fd638152310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:03.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 0x7fd638152310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7fd628006940 tx=0x7fd628008040).stop 2022-07-12T08:48:03.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 shutdown_connections 2022-07-12T08:48:03.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd638142f30 0x7fd638152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd61c0539b0 0x7fd61c055e70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6380b4090 0x7fd638143340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 --2- 172.21.15.61:0/1984109504 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6380b1080 0x7fd638152310 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 >> 172.21.15.61:0/1984109504 conn(0x7fd638067020 msgr2=0x7fd6381416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:03.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 shutdown_connections 2022-07-12T08:48:03.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.208+0000 7fd63d412700 1 -- 172.21.15.61:0/1984109504 wait complete. 2022-07-12T08:48:03.227 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:48:03.228 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:48:03.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 Processor -- start 2022-07-12T08:48:03.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 -- start start 2022-07-12T08:48:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 0x7f70e4065ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e4141b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e4066e30 con 0x7f70e40b1f40 2022-07-12T08:48:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e40b2320 con 0x7f70e405c9f0 2022-07-12T08:48:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70ea46a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e4066240 con 0x7f70e40b3d40 2022-07-12T08:48:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70e37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 0x7f70e4065ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70e8a07700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e4141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:03.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70e37fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 0x7f70e4065ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36262/0 (socket says 172.21.15.61:36262) 2022-07-12T08:48:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.344+0000 7f70e37fe700 1 -- 172.21.15.61:0/438835060 learned_addr learned my addr 172.21.15.61:0/438835060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e2ffd700 1 -- 172.21.15.61:0/438835060 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736344641 0 0) 0x7f70e4066e30 con 0x7f70e40b1f40 2022-07-12T08:48:03.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e2ffd700 1 -- 172.21.15.61:0/438835060 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d0003260 con 0x7f70e40b1f40 2022-07-12T08:48:03.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e8a07700 1 -- 172.21.15.61:0/438835060 >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 msgr2=0x7f70e4065ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:03.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e8a07700 1 --2- 172.21.15.61:0/438835060 >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 0x7f70e4065ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e8a07700 1 -- 172.21.15.61:0/438835060 >> v1:172.21.15.111:6789/0 conn(0x7f70e40b1f40 legacy=0x7f70e4066720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:03.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e8a07700 1 -- 172.21.15.61:0/438835060 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e40664e0 con 0x7f70e40b3d40 2022-07-12T08:48:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e37fe700 1 --2- 172.21.15.61:0/438835060 >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 0x7f70e4065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:03.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e8a07700 1 --2- 172.21.15.61:0/438835060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e4141b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f70dc0032f0 tx=0x7f70dc0091d0).ready entity=mon.2 client_cookie=d9c8b221272e1211 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:03.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e2ffd700 1 -- 172.21.15.61:0/438835060 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70dc00b600 con 0x7f70e40b3d40 2022-07-12T08:48:03.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e2ffd700 1 -- 172.21.15.61:0/438835060 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc00b760 con 0x7f70e40b3d40 2022-07-12T08:48:03.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70e2ffd700 1 -- 172.21.15.61:0/438835060 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70dc00b8d0 con 0x7f70e40b3d40 2022-07-12T08:48:03.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 -- 172.21.15.61:0/438835060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 msgr2=0x7f70e4141b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 --2- 172.21.15.61:0/438835060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e4141b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f70dc0032f0 tx=0x7f70dc0091d0).stop 2022-07-12T08:48:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 -- 172.21.15.61:0/438835060 shutdown_connections 2022-07-12T08:48:03.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 --2- 172.21.15.61:0/438835060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e4141b30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 --2- 172.21.15.61:0/438835060 >> v2:172.21.15.111:3300/0 conn(0x7f70e405c9f0 0x7f70e4065ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 -- 172.21.15.61:0/438835060 >> 172.21.15.61:0/438835060 conn(0x7f70e4064b90 msgr2=0x7f70e4064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:03.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 -- 172.21.15.61:0/438835060 shutdown_connections 2022-07-12T08:48:03.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 -- 172.21.15.61:0/438835060 wait complete. 2022-07-12T08:48:03.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 Processor -- start 2022-07-12T08:48:03.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.348+0000 7f70ea46a700 1 -- start start 2022-07-12T08:48:03.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 0x7f70e41520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:03.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e40b1f40 0x7f70e40b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:03.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e413fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:03.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e41525a0 con 0x7f70e405c9f0 2022-07-12T08:48:03.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e40b0170 con 0x7f70e40b1f40 2022-07-12T08:48:03.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e41527c0 con 0x7f70e40b3d40 2022-07-12T08:48:03.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 0x7f70e41520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:03.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e8a07700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e413fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:03.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e40b1f40 0x7f70e40b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:03.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 0x7f70e41520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57692/0 (socket says 172.21.15.61:57692) 2022-07-12T08:48:03.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e3fff700 1 -- 172.21.15.61:0/2365085674 learned_addr learned my addr 172.21.15.61:0/2365085674 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:03.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 -- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 msgr2=0x7f70e413fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:03.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e413fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 -- 172.21.15.61:0/2365085674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 msgr2=0x7f70e41520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:03.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 0x7f70e41520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 -- 172.21.15.61:0/2365085674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d400b1e0 con 0x7f70e40b1f40 2022-07-12T08:48:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e3fff700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 0x7f70e41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e37fe700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e40b1f40 0x7f70e40b0eb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f70e4145fb0 tx=0x7f70d4006eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70d4012600 con 0x7f70e40b1f40 2022-07-12T08:48:03.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4012760 con 0x7f70e40b1f40 2022-07-12T08:48:03.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc009470 con 0x7f70e40b1f40 2022-07-12T08:48:03.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70d4012ac0 con 0x7f70e40b1f40 2022-07-12T08:48:03.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.352+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e40b13a0 con 0x7f70e40b1f40 2022-07-12T08:48:03.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.356+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e4147250 con 0x7f70e40b1f40 2022-07-12T08:48:03.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.356+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f70d4012c20 con 0x7f70e40b1f40 2022-07-12T08:48:03.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.356+0000 7f70e17fa700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f70c40539d0 0x7f70c4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:03.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.356+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5764+0+0 (secure 0 0 0) 0x7f70d406c290 con 0x7f70e40b1f40 2022-07-12T08:48:03.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.356+0000 7f70e3fff700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f70c40539d0 0x7f70c4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:03.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.356+0000 7f70e3fff700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f70c40539d0 0x7f70c4055e90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f70d800d000 tx=0x7f70d8007b20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:03.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.364+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f70e4147250 con 0x7f70e40b1f40 2022-07-12T08:48:03.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:03.496+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f70e4147250 con 0x7f70e40b1f40 2022-07-12T08:48:04.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.216+0000 7f70e17fa700 1 -- 172.21.15.61:0/2365085674 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 173+0+0 (secure 0 0 0) 0x7f70e4147250 con 0x7f70e40b1f40 2022-07-12T08:48:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f70c40539d0 msgr2=0x7f70c4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f70c40539d0 0x7f70c4055e90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f70d800d000 tx=0x7f70d8007b20).stop 2022-07-12T08:48:04.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e40b1f40 msgr2=0x7f70e40b0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e40b1f40 0x7f70e40b0eb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f70e4145fb0 tx=0x7f70d4006eb0).stop 2022-07-12T08:48:04.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 shutdown_connections 2022-07-12T08:48:04.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70e40b3d40 0x7f70e413fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f70c40539d0 0x7f70c4055e90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70e40b1f40 0x7f70e40b0eb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 --2- 172.21.15.61:0/2365085674 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70e405c9f0 0x7f70e41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 >> 172.21.15.61:0/2365085674 conn(0x7f70e4064b90 msgr2=0x7f70e4066670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:04.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 shutdown_connections 2022-07-12T08:48:04.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.220+0000 7f70ea46a700 1 -- 172.21.15.61:0/2365085674 wait complete. 2022-07-12T08:48:04.225 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:48:04.237 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:48:04.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 Processor -- start 2022-07-12T08:48:04.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- start start 2022-07-12T08:48:04.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f691c0b20a0 0x7f691c0b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c09b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c067620 con 0x7f691c066e50 2022-07-12T08:48:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f691c0b3030 con 0x7f691c0b20a0 2022-07-12T08:48:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0b32d0 con 0x7f691c0b14a0 2022-07-12T08:48:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f691c0b20a0 0x7f691c0b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f691c0b20a0 0x7f691c0b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57700/0 (socket says 172.21.15.61:57700) 2022-07-12T08:48:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 -- 172.21.15.61:0/965060595 learned_addr learned my addr 172.21.15.61:0/965060595 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921207700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f691c066e50 0x7f691c146df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45922/0 (socket says 172.21.15.61:45922) 2022-07-12T08:48:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f691bfff700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c09b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 -- 172.21.15.61:0/965060595 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 msgr2=0x7f691c09b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c09b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 -- 172.21.15.61:0/965060595 >> v1:172.21.15.61:6789/0 conn(0x7f691c066e50 legacy=0x7f691c146df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f691b7fe700 1 -- 172.21.15.61:0/965060595 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269148655 0 0) 0x7f691c067620 con 0x7f691c066e50 2022-07-12T08:48:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 -- 172.21.15.61:0/965060595 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c067310 con 0x7f691c0b20a0 2022-07-12T08:48:04.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f691bfff700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c09b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:04.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6920a06700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c0b2480 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6910002f70 tx=0x7f6910006eb0).ready entity=mon.0 client_cookie=114fbff8d04a3c63 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:04.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f691b7fe700 1 -- 172.21.15.61:0/965060595 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691000d5c0 con 0x7f691c0b20a0 2022-07-12T08:48:04.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f691b7fe700 1 -- 172.21.15.61:0/965060595 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6910000e90 con 0x7f691c0b20a0 2022-07-12T08:48:04.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f691b7fe700 1 -- 172.21.15.61:0/965060595 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f691000d910 con 0x7f691c0b20a0 2022-07-12T08:48:04.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- 172.21.15.61:0/965060595 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 msgr2=0x7f691c0b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c0b2480 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6910002f70 tx=0x7f6910006eb0).stop 2022-07-12T08:48:04.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- 172.21.15.61:0/965060595 shutdown_connections 2022-07-12T08:48:04.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c09b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 --2- 172.21.15.61:0/965060595 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c0b2480 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- 172.21.15.61:0/965060595 >> 172.21.15.61:0/965060595 conn(0x7f691c0b5860 msgr2=0x7f691c0b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:04.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- 172.21.15.61:0/965060595 shutdown_connections 2022-07-12T08:48:04.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.352+0000 7f6921a08700 1 -- 172.21.15.61:0/965060595 wait complete. 2022-07-12T08:48:04.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 Processor -- start 2022-07-12T08:48:04.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 -- start start 2022-07-12T08:48:04.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f691c066e50 0x7f691c098b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c0b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0ab700 con 0x7f691c0b20a0 2022-07-12T08:48:04.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c0ab930 con 0x7f691c066e50 2022-07-12T08:48:04.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f691c1595c0 con 0x7f691c0b14a0 2022-07-12T08:48:04.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57710/0 (socket says 172.21.15.61:57710) 2022-07-12T08:48:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 -- 172.21.15.61:0/4294615727 learned_addr learned my addr 172.21.15.61:0/4294615727 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6920a06700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f691c066e50 0x7f691c098b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f691bfff700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c0b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 -- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 msgr2=0x7f691c0b4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c0b4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 -- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f691c066e50 msgr2=0x7f691c098b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f691c066e50 0x7f691c098b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 -- 172.21.15.61:0/4294615727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691000b1e0 con 0x7f691c0b20a0 2022-07-12T08:48:04.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921207700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c159a60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6914001360 tx=0x7f6914007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6914005ae0 con 0x7f691c0b20a0 2022-07-12T08:48:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6914005c40 con 0x7f691c0b20a0 2022-07-12T08:48:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69140070d0 con 0x7f691c0b20a0 2022-07-12T08:48:04.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c099080 con 0x7f691c0b20a0 2022-07-12T08:48:04.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.356+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c097ea0 con 0x7f691c0b20a0 2022-07-12T08:48:04.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.360+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c0abb00 con 0x7f691c0b20a0 2022-07-12T08:48:04.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.360+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6914010070 con 0x7f691c0b20a0 2022-07-12T08:48:04.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.360+0000 7f6919ffb700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68fc053a40 0x7f68fc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.360+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f691406b790 con 0x7f691c0b20a0 2022-07-12T08:48:04.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.360+0000 7f6920a06700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68fc053a40 0x7f68fc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.364+0000 7f6920a06700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68fc053a40 0x7f68fc055f00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f691001c000 tx=0x7f6910015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:04.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.364+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f69140429a0 con 0x7f691c0b20a0 2022-07-12T08:48:04.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.488+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f691c057460 con 0x7f691c0b20a0 2022-07-12T08:48:04.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.496+0000 7f6919ffb700 1 -- 172.21.15.61:0/4294615727 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6914018070 con 0x7f691c0b20a0 2022-07-12T08:48:04.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68fc053a40 msgr2=0x7f68fc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68fc053a40 0x7f68fc055f00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f691001c000 tx=0x7f6910015000).stop 2022-07-12T08:48:04.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 msgr2=0x7f691c159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c159a60 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6914001360 tx=0x7f6914007e20).stop 2022-07-12T08:48:04.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 shutdown_connections 2022-07-12T08:48:04.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f691c0b20a0 0x7f691c159a60 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f691c0b14a0 0x7f691c0b4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f68fc053a40 0x7f68fc055f00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 --2- 172.21.15.61:0/4294615727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f691c066e50 0x7f691c098b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 >> 172.21.15.61:0/4294615727 conn(0x7f691c0b5860 msgr2=0x7f691c0611a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:04.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 shutdown_connections 2022-07-12T08:48:04.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.500+0000 7f6921a08700 1 -- 172.21.15.61:0/4294615727 wait complete. 2022-07-12T08:48:04.517 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-12T08:48:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.620+0000 7f375b47c700 1 Processor -- start 2022-07-12T08:48:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.620+0000 7f375b47c700 1 -- start start 2022-07-12T08:48:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3754061f50 0x7f37541411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3754066f10 0x7f375414b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3754144220 con 0x7f3754066ab0 2022-07-12T08:48:04.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37540b3950 con 0x7f3754061f50 2022-07-12T08:48:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3754140a70 con 0x7f3754066f10 2022-07-12T08:48:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375a47a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3754061f50 0x7f37541411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759c79700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f3754066ab0 0x7f375413ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59534/0 (socket says 172.21.15.61:59534) 2022-07-12T08:48:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759c79700 1 -- 172.21.15.61:0/563745678 learned_addr learned my addr 172.21.15.61:0/563745678 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375ac7b700 1 --2- 172.21.15.61:0/563745678 >> v2:172.21.15.111:3300/0 conn(0x7f3754066f10 0x7f375414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759478700 1 -- 172.21.15.61:0/563745678 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787113844 0 0) 0x7f3754144220 con 0x7f3754066ab0 2022-07-12T08:48:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759478700 1 -- 172.21.15.61:0/563745678 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3740003260 con 0x7f3754066ab0 2022-07-12T08:48:04.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375a47a700 1 -- 172.21.15.61:0/563745678 >> v2:172.21.15.111:3300/0 conn(0x7f3754066f10 msgr2=0x7f375414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:04.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375a47a700 1 --2- 172.21.15.61:0/563745678 >> v2:172.21.15.111:3300/0 conn(0x7f3754066f10 0x7f375414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375a47a700 1 -- 172.21.15.61:0/563745678 >> v1:172.21.15.111:6789/0 conn(0x7f3754066ab0 legacy=0x7f375413ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375a47a700 1 -- 172.21.15.61:0/563745678 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3754140d10 con 0x7f3754061f50 2022-07-12T08:48:04.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375ac7b700 1 --2- 172.21.15.61:0/563745678 >> v2:172.21.15.111:3300/0 conn(0x7f3754066f10 0x7f375414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:04.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375a47a700 1 --2- 172.21.15.61:0/563745678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754061f50 0x7f37541411b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f3744002f70 tx=0x7f3744006df0).ready entity=mon.0 client_cookie=4860677640d09b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:04.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759478700 1 -- 172.21.15.61:0/563745678 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374400d4f0 con 0x7f3754061f50 2022-07-12T08:48:04.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759478700 1 -- 172.21.15.61:0/563745678 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744000e90 con 0x7f3754061f50 2022-07-12T08:48:04.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f3759478700 1 -- 172.21.15.61:0/563745678 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374400d840 con 0x7f3754061f50 2022-07-12T08:48:04.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- 172.21.15.61:0/563745678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754061f50 msgr2=0x7f37541411b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 --2- 172.21.15.61:0/563745678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754061f50 0x7f37541411b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f3744002f70 tx=0x7f3744006df0).stop 2022-07-12T08:48:04.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- 172.21.15.61:0/563745678 shutdown_connections 2022-07-12T08:48:04.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 --2- 172.21.15.61:0/563745678 >> v2:172.21.15.111:3300/0 conn(0x7f3754066f10 0x7f375414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 --2- 172.21.15.61:0/563745678 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754061f50 0x7f37541411b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- 172.21.15.61:0/563745678 >> 172.21.15.61:0/563745678 conn(0x7f37540b70c0 msgr2=0x7f37540b74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:04.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- 172.21.15.61:0/563745678 shutdown_connections 2022-07-12T08:48:04.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.624+0000 7f375b47c700 1 -- 172.21.15.61:0/563745678 wait complete. 2022-07-12T08:48:04.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 Processor -- start 2022-07-12T08:48:04.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 -- start start 2022-07-12T08:48:04.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3754061f50 0x7f3754097cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 0x7f37540981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3754066f10 0x7f375414ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375409cf10 con 0x7f3754066ab0 2022-07-12T08:48:04.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375409c950 con 0x7f3754066f10 2022-07-12T08:48:04.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3754150fb0 con 0x7f3754061f50 2022-07-12T08:48:04.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 0x7f37540981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 0x7f37540981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57722/0 (socket says 172.21.15.61:57722) 2022-07-12T08:48:04.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 -- 172.21.15.61:0/2421535904 learned_addr learned my addr 172.21.15.61:0/2421535904 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:04.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375a47a700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3754061f50 0x7f3754097cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375ac7b700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3754066f10 0x7f375414ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 -- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3754061f50 msgr2=0x7f3754097cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3754061f50 0x7f3754097cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 -- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3754066f10 msgr2=0x7f375414ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:04.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3754066f10 0x7f375414ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:04.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 -- 172.21.15.61:0/2421535904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c009470 con 0x7f3754066ab0 2022-07-12T08:48:04.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f3759c79700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 0x7f37540981c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f375409ca90 tx=0x7f3750007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:04.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f375000ce40 con 0x7f3754066ab0 2022-07-12T08:48:04.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37500072e0 con 0x7f3754066ab0 2022-07-12T08:48:04.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374400b180 con 0x7f3754066ab0 2022-07-12T08:48:04.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3750005800 con 0x7f3754066ab0 2022-07-12T08:48:04.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.628+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37540986b0 con 0x7f3754066ab0 2022-07-12T08:48:04.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.632+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37540aee30 con 0x7f3754066ab0 2022-07-12T08:48:04.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.632+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f375001a060 con 0x7f3754066ab0 2022-07-12T08:48:04.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.632+0000 7f374b7fe700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f37340539b0 0x7f3734055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:04.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.632+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f375006b070 con 0x7f3754066ab0 2022-07-12T08:48:04.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.632+0000 7f375a47a700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f37340539b0 0x7f3734055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:04.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.632+0000 7f375a47a700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f37340539b0 0x7f3734055e70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f374401a000 tx=0x7f3744014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:04.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.640+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3750068e90 con 0x7f3754066ab0 2022-07-12T08:48:04.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:04.800+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7f3754057460 con 0x7f3754066ab0 2022-07-12T08:48:05.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.216+0000 7f374b7fe700 1 -- 172.21.15.61:0/2421535904 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v301) v1 ==== 139+0+0 (secure 0 0 0) 0x7f3750018070 con 0x7f3754066ab0 2022-07-12T08:48:05.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f37340539b0 msgr2=0x7f3734055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f37340539b0 0x7f3734055e70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f374401a000 tx=0x7f3744014000).stop 2022-07-12T08:48:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 msgr2=0x7f37540981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 0x7f37540981c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f375409ca90 tx=0x7f3750007b80).stop 2022-07-12T08:48:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 shutdown_connections 2022-07-12T08:48:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f37340539b0 0x7f3734055e70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3754066f10 0x7f375414ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3754066ab0 0x7f37540981c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 --2- 172.21.15.61:0/2421535904 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3754061f50 0x7f3754097cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 >> 172.21.15.61:0/2421535904 conn(0x7f37540b70c0 msgr2=0x7f3754140130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:05.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 shutdown_connections 2022-07-12T08:48:05.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.220+0000 7f375b47c700 1 -- 172.21.15.61:0/2421535904 wait complete. 2022-07-12T08:48:05.243 INFO:tasks.cephfs_test_runner:test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:48:05.244 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:48:05.244 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-12T08:48:05.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 Processor -- start 2022-07-12T08:48:05.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 -- start start 2022-07-12T08:48:05.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c0ab290 0x7f438c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:05.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f438c05f140 0x7f438c05ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:05.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c0abc70 con 0x7f438c0afa60 2022-07-12T08:48:05.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c05f580 con 0x7f438c05f140 2022-07-12T08:48:05.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f4390e6a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c05c720 con 0x7f438c0ab290 2022-07-12T08:48:05.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f438affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f438c05f140 0x7f438c05ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:05.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f438b7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c0ab290 0x7f438c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:05.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f438affd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f438c05f140 0x7f438c05ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36302/0 (socket says 172.21.15.61:36302) 2022-07-12T08:48:05.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f438affd700 1 -- 172.21.15.61:0/4183661848 learned_addr learned my addr 172.21.15.61:0/4183661848 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:05.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.360+0000 7f438bfff700 1 --1- 172.21.15.61:0/4183661848 >> v1:172.21.15.111:6789/0 conn(0x7f438c0afa60 0x7f438c14a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59552/0 (socket says 172.21.15.61:59552) 2022-07-12T08:48:05.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438affd700 1 -- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c0ab290 msgr2=0x7f438c0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:05.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438affd700 1 --2- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c0ab290 0x7f438c0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438affd700 1 -- 172.21.15.61:0/4183661848 >> v1:172.21.15.111:6789/0 conn(0x7f438c0afa60 legacy=0x7f438c14a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:05.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438a7fc700 1 -- 172.21.15.61:0/4183661848 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468503640 0 0) 0x7f438c0abc70 con 0x7f438c0afa60 2022-07-12T08:48:05.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438affd700 1 -- 172.21.15.61:0/4183661848 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c05c9c0 con 0x7f438c05f140 2022-07-12T08:48:05.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438affd700 1 --2- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c05f140 0x7f438c05ce60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f437c0032f0 tx=0x7f437c009290).ready entity=mon.1 client_cookie=cf3f6814da9a3a25 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:05.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438a7fc700 1 -- 172.21.15.61:0/4183661848 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f437c00b8a0 con 0x7f438c05f140 2022-07-12T08:48:05.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438a7fc700 1 -- 172.21.15.61:0/4183661848 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c00ba00 con 0x7f438c05f140 2022-07-12T08:48:05.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f438a7fc700 1 -- 172.21.15.61:0/4183661848 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f437c00bb70 con 0x7f438c05f140 2022-07-12T08:48:05.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 -- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c05f140 msgr2=0x7f438c05ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:05.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 --2- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c05f140 0x7f438c05ce60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f437c0032f0 tx=0x7f437c009290).stop 2022-07-12T08:48:05.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 -- 172.21.15.61:0/4183661848 shutdown_connections 2022-07-12T08:48:05.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 --2- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c05f140 0x7f438c05ce60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 --2- 172.21.15.61:0/4183661848 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c0ab290 0x7f438c0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 -- 172.21.15.61:0/4183661848 >> 172.21.15.61:0/4183661848 conn(0x7f438c0b1950 msgr2=0x7f438c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:05.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 -- 172.21.15.61:0/4183661848 shutdown_connections 2022-07-12T08:48:05.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.364+0000 7f4390e6a700 1 -- 172.21.15.61:0/4183661848 wait complete. 2022-07-12T08:48:05.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 Processor -- start 2022-07-12T08:48:05.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 -- start start 2022-07-12T08:48:05.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c05f140 0x7f438c099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:05.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c0ab290 0x7f438c155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:05.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 0x7f438c156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c065f20 con 0x7f438c0afa60 2022-07-12T08:48:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c149db0 con 0x7f438c0ab290 2022-07-12T08:48:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c14a050 con 0x7f438c05f140 2022-07-12T08:48:05.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 0x7f438c156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:05.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 0x7f438c156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57738/0 (socket says 172.21.15.61:57738) 2022-07-12T08:48:05.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 -- 172.21.15.61:0/2220803164 learned_addr learned my addr 172.21.15.61:0/2220803164 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438b7fe700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c05f140 0x7f438c099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438affd700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c0ab290 0x7f438c155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 -- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c05f140 msgr2=0x7f438c099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c05f140 0x7f438c099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 -- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c0ab290 msgr2=0x7f438c155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c0ab290 0x7f438c155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 -- 172.21.15.61:0/2220803164 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c009530 con 0x7f438c0afa60 2022-07-12T08:48:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f438bfff700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 0x7f438c156260 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f4384000fe0 tx=0x7f4384007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43840070d0 con 0x7f438c0afa60 2022-07-12T08:48:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4384007230 con 0x7f438c0afa60 2022-07-12T08:48:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c14a2f0 con 0x7f438c0afa60 2022-07-12T08:48:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4384005740 con 0x7f438c0afa60 2022-07-12T08:48:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.368+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c156970 con 0x7f438c0afa60 2022-07-12T08:48:05.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.372+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c14a480 con 0x7f438c0afa60 2022-07-12T08:48:05.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.372+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f438401a060 con 0x7f438c0afa60 2022-07-12T08:48:05.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.372+0000 7f4388ff9700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f436c0539b0 0x7f436c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:05.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.372+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f438406b2a0 con 0x7f438c0afa60 2022-07-12T08:48:05.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.372+0000 7f438b7fe700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f436c0539b0 0x7f436c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:05.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.376+0000 7f438b7fe700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f436c0539b0 0x7f436c055e70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f438000b000 tx=0x7f4380006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:05.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.380+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43840423a0 con 0x7f438c0afa60 2022-07-12T08:48:05.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:05.552+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7f438c156970 con 0x7f438c0afa60 2022-07-12T08:48:06.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.219+0000 7f4388ff9700 1 -- 172.21.15.61:0/2220803164 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v302) v1 ==== 129+0+0 (secure 0 0 0) 0x7f438c156970 con 0x7f438c0afa60 2022-07-12T08:48:06.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f436c0539b0 msgr2=0x7f436c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f436c0539b0 0x7f436c055e70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f438000b000 tx=0x7f4380006df0).stop 2022-07-12T08:48:06.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 msgr2=0x7f438c156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 0x7f438c156260 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 rx=0x7f4384000fe0 tx=0x7f4384007bc0).stop 2022-07-12T08:48:06.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 shutdown_connections 2022-07-12T08:48:06.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f438c0afa60 0x7f438c156260 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f436c0539b0 0x7f436c055e70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f438c0ab290 0x7f438c155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 --2- 172.21.15.61:0/2220803164 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f438c05f140 0x7f438c099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 >> 172.21.15.61:0/2220803164 conn(0x7f438c0b1950 msgr2=0x7f438c0b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:06.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 shutdown_connections 2022-07-12T08:48:06.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.223+0000 7f4390e6a700 1 -- 172.21.15.61:0/2220803164 wait complete. 2022-07-12T08:48:06.244 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:48:06.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.363+0000 7f4668cfa700 1 Processor -- start 2022-07-12T08:48:06.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.363+0000 7f4668cfa700 1 -- start start 2022-07-12T08:48:06.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46640b21f0 0x7f46640b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 0x7f4664142d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46640ae3d0 con 0x7f4664143800 2022-07-12T08:48:06.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46640ae690 con 0x7f46640b21f0 2022-07-12T08:48:06.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640ae930 con 0x7f46640afb70 2022-07-12T08:48:06.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 0x7f4664142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f46637fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46640b21f0 0x7f46640b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 0x7f4664142d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:39940/0 (socket says 172.21.15.61:39940) 2022-07-12T08:48:06.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 -- 172.21.15.61:0/1725006599 learned_addr learned my addr 172.21.15.61:0/1725006599 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:06.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4663fff700 1 --1- 172.21.15.61:0/1725006599 >> v1:172.21.15.111:6789/0 conn(0x7f4664143800 0x7f46640aedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59564/0 (socket says 172.21.15.61:59564) 2022-07-12T08:48:06.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 -- 172.21.15.61:0/1725006599 >> v2:172.21.15.111:3300/0 conn(0x7f46640b21f0 msgr2=0x7f46640b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:06.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 --2- 172.21.15.61:0/1725006599 >> v2:172.21.15.111:3300/0 conn(0x7f46640b21f0 0x7f46640b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 -- 172.21.15.61:0/1725006599 >> v1:172.21.15.111:6789/0 conn(0x7f4664143800 legacy=0x7f46640aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 -- 172.21.15.61:0/1725006599 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664143280 con 0x7f46640afb70 2022-07-12T08:48:06.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f465bfff700 1 --2- 172.21.15.61:0/1725006599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 0x7f4664142d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f464c002f70 tx=0x7f464c006eb0).ready entity=mon.2 client_cookie=4c95489b874fe49a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:06.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4662ffd700 1 -- 172.21.15.61:0/1725006599 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464c00d570 con 0x7f46640afb70 2022-07-12T08:48:06.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4662ffd700 1 -- 172.21.15.61:0/1725006599 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c000e90 con 0x7f46640afb70 2022-07-12T08:48:06.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4662ffd700 1 -- 172.21.15.61:0/1725006599 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464c00d900 con 0x7f46640afb70 2022-07-12T08:48:06.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- 172.21.15.61:0/1725006599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 msgr2=0x7f4664142d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1725006599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 0x7f4664142d90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f464c002f70 tx=0x7f464c006eb0).stop 2022-07-12T08:48:06.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- 172.21.15.61:0/1725006599 shutdown_connections 2022-07-12T08:48:06.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1725006599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46640afb70 0x7f4664142d90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1725006599 >> v2:172.21.15.111:3300/0 conn(0x7f46640b21f0 0x7f46640b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- 172.21.15.61:0/1725006599 >> 172.21.15.61:0/1725006599 conn(0x7f466405f140 msgr2=0x7f466405f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.367+0000 7f4668cfa700 1 -- 172.21.15.61:0/1725006599 shutdown_connections 2022-07-12T08:48:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.371+0000 7f4668cfa700 1 -- 172.21.15.61:0/1725006599 wait complete. 2022-07-12T08:48:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.371+0000 7f4668cfa700 1 Processor -- start 2022-07-12T08:48:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.371+0000 7f4668cfa700 1 -- start start 2022-07-12T08:48:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46640afb70 0x7f46640b4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 0x7f4664065c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4664143800 0x7f46640647d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46641388c0 con 0x7f46640b21f0 2022-07-12T08:48:06.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46641536d0 con 0x7f46640afb70 2022-07-12T08:48:06.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4664066d40 con 0x7f4664143800 2022-07-12T08:48:06.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 0x7f4664065c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 0x7f4664065c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57748/0 (socket says 172.21.15.61:57748) 2022-07-12T08:48:06.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 -- 172.21.15.61:0/1582086059 learned_addr learned my addr 172.21.15.61:0/1582086059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4663fff700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4664143800 0x7f46640647d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f46637fe700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46640afb70 0x7f46640b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 -- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4664143800 msgr2=0x7f46640647d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4664143800 0x7f46640647d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 -- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46640afb70 msgr2=0x7f46640b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46640afb70 0x7f46640b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 -- 172.21.15.61:0/1582086059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c00b1e0 con 0x7f46640b21f0 2022-07-12T08:48:06.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4663fff700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4664143800 0x7f46640647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:06.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f46637fe700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46640afb70 0x7f46640b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:06.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f465bfff700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 0x7f4664065c00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f4664153810 tx=0x7f464c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:06.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464c006b40 con 0x7f46640b21f0 2022-07-12T08:48:06.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c00d8f0 con 0x7f46640b21f0 2022-07-12T08:48:06.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464c011e50 con 0x7f46640b21f0 2022-07-12T08:48:06.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4664066f70 con 0x7f46640b21f0 2022-07-12T08:48:06.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46640641f0 con 0x7f46640b21f0 2022-07-12T08:48:06.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4664152480 con 0x7f46640b21f0 2022-07-12T08:48:06.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.375+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f464c025030 con 0x7f46640b21f0 2022-07-12T08:48:06.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.379+0000 7f46617fa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46480539d0 0x7f4648055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.379+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f464c06cee0 con 0x7f46640b21f0 2022-07-12T08:48:06.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.379+0000 7f46637fe700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46480539d0 0x7f4648055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.379+0000 7f46637fe700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46480539d0 0x7f4648055e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f465400b000 tx=0x7f4654008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:06.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.383+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f464c0440f0 con 0x7f46640b21f0 2022-07-12T08:48:06.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.511+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4664067150 con 0x7f46640b21f0 2022-07-12T08:48:06.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f46617fa700 1 -- 172.21.15.61:0/1582086059 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f464c023020 con 0x7f46640b21f0 2022-07-12T08:48:06.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46480539d0 msgr2=0x7f4648055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46480539d0 0x7f4648055e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f465400b000 tx=0x7f4654008040).stop 2022-07-12T08:48:06.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 msgr2=0x7f4664065c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 0x7f4664065c00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f4664153810 tx=0x7f464c015000).stop 2022-07-12T08:48:06.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 shutdown_connections 2022-07-12T08:48:06.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4664143800 0x7f46640647d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46640b21f0 0x7f4664065c00 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46480539d0 0x7f4648055e90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 --2- 172.21.15.61:0/1582086059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46640afb70 0x7f46640b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.519+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 >> 172.21.15.61:0/1582086059 conn(0x7f466405f140 msgr2=0x7f46640b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:06.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.523+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 shutdown_connections 2022-07-12T08:48:06.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.523+0000 7f4668cfa700 1 -- 172.21.15.61:0/1582086059 wait complete. 2022-07-12T08:48:06.538 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:48:06.539 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:06.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:06.589 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:06.614 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:06.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:06.663 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:48:06.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:06.666 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:06.667 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:06.687 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:48:06.690 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:06.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:06.691 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:06.692 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:06.704 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:48:06.705 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:06.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:06.706 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:06.706 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:06.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:06.716 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:06.716 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:48:06.717 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:48:06.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 Processor -- start 2022-07-12T08:48:06.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- start start 2022-07-12T08:48:06.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd6ac0679a0 0x7fd6ac142d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac143270 0x7fd6ac13c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac0b4d40 con 0x7fd6ac0b6080 2022-07-12T08:48:06.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac144030 con 0x7fd6ac0679a0 2022-07-12T08:48:06.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0b1680 con 0x7fd6ac143270 2022-07-12T08:48:06.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1e2a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd6ac0b6080 0x7fd6ac0b1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45972/0 (socket says 172.21.15.61:45972) 2022-07-12T08:48:06.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1e2a700 1 -- 172.21.15.61:0/2421800587 learned_addr learned my addr 172.21.15.61:0/2421800587 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:06.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1629700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd6ac0679a0 0x7fd6ac142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b262b700 1 --2- 172.21.15.61:0/2421800587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac143270 0x7fd6ac13c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b0e28700 1 -- 172.21.15.61:0/2421800587 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130225541 0 0) 0x7fd6ac0b4d40 con 0x7fd6ac0b6080 2022-07-12T08:48:06.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b0e28700 1 -- 172.21.15.61:0/2421800587 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd694003260 con 0x7fd6ac0b6080 2022-07-12T08:48:06.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1629700 1 -- 172.21.15.61:0/2421800587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac143270 msgr2=0x7fd6ac13c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:06.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1629700 1 --2- 172.21.15.61:0/2421800587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac143270 0x7fd6ac13c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1629700 1 -- 172.21.15.61:0/2421800587 >> v1:172.21.15.61:6789/0 conn(0x7fd6ac0b6080 legacy=0x7fd6ac0b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1629700 1 -- 172.21.15.61:0/2421800587 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac0b1920 con 0x7fd6ac0679a0 2022-07-12T08:48:06.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b1629700 1 --2- 172.21.15.61:0/2421800587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac142d80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fd69c002f70 tx=0x7fd69c006df0).ready entity=mon.0 client_cookie=7050b491c3be6f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:06.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b0e28700 1 -- 172.21.15.61:0/2421800587 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd69c00d4a0 con 0x7fd6ac0679a0 2022-07-12T08:48:06.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b0e28700 1 -- 172.21.15.61:0/2421800587 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c000e90 con 0x7fd6ac0679a0 2022-07-12T08:48:06.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b0e28700 1 -- 172.21.15.61:0/2421800587 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd69c00d7f0 con 0x7fd6ac0679a0 2022-07-12T08:48:06.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- 172.21.15.61:0/2421800587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac0679a0 msgr2=0x7fd6ac142d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 --2- 172.21.15.61:0/2421800587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac142d80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7fd69c002f70 tx=0x7fd69c006df0).stop 2022-07-12T08:48:06.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- 172.21.15.61:0/2421800587 shutdown_connections 2022-07-12T08:48:06.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 --2- 172.21.15.61:0/2421800587 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac143270 0x7fd6ac13c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 --2- 172.21.15.61:0/2421800587 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac142d80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- 172.21.15.61:0/2421800587 >> 172.21.15.61:0/2421800587 conn(0x7fd6ac065760 msgr2=0x7fd6ac066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:06.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- 172.21.15.61:0/2421800587 shutdown_connections 2022-07-12T08:48:06.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.831+0000 7fd6b408e700 1 -- 172.21.15.61:0/2421800587 wait complete. 2022-07-12T08:48:06.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 Processor -- start 2022-07-12T08:48:06.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 -- start start 2022-07-12T08:48:06.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac0b6080 0x7fd6ac0adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 0x7fd6ac13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac142a00 con 0x7fd6ac143270 2022-07-12T08:48:06.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac142760 con 0x7fd6ac0679a0 2022-07-12T08:48:06.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b408e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0ae1f0 con 0x7fd6ac0b6080 2022-07-12T08:48:06.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 0x7fd6ac13f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 0x7fd6ac13f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57762/0 (socket says 172.21.15.61:57762) 2022-07-12T08:48:06.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 -- 172.21.15.61:0/1505081245 learned_addr learned my addr 172.21.15.61:0/1505081245 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:06.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b1e2a700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b1629700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac0b6080 0x7fd6ac0adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 -- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac0b6080 msgr2=0x7fd6ac0adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac0b6080 0x7fd6ac0adbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 -- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0679a0 msgr2=0x7fd6ac0affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:06.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac0affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:06.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b262b700 1 -- 172.21.15.61:0/1505081245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c00b180 con 0x7fd6ac143270 2022-07-12T08:48:06.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.835+0000 7fd6b1e2a700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac0affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:06.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd6b262b700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 0x7fd6ac13f9a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fd6a4006e00 tx=0x7fd6a40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6a4012600 con 0x7fd6ac143270 2022-07-12T08:48:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4012760 con 0x7fd6ac143270 2022-07-12T08:48:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac141a40 con 0x7fd6ac143270 2022-07-12T08:48:06.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6a4012ac0 con 0x7fd6ac143270 2022-07-12T08:48:06.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac0ad530 con 0x7fd6ac143270 2022-07-12T08:48:06.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac057460 con 0x7fd6ac143270 2022-07-12T08:48:06.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd6a4022070 con 0x7fd6ac143270 2022-07-12T08:48:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd69affd700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd68c053a40 0x7fd68c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fd6a406c1d0 con 0x7fd6ac143270 2022-07-12T08:48:06.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.839+0000 7fd6b1e2a700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd68c053a40 0x7fd68c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:06.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.843+0000 7fd6b1e2a700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd68c053a40 0x7fd68c055f00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fd6a800d000 tx=0x7fd6a8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:06.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.847+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd6a4043340 con 0x7fd6ac143270 2022-07-12T08:48:06.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.975+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd6ac0b41f0 con 0x7fd6ac143270 2022-07-12T08:48:06.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd69affd700 1 -- 172.21.15.61:0/1505081245 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v75) v1 ==== 81+0+28949 (secure 0 0 0) 0x7fd6a4016090 con 0x7fd6ac143270 2022-07-12T08:48:06.983 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:06.983 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:06.984 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 75, 2022-07-12T08:48:06.984 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:48:06.985 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:48:06.985 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:48:04.201842+0000", 2022-07-12T08:48:06.986 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:06.987 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:06.988 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:48:06.988 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:48:06.989 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:48:06.989 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:48:06.990 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:48:06.990 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:48:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:48:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:48:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:48:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:48:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:48:07.008 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:48:07.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 14, 2022-07-12T08:48:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:48:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:48:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:48:07.013 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:48:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:48:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:48:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:48:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:48:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:48:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:48:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:48:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:48:07.018 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:48:07.018 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:07.024 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:07.025 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:07.025 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:07.025 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:48:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:07.030 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:07.035 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:07.035 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:07.038 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:07.038 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:07.039 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:07.039 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:07.039 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:07.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:07.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.040 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:07.040 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:07.041 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:07.041 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:07.041 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:07.044 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:48:07.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.044 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:07.044 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:48:07.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.046 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:48:07.047 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:48:07.047 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:48:07.047 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:07.050 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:07.050 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:07.051 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:07.051 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:07.053 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:07.055 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.055 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:48:07.055 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:07.057 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:07.058 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:07.058 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:07.059 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:07.059 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:07.059 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:07.062 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:07.062 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.065 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:07.066 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:07.066 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:07.067 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:07.067 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:07.068 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:07.068 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:07.068 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:07.069 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:07.072 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:07.072 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:48:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:07.074 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:48:07.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.075 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:07.075 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:48:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.077 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.077 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.078 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.078 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.079 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.079 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:48:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:48:07.083 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:48:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:48:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.090 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.090 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:48:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.094 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.094 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:48:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:48:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:48:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.106 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:48:07.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:48:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:48:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:48:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:48:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:48:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 60, 2022-07-12T08:48:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.116 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.116 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:48:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:48:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:48:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:48:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:48:07.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:48:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:48:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.130 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.130 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:48:07.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:07.131 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.131 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:48:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:48:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:48:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:48:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.133 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.133 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.133 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.134 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:07.134 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:48:07.134 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.136 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.136 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.136 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:48:07.136 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.137 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.137 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:48:07.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:48:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:48:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:48:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:48:07.151 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.151 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:48:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.154 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.158 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.159 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:48:07.159 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.167 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:48:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:48:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:48:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:48:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:48:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:48:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:48:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:48:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:48:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:48:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:48:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:48:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:48:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:48:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:48:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:48:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:48:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:48:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:48:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:48:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.190 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.190 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.190 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-12T08:48:07.190 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:48:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:48:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.196 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:48:07.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.199 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:48:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.201 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.201 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.202 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.202 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:48:07.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.206 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.207 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:48:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.209 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.209 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:48:07.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.214 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:48:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.217 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:48:07.217 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:48:07.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:48:07.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:48:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.220 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.221 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.221 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:07.222 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:48:07.222 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.224 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.224 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.225 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.225 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.226 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:48:07.226 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:48:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:48:07.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.232 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.234 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.234 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:48:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.238 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:48:07.238 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.241 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:48:07.243 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:48:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.246 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.246 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.247 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:48:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.249 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:48:07.249 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:07.249 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:48:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:48:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:48:07.251 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:48:07.251 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.251 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.252 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.252 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:48:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-12T08:48:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:48:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:48:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:48:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:48:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:48:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:48:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:48:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:48:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:48:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:48:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:48:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:48:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.270 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.270 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.270 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.270 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:48:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-12T08:48:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:48:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:48:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:48:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:48:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:48:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.283 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:48:07.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:48:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:48:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.288 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:48:07.288 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:48:07.288 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:48:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:48:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:07.290 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:07.290 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:07.292 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:48:07.293 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.293 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:07.293 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.294 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.294 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.295 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.295 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.296 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:48:07.296 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.296 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.297 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.297 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:07.298 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.298 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.299 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.299 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.300 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.300 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:48:07.300 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.301 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.301 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:48:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:48:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:48:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:48:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:48:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:48:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:48:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:48:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:48:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:48:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:48:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/3265139715": "2022-07-13T08:48:00.251197+0000", 2022-07-12T08:48:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/3265139715": "2022-07-13T08:48:00.251197+0000", 2022-07-12T08:48:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.121:0/3889265026": "2022-07-12T09:47:49.061933+0000" 2022-07-12T08:48:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:48:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:48:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:48:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:48:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:48:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:48:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:48:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:48:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:48:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:48:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:48:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:48:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:48:07.324 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:07.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd68c053a40 msgr2=0x7fd68c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd68c053a40 0x7fd68c055f00 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fd6a800d000 tx=0x7fd6a8009040).stop 2022-07-12T08:48:07.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 msgr2=0x7fd6ac13f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 0x7fd6ac13f9a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7fd6a4006e00 tx=0x7fd6a40091d0).stop 2022-07-12T08:48:07.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 shutdown_connections 2022-07-12T08:48:07.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac143270 0x7fd6ac13f9a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac0b6080 0x7fd6ac0adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd68c053a40 0x7fd68c055f00 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 --2- 172.21.15.61:0/1505081245 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0679a0 0x7fd6ac0affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 >> 172.21.15.61:0/1505081245 conn(0x7fd6ac065760 msgr2=0x7fd6ac066db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 shutdown_connections 2022-07-12T08:48:07.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:06.979+0000 7fd6b408e700 1 -- 172.21.15.61:0/1505081245 wait complete. 2022-07-12T08:48:07.329 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:07.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 Processor -- start 2022-07-12T08:48:07.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 -- start start 2022-07-12T08:48:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e54060b60 0x7f2e54141f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e54142470 0x7f2e54142900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e540ab2c0 con 0x7f2e540aaee0 2022-07-12T08:48:07.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e540ab530 con 0x7f2e54060b60 2022-07-12T08:48:07.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5ad3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e540ab7d0 con 0x7f2e54142470 2022-07-12T08:48:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e59d38700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2e540aaee0 0x7f2e540b0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45988/0 (socket says 172.21.15.61:45988) 2022-07-12T08:48:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e59d38700 1 -- 172.21.15.61:0/2731500166 learned_addr learned my addr 172.21.15.61:0/2731500166 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5a539700 1 --2- 172.21.15.61:0/2731500166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e54142470 0x7f2e54142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e59537700 1 --2- 172.21.15.61:0/2731500166 >> v2:172.21.15.111:3300/0 conn(0x7f2e54060b60 0x7f2e54141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e58d36700 1 -- 172.21.15.61:0/2731500166 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3725195428 0 0) 0x7f2e540ab2c0 con 0x7f2e540aaee0 2022-07-12T08:48:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e58d36700 1 -- 172.21.15.61:0/2731500166 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e3c003260 con 0x7f2e540aaee0 2022-07-12T08:48:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.439+0000 7f2e5a539700 1 -- 172.21.15.61:0/2731500166 >> v2:172.21.15.111:3300/0 conn(0x7f2e54060b60 msgr2=0x7f2e54141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:07.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5a539700 1 --2- 172.21.15.61:0/2731500166 >> v2:172.21.15.111:3300/0 conn(0x7f2e54060b60 0x7f2e54141f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5a539700 1 -- 172.21.15.61:0/2731500166 >> v1:172.21.15.61:6789/0 conn(0x7f2e540aaee0 legacy=0x7f2e540b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e58d36700 1 -- 172.21.15.61:0/2731500166 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3647130655 0 0) 0x7f2e3c003260 con 0x7f2e540aaee0 2022-07-12T08:48:07.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5a539700 1 -- 172.21.15.61:0/2731500166 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e540b10d0 con 0x7f2e54142470 2022-07-12T08:48:07.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5a539700 1 --2- 172.21.15.61:0/2731500166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e54142470 0x7f2e54142900 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2e44001800 tx=0x7f2e4c006df0).ready entity=mon.2 client_cookie=347a80685b1f50d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:07.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e58d36700 1 -- 172.21.15.61:0/2731500166 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e44001660 con 0x7f2e54142470 2022-07-12T08:48:07.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e58d36700 1 -- 172.21.15.61:0/2731500166 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c000e90 con 0x7f2e54142470 2022-07-12T08:48:07.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e58d36700 1 -- 172.21.15.61:0/2731500166 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4c012460 con 0x7f2e54142470 2022-07-12T08:48:07.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/2731500166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e54142470 msgr2=0x7f2e54142900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/2731500166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e54142470 0x7f2e54142900 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f2e44001800 tx=0x7f2e4c006df0).stop 2022-07-12T08:48:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/2731500166 shutdown_connections 2022-07-12T08:48:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/2731500166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e54142470 0x7f2e54142900 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/2731500166 >> v2:172.21.15.111:3300/0 conn(0x7f2e54060b60 0x7f2e54141f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/2731500166 >> 172.21.15.61:0/2731500166 conn(0x7f2e540b1950 msgr2=0x7f2e540b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/2731500166 shutdown_connections 2022-07-12T08:48:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/2731500166 wait complete. 2022-07-12T08:48:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 Processor -- start 2022-07-12T08:48:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.443+0000 7f2e5ad3a700 1 -- start start 2022-07-12T08:48:07.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 0x7f2e54144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e540aaee0 0x7f2e5413dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e54142470 0x7f2e5413e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e540aee10 con 0x7f2e54060b60 2022-07-12T08:48:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e54145c40 con 0x7f2e54142470 2022-07-12T08:48:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e541528e0 con 0x7f2e540aaee0 2022-07-12T08:48:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 0x7f2e54144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 0x7f2e54144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57774/0 (socket says 172.21.15.61:57774) 2022-07-12T08:48:07.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 -- 172.21.15.61:0/1491552380 learned_addr learned my addr 172.21.15.61:0/1491552380 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:07.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5a539700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e54142470 0x7f2e5413e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59537700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e540aaee0 0x7f2e5413dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 -- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e540aaee0 msgr2=0x7f2e5413dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e540aaee0 0x7f2e5413dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 -- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e54142470 msgr2=0x7f2e5413e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e54142470 0x7f2e5413e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 -- 172.21.15.61:0/1491552380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c00b180 con 0x7f2e54060b60 2022-07-12T08:48:07.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e59d38700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 0x7f2e54144da0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f2e440018c0 tx=0x7f2e440012c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:07.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4400ce20 con 0x7f2e54060b60 2022-07-12T08:48:07.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e44007170 con 0x7f2e54060b60 2022-07-12T08:48:07.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e54152640 con 0x7f2e54060b60 2022-07-12T08:48:07.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e44004730 con 0x7f2e54060b60 2022-07-12T08:48:07.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.447+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e54152360 con 0x7f2e54060b60 2022-07-12T08:48:07.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.451+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e541459a0 con 0x7f2e54060b60 2022-07-12T08:48:07.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.451+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2e44004890 con 0x7f2e54060b60 2022-07-12T08:48:07.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.451+0000 7f2e42ffd700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e34053ad0 0x7f2e34055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.451+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f2e4406b6e0 con 0x7f2e54060b60 2022-07-12T08:48:07.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.451+0000 7f2e59537700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e34053ad0 0x7f2e34055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.451+0000 7f2e59537700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e34053ad0 0x7f2e34055f90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f2e5000b000 tx=0x7f2e500091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:07.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.459+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e541459a0 con 0x7f2e54060b60 2022-07-12T08:48:07.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.619+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e541459a0 con 0x7f2e54060b60 2022-07-12T08:48:07.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.619+0000 7f2e42ffd700 1 -- 172.21.15.61:0/1491552380 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f2e541459a0 con 0x7f2e54060b60 2022-07-12T08:48:07.633 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:07.633 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35},{"gid":6597,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3437516713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3437516713},{"type":"v1","addr":"172.21.15.61:6833","nonce":3437516713}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":40}],"filesystems":[]} 2022-07-12T08:48:07.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e34053ad0 msgr2=0x7f2e34055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e34053ad0 0x7f2e34055f90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f2e5000b000 tx=0x7f2e500091d0).stop 2022-07-12T08:48:07.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 msgr2=0x7f2e54144da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 0x7f2e54144da0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7f2e440018c0 tx=0x7f2e440012c0).stop 2022-07-12T08:48:07.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 shutdown_connections 2022-07-12T08:48:07.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e34053ad0 0x7f2e34055f90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e54142470 0x7f2e5413e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e540aaee0 0x7f2e5413dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 --2- 172.21.15.61:0/1491552380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e54060b60 0x7f2e54144da0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 >> 172.21.15.61:0/1491552380 conn(0x7f2e540b1950 msgr2=0x7f2e540b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:07.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 shutdown_connections 2022-07-12T08:48:07.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.623+0000 7f2e5ad3a700 1 -- 172.21.15.61:0/1491552380 wait complete. 2022-07-12T08:48:07.642 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-12T08:48:07.646 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 Processor -- start 2022-07-12T08:48:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 -- start start 2022-07-12T08:48:07.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f35a40628e0 0x7f35a40b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f35a4142f40 0x7f35a4143390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a40b7ee0 con 0x7f35a40b7b00 2022-07-12T08:48:07.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40acf80 con 0x7f35a40628e0 2022-07-12T08:48:07.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a9dbd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a40ae2e0 con 0x7f35a4142f40 2022-07-12T08:48:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f35a40628e0 0x7f35a40b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f35a40628e0 0x7f35a40b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57784/0 (socket says 172.21.15.61:57784) 2022-07-12T08:48:07.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a3fff700 1 -- 172.21.15.61:0/3989367899 learned_addr learned my addr 172.21.15.61:0/3989367899 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:07.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.755+0000 7f35a95bc700 1 --2- 172.21.15.61:0/3989367899 >> v2:172.21.15.111:3300/0 conn(0x7f35a4142f40 0x7f35a4143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a3fff700 1 -- 172.21.15.61:0/3989367899 >> v2:172.21.15.111:3300/0 conn(0x7f35a4142f40 msgr2=0x7f35a4143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:07.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a3fff700 1 --2- 172.21.15.61:0/3989367899 >> v2:172.21.15.111:3300/0 conn(0x7f35a4142f40 0x7f35a4143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a3fff700 1 -- 172.21.15.61:0/3989367899 >> v1:172.21.15.111:6789/0 conn(0x7f35a40b7b00 legacy=0x7f35a40ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a3fff700 1 -- 172.21.15.61:0/3989367899 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a40b2d30 con 0x7f35a40628e0 2022-07-12T08:48:07.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a3fff700 1 --2- 172.21.15.61:0/3989367899 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a40628e0 0x7f35a40b1f90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f3598002f70 tx=0x7f3598006eb0).ready entity=mon.0 client_cookie=a8bce3c56f2aa1dd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:07.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a37fe700 1 -- 172.21.15.61:0/3989367899 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f359800d5c0 con 0x7f35a40628e0 2022-07-12T08:48:07.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a37fe700 1 -- 172.21.15.61:0/3989367899 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3598000e90 con 0x7f35a40628e0 2022-07-12T08:48:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a37fe700 1 -- 172.21.15.61:0/3989367899 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f359800d950 con 0x7f35a40628e0 2022-07-12T08:48:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 -- 172.21.15.61:0/3989367899 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a40628e0 msgr2=0x7f35a40b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/3989367899 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a40628e0 0x7f35a40b1f90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7f3598002f70 tx=0x7f3598006eb0).stop 2022-07-12T08:48:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 -- 172.21.15.61:0/3989367899 shutdown_connections 2022-07-12T08:48:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/3989367899 >> v2:172.21.15.111:3300/0 conn(0x7f35a4142f40 0x7f35a4143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/3989367899 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a40628e0 0x7f35a40b1f90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 -- 172.21.15.61:0/3989367899 >> 172.21.15.61:0/3989367899 conn(0x7f35a4065550 msgr2=0x7f35a40b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 -- 172.21.15.61:0/3989367899 shutdown_connections 2022-07-12T08:48:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 -- 172.21.15.61:0/3989367899 wait complete. 2022-07-12T08:48:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.759+0000 7f35a9dbd700 1 Processor -- start 2022-07-12T08:48:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 -- start start 2022-07-12T08:48:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a40b7b00 0x7f35a40aea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35a4142f40 0x7f35a414fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 0x7f35a4099230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a40980e0 con 0x7f35a4097210 2022-07-12T08:48:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a4150210 con 0x7f35a40b7b00 2022-07-12T08:48:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a4150460 con 0x7f35a4142f40 2022-07-12T08:48:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 0x7f35a4099230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 0x7f35a4099230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57792/0 (socket says 172.21.15.61:57792) 2022-07-12T08:48:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 -- 172.21.15.61:0/835758814 learned_addr learned my addr 172.21.15.61:0/835758814 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a8dbb700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a40b7b00 0x7f35a40aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a3fff700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35a4142f40 0x7f35a414fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 -- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35a4142f40 msgr2=0x7f35a414fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35a4142f40 0x7f35a414fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 -- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a40b7b00 msgr2=0x7f35a40aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a40b7b00 0x7f35a40aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 -- 172.21.15.61:0/835758814 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359800b1e0 con 0x7f35a4097210 2022-07-12T08:48:07.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a3fff700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35a4142f40 0x7f35a414fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:07.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a8dbb700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a40b7b00 0x7f35a40aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:07.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a95bc700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 0x7f35a4099230 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f359400c000 tx=0x7f3594006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:07.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3594010780 con 0x7f35a4097210 2022-07-12T08:48:07.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35940108e0 con 0x7f35a4097210 2022-07-12T08:48:07.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a4150700 con 0x7f35a4097210 2022-07-12T08:48:07.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3594010c40 con 0x7f35a4097210 2022-07-12T08:48:07.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.763+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a4099720 con 0x7f35a4097210 2022-07-12T08:48:07.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.767+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a4057460 con 0x7f35a4097210 2022-07-12T08:48:07.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.767+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f359401c460 con 0x7f35a4097210 2022-07-12T08:48:07.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.767+0000 7f35a1ffb700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35840539b0 0x7f3584055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:07.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.767+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f359406c130 con 0x7f35a4097210 2022-07-12T08:48:07.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.767+0000 7f35a8dbb700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35840539b0 0x7f3584055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:07.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.767+0000 7f35a8dbb700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35840539b0 0x7f3584055e70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f359c00b000 tx=0x7f359c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:07.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.771+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35940431d0 con 0x7f35a4097210 2022-07-12T08:48:07.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.935+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a4150ba0 con 0x7f35a4097210 2022-07-12T08:48:07.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.935+0000 7f35a1ffb700 1 -- 172.21.15.61:0/835758814 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f35940467f0 con 0x7f35a4097210 2022-07-12T08:48:07.942 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:07.942 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5690,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/4089489454","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":4089489454},{"type":"v1","addr":"172.21.15.111:6835","nonce":4089489454}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":35},{"gid":6597,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3437516713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3437516713},{"type":"v1","addr":"172.21.15.61:6833","nonce":3437516713}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":40}],"filesystems":[]} 2022-07-12T08:48:07.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35840539b0 msgr2=0x7f3584055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35840539b0 0x7f3584055e70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f359c00b000 tx=0x7f359c009040).stop 2022-07-12T08:48:07.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 msgr2=0x7f35a4099230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:07.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 0x7f35a4099230 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 rx=0x7f359400c000 tx=0x7f3594006df0).stop 2022-07-12T08:48:07.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 shutdown_connections 2022-07-12T08:48:07.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35a4097210 0x7f35a4099230 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35a4142f40 0x7f35a414fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35840539b0 0x7f3584055e70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 --2- 172.21.15.61:0/835758814 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35a40b7b00 0x7f35a40aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:07.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 >> 172.21.15.61:0/835758814 conn(0x7f35a4065550 msgr2=0x7f35a4065a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:07.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 shutdown_connections 2022-07-12T08:48:07.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:07.939+0000 7f35a9dbd700 1 -- 172.21.15.61:0/835758814 wait complete. 2022-07-12T08:48:07.948 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-12T08:48:07.955 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:48:07.955 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:48:07.956 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:48:07.957 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:48:07.957 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:48:07.958 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:48:07.959 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:48:07.959 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:48:07.979 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:48:07.980 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:48:07.981 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:48:07.982 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:48:07.987 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:48:07.988 DEBUG:teuthology.parallel:result is None 2022-07-12T08:48:08.021 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:48:08.022 DEBUG:teuthology.parallel:result is None 2022-07-12T08:48:08.023 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:48:08.058 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:48:08.097 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:48:08.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 Processor -- start 2022-07-12T08:48:08.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 -- start start 2022-07-12T08:48:08.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8eec0af280 0x7f8eec060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:08.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8eec09bce0 0x7f8eec0af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:08.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eec0aecb0 con 0x7f8eec0b3f60 2022-07-12T08:48:08.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eec143b00 con 0x7f8eec09bce0 2022-07-12T08:48:08.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8eebfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eec1435c0 con 0x7f8eec0af280 2022-07-12T08:48:08.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.131+0000 7f8ee3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8eec09bce0 0x7f8eec0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:08.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8eec09bce0 0x7f8eec0af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57810/0 (socket says 172.21.15.61:57810) 2022-07-12T08:48:08.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 -- 172.21.15.61:0/1704538648 learned_addr learned my addr 172.21.15.61:0/1704538648 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:08.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eeaffd700 1 --2- 172.21.15.61:0/1704538648 >> v2:172.21.15.111:3300/0 conn(0x7f8eec0af280 0x7f8eec060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:08.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 -- 172.21.15.61:0/1704538648 >> v2:172.21.15.111:3300/0 conn(0x7f8eec0af280 msgr2=0x7f8eec060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:08.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 --2- 172.21.15.61:0/1704538648 >> v2:172.21.15.111:3300/0 conn(0x7f8eec0af280 0x7f8eec060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:08.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 -- 172.21.15.61:0/1704538648 >> v1:172.21.15.111:6789/0 conn(0x7f8eec0b3f60 legacy=0x7f8eec0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:08.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eea7fc700 1 -- 172.21.15.61:0/1704538648 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248665056 0 0) 0x7f8eec0aecb0 con 0x7f8eec0b3f60 2022-07-12T08:48:08.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 -- 172.21.15.61:0/1704538648 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec143860 con 0x7f8eec09bce0 2022-07-12T08:48:08.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8ee3fff700 1 --2- 172.21.15.61:0/1704538648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec09bce0 0x7f8eec0af660 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f8ed4002f70 tx=0x7f8ed4006eb0).ready entity=mon.0 client_cookie=6edbd7790ad0edef server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:08.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eea7fc700 1 -- 172.21.15.61:0/1704538648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed400d5c0 con 0x7f8eec09bce0 2022-07-12T08:48:08.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eea7fc700 1 -- 172.21.15.61:0/1704538648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed4000e90 con 0x7f8eec09bce0 2022-07-12T08:48:08.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eea7fc700 1 -- 172.21.15.61:0/1704538648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ed400d910 con 0x7f8eec09bce0 2022-07-12T08:48:08.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 -- 172.21.15.61:0/1704538648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec09bce0 msgr2=0x7f8eec0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:08.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 --2- 172.21.15.61:0/1704538648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec09bce0 0x7f8eec0af660 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f8ed4002f70 tx=0x7f8ed4006eb0).stop 2022-07-12T08:48:08.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 -- 172.21.15.61:0/1704538648 shutdown_connections 2022-07-12T08:48:08.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 --2- 172.21.15.61:0/1704538648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec09bce0 0x7f8eec0af660 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:08.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 --2- 172.21.15.61:0/1704538648 >> v2:172.21.15.111:3300/0 conn(0x7f8eec0af280 0x7f8eec060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:08.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 -- 172.21.15.61:0/1704538648 >> 172.21.15.61:0/1704538648 conn(0x7f8eec0b92c0 msgr2=0x7f8eec0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:08.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 -- 172.21.15.61:0/1704538648 shutdown_connections 2022-07-12T08:48:08.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.135+0000 7f8eebfff700 1 -- 172.21.15.61:0/1704538648 wait complete. 2022-07-12T08:48:08.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 Processor -- start 2022-07-12T08:48:08.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 -- start start 2022-07-12T08:48:08.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8eec09bce0 0x7f8eec142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:08.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eec0af280 0x7f8eec09c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:08.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 0x7f8eec0b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:08.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec09ee80 con 0x7f8eec0b3f60 2022-07-12T08:48:08.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec09d500 con 0x7f8eec0af280 2022-07-12T08:48:08.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eec09e2c0 con 0x7f8eec09bce0 2022-07-12T08:48:08.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 0x7f8eec0b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:08.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeaffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8eec09bce0 0x7f8eec142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:08.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 0x7f8eec0b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57818/0 (socket says 172.21.15.61:57818) 2022-07-12T08:48:08.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 -- 172.21.15.61:0/3254465148 learned_addr learned my addr 172.21.15.61:0/3254465148 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:08.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8ee3fff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eec0af280 0x7f8eec09c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:08.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 -- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8eec09bce0 msgr2=0x7f8eec142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:08.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8eec09bce0 0x7f8eec142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:08.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 -- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eec0af280 msgr2=0x7f8eec09c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:08.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eec0af280 0x7f8eec09c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:08.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 -- 172.21.15.61:0/3254465148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed400b1e0 con 0x7f8eec0b3f60 2022-07-12T08:48:08.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eeb7fe700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 0x7f8eec0b2ce0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f8ee4000f40 tx=0x7f8ee4007ec0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:08.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee400f470 con 0x7f8eec0b3f60 2022-07-12T08:48:08.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee400f5d0 con 0x7f8eec0b3f60 2022-07-12T08:48:08.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eec09f0b0 con 0x7f8eec0b3f60 2022-07-12T08:48:08.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ee400f930 con 0x7f8eec0b3f60 2022-07-12T08:48:08.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.139+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eec0b0700 con 0x7f8eec0b3f60 2022-07-12T08:48:08.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.143+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eec057460 con 0x7f8eec0b3f60 2022-07-12T08:48:08.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.143+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8ee4019030 con 0x7f8eec0b3f60 2022-07-12T08:48:08.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.143+0000 7f8ee8ff9700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ed0053920 0x7f8ed0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:08.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.143+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f8ee406b040 con 0x7f8eec0b3f60 2022-07-12T08:48:08.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.143+0000 7f8eeaffd700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ed0053920 0x7f8ed0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:08.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.143+0000 7f8eeaffd700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ed0053920 0x7f8ed0055de0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8edc006000 tx=0x7f8edc006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:08.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.151+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ee4042300 con 0x7f8eec0b3f60 2022-07-12T08:48:08.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:08.291+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f8eec06b2b0 con 0x7f8eec0b3f60 2022-07-12T08:48:09.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.235+0000 7f8ee8ff9700 1 -- 172.21.15.61:0/3254465148 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v77) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8ee4017030 con 0x7f8eec0b3f60 2022-07-12T08:48:09.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ed0053920 msgr2=0x7f8ed0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ed0053920 0x7f8ed0055de0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f8edc006000 tx=0x7f8edc006d90).stop 2022-07-12T08:48:09.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 msgr2=0x7f8eec0b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 0x7f8eec0b2ce0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f8ee4000f40 tx=0x7f8ee4007ec0).stop 2022-07-12T08:48:09.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 shutdown_connections 2022-07-12T08:48:09.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8eec0b3f60 0x7f8eec0b2ce0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8ed0053920 0x7f8ed0055de0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8eec0af280 0x7f8eec09c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 --2- 172.21.15.61:0/3254465148 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8eec09bce0 0x7f8eec142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 >> 172.21.15.61:0/3254465148 conn(0x7f8eec0b92c0 msgr2=0x7f8eec0b80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:09.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 shutdown_connections 2022-07-12T08:48:09.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.239+0000 7f8eebfff700 1 -- 172.21.15.61:0/3254465148 wait complete. 2022-07-12T08:48:09.251 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:48:09.260 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:48:09.261 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:48:09.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 Processor -- start 2022-07-12T08:48:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- start start 2022-07-12T08:48:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 0x7fd6ac0b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 0x7fd6ac1474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ac05d3d0 con 0x7fd6ac0b5aa0 2022-07-12T08:48:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ac14b590 con 0x7fd6ac05bfc0 2022-07-12T08:48:09.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0b5e80 con 0x7fd6ac147000 2022-07-12T08:48:09.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b0ace700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 0x7fd6ac0b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 0x7fd6ac1474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 0x7fd6ac1474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40022/0 (socket says 172.21.15.61:40022) 2022-07-12T08:48:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b0ace700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 0x7fd6ac0b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36396/0 (socket says 172.21.15.61:36396) 2022-07-12T08:48:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b12cf700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fd6ac0b5aa0 0x7fd6ac14ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59642/0 (socket says 172.21.15.61:59642) 2022-07-12T08:48:09.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/3738776100 learned_addr learned my addr 172.21.15.61:0/3738776100 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:09.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/3738776100 >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 msgr2=0x7fd6ac0b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 --2- 172.21.15.61:0/3738776100 >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 0x7fd6ac0b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/3738776100 >> v1:172.21.15.111:6789/0 conn(0x7fd6ac0b5aa0 legacy=0x7fd6ac14ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/3738776100 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac14a740 con 0x7fd6ac147000 2022-07-12T08:48:09.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b0ace700 1 --2- 172.21.15.61:0/3738776100 >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 0x7fd6ac0b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b1ad0700 1 --2- 172.21.15.61:0/3738776100 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 0x7fd6ac1474b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fd6a40032f0 tx=0x7fd6a4009290).ready entity=mon.2 client_cookie=4beabc47903457a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd69ffff700 1 -- 172.21.15.61:0/3738776100 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6a400b8a0 con 0x7fd6ac147000 2022-07-12T08:48:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd69ffff700 1 -- 172.21.15.61:0/3738776100 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a400ba00 con 0x7fd6ac147000 2022-07-12T08:48:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd69ffff700 1 -- 172.21.15.61:0/3738776100 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6a400bb70 con 0x7fd6ac147000 2022-07-12T08:48:09.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- 172.21.15.61:0/3738776100 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 msgr2=0x7fd6ac1474b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 --2- 172.21.15.61:0/3738776100 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 0x7fd6ac1474b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fd6a40032f0 tx=0x7fd6a4009290).stop 2022-07-12T08:48:09.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- 172.21.15.61:0/3738776100 shutdown_connections 2022-07-12T08:48:09.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 --2- 172.21.15.61:0/3738776100 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac147000 0x7fd6ac1474b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 --2- 172.21.15.61:0/3738776100 >> v2:172.21.15.111:3300/0 conn(0x7fd6ac05bfc0 0x7fd6ac0b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- 172.21.15.61:0/3738776100 >> 172.21.15.61:0/3738776100 conn(0x7fd6ac063520 msgr2=0x7fd6ac0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:09.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.371+0000 7fd6b3533700 1 -- 172.21.15.61:0/3738776100 shutdown_connections 2022-07-12T08:48:09.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 -- 172.21.15.61:0/3738776100 wait complete. 2022-07-12T08:48:09.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 Processor -- start 2022-07-12T08:48:09.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 -- start start 2022-07-12T08:48:09.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac05bfc0 0x7fd6ac09e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0b5aa0 0x7fd6ac159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 0x7fd6ac159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0a0b00 con 0x7fd6ac147000 2022-07-12T08:48:09.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0a0d60 con 0x7fd6ac0b5aa0 2022-07-12T08:48:09.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b3533700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac0654c0 con 0x7fd6ac05bfc0 2022-07-12T08:48:09.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 0x7fd6ac159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b0ace700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0b5aa0 0x7fd6ac159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 0x7fd6ac159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57832/0 (socket says 172.21.15.61:57832) 2022-07-12T08:48:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/690437342 learned_addr learned my addr 172.21.15.61:0/690437342 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b12cf700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac05bfc0 0x7fd6ac09e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac05bfc0 msgr2=0x7fd6ac09e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac05bfc0 0x7fd6ac09e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0b5aa0 msgr2=0x7fd6ac159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0b5aa0 0x7fd6ac159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 -- 172.21.15.61:0/690437342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a000b180 con 0x7fd6ac147000 2022-07-12T08:48:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.375+0000 7fd6b1ad0700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 0x7fd6ac159f40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7fd6a40032f0 tx=0x7fd6a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6a400b8a0 con 0x7fd6ac147000 2022-07-12T08:48:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4021440 con 0x7fd6ac147000 2022-07-12T08:48:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a4009530 con 0x7fd6ac147000 2022-07-12T08:48:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6a40217a0 con 0x7fd6ac147000 2022-07-12T08:48:09.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac0674d0 con 0x7fd6ac147000 2022-07-12T08:48:09.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd6a4027020 con 0x7fd6ac147000 2022-07-12T08:48:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac09ed60 con 0x7fd6ac147000 2022-07-12T08:48:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd69e7fc700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd694053a40 0x7fd694055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd6a406cff0 con 0x7fd6ac147000 2022-07-12T08:48:09.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd6b12cf700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd694053a40 0x7fd694055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.379+0000 7fd6b12cf700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd694053a40 0x7fd694055f00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd6a800d000 tx=0x7fd6a8007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:09.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.387+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd6a4044260 con 0x7fd6ac147000 2022-07-12T08:48:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.555+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd6ac0a0ed0 con 0x7fd6ac147000 2022-07-12T08:48:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.555+0000 7fd69e7fc700 1 -- 172.21.15.61:0/690437342 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fd6a4015020 con 0x7fd6ac147000 2022-07-12T08:48:09.559 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:09.559 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:09.560 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:48:09.561 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.562 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:48:09.562 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:48:09.562 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.563 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:09.563 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.564 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.564 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.564 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.565 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.565 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.565 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:48:09.566 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:48:09.567 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.567 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:09.568 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.569 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.569 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.570 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.570 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.571 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.571 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:48:09.572 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:48:09.572 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.574 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.574 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.576 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.576 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:48:09.577 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:48:09.577 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.579 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.579 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.580 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.580 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:48:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:48:09.582 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.582 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.583 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.583 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.583 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:48:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:48:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.586 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.588 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.588 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.589 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.590 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:48:09.590 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:48:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.593 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.593 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:48:09.593 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:48:09.593 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.594 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.594 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.594 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.594 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.595 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.595 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.595 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:48:09.595 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:48:09.596 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.596 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.596 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.596 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.596 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.597 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.597 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.597 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:48:09.597 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:48:09.598 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.598 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.598 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.598 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.598 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.599 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.599 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.599 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:48:09.600 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:48:09.600 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.601 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:09.601 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:09.601 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:09.601 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.602 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.602 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.602 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:48:09.602 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:48:09.603 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.603 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:09.603 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:09.603 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:09.604 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.604 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.604 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.604 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:48:09.604 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:48:09.605 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.605 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:09.605 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:09.605 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:09.606 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.606 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.606 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.606 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:48:09.607 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:48:09.607 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.607 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:09.607 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:09.607 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:09.608 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.608 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.608 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.608 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.609 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:48:09.609 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:48:09.609 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.609 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:09.610 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:48:09.610 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.610 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.610 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:09.610 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:09.611 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:48:09.611 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:48:09.611 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:09.611 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:09.612 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:48:09.612 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:09.612 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.612 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:09.612 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:09.613 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:09.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd694053a40 msgr2=0x7fd694055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd694053a40 0x7fd694055f00 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd6a800d000 tx=0x7fd6a8007b80).stop 2022-07-12T08:48:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 msgr2=0x7fd6ac159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 0x7fd6ac159f40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7fd6a40032f0 tx=0x7fd6a4016000).stop 2022-07-12T08:48:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 shutdown_connections 2022-07-12T08:48:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd6ac147000 0x7fd6ac159f40 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd694053a40 0x7fd694055f00 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd6ac0b5aa0 0x7fd6ac159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 --2- 172.21.15.61:0/690437342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd6ac05bfc0 0x7fd6ac09e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 >> 172.21.15.61:0/690437342 conn(0x7fd6ac063520 msgr2=0x7fd6ac148df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 shutdown_connections 2022-07-12T08:48:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.559+0000 7fd6b3533700 1 -- 172.21.15.61:0/690437342 wait complete. 2022-07-12T08:48:09.619 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 Processor -- start 2022-07-12T08:48:09.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 -- start start 2022-07-12T08:48:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07b80628e0 0x7f07b80b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 0x7f07b81433d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b80b7ee0 con 0x7f07b80b7b00 2022-07-12T08:48:09.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80acf80 con 0x7f07b80628e0 2022-07-12T08:48:09.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bedc3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80ae2e0 con 0x7f07b8142f40 2022-07-12T08:48:09.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bd5c0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07b80628e0 0x7f07b80b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07be5c2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 0x7f07b81433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bddc1700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f07b80b7b00 0x7f07b80ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59656/0 (socket says 172.21.15.61:59656) 2022-07-12T08:48:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bddc1700 1 -- 172.21.15.61:0/1656485154 learned_addr learned my addr 172.21.15.61:0/1656485154 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07bd5c0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07b80628e0 0x7f07b80b1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57840/0 (socket says 172.21.15.61:57840) 2022-07-12T08:48:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.723+0000 7f07be5c2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 0x7f07b81433d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40034/0 (socket says 172.21.15.61:40034) 2022-07-12T08:48:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bcdbf700 1 -- 172.21.15.61:0/1656485154 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481433214 0 0) 0x7f07b80b7ee0 con 0x7f07b80b7b00 2022-07-12T08:48:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bcdbf700 1 -- 172.21.15.61:0/1656485154 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a4003260 con 0x7f07b80b7b00 2022-07-12T08:48:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07be5c2700 1 -- 172.21.15.61:0/1656485154 >> v2:172.21.15.61:3300/0 conn(0x7f07b80628e0 msgr2=0x7f07b80b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07be5c2700 1 --2- 172.21.15.61:0/1656485154 >> v2:172.21.15.61:3300/0 conn(0x7f07b80628e0 0x7f07b80b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07be5c2700 1 -- 172.21.15.61:0/1656485154 >> v1:172.21.15.111:6789/0 conn(0x7f07b80b7b00 legacy=0x7f07b80ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07be5c2700 1 -- 172.21.15.61:0/1656485154 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b80b2d30 con 0x7f07b8142f40 2022-07-12T08:48:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07be5c2700 1 --2- 172.21.15.61:0/1656485154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 0x7f07b81433d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f07b0002f70 tx=0x7f07b0006df0).ready entity=mon.2 client_cookie=13d54f4be33ef19 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bcdbf700 1 -- 172.21.15.61:0/1656485154 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07b000d4a0 con 0x7f07b8142f40 2022-07-12T08:48:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bcdbf700 1 -- 172.21.15.61:0/1656485154 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0000e90 con 0x7f07b8142f40 2022-07-12T08:48:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bcdbf700 1 -- 172.21.15.61:0/1656485154 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07b000d7f0 con 0x7f07b8142f40 2022-07-12T08:48:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 -- 172.21.15.61:0/1656485154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 msgr2=0x7f07b81433d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 --2- 172.21.15.61:0/1656485154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 0x7f07b81433d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f07b0002f70 tx=0x7f07b0006df0).stop 2022-07-12T08:48:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 -- 172.21.15.61:0/1656485154 shutdown_connections 2022-07-12T08:48:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 --2- 172.21.15.61:0/1656485154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b8142f40 0x7f07b81433d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 --2- 172.21.15.61:0/1656485154 >> v2:172.21.15.61:3300/0 conn(0x7f07b80628e0 0x7f07b80b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 -- 172.21.15.61:0/1656485154 >> 172.21.15.61:0/1656485154 conn(0x7f07b8065550 msgr2=0x7f07b80b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 -- 172.21.15.61:0/1656485154 shutdown_connections 2022-07-12T08:48:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 -- 172.21.15.61:0/1656485154 wait complete. 2022-07-12T08:48:09.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 Processor -- start 2022-07-12T08:48:09.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.727+0000 7f07bedc3700 1 -- start start 2022-07-12T08:48:09.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b80628e0 0x7f07b80ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b80b7b00 0x7f07b8152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 0x7f07b81525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b81517d0 con 0x7f07b8142f40 2022-07-12T08:48:09.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b81519c0 con 0x7f07b80628e0 2022-07-12T08:48:09.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b8151c60 con 0x7f07b80b7b00 2022-07-12T08:48:09.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 0x7f07b81525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 0x7f07b81525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57846/0 (socket says 172.21.15.61:57846) 2022-07-12T08:48:09.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 -- 172.21.15.61:0/967252593 learned_addr learned my addr 172.21.15.61:0/967252593 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:09.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bddc1700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b80628e0 0x7f07b80ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bd5c0700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b80b7b00 0x7f07b8152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 -- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b80b7b00 msgr2=0x7f07b8152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b80b7b00 0x7f07b8152100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 -- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b80628e0 msgr2=0x7f07b80ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b80628e0 0x7f07b80ae890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 -- 172.21.15.61:0/967252593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b000b180 con 0x7f07b8142f40 2022-07-12T08:48:09.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bd5c0700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b80b7b00 0x7f07b8152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:09.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07be5c2700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 0x7f07b81525f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f07b0002f70 tx=0x7f07b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:09.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07b0016070 con 0x7f07b8142f40 2022-07-12T08:48:09.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b00139c0 con 0x7f07b8142f40 2022-07-12T08:48:09.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b8060eb0 con 0x7f07b8142f40 2022-07-12T08:48:09.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07b0013d20 con 0x7f07b8142f40 2022-07-12T08:48:09.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b80b0af0 con 0x7f07b8142f40 2022-07-12T08:48:09.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b80b12a0 con 0x7f07b8142f40 2022-07-12T08:48:09.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.731+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f07b0025030 con 0x7f07b8142f40 2022-07-12T08:48:09.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.735+0000 7f07b6ffd700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0798053ad0 0x7f0798055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:09.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.735+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f07b006cbe0 con 0x7f07b8142f40 2022-07-12T08:48:09.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.735+0000 7f07bddc1700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0798053ad0 0x7f0798055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:09.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.735+0000 7f07bddc1700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0798053ad0 0x7f0798055f90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f07a800b000 tx=0x7f07a8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:09.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.739+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07b80b12a0 con 0x7f07b8142f40 2022-07-12T08:48:09.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.867+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07b80b12a0 con 0x7f07b8142f40 2022-07-12T08:48:09.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07b6ffd700 1 -- 172.21.15.61:0/967252593 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13230 (secure 0 0 0) 0x7f07b80b12a0 con 0x7f07b8142f40 2022-07-12T08:48:09.877 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:09.877 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:09.223488+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:09.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0798053ad0 msgr2=0x7f0798055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0798053ad0 0x7f0798055f90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f07a800b000 tx=0x7f07a8009040).stop 2022-07-12T08:48:09.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 msgr2=0x7f07b81525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:09.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 0x7f07b81525f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f07b0002f70 tx=0x7f07b0014000).stop 2022-07-12T08:48:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 shutdown_connections 2022-07-12T08:48:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07b8142f40 0x7f07b81525f0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f07b80b7b00 0x7f07b8152100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0798053ad0 0x7f0798055f90 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 --2- 172.21.15.61:0/967252593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07b80628e0 0x7f07b80ae890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:09.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 >> 172.21.15.61:0/967252593 conn(0x7f07b8065550 msgr2=0x7f07b8066b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:09.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 shutdown_connections 2022-07-12T08:48:09.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:09.871+0000 7f07bedc3700 1 -- 172.21.15.61:0/967252593 wait complete. 2022-07-12T08:48:09.894 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:48:09.895 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:10.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 Processor -- start 2022-07-12T08:48:10.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- start start 2022-07-12T08:48:10.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d805be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7d814a650 0x7fd7d8146460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d80b5e80 con 0x7fd7d8146950 2022-07-12T08:48:10.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d805d2d0 con 0x7fd7d814a650 2022-07-12T08:48:10.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d80b05a0 con 0x7fd7d80b5aa0 2022-07-12T08:48:10.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7d814a650 0x7fd7d8146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7d814a650 0x7fd7d8146460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57850/0 (socket says 172.21.15.61:57850) 2022-07-12T08:48:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/1870308790 learned_addr learned my addr 172.21.15.61:0/1870308790 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:10.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d77fe700 1 --2- 172.21.15.61:0/1870308790 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d805be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/1870308790 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 msgr2=0x7fd7d805be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:10.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 --2- 172.21.15.61:0/1870308790 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d805be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/1870308790 >> v1:172.21.15.111:6789/0 conn(0x7fd7d8146950 legacy=0x7fd7d80ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/1870308790 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d80b0840 con 0x7fd7d814a650 2022-07-12T08:48:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d6ffd700 1 --2- 172.21.15.61:0/1870308790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d814a650 0x7fd7d8146460 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7fd7c8002f70 tx=0x7fd7c8006eb0).ready entity=mon.0 client_cookie=af3dee3158ccefc1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d67fc700 1 -- 172.21.15.61:0/1870308790 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c800d5c0 con 0x7fd7d814a650 2022-07-12T08:48:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d67fc700 1 -- 172.21.15.61:0/1870308790 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c8000e90 con 0x7fd7d814a650 2022-07-12T08:48:10.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7d67fc700 1 -- 172.21.15.61:0/1870308790 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c800d950 con 0x7fd7d814a650 2022-07-12T08:48:10.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- 172.21.15.61:0/1870308790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d814a650 msgr2=0x7fd7d8146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/1870308790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d814a650 0x7fd7d8146460 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7fd7c8002f70 tx=0x7fd7c8006eb0).stop 2022-07-12T08:48:10.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- 172.21.15.61:0/1870308790 shutdown_connections 2022-07-12T08:48:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/1870308790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d814a650 0x7fd7d8146460 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/1870308790 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d805be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- 172.21.15.61:0/1870308790 >> 172.21.15.61:0/1870308790 conn(0x7fd7d80634d0 msgr2=0x7fd7d80b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- 172.21.15.61:0/1870308790 shutdown_connections 2022-07-12T08:48:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.003+0000 7fd7ddb71700 1 -- 172.21.15.61:0/1870308790 wait complete. 2022-07-12T08:48:10.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 Processor -- start 2022-07-12T08:48:10.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 -- start start 2022-07-12T08:48:10.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d815a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 0x7fd7d809f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7d814a650 0x7fd7d8149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d80ad590 con 0x7fd7d8146950 2022-07-12T08:48:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d814a240 con 0x7fd7d814a650 2022-07-12T08:48:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d814a3b0 con 0x7fd7d80b5aa0 2022-07-12T08:48:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 0x7fd7d809f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 0x7fd7d809f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57858/0 (socket says 172.21.15.61:57858) 2022-07-12T08:48:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/202134334 learned_addr learned my addr 172.21.15.61:0/202134334 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d7fff700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7d814a650 0x7fd7d8149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d77fe700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d815a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 msgr2=0x7fd7d815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d815a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7d814a650 msgr2=0x7fd7d8149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7d814a650 0x7fd7d8149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 -- 172.21.15.61:0/202134334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c800b1e0 con 0x7fd7d8146950 2022-07-12T08:48:10.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d77fe700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:10.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d6ffd700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 0x7fd7d809f6b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fd7c8002f70 tx=0x7fd7c8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c8011860 con 0x7fd7d8146950 2022-07-12T08:48:10.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c80119c0 con 0x7fd7d8146950 2022-07-12T08:48:10.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d809e100 con 0x7fd7d8146950 2022-07-12T08:48:10.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c8011d20 con 0x7fd7d8146950 2022-07-12T08:48:10.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.007+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d80a1ee0 con 0x7fd7d8146950 2022-07-12T08:48:10.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.011+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d8057460 con 0x7fd7d8146950 2022-07-12T08:48:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.011+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd7c8025030 con 0x7fd7d8146950 2022-07-12T08:48:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.011+0000 7fd7d4ff9700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7b80539b0 0x7fd7b8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.011+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd7c806cda0 con 0x7fd7d8146950 2022-07-12T08:48:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.011+0000 7fd7d77fe700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7b80539b0 0x7fd7b8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.011+0000 7fd7d77fe700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7b80539b0 0x7fd7b8055e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fd7d000d000 tx=0x7fd7d0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.015+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd7c8044030 con 0x7fd7d8146950 2022-07-12T08:48:10.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd7d80af610 con 0x7fd7d8146950 2022-07-12T08:48:10.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7d4ff9700 1 -- 172.21.15.61:0/202134334 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v77) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd7c8023070 con 0x7fd7d8146950 2022-07-12T08:48:10.142 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:10.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7b80539b0 msgr2=0x7fd7b8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7b80539b0 0x7fd7b8055e70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fd7d000d000 tx=0x7fd7d0008040).stop 2022-07-12T08:48:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 msgr2=0x7fd7d809f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 0x7fd7d809f6b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fd7c8002f70 tx=0x7fd7c8015000).stop 2022-07-12T08:48:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 shutdown_connections 2022-07-12T08:48:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7b80539b0 0x7fd7b8055e70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7d814a650 0x7fd7d8149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7d8146950 0x7fd7d809f6b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 --2- 172.21.15.61:0/202134334 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd7d80b5aa0 0x7fd7d815a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 >> 172.21.15.61:0/202134334 conn(0x7fd7d80634d0 msgr2=0x7fd7d80b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 shutdown_connections 2022-07-12T08:48:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.139+0000 7fd7ddb71700 1 -- 172.21.15.61:0/202134334 wait complete. 2022-07-12T08:48:10.156 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 Processor -- start 2022-07-12T08:48:10.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 -- start start 2022-07-12T08:48:10.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f87440b2010 0x7f87440b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8744143d80 0x7f874405ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87441441d0 con 0x7f87440b7810 2022-07-12T08:48:10.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874405bd10 con 0x7f8744143d80 2022-07-12T08:48:10.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749949700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f874405bfb0 con 0x7f87440b2010 2022-07-12T08:48:10.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8749148700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8744143d80 0x7f874405ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8748947700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f87440b7810 0x7f874413a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46082/0 (socket says 172.21.15.61:46082) 2022-07-12T08:48:10.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.259+0000 7f8748947700 1 -- 172.21.15.61:0/3934333561 learned_addr learned my addr 172.21.15.61:0/3934333561 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8743fff700 1 --2- 172.21.15.61:0/3934333561 >> v2:172.21.15.111:3300/0 conn(0x7f87440b2010 0x7f87440b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749148700 1 -- 172.21.15.61:0/3934333561 >> v2:172.21.15.111:3300/0 conn(0x7f87440b2010 msgr2=0x7f87440b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749148700 1 --2- 172.21.15.61:0/3934333561 >> v2:172.21.15.111:3300/0 conn(0x7f87440b2010 0x7f87440b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749148700 1 -- 172.21.15.61:0/3934333561 >> v1:172.21.15.61:6789/0 conn(0x7f87440b7810 legacy=0x7f874413a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f87437fe700 1 -- 172.21.15.61:0/3934333561 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096696624 0 0) 0x7f87441441d0 con 0x7f87440b7810 2022-07-12T08:48:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749148700 1 -- 172.21.15.61:0/3934333561 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f874405c250 con 0x7f8744143d80 2022-07-12T08:48:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749148700 1 --2- 172.21.15.61:0/3934333561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8744143d80 0x7f874405ca70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f873c0032f0 tx=0x7f873c0091d0).ready entity=mon.0 client_cookie=175b982a1f57d3c0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f87437fe700 1 -- 172.21.15.61:0/3934333561 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873c00b680 con 0x7f8744143d80 2022-07-12T08:48:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f87437fe700 1 -- 172.21.15.61:0/3934333561 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c00b7e0 con 0x7f8744143d80 2022-07-12T08:48:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f87437fe700 1 -- 172.21.15.61:0/3934333561 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f873c00b950 con 0x7f8744143d80 2022-07-12T08:48:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 -- 172.21.15.61:0/3934333561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8744143d80 msgr2=0x7f874405ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 --2- 172.21.15.61:0/3934333561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8744143d80 0x7f874405ca70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f873c0032f0 tx=0x7f873c0091d0).stop 2022-07-12T08:48:10.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 -- 172.21.15.61:0/3934333561 shutdown_connections 2022-07-12T08:48:10.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 --2- 172.21.15.61:0/3934333561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8744143d80 0x7f874405ca70 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 --2- 172.21.15.61:0/3934333561 >> v2:172.21.15.111:3300/0 conn(0x7f87440b2010 0x7f87440b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 -- 172.21.15.61:0/3934333561 >> 172.21.15.61:0/3934333561 conn(0x7f8744065650 msgr2=0x7f87440b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 -- 172.21.15.61:0/3934333561 shutdown_connections 2022-07-12T08:48:10.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.263+0000 7f8749949700 1 -- 172.21.15.61:0/3934333561 wait complete. 2022-07-12T08:48:10.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 Processor -- start 2022-07-12T08:48:10.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 -- start start 2022-07-12T08:48:10.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 0x7f87440af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87440b7810 0x7f8744147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8744143d80 0x7f8744153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874413b040 con 0x7f87440b2010 2022-07-12T08:48:10.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874413b2e0 con 0x7f8744143d80 2022-07-12T08:48:10.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749949700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8744152d70 con 0x7f87440b7810 2022-07-12T08:48:10.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 0x7f87440af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 0x7f87440af580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57868/0 (socket says 172.21.15.61:57868) 2022-07-12T08:48:10.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 -- 172.21.15.61:0/2060974690 learned_addr learned my addr 172.21.15.61:0/2060974690 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:10.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8743fff700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87440b7810 0x7f8744147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749148700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8744143d80 0x7f8744153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 -- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87440b7810 msgr2=0x7f8744147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87440b7810 0x7f8744147430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 -- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8744143d80 msgr2=0x7f8744153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8744143d80 0x7f8744153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 -- 172.21.15.61:0/2060974690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c009470 con 0x7f87440b2010 2022-07-12T08:48:10.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8749148700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8744143d80 0x7f8744153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:10.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8743fff700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87440b7810 0x7f8744147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:10.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.267+0000 7f8748947700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 0x7f87440af580 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f8734000bc0 tx=0x7f8734007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87340072e0 con 0x7f87440b2010 2022-07-12T08:48:10.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874413bac0 con 0x7f87440b2010 2022-07-12T08:48:10.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8734007440 con 0x7f87440b2010 2022-07-12T08:48:10.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8734005800 con 0x7f87440b2010 2022-07-12T08:48:10.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8744153700 con 0x7f87440b2010 2022-07-12T08:48:10.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874413aea0 con 0x7f87440b2010 2022-07-12T08:48:10.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8734010070 con 0x7f87440b2010 2022-07-12T08:48:10.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8741ffb700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f871c053a40 0x7f871c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f873406b450 con 0x7f87440b2010 2022-07-12T08:48:10.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.271+0000 7f8743fff700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f871c053a40 0x7f871c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.275+0000 7f8743fff700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f871c053a40 0x7f871c055f00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f872c00b000 tx=0x7f872c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.279+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f874413aea0 con 0x7f87440b2010 2022-07-12T08:48:10.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.403+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f874413aea0 con 0x7f87440b2010 2022-07-12T08:48:10.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.403+0000 7f8741ffb700 1 -- 172.21.15.61:0/2060974690 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v77) v1 ==== 92+0+10 (secure 0 0 0) 0x7f874413aea0 con 0x7f87440b2010 2022-07-12T08:48:10.407 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:10.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f871c053a40 msgr2=0x7f871c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f871c053a40 0x7f871c055f00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f872c00b000 tx=0x7f872c008040).stop 2022-07-12T08:48:10.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 msgr2=0x7f87440af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 0x7f87440af580 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f8734000bc0 tx=0x7f8734007af0).stop 2022-07-12T08:48:10.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 shutdown_connections 2022-07-12T08:48:10.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f871c053a40 0x7f871c055f00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8744143d80 0x7f8744153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f87440b7810 0x7f8744147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 --2- 172.21.15.61:0/2060974690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f87440b2010 0x7f87440af580 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 >> 172.21.15.61:0/2060974690 conn(0x7f8744065650 msgr2=0x7f874413a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 shutdown_connections 2022-07-12T08:48:10.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.407+0000 7f8749949700 1 -- 172.21.15.61:0/2060974690 wait complete. 2022-07-12T08:48:10.420 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:48:10.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.519+0000 7f2b7ccf4700 1 Processor -- start 2022-07-12T08:48:10.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.519+0000 7f2b7ccf4700 1 -- start start 2022-07-12T08:48:10.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b780b4050 0x7f2b780b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b780b56c0 con 0x7f2b78141e50 2022-07-12T08:48:10.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b780b4bd0 con 0x7f2b780b52e0 2022-07-12T08:48:10.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b78142f00 con 0x7f2b780b4050 2022-07-12T08:48:10.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b76d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2b78141e50 0x7f2b781422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46098/0 (socket says 172.21.15.61:46098) 2022-07-12T08:48:10.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b75d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b780b4050 0x7f2b780b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b76d9d700 1 -- 172.21.15.61:0/3398306050 learned_addr learned my addr 172.21.15.61:0/3398306050 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:10.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3398306050 >> v1:172.21.15.111:6789/0 conn(0x7f2b780b52e0 legacy=0x7f2b780b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3398306050 >> v1:172.21.15.61:6789/0 conn(0x7f2b78141e50 legacy=0x7f2b781422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3398306050 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b780b0e80 con 0x7f2b780b4050 2022-07-12T08:48:10.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b75d9b700 1 --2- 172.21.15.61:0/3398306050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b4050 0x7f2b780b28c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f2b60002f70 tx=0x7f2b60006df0).ready entity=mon.0 client_cookie=5abc9b023dd879e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7559a700 1 -- 172.21.15.61:0/3398306050 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b6000d610 con 0x7f2b780b4050 2022-07-12T08:48:10.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7559a700 1 -- 172.21.15.61:0/3398306050 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b60000e90 con 0x7f2b780b4050 2022-07-12T08:48:10.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7559a700 1 -- 172.21.15.61:0/3398306050 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b6000d960 con 0x7f2b780b4050 2022-07-12T08:48:10.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3398306050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b4050 msgr2=0x7f2b780b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3398306050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b4050 0x7f2b780b28c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f2b60002f70 tx=0x7f2b60006df0).stop 2022-07-12T08:48:10.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3398306050 shutdown_connections 2022-07-12T08:48:10.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3398306050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b4050 0x7f2b780b28c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3398306050 >> 172.21.15.61:0/3398306050 conn(0x7f2b78067020 msgr2=0x7f2b78067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3398306050 shutdown_connections 2022-07-12T08:48:10.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.523+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3398306050 wait complete. 2022-07-12T08:48:10.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 Processor -- start 2022-07-12T08:48:10.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 -- start start 2022-07-12T08:48:10.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b780b4050 0x7f2b78155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 0x7f2b780aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b78141e50 0x7f2b78114240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b78113e50 con 0x7f2b780b52e0 2022-07-12T08:48:10.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b780ade90 con 0x7f2b78141e50 2022-07-12T08:48:10.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b780ae130 con 0x7f2b780b4050 2022-07-12T08:48:10.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 0x7f2b780aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 0x7f2b780aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57884/0 (socket says 172.21.15.61:57884) 2022-07-12T08:48:10.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3351716748 learned_addr learned my addr 172.21.15.61:0/3351716748 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:10.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b76d9d700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b78141e50 0x7f2b78114240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7659c700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b780b4050 0x7f2b78155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b780b4050 msgr2=0x7f2b78155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b780b4050 0x7f2b78155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b78141e50 msgr2=0x7f2b78114240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b78141e50 0x7f2b78114240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 -- 172.21.15.61:0/3351716748 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6000b180 con 0x7f2b780b52e0 2022-07-12T08:48:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7659c700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b780b4050 0x7f2b78155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b75d9b700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 0x7f2b780aefb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f2b60002f70 tx=0x7f2b60015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b6000de70 con 0x7f2b780b52e0 2022-07-12T08:48:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b60011ca0 con 0x7f2b780b52e0 2022-07-12T08:48:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b780ae3d0 con 0x7f2b780b52e0 2022-07-12T08:48:10.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b6001e460 con 0x7f2b780b52e0 2022-07-12T08:48:10.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.527+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b780af6c0 con 0x7f2b780b52e0 2022-07-12T08:48:10.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.531+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b780b00f0 con 0x7f2b780b52e0 2022-07-12T08:48:10.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.531+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b60025030 con 0x7f2b780b52e0 2022-07-12T08:48:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.531+0000 7f2b6f7fe700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b5c0539b0 0x7f2b5c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.531+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f2b6006d090 con 0x7f2b780b52e0 2022-07-12T08:48:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.531+0000 7f2b7659c700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b5c0539b0 0x7f2b5c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.531+0000 7f2b7659c700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b5c0539b0 0x7f2b5c055e70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f2b68008000 tx=0x7f2b68009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.539+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b780b00f0 con 0x7f2b780b52e0 2022-07-12T08:48:10.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.667+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2b780b00f0 con 0x7f2b780b52e0 2022-07-12T08:48:10.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.667+0000 7f2b6f7fe700 1 -- 172.21.15.61:0/3351716748 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+143 (secure 0 0 0) 0x7f2b780b00f0 con 0x7f2b780b52e0 2022-07-12T08:48:10.672 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:10.673 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:48:10.673 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:10.673 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:48:10.674 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:48:10.674 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:10.675 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:10.675 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:48:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:48:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:10.677 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:48:10.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b5c0539b0 msgr2=0x7f2b5c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b5c0539b0 0x7f2b5c055e70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f2b68008000 tx=0x7f2b68009040).stop 2022-07-12T08:48:10.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 msgr2=0x7f2b780aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 0x7f2b780aefb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f2b60002f70 tx=0x7f2b60015000).stop 2022-07-12T08:48:10.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 shutdown_connections 2022-07-12T08:48:10.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2b5c0539b0 0x7f2b5c055e70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b78141e50 0x7f2b78114240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b52e0 0x7f2b780aefb0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 --2- 172.21.15.61:0/3351716748 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b780b4050 0x7f2b78155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 >> 172.21.15.61:0/3351716748 conn(0x7f2b78067020 msgr2=0x7f2b781422c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 shutdown_connections 2022-07-12T08:48:10.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.671+0000 7f2b7ccf4700 1 -- 172.21.15.61:0/3351716748 wait complete. 2022-07-12T08:48:10.687 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:48:10.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:48:10.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.795+0000 7fc2c1718700 1 Processor -- start 2022-07-12T08:48:10.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.795+0000 7fc2c1718700 1 -- start start 2022-07-12T08:48:10.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2c1718700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2c1718700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0b3560 con 0x7fc2bc0b3730 2022-07-12T08:48:10.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2c1718700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc0b5b80 con 0x7fc2bc0b5480 2022-07-12T08:48:10.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2c1718700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b5e20 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2bb7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc2bc0b3730 0x7fc2bc142f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46110/0 (socket says 172.21.15.61:46110) 2022-07-12T08:48:10.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2bb7fe700 1 -- 172.21.15.61:0/3573008788 learned_addr learned my addr 172.21.15.61:0/3573008788 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:10.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2ba7fc700 1 --2- 172.21.15.61:0/3573008788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208749119 0 0) 0x7fc2bc0b3560 con 0x7fc2bc0b3730 2022-07-12T08:48:10.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc29c003260 con 0x7fc2bc0b3730 2022-07-12T08:48:10.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295390132 0 0) 0x7fc2bc0b5b80 con 0x7fc2bc0b5480 2022-07-12T08:48:10.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2bc0b3560 con 0x7fc2bc0b5480 2022-07-12T08:48:10.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2036461557 0 0) 0x7fc29c003260 con 0x7fc2bc0b3730 2022-07-12T08:48:10.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2ba7fc700 1 -- 172.21.15.61:0/3573008788 >> v1:172.21.15.111:6789/0 conn(0x7fc2bc0b5480 legacy=0x7fc2bc0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2ba7fc700 1 -- 172.21.15.61:0/3573008788 >> v1:172.21.15.61:6789/0 conn(0x7fc2bc0b3730 legacy=0x7fc2bc142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2ba7fc700 1 -- 172.21.15.61:0/3573008788 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0b5860 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2ba7fc700 1 --2- 172.21.15.61:0/3573008788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc0b45d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc2b4001110 tx=0x7fc2a4006df0).ready entity=mon.2 client_cookie=39232526002bece4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2b4001660 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a4000e90 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.799+0000 7fc2b9ffb700 1 -- 172.21.15.61:0/3573008788 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2a4012460 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- 172.21.15.61:0/3573008788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 msgr2=0x7fc2bc0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 --2- 172.21.15.61:0/3573008788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc0b45d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc2b4001110 tx=0x7fc2a4006df0).stop 2022-07-12T08:48:10.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- 172.21.15.61:0/3573008788 shutdown_connections 2022-07-12T08:48:10.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 --2- 172.21.15.61:0/3573008788 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc0b45d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- 172.21.15.61:0/3573008788 >> 172.21.15.61:0/3573008788 conn(0x7fc2bc0b96b0 msgr2=0x7fc2bc0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:10.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- 172.21.15.61:0/3573008788 shutdown_connections 2022-07-12T08:48:10.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- 172.21.15.61:0/3573008788 wait complete. 2022-07-12T08:48:10.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 Processor -- start 2022-07-12T08:48:10.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- start start 2022-07-12T08:48:10.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc147160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2bc0b3730 0x7fc2bc146560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2bc0b5480 0x7fc2bc13e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc149630 con 0x7fc2bc0b5480 2022-07-12T08:48:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc1450f0 con 0x7fc2bc0b3730 2022-07-12T08:48:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2c1718700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc145320 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2bc0b5480 0x7fc2bc13e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2baffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc147160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2ba7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2bc0b3730 0x7fc2bc146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2baffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc147160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40092/0 (socket says 172.21.15.61:40092) 2022-07-12T08:48:10.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.803+0000 7fc2baffd700 1 -- 172.21.15.61:0/4142051947 learned_addr learned my addr 172.21.15.61:0/4142051947 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:10.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2baffd700 1 -- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2bc0b3730 msgr2=0x7fc2bc146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2baffd700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2bc0b3730 0x7fc2bc146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2baffd700 1 -- 172.21.15.61:0/4142051947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2bc0b5480 msgr2=0x7fc2bc13e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:10.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2baffd700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2bc0b5480 0x7fc2bc13e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:10.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2baffd700 1 -- 172.21.15.61:0/4142051947 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a400b180 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2baffd700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc147160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fc2ac001490 tx=0x7fc2ac007e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac002240 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac0023a0 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2ac0017a0 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc1455c0 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc145a60 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc145750 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc2ac010030 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.807+0000 7fc2b3fff700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc2a0053a60 0x7fc2a0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:10.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.811+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc2ac06ba30 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.811+0000 7fc2ba7fc700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc2a0053a60 0x7fc2a0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:10.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.811+0000 7fc2ba7fc700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc2a0053a60 0x7fc2a0055f20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fc2a401a000 tx=0x7fc2a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:10.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.819+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2ac042c20 con 0x7fc2bc0b05f0 2022-07-12T08:48:10.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:10.943+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fc2bc057460 con 0x7fc2bc0b05f0 2022-07-12T08:48:11.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.251+0000 7fc2b3fff700 1 -- 172.21.15.61:0/4142051947 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc2ac046240 con 0x7fc2bc0b05f0 2022-07-12T08:48:11.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc2a0053a60 msgr2=0x7fc2a0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:11.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc2a0053a60 0x7fc2a0055f20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fc2a401a000 tx=0x7fc2a4014000).stop 2022-07-12T08:48:11.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 msgr2=0x7fc2bc147160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:11.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc147160 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fc2ac001490 tx=0x7fc2ac007e60).stop 2022-07-12T08:48:11.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 shutdown_connections 2022-07-12T08:48:11.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc2bc0b5480 0x7fc2bc13e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc2a0053a60 0x7fc2a0055f20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc2bc0b3730 0x7fc2bc146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 --2- 172.21.15.61:0/4142051947 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc2bc0b05f0 0x7fc2bc147160 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 >> 172.21.15.61:0/4142051947 conn(0x7fc2bc0b96b0 msgr2=0x7fc2bc05bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:11.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 shutdown_connections 2022-07-12T08:48:11.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.255+0000 7fc2c1718700 1 -- 172.21.15.61:0/4142051947 wait complete. 2022-07-12T08:48:11.261 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:48:11.272 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:48:11.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 Processor -- start 2022-07-12T08:48:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 -- start start 2022-07-12T08:48:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc3805d130 0x7efc3805c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7efc380b6130 0x7efc380b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc38143500 con 0x7efc38142eb0 2022-07-12T08:48:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc380b65c0 con 0x7efc3805d130 2022-07-12T08:48:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3da78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc38143290 con 0x7efc380b6130 2022-07-12T08:48:11.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc37fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efc3805d130 0x7efc3805c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:11.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3ca76700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efc38142eb0 0x7efc380b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46122/0 (socket says 172.21.15.61:46122) 2022-07-12T08:48:11.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3ca76700 1 -- 172.21.15.61:0/3797463877 learned_addr learned my addr 172.21.15.61:0/3797463877 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:11.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc3d277700 1 --2- 172.21.15.61:0/3797463877 >> v2:172.21.15.111:3300/0 conn(0x7efc380b6130 0x7efc380b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:11.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc37fff700 1 -- 172.21.15.61:0/3797463877 >> v2:172.21.15.111:3300/0 conn(0x7efc380b6130 msgr2=0x7efc380b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:11.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc37fff700 1 --2- 172.21.15.61:0/3797463877 >> v2:172.21.15.111:3300/0 conn(0x7efc380b6130 0x7efc380b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc37fff700 1 -- 172.21.15.61:0/3797463877 >> v1:172.21.15.61:6789/0 conn(0x7efc38142eb0 legacy=0x7efc380b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:11.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.387+0000 7efc37fff700 1 -- 172.21.15.61:0/3797463877 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc38143760 con 0x7efc3805d130 2022-07-12T08:48:11.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc37fff700 1 --2- 172.21.15.61:0/3797463877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc3805d130 0x7efc3805c6d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7efc38143c30 tx=0x7efc20006df0).ready entity=mon.0 client_cookie=c815b16ef01474d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:11.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc377fe700 1 -- 172.21.15.61:0/3797463877 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2000d5c0 con 0x7efc3805d130 2022-07-12T08:48:11.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc377fe700 1 -- 172.21.15.61:0/3797463877 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc20000e90 con 0x7efc3805d130 2022-07-12T08:48:11.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc377fe700 1 -- 172.21.15.61:0/3797463877 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc2000d910 con 0x7efc3805d130 2022-07-12T08:48:11.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- 172.21.15.61:0/3797463877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc3805d130 msgr2=0x7efc3805c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:11.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 --2- 172.21.15.61:0/3797463877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc3805d130 0x7efc3805c6d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7efc38143c30 tx=0x7efc20006df0).stop 2022-07-12T08:48:11.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- 172.21.15.61:0/3797463877 shutdown_connections 2022-07-12T08:48:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 --2- 172.21.15.61:0/3797463877 >> v2:172.21.15.111:3300/0 conn(0x7efc380b6130 0x7efc380b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 --2- 172.21.15.61:0/3797463877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc3805d130 0x7efc3805c6d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- 172.21.15.61:0/3797463877 >> 172.21.15.61:0/3797463877 conn(0x7efc38066d70 msgr2=0x7efc38067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- 172.21.15.61:0/3797463877 shutdown_connections 2022-07-12T08:48:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- 172.21.15.61:0/3797463877 wait complete. 2022-07-12T08:48:11.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 Processor -- start 2022-07-12T08:48:11.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- start start 2022-07-12T08:48:11.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3805d130 0x7efc380af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:11.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc380b6130 0x7efc38141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:11.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 0x7efc38141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:11.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc380ae790 con 0x7efc38142eb0 2022-07-12T08:48:11.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc38142230 con 0x7efc3805d130 2022-07-12T08:48:11.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3da78700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc38151d70 con 0x7efc380b6130 2022-07-12T08:48:11.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3d277700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 0x7efc38141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:11.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3d277700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 0x7efc38141d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57914/0 (socket says 172.21.15.61:57914) 2022-07-12T08:48:11.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3d277700 1 -- 172.21.15.61:0/2608977614 learned_addr learned my addr 172.21.15.61:0/2608977614 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:11.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc3ca76700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3805d130 0x7efc380af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:11.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.391+0000 7efc37fff700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc380b6130 0x7efc38141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:11.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3d277700 1 -- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc380b6130 msgr2=0x7efc38141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:11.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3d277700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc380b6130 0x7efc38141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3d277700 1 -- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3805d130 msgr2=0x7efc380af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:11.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3d277700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3805d130 0x7efc380af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:11.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3d277700 1 -- 172.21.15.61:0/2608977614 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2000b180 con 0x7efc38142eb0 2022-07-12T08:48:11.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3d277700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 0x7efc38141d40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7efc38136410 tx=0x7efc300091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:11.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3000c640 con 0x7efc38142eb0 2022-07-12T08:48:11.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc3000c7a0 con 0x7efc38142eb0 2022-07-12T08:48:11.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc38152010 con 0x7efc38142eb0 2022-07-12T08:48:11.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc3000cb00 con 0x7efc38142eb0 2022-07-12T08:48:11.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc38151a20 con 0x7efc38142eb0 2022-07-12T08:48:11.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc38057460 con 0x7efc38142eb0 2022-07-12T08:48:11.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.395+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efc3000cc60 con 0x7efc38142eb0 2022-07-12T08:48:11.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.399+0000 7efc35ffb700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:11.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.399+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5607+0+0 (secure 0 0 0) 0x7efc3006be80 con 0x7efc38142eb0 2022-07-12T08:48:11.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.399+0000 7efc3ca76700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:11.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.399+0000 7efc3ca76700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7efc2800d000 tx=0x7efc28009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:11.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.403+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc30042fb0 con 0x7efc38142eb0 2022-07-12T08:48:11.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:11.527+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7efc3806b2b0 con 0x7efc38142eb0 2022-07-12T08:48:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.111+0000 7efc35ffb700 1 -- 172.21.15.61:0/2608977614 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v79) v1 ==== 125+0+0 (secure 0 0 0) 0x7efc300465d0 con 0x7efc38142eb0 2022-07-12T08:48:12.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 msgr2=0x7efc10055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7efc2800d000 tx=0x7efc28009040).stop 2022-07-12T08:48:12.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 msgr2=0x7efc38141d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 0x7efc38141d40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7efc38136410 tx=0x7efc300091d0).stop 2022-07-12T08:48:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 shutdown_connections 2022-07-12T08:48:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc38142eb0 0x7efc38141d40 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc380b6130 0x7efc38141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 --2- 172.21.15.61:0/2608977614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3805d130 0x7efc380af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 >> 172.21.15.61:0/2608977614 conn(0x7efc38066d70 msgr2=0x7efc380b5140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 shutdown_connections 2022-07-12T08:48:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.115+0000 7efc3da78700 1 -- 172.21.15.61:0/2608977614 wait complete. 2022-07-12T08:48:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:48:12.132 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:48:12.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 Processor -- start 2022-07-12T08:48:12.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 -- start start 2022-07-12T08:48:12.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0060a70 0x7f5dd0060e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5dd0066b80 0x7f5dd01409a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd0144850 con 0x7f5dd00b2fc0 2022-07-12T08:48:12.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd0061340 con 0x7f5dd0066b80 2022-07-12T08:48:12.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dd489a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd0061a40 con 0x7f5dd0060a70 2022-07-12T08:48:12.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dced9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0060a70 0x7f5dd0060e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5dd0066b80 0x7f5dd01409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5dd0066b80 0x7f5dd01409a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36496/0 (socket says 172.21.15.61:36496) 2022-07-12T08:48:12.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dced9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0060a70 0x7f5dd0060e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40114/0 (socket says 172.21.15.61:40114) 2022-07-12T08:48:12.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 -- 172.21.15.61:0/805696791 learned_addr learned my addr 172.21.15.61:0/805696791 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:12.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 -- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0060a70 msgr2=0x7f5dd0060e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:12.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 --2- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0060a70 0x7f5dd0060e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 -- 172.21.15.61:0/805696791 >> v1:172.21.15.111:6789/0 conn(0x7f5dd00b2fc0 legacy=0x7f5dd0066470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.243+0000 7f5dcf59e700 1 -- 172.21.15.61:0/805696791 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd0144a80 con 0x7f5dd0066b80 2022-07-12T08:48:12.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.247+0000 7f5dcf59e700 1 --2- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd0066b80 0x7f5dd01409a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f5dc80032f0 tx=0x7f5dc8009290).ready entity=mon.1 client_cookie=c7c7b402f0daef48 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dcdd9b700 1 -- 172.21.15.61:0/805696791 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc800b8a0 con 0x7f5dd0066b80 2022-07-12T08:48:12.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dcdd9b700 1 -- 172.21.15.61:0/805696791 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc800ba00 con 0x7f5dd0066b80 2022-07-12T08:48:12.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dcdd9b700 1 -- 172.21.15.61:0/805696791 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc800bb70 con 0x7f5dd0066b80 2022-07-12T08:48:12.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 -- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd0066b80 msgr2=0x7f5dd01409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 --2- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd0066b80 0x7f5dd01409a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f5dc80032f0 tx=0x7f5dc8009290).stop 2022-07-12T08:48:12.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 -- 172.21.15.61:0/805696791 shutdown_connections 2022-07-12T08:48:12.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 --2- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd0066b80 0x7f5dd01409a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 --2- 172.21.15.61:0/805696791 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0060a70 0x7f5dd0060e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 -- 172.21.15.61:0/805696791 >> 172.21.15.61:0/805696791 conn(0x7f5dd00b5860 msgr2=0x7f5dd00b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:12.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 -- 172.21.15.61:0/805696791 shutdown_connections 2022-07-12T08:48:12.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.251+0000 7f5dd489a700 1 -- 172.21.15.61:0/805696791 wait complete. 2022-07-12T08:48:12.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 Processor -- start 2022-07-12T08:48:12.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 -- start start 2022-07-12T08:48:12.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 0x7f5dd01503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0066b80 0x7f5dd0142c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd00b2fc0 0x7f5dd00ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd0150890 con 0x7f5dd0060a70 2022-07-12T08:48:12.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00aff20 con 0x7f5dd00b2fc0 2022-07-12T08:48:12.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd00af540 con 0x7f5dd0066b80 2022-07-12T08:48:12.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 0x7f5dd01503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 0x7f5dd01503a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57924/0 (socket says 172.21.15.61:57924) 2022-07-12T08:48:12.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 -- 172.21.15.61:0/1287317719 learned_addr learned my addr 172.21.15.61:0/1287317719 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dcf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd00b2fc0 0x7f5dd00ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dce59c700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0066b80 0x7f5dd0142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 -- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0066b80 msgr2=0x7f5dd0142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0066b80 0x7f5dd0142c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 -- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd00b2fc0 msgr2=0x7f5dd00ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd00b2fc0 0x7f5dd00ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 -- 172.21.15.61:0/1287317719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8009530 con 0x7f5dd0060a70 2022-07-12T08:48:12.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dced9d700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 0x7f5dd01503a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f5dc000c000 tx=0x7f5dc0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc0010780 con 0x7f5dd0060a70 2022-07-12T08:48:12.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc00108e0 con 0x7f5dd0060a70 2022-07-12T08:48:12.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd00af7e0 con 0x7f5dd0060a70 2022-07-12T08:48:12.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc0010c40 con 0x7f5dd0060a70 2022-07-12T08:48:12.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.255+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd009b960 con 0x7f5dd0060a70 2022-07-12T08:48:12.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.259+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd0057460 con 0x7f5dd0060a70 2022-07-12T08:48:12.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.259+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5dc001c460 con 0x7f5dd0060a70 2022-07-12T08:48:12.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.259+0000 7f5dbffff700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5dac053a60 0x7f5dac055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.259+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f5dc006c210 con 0x7f5dd0060a70 2022-07-12T08:48:12.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.259+0000 7f5dce59c700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5dac053a60 0x7f5dac055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.259+0000 7f5dce59c700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5dac053a60 0x7f5dac055f20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f5dc400d000 tx=0x7f5dc4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.267+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5dc00430e0 con 0x7f5dd0060a70 2022-07-12T08:48:12.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.427+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f5dd00b6160 con 0x7f5dd0060a70 2022-07-12T08:48:12.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.443+0000 7f5dbffff700 1 -- 172.21.15.61:0/1287317719 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 15 and data pool 16 v43) v1 ==== 189+0+0 (secure 0 0 0) 0x7f5dc0046700 con 0x7f5dd0060a70 2022-07-12T08:48:12.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5dac053a60 msgr2=0x7f5dac055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5dac053a60 0x7f5dac055f20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f5dc400d000 tx=0x7f5dc4007b90).stop 2022-07-12T08:48:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 msgr2=0x7f5dd01503a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 0x7f5dd01503a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7f5dc000c000 tx=0x7f5dc0006df0).stop 2022-07-12T08:48:12.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 shutdown_connections 2022-07-12T08:48:12.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5dac053a60 0x7f5dac055f20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5dd00b2fc0 0x7f5dd00ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5dd0066b80 0x7f5dd0142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 --2- 172.21.15.61:0/1287317719 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd0060a70 0x7f5dd01503a0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 >> 172.21.15.61:0/1287317719 conn(0x7f5dd00b5860 msgr2=0x7f5dd00b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:12.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 shutdown_connections 2022-07-12T08:48:12.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.447+0000 7f5dd489a700 1 -- 172.21.15.61:0/1287317719 wait complete. 2022-07-12T08:48:12.458 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 15 and data pool 16 2022-07-12T08:48:12.472 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:12.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03fdc91700 1 Processor -- start 2022-07-12T08:48:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03fdc91700 1 -- start start 2022-07-12T08:48:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03fdc91700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 0x7f03f80ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03fdc91700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f80ad2f0 con 0x7f03f805c050 2022-07-12T08:48:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03fdc91700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f805c430 con 0x7f03f80b01f0 2022-07-12T08:48:12.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03fdc91700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80b0a40 con 0x7f03f8144ad0 2022-07-12T08:48:12.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f03f805c050 0x7f03f805cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46164/0 (socket says 172.21.15.61:46164) 2022-07-12T08:48:12.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f77fe700 1 -- 172.21.15.61:0/3482794646 learned_addr learned my addr 172.21.15.61:0/3482794646 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:12.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 0x7f03f80ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019510649 0 0) 0x7f03f80ad2f0 con 0x7f03f805c050 2022-07-12T08:48:12.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d8003260 con 0x7f03f805c050 2022-07-12T08:48:12.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782746041 0 0) 0x7f03f805c430 con 0x7f03f80b01f0 2022-07-12T08:48:12.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f80ad2f0 con 0x7f03f80b01f0 2022-07-12T08:48:12.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 484823669 0 0) 0x7f03d8003260 con 0x7f03f805c050 2022-07-12T08:48:12.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 msgr2=0x7f03f80ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:12.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 0x7f03f80ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 >> v1:172.21.15.111:6789/0 conn(0x7f03f80b01f0 legacy=0x7f03f81443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80b2420 con 0x7f03f805c050 2022-07-12T08:48:12.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f03e8001660 con 0x7f03f805c050 2022-07-12T08:48:12.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 >> v1:172.21.15.61:6789/0 conn(0x7f03f805c050 legacy=0x7f03f805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 0x7f03d8003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03d8003e20 0x7f03d8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03d80049f0 0x7f03d8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80b0a40 con 0x7f03d80049f0 2022-07-12T08:48:12.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d800b800 con 0x7f03d8003e20 2022-07-12T08:48:12.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d800b9c0 con 0x7f03f8144ad0 2022-07-12T08:48:12.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f77fe700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 0x7f03d8003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03d80049f0 0x7f03d8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03eeffd700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03d8003e20 0x7f03d8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 -- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 msgr2=0x7f03d8003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8144ad0 0x7f03d8003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 -- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03d8003e20 msgr2=0x7f03d8004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:12.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03d8003e20 0x7f03d8004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.571+0000 7f03f7fff700 1 -- 172.21.15.61:0/3482794646 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8003260 con 0x7f03d80049f0 2022-07-12T08:48:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f7fff700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03d80049f0 0x7f03d8004e70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f03f000c000 tx=0x7f03f00091d0).ready entity=mon.0 client_cookie=abbb7ad2940df5b1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03f000e7e0 con 0x7f03d80049f0 2022-07-12T08:48:12.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f000e940 con 0x7f03d80049f0 2022-07-12T08:48:12.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f6ffd700 1 -- 172.21.15.61:0/3482794646 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03f000eca0 con 0x7f03d80049f0 2022-07-12T08:48:12.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- 172.21.15.61:0/3482794646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03d80049f0 msgr2=0x7f03d8004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 --2- 172.21.15.61:0/3482794646 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03d80049f0 0x7f03d8004e70 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7f03f000c000 tx=0x7f03f00091d0).stop 2022-07-12T08:48:12.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f7fff700 1 -- 172.21.15.61:0/3482794646 reap_dead start 2022-07-12T08:48:12.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- 172.21.15.61:0/3482794646 shutdown_connections 2022-07-12T08:48:12.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- 172.21.15.61:0/3482794646 >> 172.21.15.61:0/3482794646 conn(0x7f03f8062c00 msgr2=0x7f03f8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:12.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- 172.21.15.61:0/3482794646 shutdown_connections 2022-07-12T08:48:12.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- 172.21.15.61:0/3482794646 wait complete. 2022-07-12T08:48:12.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 Processor -- start 2022-07-12T08:48:12.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- start start 2022-07-12T08:48:12.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8147870 0x7f03f8147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 0x7f03f8146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8146770 0x7f03f8159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8148140 con 0x7f03f80a0980 2022-07-12T08:48:12.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f81448f0 con 0x7f03f8146770 2022-07-12T08:48:12.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03fdc91700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f809e430 con 0x7f03f8147870 2022-07-12T08:48:12.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03eeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 0x7f03f8146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03eeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 0x7f03f8146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57958/0 (socket says 172.21.15.61:57958) 2022-07-12T08:48:12.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03eeffd700 1 -- 172.21.15.61:0/158727968 learned_addr learned my addr 172.21.15.61:0/158727968 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:12.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f77fe700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8147870 0x7f03f8147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.575+0000 7f03f7fff700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8146770 0x7f03f8159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03eeffd700 1 -- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8147870 msgr2=0x7f03f8147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03eeffd700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8147870 0x7f03f8147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03eeffd700 1 -- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8146770 msgr2=0x7f03f8159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03eeffd700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8146770 0x7f03f8159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03eeffd700 1 -- 172.21.15.61:0/158727968 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f809e6d0 con 0x7f03f80a0980 2022-07-12T08:48:12.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f7fff700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8146770 0x7f03f8159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:12.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03eeffd700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 0x7f03f8146280 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f03e000d000 tx=0x7f03e0007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03e0011780 con 0x7f03f80a0980 2022-07-12T08:48:12.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e00118e0 con 0x7f03f80a0980 2022-07-12T08:48:12.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f809e970 con 0x7f03f80a0980 2022-07-12T08:48:12.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03e0011c40 con 0x7f03f80a0980 2022-07-12T08:48:12.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f8146d90 con 0x7f03f80a0980 2022-07-12T08:48:12.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f809eb00 con 0x7f03f80a0980 2022-07-12T08:48:12.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f03e001b030 con 0x7f03f80a0980 2022-07-12T08:48:12.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f57fa700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03dc053940 0x7f03dc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f03e006c050 con 0x7f03f80a0980 2022-07-12T08:48:12.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.579+0000 7f03f77fe700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03dc053940 0x7f03dc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.583+0000 7f03f77fe700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03dc053940 0x7f03dc055e00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f03e8005000 tx=0x7f03e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.587+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03e0042f10 con 0x7f03f80a0980 2022-07-12T08:48:12.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.711+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03f8057460 con 0x7f03f80a0980 2022-07-12T08:48:12.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03f57fa700 1 -- 172.21.15.61:0/158727968 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f03e0046530 con 0x7f03f80a0980 2022-07-12T08:48:12.717 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:12.717 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:12.433374+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:11.002774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:11.580821+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:12.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03dc053940 msgr2=0x7f03dc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03dc053940 0x7f03dc055e00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f03e8005000 tx=0x7f03e8009040).stop 2022-07-12T08:48:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 msgr2=0x7f03f8146280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 0x7f03f8146280 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f03e000d000 tx=0x7f03e0007a00).stop 2022-07-12T08:48:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 shutdown_connections 2022-07-12T08:48:12.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03dc053940 0x7f03dc055e00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03f8146770 0x7f03f8159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03f80a0980 0x7f03f8146280 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 --2- 172.21.15.61:0/158727968 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03f8147870 0x7f03f8147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 >> 172.21.15.61:0/158727968 conn(0x7f03f8062c00 msgr2=0x7f03f8141f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:12.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 shutdown_connections 2022-07-12T08:48:12.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.715+0000 7f03fdc91700 1 -- 172.21.15.61:0/158727968 wait complete. 2022-07-12T08:48:12.736 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:12.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.835+0000 7f3195de7700 1 Processor -- start 2022-07-12T08:48:12.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.835+0000 7f3195de7700 1 -- start start 2022-07-12T08:48:12.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f31900aa7f0 0x7f3190098200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f31900986f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f319005f140 0x7f3190142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319005f650 con 0x7f31900aa7f0 2022-07-12T08:48:12.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31900aabd0 con 0x7f319005f140 2022-07-12T08:48:12.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3190067c80 con 0x7f31900b1450 2022-07-12T08:48:12.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f31900aa7f0 0x7f3190098200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f31900aa7f0 0x7f3190098200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57964/0 (socket says 172.21.15.61:57964) 2022-07-12T08:48:12.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 -- 172.21.15.61:0/1687332542 learned_addr learned my addr 172.21.15.61:0/1687332542 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:12.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f31955e6700 1 --2- 172.21.15.61:0/1687332542 >> v2:172.21.15.111:3300/0 conn(0x7f319005f140 0x7f3190142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 -- 172.21.15.61:0/1687332542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 msgr2=0x7f31900986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:12.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f318ffff700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f31900986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f31900986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 -- 172.21.15.61:0/1687332542 >> v2:172.21.15.111:3300/0 conn(0x7f319005f140 msgr2=0x7f3190142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:12.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 --2- 172.21.15.61:0/1687332542 >> v2:172.21.15.111:3300/0 conn(0x7f319005f140 0x7f3190142fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 -- 172.21.15.61:0/1687332542 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190098be0 con 0x7f31900aa7f0 2022-07-12T08:48:12.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f31955e6700 1 --2- 172.21.15.61:0/1687332542 >> v2:172.21.15.111:3300/0 conn(0x7f319005f140 0x7f3190142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:12.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f318ffff700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f31900986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:12.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3194de5700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31900aa7f0 0x7f3190098200 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f3184002f70 tx=0x7f3184006eb0).ready entity=mon.0 client_cookie=fb4703e5798964b9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f318f7fe700 1 -- 172.21.15.61:0/1687332542 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f318400d570 con 0x7f31900aa7f0 2022-07-12T08:48:12.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f318f7fe700 1 -- 172.21.15.61:0/1687332542 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184000e90 con 0x7f31900aa7f0 2022-07-12T08:48:12.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f318f7fe700 1 -- 172.21.15.61:0/1687332542 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f318400d900 con 0x7f31900aa7f0 2022-07-12T08:48:12.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- 172.21.15.61:0/1687332542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31900aa7f0 msgr2=0x7f3190098200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31900aa7f0 0x7f3190098200 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f3184002f70 tx=0x7f3184006eb0).stop 2022-07-12T08:48:12.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- 172.21.15.61:0/1687332542 shutdown_connections 2022-07-12T08:48:12.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f31900986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- 172.21.15.61:0/1687332542 >> v2:172.21.15.111:3300/0 conn(0x7f319005f140 0x7f3190142fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 --2- 172.21.15.61:0/1687332542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31900aa7f0 0x7f3190098200 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- 172.21.15.61:0/1687332542 >> 172.21.15.61:0/1687332542 conn(0x7f31900b1950 msgr2=0x7f31900b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:12.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- 172.21.15.61:0/1687332542 shutdown_connections 2022-07-12T08:48:12.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.839+0000 7f3195de7700 1 -- 172.21.15.61:0/1687332542 wait complete. 2022-07-12T08:48:12.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 Processor -- start 2022-07-12T08:48:12.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 -- start start 2022-07-12T08:48:12.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f319005f140 0x7f31900ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31900aa7f0 0x7f31900ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f3190066640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3190150930 con 0x7f319005f140 2022-07-12T08:48:12.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31900671f0 con 0x7f31900aa7f0 2022-07-12T08:48:12.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3195de7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3190150490 con 0x7f31900b1450 2022-07-12T08:48:12.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3194de5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f319005f140 0x7f31900ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f318ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31900aa7f0 0x7f31900ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3194de5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f319005f140 0x7f31900ad300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57970/0 (socket says 172.21.15.61:57970) 2022-07-12T08:48:12.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f3194de5700 1 -- 172.21.15.61:0/3155941809 learned_addr learned my addr 172.21.15.61:0/3155941809 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:12.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f31955e6700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f3190066640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f318ffff700 1 -- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 msgr2=0x7f3190066640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f318ffff700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f3190066640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f318ffff700 1 -- 172.21.15.61:0/3155941809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f319005f140 msgr2=0x7f31900ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f318ffff700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f319005f140 0x7f31900ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:12.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.843+0000 7f318ffff700 1 -- 172.21.15.61:0/3155941809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318400b1e0 con 0x7f31900aa7f0 2022-07-12T08:48:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318ffff700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31900aa7f0 0x7f31900ad7f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f317c007000 tx=0x7f317c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c00ae80 con 0x7f31900aa7f0 2022-07-12T08:48:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c00c030 con 0x7f31900aa7f0 2022-07-12T08:48:12.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31901501a0 con 0x7f31900aa7f0 2022-07-12T08:48:12.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317c00f6d0 con 0x7f31900aa7f0 2022-07-12T08:48:12.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31900ae880 con 0x7f31900aa7f0 2022-07-12T08:48:12.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3190150650 con 0x7f31900aa7f0 2022-07-12T08:48:12.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f317c019030 con 0x7f31900aa7f0 2022-07-12T08:48:12.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318dffb700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3170053940 0x7f3170055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:12.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f317c06b570 con 0x7f31900aa7f0 2022-07-12T08:48:12.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.847+0000 7f3194de5700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3170053940 0x7f3170055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:12.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.851+0000 7f3194de5700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3170053940 0x7f3170055e00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f318401c000 tx=0x7f3184015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:12.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.859+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f317c042410 con 0x7f31900aa7f0 2022-07-12T08:48:12.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.987+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f319006b2b0 con 0x7f31900aa7f0 2022-07-12T08:48:12.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.987+0000 7f318dffb700 1 -- 172.21.15.61:0/3155941809 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f317c045a30 con 0x7f31900aa7f0 2022-07-12T08:48:12.991 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:12.993 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:12.433374+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:11.002774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:11.580821+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:12.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3170053940 msgr2=0x7f3170055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3170053940 0x7f3170055e00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f318401c000 tx=0x7f3184015000).stop 2022-07-12T08:48:12.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31900aa7f0 msgr2=0x7f31900ad7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:12.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31900aa7f0 0x7f31900ad7f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f317c007000 tx=0x7f317c008040).stop 2022-07-12T08:48:12.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 shutdown_connections 2022-07-12T08:48:13.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31900b1450 0x7f3190066640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3170053940 0x7f3170055e00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31900aa7f0 0x7f31900ad7f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 --2- 172.21.15.61:0/3155941809 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f319005f140 0x7f31900ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 >> 172.21.15.61:0/3155941809 conn(0x7f31900b1950 msgr2=0x7f319009a750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:13.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 shutdown_connections 2022-07-12T08:48:13.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:12.991+0000 7f3195de7700 1 -- 172.21.15.61:0/3155941809 wait complete. 2022-07-12T08:48:13.005 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:48:13.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 Processor -- start 2022-07-12T08:48:13.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 -- start start 2022-07-12T08:48:13.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5bfc0b9db0 0x7f5bfc0b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5bfc05c810 0x7f5bfc0b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0b3120 con 0x7f5bfc0b3ea0 2022-07-12T08:48:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc05bd10 con 0x7f5bfc0b9db0 2022-07-12T08:48:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5c01ec0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b33c0 con 0x7f5bfc05c810 2022-07-12T08:48:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5bf2ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5bfc0b3ea0 0x7f5bfc05c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46196/0 (socket says 172.21.15.61:46196) 2022-07-12T08:48:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5bf2ffd700 1 -- 172.21.15.61:0/1944042369 learned_addr learned my addr 172.21.15.61:0/1944042369 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:13.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5bfb7fe700 1 --2- 172.21.15.61:0/1944042369 >> v2:172.21.15.61:3300/0 conn(0x7f5bfc0b9db0 0x7f5bfc0b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5bfbfff700 1 --2- 172.21.15.61:0/1944042369 >> v2:172.21.15.111:3300/0 conn(0x7f5bfc05c810 0x7f5bfc0b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5bfaffd700 1 -- 172.21.15.61:0/1944042369 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268919067 0 0) 0x7f5bfc0b3120 con 0x7f5bfc0b3ea0 2022-07-12T08:48:13.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.131+0000 7f5bfaffd700 1 -- 172.21.15.61:0/1944042369 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc003260 con 0x7f5bfc0b3ea0 2022-07-12T08:48:13.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/1944042369 >> v2:172.21.15.111:3300/0 conn(0x7f5bfc05c810 msgr2=0x7f5bfc0b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:13.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfb7fe700 1 --2- 172.21.15.61:0/1944042369 >> v2:172.21.15.111:3300/0 conn(0x7f5bfc05c810 0x7f5bfc0b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/1944042369 >> v1:172.21.15.61:6789/0 conn(0x7f5bfc0b3ea0 legacy=0x7f5bfc05c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/1944042369 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc05cc60 con 0x7f5bfc0b9db0 2022-07-12T08:48:13.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfb7fe700 1 --2- 172.21.15.61:0/1944042369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc0b2a40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f5bfc0b1230 tx=0x7f5be4006df0).ready entity=mon.0 client_cookie=3a521399dbc237a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfaffd700 1 -- 172.21.15.61:0/1944042369 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be400d4f0 con 0x7f5bfc0b9db0 2022-07-12T08:48:13.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfaffd700 1 -- 172.21.15.61:0/1944042369 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4000e90 con 0x7f5bfc0b9db0 2022-07-12T08:48:13.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfaffd700 1 -- 172.21.15.61:0/1944042369 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be400d840 con 0x7f5bfc0b9db0 2022-07-12T08:48:13.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- 172.21.15.61:0/1944042369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc0b9db0 msgr2=0x7f5bfc0b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/1944042369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc0b2a40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f5bfc0b1230 tx=0x7f5be4006df0).stop 2022-07-12T08:48:13.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- 172.21.15.61:0/1944042369 shutdown_connections 2022-07-12T08:48:13.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/1944042369 >> v2:172.21.15.111:3300/0 conn(0x7f5bfc05c810 0x7f5bfc0b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/1944042369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc0b2a40 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- 172.21.15.61:0/1944042369 >> 172.21.15.61:0/1944042369 conn(0x7f5bfc067960 msgr2=0x7f5bfc0b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:13.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- 172.21.15.61:0/1944042369 shutdown_connections 2022-07-12T08:48:13.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- 172.21.15.61:0/1944042369 wait complete. 2022-07-12T08:48:13.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 Processor -- start 2022-07-12T08:48:13.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- start start 2022-07-12T08:48:13.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 0x7f5bfc143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5bfc0b3ea0 0x7f5bfc141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc1422f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0b0110 con 0x7f5bfc05c810 2022-07-12T08:48:13.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0aefc0 con 0x7f5bfc0b9db0 2022-07-12T08:48:13.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5c01ec0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1427e0 con 0x7f5bfc0b3ea0 2022-07-12T08:48:13.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.135+0000 7f5bfb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 0x7f5bfc143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 0x7f5bfc143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:57984/0 (socket says 172.21.15.61:57984) 2022-07-12T08:48:13.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/2314622380 learned_addr learned my addr 172.21.15.61:0/2314622380 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:13.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfbfff700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc1422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf2ffd700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5bfc0b3ea0 0x7f5bfc141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5bfc0b3ea0 msgr2=0x7f5bfc141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5bfc0b3ea0 0x7f5bfc141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bfc0b9db0 msgr2=0x7f5bfc1422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc1422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 -- 172.21.15.61:0/2314622380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be400b180 con 0x7f5bfc05c810 2022-07-12T08:48:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf2ffd700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5bfc0b3ea0 0x7f5bfc141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bfb7fe700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 0x7f5bfc143b40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f5bfc0b1230 tx=0x7f5be4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be4013e90 con 0x7f5bfc05c810 2022-07-12T08:48:13.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be4002f70 con 0x7f5bfc05c810 2022-07-12T08:48:13.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc152fa0 con 0x7f5bfc05c810 2022-07-12T08:48:13.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5be401e5f0 con 0x7f5bfc05c810 2022-07-12T08:48:13.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc0adf60 con 0x7f5bfc05c810 2022-07-12T08:48:13.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc0aff30 con 0x7f5bfc05c810 2022-07-12T08:48:13.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5be401c070 con 0x7f5bfc05c810 2022-07-12T08:48:13.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf97fa700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5be00539d0 0x7f5be0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.139+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5be406c6d0 con 0x7f5bfc05c810 2022-07-12T08:48:13.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.143+0000 7f5bf2ffd700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5be00539d0 0x7f5be0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.143+0000 7f5bf2ffd700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5be00539d0 0x7f5be0055e90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5bec00b000 tx=0x7f5bec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.151+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5be4043630 con 0x7f5bfc05c810 2022-07-12T08:48:13.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.315+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5bfc057460 con 0x7f5bfc05c810 2022-07-12T08:48:13.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.451+0000 7f5bf97fa700 1 -- 172.21.15.61:0/2314622380 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5be4023020 con 0x7f5bfc05c810 2022-07-12T08:48:13.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5be00539d0 msgr2=0x7f5be0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5be00539d0 0x7f5be0055e90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f5bec00b000 tx=0x7f5bec009040).stop 2022-07-12T08:48:13.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 msgr2=0x7f5bfc143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 0x7f5bfc143b40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f5bfc0b1230 tx=0x7f5be4014000).stop 2022-07-12T08:48:13.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 shutdown_connections 2022-07-12T08:48:13.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5be00539d0 0x7f5be0055e90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5bfc0b9db0 0x7f5bfc1422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5bfc0b3ea0 0x7f5bfc141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 --2- 172.21.15.61:0/2314622380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bfc05c810 0x7f5bfc143b40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 >> 172.21.15.61:0/2314622380 conn(0x7f5bfc067960 msgr2=0x7f5bfc0b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 shutdown_connections 2022-07-12T08:48:13.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.455+0000 7f5c01ec0700 1 -- 172.21.15.61:0/2314622380 wait complete. 2022-07-12T08:48:13.474 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:13.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcf383700 1 Processor -- start 2022-07-12T08:48:13.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcf383700 1 -- start start 2022-07-12T08:48:13.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcf383700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fedc805f140 0x7fedc8151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcf383700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc80abc70 con 0x7fedc80ab290 2022-07-12T08:48:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcf383700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc805c720 con 0x7fedc80ab670 2022-07-12T08:48:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcf383700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedc80aba90 con 0x7fedc805f140 2022-07-12T08:48:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fedc80ab290 0x7fedc805ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46210/0 (socket says 172.21.15.61:46210) 2022-07-12T08:48:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 -- 172.21.15.61:0/2388451499 learned_addr learned my addr 172.21.15.61:0/2388451499 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedceb82700 1 --2- 172.21.15.61:0/2388451499 >> v2:172.21.15.111:3300/0 conn(0x7fedc805f140 0x7fedc8151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683373920 0 0) 0x7fedc80abc70 con 0x7fedc80ab290 2022-07-12T08:48:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb0003260 con 0x7fedc80ab290 2022-07-12T08:48:13.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 710415923 0 0) 0x7fedb0003260 con 0x7fedc80ab290 2022-07-12T08:48:13.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 >> v2:172.21.15.111:3300/0 conn(0x7fedc805f140 msgr2=0x7fedc8151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:13.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 --2- 172.21.15.61:0/2388451499 >> v2:172.21.15.111:3300/0 conn(0x7fedc805f140 0x7fedc8151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 >> v1:172.21.15.111:6789/0 conn(0x7fedc80ab670 legacy=0x7fedc80afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc805c9c0 con 0x7fedc80ab290 2022-07-12T08:48:13.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fedb8001660 con 0x7fedc80ab290 2022-07-12T08:48:13.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 >> v1:172.21.15.61:6789/0 conn(0x7fedc80ab290 legacy=0x7fedc805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb0003750 0x7fedb0003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0004500 0x7fedb0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedb0004e70 0x7fedb000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc80aba90 con 0x7fedb0003750 2022-07-12T08:48:13.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb000bc70 con 0x7fedb0004500 2022-07-12T08:48:13.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb000be30 con 0x7fedb0004e70 2022-07-12T08:48:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb0003750 0x7fedb0003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcdb80700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0004500 0x7fedb0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedceb82700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedb0004e70 0x7fedb000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 -- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedb0004e70 msgr2=0x7fedb000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:13.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedb0004e70 0x7fedb000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 -- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0004500 msgr2=0x7fedb0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:13.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0004500 0x7fedb0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 -- 172.21.15.61:0/2388451499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb0003260 con 0x7fedb0003750 2022-07-12T08:48:13.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedceb82700 1 -- 172.21.15.61:0/2388451499 reap_dead start 2022-07-12T08:48:13.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedce381700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb0003750 0x7fedb0003b30 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fedb8001370 tx=0x7fedc80aadc0).ready entity=mon.0 client_cookie=f06e5dd1f9ed5d76 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedb8007270 con 0x7fedb0003750 2022-07-12T08:48:13.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb80073d0 con 0x7fedb0003750 2022-07-12T08:48:13.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.579+0000 7fedcd37f700 1 -- 172.21.15.61:0/2388451499 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedb800ca60 con 0x7fedb0003750 2022-07-12T08:48:13.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/2388451499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb0003750 msgr2=0x7fedb0003b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb0003750 0x7fedb0003b30 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fedb8001370 tx=0x7fedc80aadc0).stop 2022-07-12T08:48:13.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/2388451499 shutdown_connections 2022-07-12T08:48:13.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 --2- 172.21.15.61:0/2388451499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb0003750 0x7fedb0003b30 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/2388451499 >> 172.21.15.61:0/2388451499 conn(0x7fedc80b1950 msgr2=0x7fedc80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:13.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/2388451499 shutdown_connections 2022-07-12T08:48:13.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/2388451499 wait complete. 2022-07-12T08:48:13.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 Processor -- start 2022-07-12T08:48:13.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- start start 2022-07-12T08:48:13.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0003750 0x7fedc8155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 0x7fedc8066c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedc80970a0 0x7fedc8097520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8156b60 con 0x7fedc8092050 2022-07-12T08:48:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8147800 con 0x7fedb0003750 2022-07-12T08:48:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc8156180 con 0x7fedc80970a0 2022-07-12T08:48:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 0x7fedc8066c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 0x7fedc8066c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58006/0 (socket says 172.21.15.61:58006) 2022-07-12T08:48:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 -- 172.21.15.61:0/343313558 learned_addr learned my addr 172.21.15.61:0/343313558 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedce381700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0003750 0x7fedc8155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedceb82700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedc80970a0 0x7fedc8097520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 -- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedc80970a0 msgr2=0x7fedc8097520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedc80970a0 0x7fedc8097520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 -- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0003750 msgr2=0x7fedc8155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0003750 0x7fedc8155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 -- 172.21.15.61:0/343313558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc8156420 con 0x7fedc8092050 2022-07-12T08:48:13.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedceb82700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedc80970a0 0x7fedc8097520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:13.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcdb80700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 0x7fedc8066c00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7fedc8156ca0 tx=0x7fedc4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedc4001640 con 0x7fedc8092050 2022-07-12T08:48:13.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc40017a0 con 0x7fedc8092050 2022-07-12T08:48:13.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc81566c0 con 0x7fedc8092050 2022-07-12T08:48:13.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedc4001b00 con 0x7fedc8092050 2022-07-12T08:48:13.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.583+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc8097a10 con 0x7fedc8092050 2022-07-12T08:48:13.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.587+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc8156850 con 0x7fedc8092050 2022-07-12T08:48:13.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.587+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fedc4001c60 con 0x7fedc8092050 2022-07-12T08:48:13.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.587+0000 7fedb77fe700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feda8053a40 0x7feda8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.587+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fedc406c120 con 0x7fedc8092050 2022-07-12T08:48:13.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.587+0000 7fedce381700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feda8053a40 0x7feda8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.587+0000 7fedce381700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feda8053a40 0x7feda8055f00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fedb8019000 tx=0x7fedb8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.595+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fedc4046e60 con 0x7fedc8092050 2022-07-12T08:48:13.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.759+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedc8057460 con 0x7fedc8092050 2022-07-12T08:48:13.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.759+0000 7fedb77fe700 1 -- 172.21.15.61:0/343313558 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fedc4042f70 con 0x7fedc8092050 2022-07-12T08:48:13.764 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:13.765 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:13.437914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3665156048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3665156048},{"type":"v1","addr":"172.21.15.61:6833","nonce":3665156048}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:13.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feda8053a40 msgr2=0x7feda8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feda8053a40 0x7feda8055f00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fedb8019000 tx=0x7fedb8015000).stop 2022-07-12T08:48:13.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 msgr2=0x7fedc8066c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 0x7fedc8066c00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7fedc8156ca0 tx=0x7fedc4009040).stop 2022-07-12T08:48:13.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 shutdown_connections 2022-07-12T08:48:13.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fedc80970a0 0x7fedc8097520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedc8092050 0x7fedc8066c00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feda8053a40 0x7feda8055f00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 --2- 172.21.15.61:0/343313558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fedb0003750 0x7fedc8155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 >> 172.21.15.61:0/343313558 conn(0x7fedc80b1950 msgr2=0x7fedc80b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:13.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 shutdown_connections 2022-07-12T08:48:13.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.763+0000 7fedcf383700 1 -- 172.21.15.61:0/343313558 wait complete. 2022-07-12T08:48:13.775 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 44 2022-07-12T08:48:13.777 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:13.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 Processor -- start 2022-07-12T08:48:13.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 -- start start 2022-07-12T08:48:13.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7e840ab670 0x7f7e8405f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7e840afa60 0x7f7e840afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840abc70 con 0x7f7e840ab290 2022-07-12T08:48:13.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e840aba90 con 0x7f7e840afa60 2022-07-12T08:48:13.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8a67d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8405c720 con 0x7f7e840ab670 2022-07-12T08:48:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e89e7c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7e840afa60 0x7f7e840afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e89e7c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7e840afa60 0x7f7e840afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58016/0 (socket says 172.21.15.61:58016) 2022-07-12T08:48:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e8967b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7e840ab290 0x7f7e8405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46232/0 (socket says 172.21.15.61:46232) 2022-07-12T08:48:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e89e7c700 1 -- 172.21.15.61:0/1238675033 learned_addr learned my addr 172.21.15.61:0/1238675033 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.879+0000 7f7e88e7a700 1 --2- 172.21.15.61:0/1238675033 >> v2:172.21.15.111:3300/0 conn(0x7f7e840ab670 0x7f7e8405f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e83fff700 1 -- 172.21.15.61:0/1238675033 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513900664 0 0) 0x7f7e840abc70 con 0x7f7e840ab290 2022-07-12T08:48:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e89e7c700 1 -- 172.21.15.61:0/1238675033 >> v2:172.21.15.111:3300/0 conn(0x7f7e840ab670 msgr2=0x7f7e8405f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e89e7c700 1 --2- 172.21.15.61:0/1238675033 >> v2:172.21.15.111:3300/0 conn(0x7f7e840ab670 0x7f7e8405f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e89e7c700 1 -- 172.21.15.61:0/1238675033 >> v1:172.21.15.61:6789/0 conn(0x7f7e840ab290 legacy=0x7f7e8405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e89e7c700 1 -- 172.21.15.61:0/1238675033 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8405c9c0 con 0x7f7e840afa60 2022-07-12T08:48:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e89e7c700 1 --2- 172.21.15.61:0/1238675033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840afa60 0x7f7e840afe90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f7e7c002f70 tx=0x7f7e7c006eb0).ready entity=mon.0 client_cookie=2d674b2efb557a75 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e83fff700 1 -- 172.21.15.61:0/1238675033 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c00d5c0 con 0x7f7e840afa60 2022-07-12T08:48:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e83fff700 1 -- 172.21.15.61:0/1238675033 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c000e90 con 0x7f7e840afa60 2022-07-12T08:48:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e83fff700 1 -- 172.21.15.61:0/1238675033 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c00d910 con 0x7f7e840afa60 2022-07-12T08:48:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- 172.21.15.61:0/1238675033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840afa60 msgr2=0x7f7e840afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/1238675033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840afa60 0x7f7e840afe90 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7f7e7c002f70 tx=0x7f7e7c006eb0).stop 2022-07-12T08:48:13.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- 172.21.15.61:0/1238675033 shutdown_connections 2022-07-12T08:48:13.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/1238675033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840afa60 0x7f7e840afe90 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/1238675033 >> v2:172.21.15.111:3300/0 conn(0x7f7e840ab670 0x7f7e8405f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- 172.21.15.61:0/1238675033 >> 172.21.15.61:0/1238675033 conn(0x7f7e840b1950 msgr2=0x7f7e840b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:13.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- 172.21.15.61:0/1238675033 shutdown_connections 2022-07-12T08:48:13.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- 172.21.15.61:0/1238675033 wait complete. 2022-07-12T08:48:13.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 Processor -- start 2022-07-12T08:48:13.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- start start 2022-07-12T08:48:13.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e840ab290 0x7f7e84063f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 0x7f7e84159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7e840afa60 0x7f7e84159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840910a0 con 0x7f7e840ab670 2022-07-12T08:48:13.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840667f0 con 0x7f7e840ab290 2022-07-12T08:48:13.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e8a67d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e84066a90 con 0x7f7e840afa60 2022-07-12T08:48:13.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e88e7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 0x7f7e84159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e88e7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 0x7f7e84159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58020/0 (socket says 172.21.15.61:58020) 2022-07-12T08:48:13.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.883+0000 7f7e88e7a700 1 -- 172.21.15.61:0/248401938 learned_addr learned my addr 172.21.15.61:0/248401938 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:13.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e89e7c700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7e840afa60 0x7f7e84159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e8967b700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e840ab290 0x7f7e84063f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e88e7a700 1 -- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7e840afa60 msgr2=0x7f7e84159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e88e7a700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7e840afa60 0x7f7e84159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e88e7a700 1 -- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e840ab290 msgr2=0x7f7e84063f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:13.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e88e7a700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e840ab290 0x7f7e84063f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:13.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e88e7a700 1 -- 172.21.15.61:0/248401938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c00b1e0 con 0x7f7e840ab670 2022-07-12T08:48:13.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e88e7a700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 0x7f7e84159650 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f7e7400c000 tx=0x7f7e740091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7400c6a0 con 0x7f7e840ab670 2022-07-12T08:48:13.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7400c800 con 0x7f7e840ab670 2022-07-12T08:48:13.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e84066d30 con 0x7f7e840ab670 2022-07-12T08:48:13.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7400cb60 con 0x7f7e840ab670 2022-07-12T08:48:13.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e84092930 con 0x7f7e840ab670 2022-07-12T08:48:13.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e84092c40 con 0x7f7e840ab670 2022-07-12T08:48:13.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7e74022070 con 0x7f7e840ab670 2022-07-12T08:48:13.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.887+0000 7f7e827fc700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7e64053a40 0x7f7e64055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:13.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.891+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7e7406c4b0 con 0x7f7e840ab670 2022-07-12T08:48:13.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.891+0000 7f7e8967b700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7e64053a40 0x7f7e64055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:13.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.891+0000 7f7e8967b700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7e64053a40 0x7f7e64055f00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f7e7800d000 tx=0x7f7e78007a90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:13.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:13.895+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7e74043410 con 0x7f7e840ab670 2022-07-12T08:48:14.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.019+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e84065200 con 0x7f7e840ab670 2022-07-12T08:48:14.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.019+0000 7f7e827fc700 1 -- 172.21.15.61:0/248401938 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f7e74046a30 con 0x7f7e840ab670 2022-07-12T08:48:14.023 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:14.024 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":81,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:13.437499+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:11.002774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:11.580821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:14.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.019+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7e64053a40 msgr2=0x7f7e64055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.019+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7e64053a40 0x7f7e64055f00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f7e7800d000 tx=0x7f7e78007a90).stop 2022-07-12T08:48:14.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.019+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 msgr2=0x7f7e84159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.019+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 0x7f7e84159650 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7f7e7400c000 tx=0x7f7e740091d0).stop 2022-07-12T08:48:14.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 shutdown_connections 2022-07-12T08:48:14.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7e840afa60 0x7f7e84159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e840ab670 0x7f7e84159650 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7e64053a40 0x7f7e64055f00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 --2- 172.21.15.61:0/248401938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e840ab290 0x7f7e84063f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 >> 172.21.15.61:0/248401938 conn(0x7f7e840b1950 msgr2=0x7f7e840b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:14.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 shutdown_connections 2022-07-12T08:48:14.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.023+0000 7f7e8a67d700 1 -- 172.21.15.61:0/248401938 wait complete. 2022-07-12T08:48:14.039 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:48:14.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 Processor -- start 2022-07-12T08:48:14.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 -- start start 2022-07-12T08:48:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9ecc0b4050 0x7f9ecc0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9ecc141e50 0x7f9ecc1422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc0b56c0 con 0x7f9ecc0b52e0 2022-07-12T08:48:14.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc142f00 con 0x7f9ecc0b4050 2022-07-12T08:48:14.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ed1d13700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc0b1320 con 0x7f9ecc141e50 2022-07-12T08:48:14.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9ecc0b4050 0x7f9ecc0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9ecc0b4050 0x7f9ecc0b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58026/0 (socket says 172.21.15.61:58026) 2022-07-12T08:48:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecb7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9ecc0b52e0 0x7f9ecc0b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46244/0 (socket says 172.21.15.61:46244) 2022-07-12T08:48:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 -- 172.21.15.61:0/2803345589 learned_addr learned my addr 172.21.15.61:0/2803345589 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:14.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecbfff700 1 --2- 172.21.15.61:0/2803345589 >> v2:172.21.15.111:3300/0 conn(0x7f9ecc141e50 0x7f9ecc1422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 -- 172.21.15.61:0/2803345589 >> v2:172.21.15.111:3300/0 conn(0x7f9ecc141e50 msgr2=0x7f9ecc1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 --2- 172.21.15.61:0/2803345589 >> v2:172.21.15.111:3300/0 conn(0x7f9ecc141e50 0x7f9ecc1422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 -- 172.21.15.61:0/2803345589 >> v1:172.21.15.61:6789/0 conn(0x7f9ecc0b52e0 legacy=0x7f9ecc0b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9eca7fc700 1 -- 172.21.15.61:0/2803345589 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229036932 0 0) 0x7f9ecc0b56c0 con 0x7f9ecc0b52e0 2022-07-12T08:48:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 -- 172.21.15.61:0/2803345589 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0b4bd0 con 0x7f9ecc0b4050 2022-07-12T08:48:14.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9ecaffd700 1 --2- 172.21.15.61:0/2803345589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc0b28c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f9ecc0b1460 tx=0x7f9ec0006eb0).ready entity=mon.0 client_cookie=60ad30d3120dc167 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9eca7fc700 1 -- 172.21.15.61:0/2803345589 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec000d610 con 0x7f9ecc0b4050 2022-07-12T08:48:14.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9eca7fc700 1 -- 172.21.15.61:0/2803345589 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec0000e90 con 0x7f9ecc0b4050 2022-07-12T08:48:14.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.151+0000 7f9eca7fc700 1 -- 172.21.15.61:0/2803345589 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec000d960 con 0x7f9ecc0b4050 2022-07-12T08:48:14.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 -- 172.21.15.61:0/2803345589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc0b4050 msgr2=0x7f9ecc0b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/2803345589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc0b28c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f9ecc0b1460 tx=0x7f9ec0006eb0).stop 2022-07-12T08:48:14.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 -- 172.21.15.61:0/2803345589 shutdown_connections 2022-07-12T08:48:14.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/2803345589 >> v2:172.21.15.111:3300/0 conn(0x7f9ecc141e50 0x7f9ecc1422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/2803345589 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc0b28c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 -- 172.21.15.61:0/2803345589 >> 172.21.15.61:0/2803345589 conn(0x7f9ecc067020 msgr2=0x7f9ecc067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 -- 172.21.15.61:0/2803345589 shutdown_connections 2022-07-12T08:48:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 -- 172.21.15.61:0/2803345589 wait complete. 2022-07-12T08:48:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 Processor -- start 2022-07-12T08:48:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.155+0000 7f9ed1d13700 1 -- start start 2022-07-12T08:48:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ecc0b52e0 0x7f9ecc152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 0x7f9ecc13db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1526f0 con 0x7f9ecc141e50 2022-07-12T08:48:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc152910 con 0x7f9ecc0b4050 2022-07-12T08:48:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc152bb0 con 0x7f9ecc0b52e0 2022-07-12T08:48:14.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 0x7f9ecc13db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 0x7f9ecc13db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58036/0 (socket says 172.21.15.61:58036) 2022-07-12T08:48:14.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 -- 172.21.15.61:0/3101647096 learned_addr learned my addr 172.21.15.61:0/3101647096 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:14.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecb7fe700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecaffd700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ecc0b52e0 0x7f9ecc152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 -- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ecc0b52e0 msgr2=0x7f9ecc152200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ecc0b52e0 0x7f9ecc152200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 -- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc0b4050 msgr2=0x7f9ecc147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 -- 172.21.15.61:0/3101647096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec000b1e0 con 0x7f9ecc141e50 2022-07-12T08:48:14.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ecbfff700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 0x7f9ecc13db50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f9ec4006e00 tx=0x7f9ec40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec4012600 con 0x7f9ecc141e50 2022-07-12T08:48:14.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec4012760 con 0x7f9ecc141e50 2022-07-12T08:48:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec4012ac0 con 0x7f9ecc141e50 2022-07-12T08:48:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc152e50 con 0x7f9ecc141e50 2022-07-12T08:48:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.159+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc13e1e0 con 0x7f9ecc141e50 2022-07-12T08:48:14.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.163+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc152fe0 con 0x7f9ecc141e50 2022-07-12T08:48:14.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.163+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9ec4022070 con 0x7f9ecc141e50 2022-07-12T08:48:14.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.163+0000 7f9ec8ff9700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9eac053af0 0x7f9eac055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.163+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ec406c430 con 0x7f9ecc141e50 2022-07-12T08:48:14.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.163+0000 7f9ecb7fe700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9eac053af0 0x7f9eac055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.163+0000 7f9ecb7fe700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9eac053af0 0x7f9eac055fb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f9ebc00d000 tx=0x7f9ebc007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.167+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ec40433a0 con 0x7f9ecc141e50 2022-07-12T08:48:14.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.339+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9ecc057460 con 0x7f9ecc141e50 2022-07-12T08:48:14.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ec8ff9700 1 -- 172.21.15.61:0/3101647096 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9ec4016020 con 0x7f9ecc141e50 2022-07-12T08:48:14.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9eac053af0 msgr2=0x7f9eac055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9eac053af0 0x7f9eac055fb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f9ebc00d000 tx=0x7f9ebc007a00).stop 2022-07-12T08:48:14.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 msgr2=0x7f9ecc13db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 0x7f9ecc13db50 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f9ec4006e00 tx=0x7f9ec40091d0).stop 2022-07-12T08:48:14.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 shutdown_connections 2022-07-12T08:48:14.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ecc141e50 0x7f9ecc13db50 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ecc0b52e0 0x7f9ecc152200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9eac053af0 0x7f9eac055fb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 --2- 172.21.15.61:0/3101647096 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ecc0b4050 0x7f9ecc147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 >> 172.21.15.61:0/3101647096 conn(0x7f9ecc067020 msgr2=0x7f9ecc05bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:14.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 shutdown_connections 2022-07-12T08:48:14.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.347+0000 7f9ed1d13700 1 -- 172.21.15.61:0/3101647096 wait complete. 2022-07-12T08:48:14.357 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:48:14.376 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:48:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 Processor -- start 2022-07-12T08:48:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 -- start start 2022-07-12T08:48:14.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f61e4060280 0x7f61e40b4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f61e40b52a0 0x7f61e405c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e41428a0 con 0x7f61e40b49d0 2022-07-12T08:48:14.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e4143510 con 0x7f61e40b52a0 2022-07-12T08:48:14.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61ea1d2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e41437b0 con 0x7f61e4060280 2022-07-12T08:48:14.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61e91d0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f61e40b49d0 0x7f61e40b8430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46258/0 (socket says 172.21.15.61:46258) 2022-07-12T08:48:14.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61e99d1700 1 --2- 172.21.15.61:0/2802338572 >> v2:172.21.15.61:3300/0 conn(0x7f61e40b52a0 0x7f61e405c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.499+0000 7f61e91d0700 1 -- 172.21.15.61:0/2802338572 learned_addr learned my addr 172.21.15.61:0/2802338572 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:14.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e89cf700 1 --2- 172.21.15.61:0/2802338572 >> v2:172.21.15.111:3300/0 conn(0x7f61e4060280 0x7f61e40b4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e99d1700 1 -- 172.21.15.61:0/2802338572 >> v2:172.21.15.111:3300/0 conn(0x7f61e4060280 msgr2=0x7f61e40b4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:14.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e99d1700 1 --2- 172.21.15.61:0/2802338572 >> v2:172.21.15.111:3300/0 conn(0x7f61e4060280 0x7f61e40b4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61dbfff700 1 -- 172.21.15.61:0/2802338572 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584911236 0 0) 0x7f61e41428a0 con 0x7f61e40b49d0 2022-07-12T08:48:14.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e99d1700 1 -- 172.21.15.61:0/2802338572 >> v1:172.21.15.61:6789/0 conn(0x7f61e40b49d0 legacy=0x7f61e40b8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e99d1700 1 -- 172.21.15.61:0/2802338572 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e40b5710 con 0x7f61e40b52a0 2022-07-12T08:48:14.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e89cf700 1 --2- 172.21.15.61:0/2802338572 >> v2:172.21.15.111:3300/0 conn(0x7f61e4060280 0x7f61e40b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:14.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61e99d1700 1 --2- 172.21.15.61:0/2802338572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b52a0 0x7f61e405c020 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f61dc0032f0 tx=0x7f61dc0091d0).ready entity=mon.0 client_cookie=6d236838aa95529 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61dbfff700 1 -- 172.21.15.61:0/2802338572 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61dc00b680 con 0x7f61e40b52a0 2022-07-12T08:48:14.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61dbfff700 1 -- 172.21.15.61:0/2802338572 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc00b7e0 con 0x7f61e40b52a0 2022-07-12T08:48:14.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61dbfff700 1 -- 172.21.15.61:0/2802338572 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61dc00b950 con 0x7f61e40b52a0 2022-07-12T08:48:14.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 -- 172.21.15.61:0/2802338572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b52a0 msgr2=0x7f61e405c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/2802338572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b52a0 0x7f61e405c020 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7f61dc0032f0 tx=0x7f61dc0091d0).stop 2022-07-12T08:48:14.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 -- 172.21.15.61:0/2802338572 shutdown_connections 2022-07-12T08:48:14.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/2802338572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b52a0 0x7f61e405c020 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/2802338572 >> v2:172.21.15.111:3300/0 conn(0x7f61e4060280 0x7f61e40b4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 -- 172.21.15.61:0/2802338572 >> 172.21.15.61:0/2802338572 conn(0x7f61e40b9310 msgr2=0x7f61e40b96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:14.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 -- 172.21.15.61:0/2802338572 shutdown_connections 2022-07-12T08:48:14.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 -- 172.21.15.61:0/2802338572 wait complete. 2022-07-12T08:48:14.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 Processor -- start 2022-07-12T08:48:14.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.503+0000 7f61ea1d2700 1 -- start start 2022-07-12T08:48:14.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61e4060280 0x7f61e413f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 0x7f61e413e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61e40b52a0 0x7f61e4146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40b1cc0 con 0x7f61e40b49d0 2022-07-12T08:48:14.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40b30c0 con 0x7f61e40b52a0 2022-07-12T08:48:14.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e40b1ef0 con 0x7f61e4060280 2022-07-12T08:48:14.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 0x7f61e413e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 0x7f61e413e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58048/0 (socket says 172.21.15.61:58048) 2022-07-12T08:48:14.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 -- 172.21.15.61:0/181287738 learned_addr learned my addr 172.21.15.61:0/181287738 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:14.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e91d0700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61e4060280 0x7f61e413f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 -- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61e4060280 msgr2=0x7f61e413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61e4060280 0x7f61e413f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e99d1700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61e40b52a0 0x7f61e4146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 -- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61e40b52a0 msgr2=0x7f61e4146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61e40b52a0 0x7f61e4146060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 -- 172.21.15.61:0/181287738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e000b180 con 0x7f61e40b49d0 2022-07-12T08:48:14.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e99d1700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61e40b52a0 0x7f61e4146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:14.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61e89cf700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 0x7f61e413e130 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7f61e000c000 tx=0x7f61e0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e000ca40 con 0x7f61e40b49d0 2022-07-12T08:48:14.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e000cba0 con 0x7f61e40b49d0 2022-07-12T08:48:14.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc009470 con 0x7f61e40b49d0 2022-07-12T08:48:14.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e00123f0 con 0x7f61e40b49d0 2022-07-12T08:48:14.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e40b2db0 con 0x7f61e40b49d0 2022-07-12T08:48:14.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e40b20f0 con 0x7f61e40b49d0 2022-07-12T08:48:14.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.507+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f61e0012550 con 0x7f61e40b49d0 2022-07-12T08:48:14.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.511+0000 7f61da7fc700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f61cc053af0 0x7f61cc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.511+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f61e006c5f0 con 0x7f61e40b49d0 2022-07-12T08:48:14.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.511+0000 7f61e91d0700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f61cc053af0 0x7f61cc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.511+0000 7f61e91d0700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f61cc053af0 0x7f61cc055fb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f61d4008000 tx=0x7f61d4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.515+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61e00433e0 con 0x7f61e40b49d0 2022-07-12T08:48:14.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.679+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f61e4057460 con 0x7f61e40b49d0 2022-07-12T08:48:14.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.687+0000 7f61da7fc700 1 -- 172.21.15.61:0/181287738 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v27) v1 ==== 184+0+0 (secure 0 0 0) 0x7f61e0046a00 con 0x7f61e40b49d0 2022-07-12T08:48:14.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f61cc053af0 msgr2=0x7f61cc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f61cc053af0 0x7f61cc055fb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f61d4008000 tx=0x7f61d4009040).stop 2022-07-12T08:48:14.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 msgr2=0x7f61e413e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 0x7f61e413e130 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7f61e000c000 tx=0x7f61e0006df0).stop 2022-07-12T08:48:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 shutdown_connections 2022-07-12T08:48:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f61cc053af0 0x7f61cc055fb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61e40b52a0 0x7f61e4146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61e40b49d0 0x7f61e413e130 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 --2- 172.21.15.61:0/181287738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61e4060280 0x7f61e413f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 >> 172.21.15.61:0/181287738 conn(0x7f61e40b9310 msgr2=0x7f61e40aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 shutdown_connections 2022-07-12T08:48:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.691+0000 7f61ea1d2700 1 -- 172.21.15.61:0/181287738 wait complete. 2022-07-12T08:48:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:48:14.708 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:48:14.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.815+0000 7f1716f5f700 1 Processor -- start 2022-07-12T08:48:14.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.815+0000 7f1716f5f700 1 -- start start 2022-07-12T08:48:14.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1716f5f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1710067030 0x7f17101447e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1716f5f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 0x7f1710140920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1716f5f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1710067410 con 0x7f1710140e10 2022-07-12T08:48:14.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1716f5f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17101440a0 con 0x7f1710067030 2022-07-12T08:48:14.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1716f5f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1710144340 con 0x7f17100b22c0 2022-07-12T08:48:14.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 0x7f1710140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1715f5d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1710067030 0x7f17101447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 0x7f1710140920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40254/0 (socket says 172.21.15.61:40254) 2022-07-12T08:48:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171675e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f1710140e10 0x7f1710129830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:59876/0 (socket says 172.21.15.61:59876) 2022-07-12T08:48:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1715f5d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1710067030 0x7f17101447e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36628/0 (socket says 172.21.15.61:36628) 2022-07-12T08:48:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 -- 172.21.15.61:0/3659839737 learned_addr learned my addr 172.21.15.61:0/3659839737 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1714f5b700 1 -- 172.21.15.61:0/3659839737 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576827291 0 0) 0x7f1710067410 con 0x7f1710140e10 2022-07-12T08:48:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1714f5b700 1 -- 172.21.15.61:0/3659839737 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f8003260 con 0x7f1710140e10 2022-07-12T08:48:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 -- 172.21.15.61:0/3659839737 >> v2:172.21.15.111:3300/0 conn(0x7f1710067030 msgr2=0x7f17101447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 --2- 172.21.15.61:0/3659839737 >> v2:172.21.15.111:3300/0 conn(0x7f1710067030 0x7f17101447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 -- 172.21.15.61:0/3659839737 >> v1:172.21.15.111:6789/0 conn(0x7f1710140e10 legacy=0x7f1710129830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 -- 172.21.15.61:0/3659839737 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171005d3c0 con 0x7f17100b22c0 2022-07-12T08:48:14.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f171575c700 1 --2- 172.21.15.61:0/3659839737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 0x7f1710140920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1700002f70 tx=0x7f1700006eb0).ready entity=mon.2 client_cookie=6c2bf6cfc4033dae server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1714f5b700 1 -- 172.21.15.61:0/3659839737 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f170000d5c0 con 0x7f17100b22c0 2022-07-12T08:48:14.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1714f5b700 1 -- 172.21.15.61:0/3659839737 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700000e90 con 0x7f17100b22c0 2022-07-12T08:48:14.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.819+0000 7f1714f5b700 1 -- 172.21.15.61:0/3659839737 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f170000d950 con 0x7f17100b22c0 2022-07-12T08:48:14.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- 172.21.15.61:0/3659839737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 msgr2=0x7f1710140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 --2- 172.21.15.61:0/3659839737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 0x7f1710140920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1700002f70 tx=0x7f1700006eb0).stop 2022-07-12T08:48:14.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- 172.21.15.61:0/3659839737 shutdown_connections 2022-07-12T08:48:14.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 --2- 172.21.15.61:0/3659839737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17100b22c0 0x7f1710140920 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 --2- 172.21.15.61:0/3659839737 >> v2:172.21.15.111:3300/0 conn(0x7f1710067030 0x7f17101447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- 172.21.15.61:0/3659839737 >> 172.21.15.61:0/3659839737 conn(0x7f1710063ad0 msgr2=0x7f1710063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- 172.21.15.61:0/3659839737 shutdown_connections 2022-07-12T08:48:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- 172.21.15.61:0/3659839737 wait complete. 2022-07-12T08:48:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 Processor -- start 2022-07-12T08:48:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- start start 2022-07-12T08:48:14.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1710067030 0x7f17100b6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 0x7f1710126f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1710140e10 0x7f1710127470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1710151a50 con 0x7f17100b22c0 2022-07-12T08:48:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17100b0000 con 0x7f1710140e10 2022-07-12T08:48:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1716f5f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17101405f0 con 0x7f1710067030 2022-07-12T08:48:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f171575c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 0x7f1710126f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f171575c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 0x7f1710126f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58064/0 (socket says 172.21.15.61:58064) 2022-07-12T08:48:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f1715f5d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1710067030 0x7f17100b6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f171575c700 1 -- 172.21.15.61:0/2843248681 learned_addr learned my addr 172.21.15.61:0/2843248681 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.823+0000 7f171675e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1710140e10 0x7f1710127470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f171575c700 1 -- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1710067030 msgr2=0x7f17100b6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f171575c700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1710067030 0x7f17100b6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f171575c700 1 -- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1710140e10 msgr2=0x7f1710127470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:14.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f171575c700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1710140e10 0x7f1710127470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:14.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f171575c700 1 -- 172.21.15.61:0/2843248681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170000b1e0 con 0x7f17100b22c0 2022-07-12T08:48:14.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f171575c700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 0x7f1710126f80 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f171012fb00 tx=0x7f1700015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f170000de80 con 0x7f17100b22c0 2022-07-12T08:48:14.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700011bf0 con 0x7f17100b22c0 2022-07-12T08:48:14.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17100669e0 con 0x7f17100b22c0 2022-07-12T08:48:14.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f170001e460 con 0x7f17100b22c0 2022-07-12T08:48:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1710127960 con 0x7f17100b22c0 2022-07-12T08:48:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1710066840 con 0x7f17100b22c0 2022-07-12T08:48:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.827+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1700025030 con 0x7f17100b22c0 2022-07-12T08:48:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.831+0000 7f16feffd700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16f00539b0 0x7f16f0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.831+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f170006ce70 con 0x7f17100b22c0 2022-07-12T08:48:14.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.831+0000 7f1715f5d700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16f00539b0 0x7f16f0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:14.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.831+0000 7f1715f5d700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16f00539b0 0x7f16f0055e70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f170c00b000 tx=0x7f170c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:14.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.839+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1710066840 con 0x7f17100b22c0 2022-07-12T08:48:15.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:14.999+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1710066840 con 0x7f17100b22c0 2022-07-12T08:48:15.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.007+0000 7f16feffd700 1 -- 172.21.15.61:0/2843248681 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1710066840 con 0x7f17100b22c0 2022-07-12T08:48:15.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16f00539b0 msgr2=0x7f16f0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16f00539b0 0x7f16f0055e70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f170c00b000 tx=0x7f170c006df0).stop 2022-07-12T08:48:15.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 msgr2=0x7f1710126f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 0x7f1710126f80 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 rx=0x7f171012fb00 tx=0x7f1700015000).stop 2022-07-12T08:48:15.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 shutdown_connections 2022-07-12T08:48:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f16f00539b0 0x7f16f0055e70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1710140e10 0x7f1710127470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17100b22c0 0x7f1710126f80 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 --2- 172.21.15.61:0/2843248681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1710067030 0x7f17100b6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 >> 172.21.15.61:0/2843248681 conn(0x7f1710063ad0 msgr2=0x7f1710129bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 shutdown_connections 2022-07-12T08:48:15.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.011+0000 7f1716f5f700 1 -- 172.21.15.61:0/2843248681 wait complete. 2022-07-12T08:48:15.021 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:48:15.032 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:15.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.131+0000 7fe7b0ad8700 1 Processor -- start 2022-07-12T08:48:15.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- start start 2022-07-12T08:48:15.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac0aa7f0 0x7fe7ac098200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe7ac0b1450 0x7fe7ac0986f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7ac05f650 con 0x7fe7ac05f140 2022-07-12T08:48:15.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7ac0aabd0 con 0x7fe7ac0b1450 2022-07-12T08:48:15.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac067c80 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe7ac0b1450 0x7fe7ac0986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe7ac0b1450 0x7fe7ac0986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58068/0 (socket says 172.21.15.61:58068) 2022-07-12T08:48:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1394839005 learned_addr learned my addr 172.21.15.61:0/1394839005 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7ab7fe700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac0aa7f0 0x7fe7ac098200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1394839005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac0aa7f0 msgr2=0x7fe7ac098200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:15.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac0aa7f0 0x7fe7ac098200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1394839005 >> v1:172.21.15.111:6789/0 conn(0x7fe7ac05f140 legacy=0x7fe7ac142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1394839005 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ac098be0 con 0x7fe7ac0b1450 2022-07-12T08:48:15.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7ab7fe700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac0aa7f0 0x7fe7ac098200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:15.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7a3fff700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac0986f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fe794002f70 tx=0x7fe794006eb0).ready entity=mon.0 client_cookie=d88de59f358a38d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:15.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7aaffd700 1 -- 172.21.15.61:0/1394839005 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe79400d5c0 con 0x7fe7ac0b1450 2022-07-12T08:48:15.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7aaffd700 1 -- 172.21.15.61:0/1394839005 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe794000e90 con 0x7fe7ac0b1450 2022-07-12T08:48:15.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7aaffd700 1 -- 172.21.15.61:0/1394839005 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe79400d950 con 0x7fe7ac0b1450 2022-07-12T08:48:15.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1394839005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 msgr2=0x7fe7ac0986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac0986f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fe794002f70 tx=0x7fe794006eb0).stop 2022-07-12T08:48:15.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1394839005 shutdown_connections 2022-07-12T08:48:15.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac0986f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1394839005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac0aa7f0 0x7fe7ac098200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1394839005 >> 172.21.15.61:0/1394839005 conn(0x7fe7ac0b1950 msgr2=0x7fe7ac0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:15.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1394839005 shutdown_connections 2022-07-12T08:48:15.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.135+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1394839005 wait complete. 2022-07-12T08:48:15.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 Processor -- start 2022-07-12T08:48:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 -- start start 2022-07-12T08:48:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac05f140 0x7fe7ac063ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 0x7fe7ac137370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0ae240 con 0x7fe7ac0b1450 2022-07-12T08:48:15.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0ae490 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ac0ae730 con 0x7fe7ac05f140 2022-07-12T08:48:15.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7ab7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac05f140 0x7fe7ac063ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 0x7fe7ac137370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 0x7fe7ac137370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36650/0 (socket says 172.21.15.61:36650) 2022-07-12T08:48:15.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7ab7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac05f140 0x7fe7ac063ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40270/0 (socket says 172.21.15.61:40270) 2022-07-12T08:48:15.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac137860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58078/0 (socket says 172.21.15.61:58078) 2022-07-12T08:48:15.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1005342926 learned_addr learned my addr 172.21.15.61:0/1005342926 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:15.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac05f140 msgr2=0x7fe7ac063ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac05f140 0x7fe7ac063ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1005342926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 msgr2=0x7fe7ac137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 -- 172.21.15.61:0/1005342926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79400b1e0 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a3fff700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 0x7fe7ac137370 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe794002f70 tx=0x7fe794015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:15.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe79400d940 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe794006b40 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ac137d50 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe794011d30 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.139+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ac0aebd0 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.143+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ac0ae8f0 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.143+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe794025030 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.143+0000 7fe7a97fa700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe790053a60 0x7fe790055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.143+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe79406d190 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.143+0000 7fe7ab7fe700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe790053a60 0x7fe790055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.143+0000 7fe7ab7fe700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe790053a60 0x7fe790055f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fe79c00c000 tx=0x7fe79c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.151+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe794043fe0 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.315+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7ac0ad920 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.315+0000 7fe7a97fa700 1 -- 172.21.15.61:0/1005342926 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fe794047580 con 0x7fe7ac0aa7f0 2022-07-12T08:48:15.320 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:15.321 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:13.437914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3665156048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3665156048},{"type":"v1","addr":"172.21.15.61:6833","nonce":3665156048}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:15.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe790053a60 msgr2=0x7fe790055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe790053a60 0x7fe790055f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fe79c00c000 tx=0x7fe79c008040).stop 2022-07-12T08:48:15.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 msgr2=0x7fe7ac137370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 0x7fe7ac137370 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe794002f70 tx=0x7fe794015000).stop 2022-07-12T08:48:15.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 shutdown_connections 2022-07-12T08:48:15.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7ac0b1450 0x7fe7ac137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe790053a60 0x7fe790055f20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7ac0aa7f0 0x7fe7ac137370 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 --2- 172.21.15.61:0/1005342926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe7ac05f140 0x7fe7ac063ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 >> 172.21.15.61:0/1005342926 conn(0x7fe7ac0b1950 msgr2=0x7fe7ac05bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:15.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 shutdown_connections 2022-07-12T08:48:15.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.319+0000 7fe7b0ad8700 1 -- 172.21.15.61:0/1005342926 wait complete. 2022-07-12T08:48:15.326 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 44 2022-07-12T08:48:15.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:12.433872+0000', 'modified': '2022-07-12T08:48:13.437914+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'a', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3665156048', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3665156048}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3665156048}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:48:15.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:48:15.543 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:48:15.662 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5981,"want_state":"up:active","state":"up:active","rank_uptime":3.213851456,"mdsmap_epoch":44,"osdmap_epoch":81,"osdmap_epoch_barrier":80,"uptime":7.5552670129999999} 2022-07-12T08:48:15.663 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5981, "mdsmap_epoch": 44, "osdmap_epoch": 81, "osdmap_epoch_barrier": 80, "rank_uptime": 3.213851456, "state": "up:active", "uptime": 7.555267013, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:48:15.663 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:15.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70abfff700 1 Processor -- start 2022-07-12T08:48:15.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70abfff700 1 -- start start 2022-07-12T08:48:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70abfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f70ac14b070 0x7f70ac0b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70abfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac0b29b0 con 0x7f70ac0b2090 2022-07-12T08:48:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70abfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac061e60 con 0x7f70ac066ab0 2022-07-12T08:48:15.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70abfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac062100 con 0x7f70ac14b070 2022-07-12T08:48:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70aaffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f70ac0b2090 0x7f70ac0672c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46302/0 (socket says 172.21.15.61:46302) 2022-07-12T08:48:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70aaffd700 1 -- 172.21.15.61:0/1495991521 learned_addr learned my addr 172.21.15.61:0/1495991521 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.767+0000 7f70ab7fe700 1 --2- 172.21.15.61:0/1495991521 >> v2:172.21.15.111:3300/0 conn(0x7f70ac14b070 0x7f70ac0b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70a9ffb700 1 -- 172.21.15.61:0/1495991521 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949997690 0 0) 0x7f70ac0b29b0 con 0x7f70ac0b2090 2022-07-12T08:48:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70a9ffb700 1 -- 172.21.15.61:0/1495991521 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708c003260 con 0x7f70ac0b2090 2022-07-12T08:48:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70ab7fe700 1 -- 172.21.15.61:0/1495991521 >> v1:172.21.15.111:6789/0 conn(0x7f70ac066ab0 legacy=0x7f70ac0b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70ab7fe700 1 -- 172.21.15.61:0/1495991521 >> v1:172.21.15.61:6789/0 conn(0x7f70ac0b2090 legacy=0x7f70ac0672c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70ab7fe700 1 -- 172.21.15.61:0/1495991521 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac0623a0 con 0x7f70ac14b070 2022-07-12T08:48:15.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70ab7fe700 1 --2- 172.21.15.61:0/1495991521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b2470 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f70a4002f70 tx=0x7f70a4006df0).ready entity=mon.1 client_cookie=236c4024a21ddfaf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70a9ffb700 1 -- 172.21.15.61:0/1495991521 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70a400d610 con 0x7f70ac14b070 2022-07-12T08:48:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70a9ffb700 1 -- 172.21.15.61:0/1495991521 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4000e90 con 0x7f70ac14b070 2022-07-12T08:48:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70a9ffb700 1 -- 172.21.15.61:0/1495991521 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f70a400d960 con 0x7f70ac14b070 2022-07-12T08:48:15.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 -- 172.21.15.61:0/1495991521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 msgr2=0x7f70ac0b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 --2- 172.21.15.61:0/1495991521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b2470 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f70a4002f70 tx=0x7f70a4006df0).stop 2022-07-12T08:48:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 -- 172.21.15.61:0/1495991521 shutdown_connections 2022-07-12T08:48:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 --2- 172.21.15.61:0/1495991521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b2470 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 -- 172.21.15.61:0/1495991521 >> 172.21.15.61:0/1495991521 conn(0x7f70ac0b70c0 msgr2=0x7f70ac0b74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 -- 172.21.15.61:0/1495991521 shutdown_connections 2022-07-12T08:48:15.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.771+0000 7f70abfff700 1 -- 172.21.15.61:0/1495991521 wait complete. 2022-07-12T08:48:15.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 Processor -- start 2022-07-12T08:48:15.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 -- start start 2022-07-12T08:48:15.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 0x7f70ac09a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70ac0b2090 0x7f70ac09bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac099bf0 con 0x7f70ac066ab0 2022-07-12T08:48:15.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac09c9c0 con 0x7f70ac14b070 2022-07-12T08:48:15.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70abfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac09ce50 con 0x7f70ac0b2090 2022-07-12T08:48:15.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 0x7f70ac09a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 0x7f70ac09a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58088/0 (socket says 172.21.15.61:58088) 2022-07-12T08:48:15.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 -- 172.21.15.61:0/3110020628 learned_addr learned my addr 172.21.15.61:0/3110020628 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:15.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70ab7fe700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aa7fc700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70ac0b2090 0x7f70ac09bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 -- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70ac0b2090 msgr2=0x7f70ac09bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70ac0b2090 0x7f70ac09bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 -- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 msgr2=0x7f70ac0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.775+0000 7f70aaffd700 1 -- 172.21.15.61:0/3110020628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a400b180 con 0x7f70ac066ab0 2022-07-12T08:48:15.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70aaffd700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 0x7f70ac09a1f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f709c001110 tx=0x7f709c007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:15.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f709c0070d0 con 0x7f70ac066ab0 2022-07-12T08:48:15.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c007230 con 0x7f70ac066ab0 2022-07-12T08:48:15.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ac09c3e0 con 0x7f70ac066ab0 2022-07-12T08:48:15.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f709c005610 con 0x7f70ac066ab0 2022-07-12T08:48:15.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac0af7f0 con 0x7f70ac066ab0 2022-07-12T08:48:15.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac09c5d0 con 0x7f70ac066ab0 2022-07-12T08:48:15.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f709c01a060 con 0x7f70ac066ab0 2022-07-12T08:48:15.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70a3fff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7090053af0 0x7f7090055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:15.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f709c06b6d0 con 0x7f70ac066ab0 2022-07-12T08:48:15.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.779+0000 7f70aa7fc700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7090053af0 0x7f7090055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:15.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.783+0000 7f70aa7fc700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7090053af0 0x7f7090055fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f709400d000 tx=0x7f7094000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:15.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.787+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f709c042630 con 0x7f70ac066ab0 2022-07-12T08:48:15.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.915+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70ac09c800 con 0x7f70ac066ab0 2022-07-12T08:48:15.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70a3fff700 1 -- 172.21.15.61:0/3110020628 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f709c018020 con 0x7f70ac066ab0 2022-07-12T08:48:15.925 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:15.925 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":81,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:13.437499+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:11.002774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:11.580821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:15.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7090053af0 msgr2=0x7f7090055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7090053af0 0x7f7090055fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f709400d000 tx=0x7f7094000f40).stop 2022-07-12T08:48:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 msgr2=0x7f70ac09a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 0x7f70ac09a1f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f709c001110 tx=0x7f709c007b80).stop 2022-07-12T08:48:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 shutdown_connections 2022-07-12T08:48:15.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7090053af0 0x7f7090055fb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70ac14b070 0x7f70ac0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f70ac0b2090 0x7f70ac09bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 --2- 172.21.15.61:0/3110020628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70ac066ab0 0x7f70ac09a1f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:15.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 >> 172.21.15.61:0/3110020628 conn(0x7f70ac0b70c0 msgr2=0x7f70ac143db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:15.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 shutdown_connections 2022-07-12T08:48:15.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:15.919+0000 7f70abfff700 1 -- 172.21.15.61:0/3110020628 wait complete. 2022-07-12T08:48:15.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:48:11.002774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:48:11.580821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:48:15.947 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:16.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 Processor -- start 2022-07-12T08:48:16.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 -- start start 2022-07-12T08:48:16.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe4f005d060 0x7fe4f00b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f00afb50 0x7fe4f0143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f005d440 con 0x7fe4f0142ab0 2022-07-12T08:48:16.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f0143d80 con 0x7fe4f005d060 2022-07-12T08:48:16.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f6109700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00674b0 con 0x7fe4f00afb50 2022-07-12T08:48:16.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f4906700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe4f0142ab0 0x7fe4f0142f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46314/0 (socket says 172.21.15.61:46314) 2022-07-12T08:48:16.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f4906700 1 -- 172.21.15.61:0/2089004086 learned_addr learned my addr 172.21.15.61:0/2089004086 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:16.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f5908700 1 --2- 172.21.15.61:0/2089004086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f00afb50 0x7fe4f0143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4f5107700 1 --2- 172.21.15.61:0/2089004086 >> v2:172.21.15.111:3300/0 conn(0x7fe4f005d060 0x7fe4f00b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4dbfff700 1 -- 172.21.15.61:0/2089004086 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470607097 0 0) 0x7fe4f005d440 con 0x7fe4f0142ab0 2022-07-12T08:48:16.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.059+0000 7fe4dbfff700 1 -- 172.21.15.61:0/2089004086 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d4003260 con 0x7fe4f0142ab0 2022-07-12T08:48:16.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f5908700 1 -- 172.21.15.61:0/2089004086 >> v2:172.21.15.111:3300/0 conn(0x7fe4f005d060 msgr2=0x7fe4f00b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:16.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f5908700 1 --2- 172.21.15.61:0/2089004086 >> v2:172.21.15.111:3300/0 conn(0x7fe4f005d060 0x7fe4f00b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f5908700 1 -- 172.21.15.61:0/2089004086 >> v1:172.21.15.61:6789/0 conn(0x7fe4f0142ab0 legacy=0x7fe4f0142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f5908700 1 -- 172.21.15.61:0/2089004086 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0067750 con 0x7fe4f00afb50 2022-07-12T08:48:16.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f5908700 1 --2- 172.21.15.61:0/2089004086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f00afb50 0x7fe4f0143620 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe4e80032f0 tx=0x7fe4e80091d0).ready entity=mon.2 client_cookie=88cbc6d151a6c346 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4dbfff700 1 -- 172.21.15.61:0/2089004086 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800b600 con 0x7fe4f00afb50 2022-07-12T08:48:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4dbfff700 1 -- 172.21.15.61:0/2089004086 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e800b760 con 0x7fe4f00afb50 2022-07-12T08:48:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4dbfff700 1 -- 172.21.15.61:0/2089004086 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800b8d0 con 0x7fe4f00afb50 2022-07-12T08:48:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 -- 172.21.15.61:0/2089004086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f00afb50 msgr2=0x7fe4f0143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 --2- 172.21.15.61:0/2089004086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f00afb50 0x7fe4f0143620 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe4e80032f0 tx=0x7fe4e80091d0).stop 2022-07-12T08:48:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 -- 172.21.15.61:0/2089004086 shutdown_connections 2022-07-12T08:48:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 --2- 172.21.15.61:0/2089004086 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f00afb50 0x7fe4f0143620 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 --2- 172.21.15.61:0/2089004086 >> v2:172.21.15.111:3300/0 conn(0x7fe4f005d060 0x7fe4f00b0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 -- 172.21.15.61:0/2089004086 >> 172.21.15.61:0/2089004086 conn(0x7fe4f00b6900 msgr2=0x7fe4f0066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 -- 172.21.15.61:0/2089004086 shutdown_connections 2022-07-12T08:48:16.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.063+0000 7fe4f6109700 1 -- 172.21.15.61:0/2089004086 wait complete. 2022-07-12T08:48:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 Processor -- start 2022-07-12T08:48:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 -- start start 2022-07-12T08:48:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f005d060 0x7fe4f009cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00afb50 0x7fe4f00b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 0x7fe4f00b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00b2d40 con 0x7fe4f009ef20 2022-07-12T08:48:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f01529a0 con 0x7fe4f00afb50 2022-07-12T08:48:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0152700 con 0x7fe4f005d060 2022-07-12T08:48:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 0x7fe4f00b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 0x7fe4f00b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58106/0 (socket says 172.21.15.61:58106) 2022-07-12T08:48:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 -- 172.21.15.61:0/4203828645 learned_addr learned my addr 172.21.15.61:0/4203828645 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f4906700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00afb50 0x7fe4f00b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5107700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f005d060 0x7fe4f009cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 -- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f005d060 msgr2=0x7fe4f009cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f005d060 0x7fe4f009cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 -- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00afb50 msgr2=0x7fe4f00b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00afb50 0x7fe4f00b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 -- 172.21.15.61:0/4203828645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e8009470 con 0x7fe4f009ef20 2022-07-12T08:48:16.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f5908700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 0x7fe4f00b2850 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fe4e80032f0 tx=0x7fe4e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800b8d0 con 0x7fe4f009ef20 2022-07-12T08:48:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e801f3f0 con 0x7fe4f009ef20 2022-07-12T08:48:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f0152390 con 0x7fe4f009ef20 2022-07-12T08:48:16.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4e800b600 con 0x7fe4f009ef20 2022-07-12T08:48:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.067+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f0149d30 con 0x7fe4f009ef20 2022-07-12T08:48:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.071+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f009d4a0 con 0x7fe4f009ef20 2022-07-12T08:48:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.071+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe4e8026030 con 0x7fe4f009ef20 2022-07-12T08:48:16.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.071+0000 7fe4e2ffd700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4cc053940 0x7fe4cc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.071+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe4e806c8d0 con 0x7fe4f009ef20 2022-07-12T08:48:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.071+0000 7fe4f5107700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4cc053940 0x7fe4cc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.071+0000 7fe4f5107700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4cc053940 0x7fe4cc055e00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fe4e400b000 tx=0x7fe4e4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.075+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4e8043840 con 0x7fe4f009ef20 2022-07-12T08:48:16.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.203+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe4f0057460 con 0x7fe4f009ef20 2022-07-12T08:48:16.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.203+0000 7fe4e2ffd700 1 -- 172.21.15.61:0/4203828645 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe4e80240b0 con 0x7fe4f009ef20 2022-07-12T08:48:16.209 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:16.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4cc053940 msgr2=0x7fe4cc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4cc053940 0x7fe4cc055e00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fe4e400b000 tx=0x7fe4e4006df0).stop 2022-07-12T08:48:16.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 msgr2=0x7fe4f00b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 0x7fe4f00b2850 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7fe4e80032f0 tx=0x7fe4e8016000).stop 2022-07-12T08:48:16.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 shutdown_connections 2022-07-12T08:48:16.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4f009ef20 0x7fe4f00b2850 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4cc053940 0x7fe4cc055e00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4f00afb50 0x7fe4f00b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 --2- 172.21.15.61:0/4203828645 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4f005d060 0x7fe4f009cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 >> 172.21.15.61:0/4203828645 conn(0x7fe4f00b6900 msgr2=0x7fe4f00b7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 shutdown_connections 2022-07-12T08:48:16.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.207+0000 7fe4f6109700 1 -- 172.21.15.61:0/4203828645 wait complete. 2022-07-12T08:48:16.222 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:16.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.331+0000 7f5cda470700 1 Processor -- start 2022-07-12T08:48:16.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.331+0000 7f5cda470700 1 -- start start 2022-07-12T08:48:16.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.331+0000 7f5cda470700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.331+0000 7f5cda470700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5cd40b3730 0x7f5cd4142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.331+0000 7f5cda470700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cd40b3560 con 0x7f5cd40b5480 2022-07-12T08:48:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cd40b5b80 con 0x7f5cd40b3730 2022-07-12T08:48:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd40b5e20 con 0x7f5cd40b05f0 2022-07-12T08:48:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5cd40b3730 0x7f5cd4142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5cd40b3730 0x7f5cd4142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58114/0 (socket says 172.21.15.61:58114) 2022-07-12T08:48:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2412370763 learned_addr learned my addr 172.21.15.61:0/2412370763 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd37fe700 1 --2- 172.21.15.61:0/2412370763 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2412370763 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 msgr2=0x7f5cd40b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 --2- 172.21.15.61:0/2412370763 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd40b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2412370763 >> v1:172.21.15.111:6789/0 conn(0x7f5cd40b5480 legacy=0x7f5cd40afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2412370763 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd40b5860 con 0x7f5cd40b3730 2022-07-12T08:48:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd8a0d700 1 --2- 172.21.15.61:0/2412370763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b3730 0x7f5cd4142f90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f5ccc0032f0 tx=0x7f5ccc009290).ready entity=mon.0 client_cookie=eb7115486c980419 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd2ffd700 1 -- 172.21.15.61:0/2412370763 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ccc00b8a0 con 0x7f5cd40b3730 2022-07-12T08:48:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd2ffd700 1 -- 172.21.15.61:0/2412370763 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc00ba00 con 0x7f5cd40b3730 2022-07-12T08:48:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cd2ffd700 1 -- 172.21.15.61:0/2412370763 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ccc00bb70 con 0x7f5cd40b3730 2022-07-12T08:48:16.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- 172.21.15.61:0/2412370763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b3730 msgr2=0x7f5cd4142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 --2- 172.21.15.61:0/2412370763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b3730 0x7f5cd4142f90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 rx=0x7f5ccc0032f0 tx=0x7f5ccc009290).stop 2022-07-12T08:48:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- 172.21.15.61:0/2412370763 shutdown_connections 2022-07-12T08:48:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 --2- 172.21.15.61:0/2412370763 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b3730 0x7f5cd4142f90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 --2- 172.21.15.61:0/2412370763 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd40b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- 172.21.15.61:0/2412370763 >> 172.21.15.61:0/2412370763 conn(0x7f5cd40b96b0 msgr2=0x7f5cd40b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- 172.21.15.61:0/2412370763 shutdown_connections 2022-07-12T08:48:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- 172.21.15.61:0/2412370763 wait complete. 2022-07-12T08:48:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 Processor -- start 2022-07-12T08:48:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.335+0000 7f5cda470700 1 -- start start 2022-07-12T08:48:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd41450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cd40b3730 0x7f5cd40acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 0x7f5cd4152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd4113cf0 con 0x7f5cd40b5480 2022-07-12T08:48:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd40aeeb0 con 0x7f5cd40b3730 2022-07-12T08:48:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd4151fc0 con 0x7f5cd40b05f0 2022-07-12T08:48:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 0x7f5cd4152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 0x7f5cd4152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58118/0 (socket says 172.21.15.61:58118) 2022-07-12T08:48:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2970119681 learned_addr learned my addr 172.21.15.61:0/2970119681 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd37fe700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cd40b3730 0x7f5cd40acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd3fff700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd41450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 msgr2=0x7f5cd41450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd41450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cd40b3730 msgr2=0x7f5cd40acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cd40b3730 0x7f5cd40acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 -- 172.21.15.61:0/2970119681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc009530 con 0x7f5cd40b5480 2022-07-12T08:48:16.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd8a0d700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 0x7f5cd4152750 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f5ccc0032f0 tx=0x7f5ccc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ccc021440 con 0x7f5cd40b5480 2022-07-12T08:48:16.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ccc0215a0 con 0x7f5cd40b5480 2022-07-12T08:48:16.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd40ad050 con 0x7f5cd40b5480 2022-07-12T08:48:16.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ccc021900 con 0x7f5cd40b5480 2022-07-12T08:48:16.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd40aea80 con 0x7f5cd40b5480 2022-07-12T08:48:16.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd4152f80 con 0x7f5cd40b5480 2022-07-12T08:48:16.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.339+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5ccc027020 con 0x7f5cd40b5480 2022-07-12T08:48:16.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.343+0000 7f5cd17fa700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cb8053ad0 0x7f5cb8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.343+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ccc06d390 con 0x7f5cd40b5480 2022-07-12T08:48:16.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.343+0000 7f5cd3fff700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cb8053ad0 0x7f5cb8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.343+0000 7f5cd3fff700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cb8053ad0 0x7f5cb8055f90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f5cbc00b000 tx=0x7f5cbc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.347+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5cd4152f80 con 0x7f5cd40b5480 2022-07-12T08:48:16.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5cd4152f80 con 0x7f5cd40b5480 2022-07-12T08:48:16.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cd17fa700 1 -- 172.21.15.61:0/2970119681 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5cd4152f80 con 0x7f5cd40b5480 2022-07-12T08:48:16.481 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cb8053ad0 msgr2=0x7f5cb8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cb8053ad0 0x7f5cb8055f90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f5cbc00b000 tx=0x7f5cbc009040).stop 2022-07-12T08:48:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 msgr2=0x7f5cd4152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 0x7f5cd4152750 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7f5ccc0032f0 tx=0x7f5ccc016000).stop 2022-07-12T08:48:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 shutdown_connections 2022-07-12T08:48:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5cd40b5480 0x7f5cd4152750 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cb8053ad0 0x7f5cb8055f90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5cd40b3730 0x7f5cd40acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 --2- 172.21.15.61:0/2970119681 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5cd40b05f0 0x7f5cd41450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.475+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 >> 172.21.15.61:0/2970119681 conn(0x7f5cd40b96b0 msgr2=0x7f5cd4148640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.479+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 shutdown_connections 2022-07-12T08:48:16.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.479+0000 7f5cda470700 1 -- 172.21.15.61:0/2970119681 wait complete. 2022-07-12T08:48:16.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:48:16.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f819690a700 1 Processor -- start 2022-07-12T08:48:16.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f819690a700 1 -- start start 2022-07-12T08:48:16.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f819690a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b5480 0x7f81900b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f819690a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900b3560 con 0x7f81900b3730 2022-07-12T08:48:16.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f819690a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81900b5b80 con 0x7f81900b05f0 2022-07-12T08:48:16.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f819690a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81900b5e20 con 0x7f81900b5480 2022-07-12T08:48:16.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f81900b3730 0x7f81900afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46344/0 (socket says 172.21.15.61:46344) 2022-07-12T08:48:16.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818f7fe700 1 -- 172.21.15.61:0/1175206023 learned_addr learned my addr 172.21.15.61:0/1175206023 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:16.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818ffff700 1 --2- 172.21.15.61:0/1175206023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b5480 0x7f81900b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818effd700 1 -- 172.21.15.61:0/1175206023 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664028920 0 0) 0x7f81900b3560 con 0x7f81900b3730 2022-07-12T08:48:16.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818effd700 1 -- 172.21.15.61:0/1175206023 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c003260 con 0x7f81900b3730 2022-07-12T08:48:16.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818ffff700 1 -- 172.21.15.61:0/1175206023 >> v1:172.21.15.111:6789/0 conn(0x7f81900b05f0 legacy=0x7f8190142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818ffff700 1 -- 172.21.15.61:0/1175206023 >> v1:172.21.15.61:6789/0 conn(0x7f81900b3730 legacy=0x7f81900afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818ffff700 1 -- 172.21.15.61:0/1175206023 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81900b5860 con 0x7f81900b5480 2022-07-12T08:48:16.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818ffff700 1 --2- 172.21.15.61:0/1175206023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b5480 0x7f81900b45d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8180002f70 tx=0x7f8180006df0).ready entity=mon.2 client_cookie=7bb7789425be9f93 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818effd700 1 -- 172.21.15.61:0/1175206023 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818000d570 con 0x7f81900b5480 2022-07-12T08:48:16.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818effd700 1 -- 172.21.15.61:0/1175206023 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8180000e90 con 0x7f81900b5480 2022-07-12T08:48:16.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.607+0000 7f818effd700 1 -- 172.21.15.61:0/1175206023 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818000d8c0 con 0x7f81900b5480 2022-07-12T08:48:16.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- 172.21.15.61:0/1175206023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b5480 msgr2=0x7f81900b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 --2- 172.21.15.61:0/1175206023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b5480 0x7f81900b45d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f8180002f70 tx=0x7f8180006df0).stop 2022-07-12T08:48:16.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- 172.21.15.61:0/1175206023 shutdown_connections 2022-07-12T08:48:16.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 --2- 172.21.15.61:0/1175206023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b5480 0x7f81900b45d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- 172.21.15.61:0/1175206023 >> 172.21.15.61:0/1175206023 conn(0x7f81900b96b0 msgr2=0x7f81900b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- 172.21.15.61:0/1175206023 shutdown_connections 2022-07-12T08:48:16.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- 172.21.15.61:0/1175206023 wait complete. 2022-07-12T08:48:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 Processor -- start 2022-07-12T08:48:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- start start 2022-07-12T08:48:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81900b05f0 0x7f8190152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b3730 0x7f81901419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 0x7f8190141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8190140bd0 con 0x7f81900b5480 2022-07-12T08:48:16.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8190140e70 con 0x7f81900b05f0 2022-07-12T08:48:16.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f819690a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8190141110 con 0x7f81900b3730 2022-07-12T08:48:16.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f8194ea7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 0x7f8190141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f8194ea7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 0x7f8190141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58134/0 (socket says 172.21.15.61:58134) 2022-07-12T08:48:16.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f8194ea7700 1 -- 172.21.15.61:0/2358807665 learned_addr learned my addr 172.21.15.61:0/2358807665 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:16.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f818ffff700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81900b05f0 0x7f8190152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.611+0000 7f818f7fe700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b3730 0x7f81901419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818f7fe700 1 -- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81900b05f0 msgr2=0x7f8190152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818f7fe700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81900b05f0 0x7f8190152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818f7fe700 1 -- 172.21.15.61:0/2358807665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 msgr2=0x7f8190141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818f7fe700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 0x7f8190141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818f7fe700 1 -- 172.21.15.61:0/2358807665 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818000b180 con 0x7f81900b3730 2022-07-12T08:48:16.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f8194ea7700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 0x7f8190141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:16.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818f7fe700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b3730 0x7f81901419a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8184001110 tx=0x7f8184007a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818400ce80 con 0x7f81900b3730 2022-07-12T08:48:16.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81840070d0 con 0x7f81900b3730 2022-07-12T08:48:16.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8190153040 con 0x7f81900b3730 2022-07-12T08:48:16.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8184007430 con 0x7f81900b3730 2022-07-12T08:48:16.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8190152090 con 0x7f81900b3730 2022-07-12T08:48:16.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81900aeb20 con 0x7f81900b3730 2022-07-12T08:48:16.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.615+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f818401a060 con 0x7f81900b3730 2022-07-12T08:48:16.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.619+0000 7f818d7fa700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8170053ad0 0x7f8170055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.619+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f818406b460 con 0x7f81900b3730 2022-07-12T08:48:16.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.619+0000 7f818ffff700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8170053ad0 0x7f8170055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.619+0000 7f818ffff700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8170053ad0 0x7f8170055f90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f818001a000 tx=0x7f8180014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.627+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8184042360 con 0x7f81900b3730 2022-07-12T08:48:16.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.755+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8190057460 con 0x7f81900b3730 2022-07-12T08:48:16.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.755+0000 7f818d7fa700 1 -- 172.21.15.61:0/2358807665 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7f81840180a0 con 0x7f81900b3730 2022-07-12T08:48:16.760 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:16.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8170053ad0 msgr2=0x7f8170055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8170053ad0 0x7f8170055f90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f818001a000 tx=0x7f8180014000).stop 2022-07-12T08:48:16.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b3730 msgr2=0x7f81901419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b3730 0x7f81901419a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f8184001110 tx=0x7f8184007a20).stop 2022-07-12T08:48:16.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 shutdown_connections 2022-07-12T08:48:16.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81900b5480 0x7f8190141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81900b3730 0x7f81901419a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8170053ad0 0x7f8170055f90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 --2- 172.21.15.61:0/2358807665 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81900b05f0 0x7f8190152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 >> 172.21.15.61:0/2358807665 conn(0x7f81900b96b0 msgr2=0x7f81900b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 shutdown_connections 2022-07-12T08:48:16.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.759+0000 7f819690a700 1 -- 172.21.15.61:0/2358807665 wait complete. 2022-07-12T08:48:16.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:48:16.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 Processor -- start 2022-07-12T08:48:16.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- start start 2022-07-12T08:48:16.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480b0120 0x7f1b48144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1b480ab730 0x7f1b48144640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1b4814aad0 0x7f1b4814af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b480b1040 con 0x7f1b480ab730 2022-07-12T08:48:16.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b480b0500 con 0x7f1b4814aad0 2022-07-12T08:48:16.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4805d0b0 con 0x7f1b480b0120 2022-07-12T08:48:16.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1b480ab730 0x7f1b48144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1b480ab730 0x7f1b48144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58140/0 (socket says 172.21.15.61:58140) 2022-07-12T08:48:16.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 -- 172.21.15.61:0/4030762221 learned_addr learned my addr 172.21.15.61:0/4030762221 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:16.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4ed1d700 1 --2- 172.21.15.61:0/4030762221 >> v2:172.21.15.111:3300/0 conn(0x7f1b4814aad0 0x7f1b4814af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4e51c700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480b0120 0x7f1b48144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 -- 172.21.15.61:0/4030762221 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480b0120 msgr2=0x7f1b48144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:16.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480b0120 0x7f1b48144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 -- 172.21.15.61:0/4030762221 >> v2:172.21.15.111:3300/0 conn(0x7f1b4814aad0 msgr2=0x7f1b4814af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:16.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 --2- 172.21.15.61:0/4030762221 >> v2:172.21.15.111:3300/0 conn(0x7f1b4814aad0 0x7f1b4814af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 -- 172.21.15.61:0/4030762221 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4805d350 con 0x7f1b480ab730 2022-07-12T08:48:16.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4ed1d700 1 --2- 172.21.15.61:0/4030762221 >> v2:172.21.15.111:3300/0 conn(0x7f1b4814aad0 0x7f1b4814af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:16.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4e51c700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480b0120 0x7f1b48144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:16.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4dd1b700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b480ab730 0x7f1b48144640 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f1b44002f70 tx=0x7f1b44006eb0).ready entity=mon.0 client_cookie=8c87153bbeebf163 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4d51a700 1 -- 172.21.15.61:0/4030762221 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b4400d570 con 0x7f1b480ab730 2022-07-12T08:48:16.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4d51a700 1 -- 172.21.15.61:0/4030762221 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44000e90 con 0x7f1b480ab730 2022-07-12T08:48:16.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b4d51a700 1 -- 172.21.15.61:0/4030762221 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b4400d900 con 0x7f1b480ab730 2022-07-12T08:48:16.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- 172.21.15.61:0/4030762221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b480ab730 msgr2=0x7f1b48144640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b480ab730 0x7f1b48144640 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f1b44002f70 tx=0x7f1b44006eb0).stop 2022-07-12T08:48:16.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- 172.21.15.61:0/4030762221 shutdown_connections 2022-07-12T08:48:16.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- 172.21.15.61:0/4030762221 >> v2:172.21.15.111:3300/0 conn(0x7f1b4814aad0 0x7f1b4814af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b480ab730 0x7f1b48144640 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 --2- 172.21.15.61:0/4030762221 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480b0120 0x7f1b48144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- 172.21.15.61:0/4030762221 >> 172.21.15.61:0/4030762221 conn(0x7f1b480b5290 msgr2=0x7f1b480b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:16.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- 172.21.15.61:0/4030762221 shutdown_connections 2022-07-12T08:48:16.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.879+0000 7f1b50780700 1 -- 172.21.15.61:0/4030762221 wait complete. 2022-07-12T08:48:16.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 Processor -- start 2022-07-12T08:48:16.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 -- start start 2022-07-12T08:48:16.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480ab730 0x7f1b48149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480b0120 0x7f1b48065f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 0x7f1b48064380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b480afa20 con 0x7f1b4814aad0 2022-07-12T08:48:16.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b480af780 con 0x7f1b480b0120 2022-07-12T08:48:16.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48063ee0 con 0x7f1b480ab730 2022-07-12T08:48:16.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 0x7f1b48064380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 0x7f1b48064380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58148/0 (socket says 172.21.15.61:58148) 2022-07-12T08:48:16.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 -- 172.21.15.61:0/2912402067 learned_addr learned my addr 172.21.15.61:0/2912402067 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:16.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4e51c700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480ab730 0x7f1b48149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4dd1b700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480b0120 0x7f1b48065f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 -- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480ab730 msgr2=0x7f1b48149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480ab730 0x7f1b48149e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 -- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480b0120 msgr2=0x7f1b48065f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:16.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480b0120 0x7f1b48065f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 -- 172.21.15.61:0/2912402067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4400b1e0 con 0x7f1b4814aad0 2022-07-12T08:48:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b4ed1d700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 0x7f1b48064380 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f1b40006cc0 tx=0x7f1b40008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b40011780 con 0x7f1b4814aad0 2022-07-12T08:48:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b400118e0 con 0x7f1b4814aad0 2022-07-12T08:48:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b48066470 con 0x7f1b4814aad0 2022-07-12T08:48:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b40011c40 con 0x7f1b4814aad0 2022-07-12T08:48:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.883+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b480af350 con 0x7f1b4814aad0 2022-07-12T08:48:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.887+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1b4001c460 con 0x7f1b4814aad0 2022-07-12T08:48:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.887+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b480af5a0 con 0x7f1b4814aad0 2022-07-12T08:48:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.887+0000 7f1b3b7fe700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b28053a60 0x7f1b28055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.887+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1b4006c320 con 0x7f1b4814aad0 2022-07-12T08:48:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.887+0000 7f1b4e51c700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b28053a60 0x7f1b28055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.887+0000 7f1b4e51c700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b28053a60 0x7f1b28055f20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1b3c00c000 tx=0x7f1b3c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:16.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:16.891+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b40043120 con 0x7f1b4814aad0 2022-07-12T08:48:17.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.015+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b48057460 con 0x7f1b4814aad0 2022-07-12T08:48:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.015+0000 7f1b3b7fe700 1 -- 172.21.15.61:0/2912402067 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f1b40046740 con 0x7f1b4814aad0 2022-07-12T08:48:17.020 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:17.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b28053a60 msgr2=0x7f1b28055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b28053a60 0x7f1b28055f20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f1b3c00c000 tx=0x7f1b3c008040).stop 2022-07-12T08:48:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 msgr2=0x7f1b48064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 0x7f1b48064380 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 rx=0x7f1b40006cc0 tx=0x7f1b40008040).stop 2022-07-12T08:48:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 shutdown_connections 2022-07-12T08:48:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b4814aad0 0x7f1b48064380 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1b28053a60 0x7f1b28055f20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480b0120 0x7f1b48065f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 --2- 172.21.15.61:0/2912402067 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1b480ab730 0x7f1b48149e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 >> 172.21.15.61:0/2912402067 conn(0x7f1b480b5290 msgr2=0x7f1b4814af60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 shutdown_connections 2022-07-12T08:48:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.019+0000 7f1b50780700 1 -- 172.21.15.61:0/2912402067 wait complete. 2022-07-12T08:48:17.033 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:48:17.034 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:17.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.135+0000 7faaeeb5f700 1 Processor -- start 2022-07-12T08:48:17.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.135+0000 7faaeeb5f700 1 -- start start 2022-07-12T08:48:17.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faae805f480 0x7faae805f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:17.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae813c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:17.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae805fd50 con 0x7faae813cc00 2022-07-12T08:48:17.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae8142420 con 0x7faae805f480 2022-07-12T08:48:17.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae8140680 con 0x7faae8140160 2022-07-12T08:48:17.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaee35e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7faae813cc00 0x7faae8142d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46374/0 (socket says 172.21.15.61:46374) 2022-07-12T08:48:17.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaee35e700 1 -- 172.21.15.61:0/1419921709 learned_addr learned my addr 172.21.15.61:0/1419921709 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaed35c700 1 --2- 172.21.15.61:0/1419921709 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae813c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaedb5d700 1 --2- 172.21.15.61:0/1419921709 >> v2:172.21.15.111:3300/0 conn(0x7faae805f480 0x7faae805f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaecb5b700 1 -- 172.21.15.61:0/1419921709 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084617159 0 0) 0x7faae805fd50 con 0x7faae813cc00 2022-07-12T08:48:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaecb5b700 1 -- 172.21.15.61:0/1419921709 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad0003260 con 0x7faae813cc00 2022-07-12T08:48:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaed35c700 1 -- 172.21.15.61:0/1419921709 >> v2:172.21.15.111:3300/0 conn(0x7faae805f480 msgr2=0x7faae805f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaed35c700 1 --2- 172.21.15.61:0/1419921709 >> v2:172.21.15.111:3300/0 conn(0x7faae805f480 0x7faae805f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaed35c700 1 -- 172.21.15.61:0/1419921709 >> v1:172.21.15.61:6789/0 conn(0x7faae813cc00 legacy=0x7faae8142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaed35c700 1 -- 172.21.15.61:0/1419921709 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae8140cd0 con 0x7faae8140160 2022-07-12T08:48:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaedb5d700 1 --2- 172.21.15.61:0/1419921709 >> v2:172.21.15.111:3300/0 conn(0x7faae805f480 0x7faae805f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaed35c700 1 --2- 172.21.15.61:0/1419921709 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae813c710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7faad8002f70 tx=0x7faad8006df0).ready entity=mon.2 client_cookie=9caf63f38adaeb26 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaecb5b700 1 -- 172.21.15.61:0/1419921709 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faad800d4a0 con 0x7faae8140160 2022-07-12T08:48:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaecb5b700 1 -- 172.21.15.61:0/1419921709 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8000e90 con 0x7faae8140160 2022-07-12T08:48:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaecb5b700 1 -- 172.21.15.61:0/1419921709 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faad800d7f0 con 0x7faae8140160 2022-07-12T08:48:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1419921709 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 msgr2=0x7faae813c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.139+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1419921709 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae813c710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7faad8002f70 tx=0x7faad8006df0).stop 2022-07-12T08:48:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1419921709 shutdown_connections 2022-07-12T08:48:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1419921709 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae813c710 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1419921709 >> v2:172.21.15.111:3300/0 conn(0x7faae805f480 0x7faae805f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1419921709 >> 172.21.15.61:0/1419921709 conn(0x7faae8063a60 msgr2=0x7faae8063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1419921709 shutdown_connections 2022-07-12T08:48:17.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1419921709 wait complete. 2022-07-12T08:48:17.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 Processor -- start 2022-07-12T08:48:17.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 -- start start 2022-07-12T08:48:17.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.143+0000 7faaeeb5f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae805f480 0x7faae8137e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 0x7faae8066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae8152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8065cb0 con 0x7faae813cc00 2022-07-12T08:48:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae8065f10 con 0x7faae805f480 2022-07-12T08:48:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae80661b0 con 0x7faae8140160 2022-07-12T08:48:17.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 0x7faae8066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:17.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 0x7faae8066650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58160/0 (socket says 172.21.15.61:58160) 2022-07-12T08:48:17.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 -- 172.21.15.61:0/1380060189 learned_addr learned my addr 172.21.15.61:0/1380060189 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaedb5d700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae805f480 0x7faae8137e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaee35e700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae8152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 -- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 msgr2=0x7faae8152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae8152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 -- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae805f480 msgr2=0x7faae8137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae805f480 0x7faae8137e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 -- 172.21.15.61:0/1380060189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae400b180 con 0x7faae813cc00 2022-07-12T08:48:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaed35c700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 0x7faae8066650 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7faad8002f70 tx=0x7faad8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faad80139c0 con 0x7faae813cc00 2022-07-12T08:48:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8013b20 con 0x7faae813cc00 2022-07-12T08:48:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad800b180 con 0x7faae813cc00 2022-07-12T08:48:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faad8013e80 con 0x7faae813cc00 2022-07-12T08:48:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae80676c0 con 0x7faae813cc00 2022-07-12T08:48:17.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faad8025030 con 0x7faae813cc00 2022-07-12T08:48:17.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.147+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae8066370 con 0x7faae813cc00 2022-07-12T08:48:17.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.151+0000 7faad6ffd700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faac8053ad0 0x7faac8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:17.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.151+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7faad806cc00 con 0x7faae813cc00 2022-07-12T08:48:17.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.151+0000 7faaedb5d700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faac8053ad0 0x7faac8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:17.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.151+0000 7faaedb5d700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faac8053ad0 0x7faac8055f90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7faae400b000 tx=0x7faae4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:17.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.155+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faad8043a50 con 0x7faae813cc00 2022-07-12T08:48:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.319+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faae8057460 con 0x7faae813cc00 2022-07-12T08:48:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.319+0000 7faad6ffd700 1 -- 172.21.15.61:0/1380060189 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2143 (secure 0 0 0) 0x7faad8023070 con 0x7faae813cc00 2022-07-12T08:48:17.323 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:17.323 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:13.437914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3665156048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3665156048},{"type":"v1","addr":"172.21.15.61:6833","nonce":3665156048}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:17.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faac8053ad0 msgr2=0x7faac8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faac8053ad0 0x7faac8055f90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7faae400b000 tx=0x7faae4006df0).stop 2022-07-12T08:48:17.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 msgr2=0x7faae8066650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:17.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 0x7faae8066650 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7faad8002f70 tx=0x7faad8014000).stop 2022-07-12T08:48:17.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 shutdown_connections 2022-07-12T08:48:17.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faae8140160 0x7faae8152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faae813cc00 0x7faae8066650 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faac8053ad0 0x7faac8055f90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 --2- 172.21.15.61:0/1380060189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faae805f480 0x7faae8137e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:17.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 >> 172.21.15.61:0/1380060189 conn(0x7faae8063a60 msgr2=0x7faae80b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:17.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 shutdown_connections 2022-07-12T08:48:17.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:17.323+0000 7faaeeb5f700 1 -- 172.21.15.61:0/1380060189 wait complete. 2022-07-12T08:48:17.332 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 44 2022-07-12T08:48:17.543 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:12.433872+0000', 'modified': '2022-07-12T08:48:13.437914+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'a', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3665156048', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3665156048}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3665156048}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:48:17.544 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:48:17.544 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:48:17.662 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5981,"want_state":"up:active","state":"up:active","rank_uptime":5.2130815420000003,"mdsmap_epoch":44,"osdmap_epoch":81,"osdmap_epoch_barrier":80,"uptime":9.5544966650000003} 2022-07-12T08:48:17.663 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5981, "mdsmap_epoch": 44, "osdmap_epoch": 81, "osdmap_epoch_barrier": 80, "rank_uptime": 5.213081542, "state": "up:active", "uptime": 9.554496665, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:48:17.663 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:48:17.664 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:48:17.664 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:48:17.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:17.665 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:17.674 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:48:17.692 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:17.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:17.749 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:17.753 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:17.754 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:48:17.761 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:48:17.763 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:17.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:48:17.791 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:17.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:17.843 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:48:17.838+0000 7f73c9dc6080 -1 init, newargv = 0x55642d98f410 newargc=9ceph-fuse[ 2022-07-12T08:48:17.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:21764]: starting ceph client 2022-07-12T08:48:17.853 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21764]: starting fuse 2022-07-12T08:48:17.853 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:17.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:17.856 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:48:17.867 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:48:17.867 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:48:17.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:17.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:48:17.869 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:48:17.870 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:48:17.871 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:48:17.872 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:48:17.873 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:17.873 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:48:17.874 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:48:17.874 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:48:17.875 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:17.875 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:48:17.875 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:48:17.876 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:48:17.876 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:48:17.877 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:17.878 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:48:17.878 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:48:17.878 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:48:17.879 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:48:17.880 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:48:17.880 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:17.880 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:48:17.882 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:48:17.927 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21764.asok 2022-07-12T08:48:17.933 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21764.asok 2022-07-12T08:48:17.935 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:17.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21764.asok status 2022-07-12T08:48:18.067 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:48:18.068 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:48:18.068 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:48:18.068 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:48:18.068 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:48:18.069 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:48:18.069 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:48:18.069 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21764", 2022-07-12T08:48:18.069 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:48:18.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:18.070 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:48:18.070 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:48:18.071 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6840, 2022-07-12T08:48:18.071 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:48:18.071 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:48:18.071 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:48:18.071 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6840 2022-07-12T08:48:18.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:18.072 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:48:18.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:48:18.072 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:48:18.073 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3913450258 2022-07-12T08:48:18.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:48:18.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:18.074 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:48:18.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:48:18.074 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:48:18.074 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3913450258 2022-07-12T08:48:18.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:18.075 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6840 172.21.15.121:0/3913450258", 2022-07-12T08:48:18.075 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/3913450258", 2022-07-12T08:48:18.076 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:48:18.076 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 44, 2022-07-12T08:48:18.076 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 81, 2022-07-12T08:48:18.076 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:48:18.077 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:48:18.077 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:48:18.077 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:48:18.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:18.079 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:18.080 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:18.087 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:48:18.089 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:18.089 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:18.090 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:18.149 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:18.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 Processor -- start 2022-07-12T08:48:18.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 -- start start 2022-07-12T08:48:18.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f07900b4090 0x7f07900b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0790142f30 0x7f07900afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07900b5740 con 0x7f07900b1080 2022-07-12T08:48:18.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079005cc00 con 0x7f0790142f30 2022-07-12T08:48:18.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0797874700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07900b1460 con 0x7f07900b4090 2022-07-12T08:48:18.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0795610700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f07900b1080 0x7f079005c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46384/0 (socket says 172.21.15.61:46384) 2022-07-12T08:48:18.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0795610700 1 -- 172.21.15.61:0/4030588088 learned_addr learned my addr 172.21.15.61:0/4030588088 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:18.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0795e11700 1 --2- 172.21.15.61:0/4030588088 >> v2:172.21.15.61:3300/0 conn(0x7f0790142f30 0x7f07900afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0794e0f700 1 --2- 172.21.15.61:0/4030588088 >> v2:172.21.15.111:3300/0 conn(0x7f07900b4090 0x7f07900b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0783fff700 1 -- 172.21.15.61:0/4030588088 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227693473 0 0) 0x7f07900b5740 con 0x7f07900b1080 2022-07-12T08:48:18.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0783fff700 1 -- 172.21.15.61:0/4030588088 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0774003260 con 0x7f07900b1080 2022-07-12T08:48:18.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0794e0f700 1 -- 172.21.15.61:0/4030588088 >> v2:172.21.15.61:3300/0 conn(0x7f0790142f30 msgr2=0x7f07900afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:18.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0794e0f700 1 --2- 172.21.15.61:0/4030588088 >> v2:172.21.15.61:3300/0 conn(0x7f0790142f30 0x7f07900afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0794e0f700 1 -- 172.21.15.61:0/4030588088 >> v1:172.21.15.61:6789/0 conn(0x7f07900b1080 legacy=0x7f079005c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0794e0f700 1 -- 172.21.15.61:0/4030588088 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07900b6880 con 0x7f07900b4090 2022-07-12T08:48:18.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0794e0f700 1 --2- 172.21.15.61:0/4030588088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900b62a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0784002f70 tx=0x7f0784006df0).ready entity=mon.1 client_cookie=3564b05d755b8d98 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0783fff700 1 -- 172.21.15.61:0/4030588088 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078400d4f0 con 0x7f07900b4090 2022-07-12T08:48:18.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0783fff700 1 -- 172.21.15.61:0/4030588088 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0784000e90 con 0x7f07900b4090 2022-07-12T08:48:18.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.259+0000 7f0783fff700 1 -- 172.21.15.61:0/4030588088 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078400d840 con 0x7f07900b4090 2022-07-12T08:48:18.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- 172.21.15.61:0/4030588088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 msgr2=0x7f07900b62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 --2- 172.21.15.61:0/4030588088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900b62a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0784002f70 tx=0x7f0784006df0).stop 2022-07-12T08:48:18.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- 172.21.15.61:0/4030588088 shutdown_connections 2022-07-12T08:48:18.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 --2- 172.21.15.61:0/4030588088 >> v2:172.21.15.61:3300/0 conn(0x7f0790142f30 0x7f07900afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 --2- 172.21.15.61:0/4030588088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900b62a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- 172.21.15.61:0/4030588088 >> 172.21.15.61:0/4030588088 conn(0x7f0790067020 msgr2=0x7f0790067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:18.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- 172.21.15.61:0/4030588088 shutdown_connections 2022-07-12T08:48:18.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- 172.21.15.61:0/4030588088 wait complete. 2022-07-12T08:48:18.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 Processor -- start 2022-07-12T08:48:18.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- start start 2022-07-12T08:48:18.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 0x7f0790153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0790142f30 0x7f07900aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0790153580 con 0x7f07900b1080 2022-07-12T08:48:18.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07900aceb0 con 0x7f07900b4090 2022-07-12T08:48:18.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0797874700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07900ad150 con 0x7f0790142f30 2022-07-12T08:48:18.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 0x7f0790153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 0x7f0790153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58172/0 (socket says 172.21.15.61:58172) 2022-07-12T08:48:18.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 -- 172.21.15.61:0/1196243737 learned_addr learned my addr 172.21.15.61:0/1196243737 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:18.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0794e0f700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795e11700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0790142f30 0x7f07900aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 -- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0790142f30 msgr2=0x7f07900aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0790142f30 0x7f07900aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 -- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 msgr2=0x7f07900aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 -- 172.21.15.61:0/1196243737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078400b180 con 0x7f07900b1080 2022-07-12T08:48:18.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.263+0000 7f0795610700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 0x7f0790153090 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f078c001490 tx=0x7f078c007f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078c005ba0 con 0x7f07900b1080 2022-07-12T08:48:18.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c005d00 con 0x7f07900b1080 2022-07-12T08:48:18.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07900ad3f0 con 0x7f07900b1080 2022-07-12T08:48:18.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f078c007300 con 0x7f07900b1080 2022-07-12T08:48:18.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07900ae230 con 0x7f07900b1080 2022-07-12T08:48:18.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0790057460 con 0x7f07900b1080 2022-07-12T08:48:18.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f078c010030 con 0x7f07900b1080 2022-07-12T08:48:18.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f07827fc700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0778053ad0 0x7f0778055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f078c06bf50 con 0x7f07900b1080 2022-07-12T08:48:18.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.267+0000 7f0794e0f700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0778053ad0 0x7f0778055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.271+0000 7f0794e0f700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0778053ad0 0x7f0778055f90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f078401a000 tx=0x7f0784014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.275+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f078c042eb0 con 0x7f07900b1080 2022-07-12T08:48:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.403+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07900b4950 con 0x7f07900b1080 2022-07-12T08:48:18.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.403+0000 7f07827fc700 1 -- 172.21.15.61:0/1196243737 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f078c0464d0 con 0x7f07900b1080 2022-07-12T08:48:18.410 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:18.411 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":81,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:13.437499+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:11.002774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:11.580821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0778053ad0 msgr2=0x7f0778055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0778053ad0 0x7f0778055f90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f078401a000 tx=0x7f0784014000).stop 2022-07-12T08:48:18.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 msgr2=0x7f0790153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 0x7f0790153090 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f078c001490 tx=0x7f078c007f00).stop 2022-07-12T08:48:18.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 shutdown_connections 2022-07-12T08:48:18.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0790142f30 0x7f07900aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0778053ad0 0x7f0778055f90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f07900b4090 0x7f07900aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 --2- 172.21.15.61:0/1196243737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07900b1080 0x7f0790153090 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 >> 172.21.15.61:0/1196243737 conn(0x7f0790067020 msgr2=0x7f07900b7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:18.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 shutdown_connections 2022-07-12T08:48:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.407+0000 7f0797874700 1 -- 172.21.15.61:0/1196243737 wait complete. 2022-07-12T08:48:18.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:48:11.002774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:48:11.580821+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:48:18.423 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:18.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 Processor -- start 2022-07-12T08:48:18.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 -- start start 2022-07-12T08:48:18.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce00060a70 0x7fce0009a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce0009ad10 0x7fce001447c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0009b160 con 0x7fce0005cf30 2022-07-12T08:48:18.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000b14e0 con 0x7fce0009ad10 2022-07-12T08:48:18.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.531+0000 7fce07541700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b1780 con 0x7fce00060a70 2022-07-12T08:48:18.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce0009ad10 0x7fce001447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce0653f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce00060a70 0x7fce0009a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fce0009ad10 0x7fce001447c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36756/0 (socket says 172.21.15.61:36756) 2022-07-12T08:48:18.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce05d3e700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fce0005cf30 0x7fce001440b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:60002/0 (socket says 172.21.15.61:60002) 2022-07-12T08:48:18.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 -- 172.21.15.61:0/2991933139 learned_addr learned my addr 172.21.15.61:0/2991933139 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:18.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 -- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce00060a70 msgr2=0x7fce0009a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:18.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 --2- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce00060a70 0x7fce0009a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 -- 172.21.15.61:0/2991933139 >> v1:172.21.15.111:6789/0 conn(0x7fce0005cf30 legacy=0x7fce001440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 -- 172.21.15.61:0/2991933139 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0009a380 con 0x7fce0009ad10 2022-07-12T08:48:18.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce06d40700 1 --2- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce0009ad10 0x7fce001447c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fcdf8002f70 tx=0x7fcdf8006eb0).ready entity=mon.1 client_cookie=dfa03f2924a7f45f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce0553d700 1 -- 172.21.15.61:0/2991933139 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf800d570 con 0x7fce0009ad10 2022-07-12T08:48:18.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce0553d700 1 -- 172.21.15.61:0/2991933139 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8000e90 con 0x7fce0009ad10 2022-07-12T08:48:18.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce0553d700 1 -- 172.21.15.61:0/2991933139 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf800d900 con 0x7fce0009ad10 2022-07-12T08:48:18.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 -- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce0009ad10 msgr2=0x7fce001447c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 --2- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce0009ad10 0x7fce001447c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fcdf8002f70 tx=0x7fcdf8006eb0).stop 2022-07-12T08:48:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 -- 172.21.15.61:0/2991933139 shutdown_connections 2022-07-12T08:48:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 --2- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce0009ad10 0x7fce001447c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 --2- 172.21.15.61:0/2991933139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce00060a70 0x7fce0009a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 -- 172.21.15.61:0/2991933139 >> 172.21.15.61:0/2991933139 conn(0x7fce0005bd10 msgr2=0x7fce0005c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:18.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 -- 172.21.15.61:0/2991933139 shutdown_connections 2022-07-12T08:48:18.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 -- 172.21.15.61:0/2991933139 wait complete. 2022-07-12T08:48:18.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 Processor -- start 2022-07-12T08:48:18.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.535+0000 7fce07541700 1 -- start start 2022-07-12T08:48:18.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce0005cf30 0x7fce00144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce00060a70 0x7fce00144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 0x7fce00146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00143520 con 0x7fce0009ad10 2022-07-12T08:48:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00145130 con 0x7fce00060a70 2022-07-12T08:48:18.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce00063750 con 0x7fce0005cf30 2022-07-12T08:48:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 0x7fce00146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 0x7fce00146670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58188/0 (socket says 172.21.15.61:58188) 2022-07-12T08:48:18.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 -- 172.21.15.61:0/2461042685 learned_addr learned my addr 172.21.15.61:0/2461042685 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce05d3e700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce00060a70 0x7fce00144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce0653f700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce0005cf30 0x7fce00144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 -- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce0005cf30 msgr2=0x7fce00144560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce0005cf30 0x7fce00144560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 -- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce00060a70 msgr2=0x7fce00144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce00060a70 0x7fce00144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 -- 172.21.15.61:0/2461042685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf800b1e0 con 0x7fce0009ad10 2022-07-12T08:48:18.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce06d40700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 0x7fce00146670 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7fcdf8002f70 tx=0x7fcdf8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf800d8f0 con 0x7fce0009ad10 2022-07-12T08:48:18.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf8011c30 con 0x7fce0009ad10 2022-07-12T08:48:18.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce00067300 con 0x7fce0009ad10 2022-07-12T08:48:18.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf801e460 con 0x7fce0009ad10 2022-07-12T08:48:18.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.539+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce00146b60 con 0x7fce0009ad10 2022-07-12T08:48:18.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.543+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce000674c0 con 0x7fce0009ad10 2022-07-12T08:48:18.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.543+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcdf8025030 con 0x7fce0009ad10 2022-07-12T08:48:18.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.543+0000 7fcdef7fe700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcde0053af0 0x7fcde0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.543+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcdf806d0c0 con 0x7fce0009ad10 2022-07-12T08:48:18.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.543+0000 7fce0653f700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcde0053af0 0x7fcde0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.543+0000 7fce0653f700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcde0053af0 0x7fcde0055fb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fcdf000b000 tx=0x7fcdf0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.547+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcdf8044020 con 0x7fce0009ad10 2022-07-12T08:48:18.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.675+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fce00066770 con 0x7fce0009ad10 2022-07-12T08:48:18.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fcdef7fe700 1 -- 172.21.15.61:0/2461042685 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7fcdf8023020 con 0x7fce0009ad10 2022-07-12T08:48:18.683 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:18.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcde0053af0 msgr2=0x7fcde0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcde0053af0 0x7fcde0055fb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fcdf000b000 tx=0x7fcdf0008040).stop 2022-07-12T08:48:18.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 msgr2=0x7fce00146670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 0x7fce00146670 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 rx=0x7fcdf8002f70 tx=0x7fcdf8015000).stop 2022-07-12T08:48:18.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 shutdown_connections 2022-07-12T08:48:18.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce0009ad10 0x7fce00146670 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcde0053af0 0x7fcde0055fb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fce00060a70 0x7fce00144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 --2- 172.21.15.61:0/2461042685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fce0005cf30 0x7fce00144560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 >> 172.21.15.61:0/2461042685 conn(0x7fce0005bd10 msgr2=0x7fce0013fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:18.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 shutdown_connections 2022-07-12T08:48:18.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.679+0000 7fce07541700 1 -- 172.21.15.61:0/2461042685 wait complete. 2022-07-12T08:48:18.699 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:18.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc8072700 1 Processor -- start 2022-07-12T08:48:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc8072700 1 -- start start 2022-07-12T08:48:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc8072700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbfc00b05f0 0x7fbfc00b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc8072700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00b3560 con 0x7fbfc00b5480 2022-07-12T08:48:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc8072700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfc00b5b80 con 0x7fbfc00b3730 2022-07-12T08:48:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc8072700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfc00b5e20 con 0x7fbfc00b05f0 2022-07-12T08:48:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc560d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbfc00b05f0 0x7fbfc00b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc5e0e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbfc00b5480 0x7fbfc00afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46412/0 (socket says 172.21.15.61:46412) 2022-07-12T08:48:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc5e0e700 1 -- 172.21.15.61:0/545680864 learned_addr learned my addr 172.21.15.61:0/545680864 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc560d700 1 -- 172.21.15.61:0/545680864 >> v1:172.21.15.111:6789/0 conn(0x7fbfc00b3730 legacy=0x7fbfc0142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc560d700 1 -- 172.21.15.61:0/545680864 >> v1:172.21.15.61:6789/0 conn(0x7fbfc00b5480 legacy=0x7fbfc00afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc560d700 1 -- 172.21.15.61:0/545680864 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc00b5860 con 0x7fbfc00b05f0 2022-07-12T08:48:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc560d700 1 --2- 172.21.15.61:0/545680864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc00b45d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fbfb0002f70 tx=0x7fbfb0006df0).ready entity=mon.0 client_cookie=380de15dce56fbf8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc4e0c700 1 -- 172.21.15.61:0/545680864 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfb000d610 con 0x7fbfc00b05f0 2022-07-12T08:48:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc4e0c700 1 -- 172.21.15.61:0/545680864 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfb0000e90 con 0x7fbfc00b05f0 2022-07-12T08:48:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.819+0000 7fbfc4e0c700 1 -- 172.21.15.61:0/545680864 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfb000d960 con 0x7fbfc00b05f0 2022-07-12T08:48:18.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- 172.21.15.61:0/545680864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 msgr2=0x7fbfc00b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 --2- 172.21.15.61:0/545680864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc00b45d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7fbfb0002f70 tx=0x7fbfb0006df0).stop 2022-07-12T08:48:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- 172.21.15.61:0/545680864 shutdown_connections 2022-07-12T08:48:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 --2- 172.21.15.61:0/545680864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc00b45d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- 172.21.15.61:0/545680864 >> 172.21.15.61:0/545680864 conn(0x7fbfc00b96b0 msgr2=0x7fbfc00b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:18.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- 172.21.15.61:0/545680864 shutdown_connections 2022-07-12T08:48:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- 172.21.15.61:0/545680864 wait complete. 2022-07-12T08:48:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 Processor -- start 2022-07-12T08:48:18.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- start start 2022-07-12T08:48:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc0149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc00b3730 0x7fbfc0141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfc00b5480 0x7fbfc0142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc00ae9a0 con 0x7fbfc00b05f0 2022-07-12T08:48:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc00aec40 con 0x7fbfc00b3730 2022-07-12T08:48:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc8072700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc00aeee0 con 0x7fbfc00b5480 2022-07-12T08:48:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc0149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc0149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58200/0 (socket says 172.21.15.61:58200) 2022-07-12T08:48:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 -- 172.21.15.61:0/2912913718 learned_addr learned my addr 172.21.15.61:0/2912913718 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc660f700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfc00b5480 0x7fbfc0142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc560d700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc00b3730 0x7fbfc0141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 -- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfc00b5480 msgr2=0x7fbfc0142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfc00b5480 0x7fbfc0142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 -- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc00b3730 msgr2=0x7fbfc0141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc00b3730 0x7fbfc0141f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 -- 172.21.15.61:0/2912913718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb000b180 con 0x7fbfc00b05f0 2022-07-12T08:48:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc660f700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfc00b5480 0x7fbfc0142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfc5e0e700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc0149040 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fbfc0066150 tx=0x7fbfbc007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.823+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfbc0070d0 con 0x7fbfc00b05f0 2022-07-12T08:48:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc007230 con 0x7fbfc00b05f0 2022-07-12T08:48:18.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfbc005610 con 0x7fbfc00b05f0 2022-07-12T08:48:18.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc00af360 con 0x7fbfc00b05f0 2022-07-12T08:48:18.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc0152260 con 0x7fbfc00b05f0 2022-07-12T08:48:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc00b0100 con 0x7fbfc00b05f0 2022-07-12T08:48:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbfbc010070 con 0x7fbfc00b05f0 2022-07-12T08:48:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfaeffd700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfa00539b0 0x7fbfa0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbfbc06b4d0 con 0x7fbfc00b05f0 2022-07-12T08:48:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.827+0000 7fbfc560d700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfa00539b0 0x7fbfa0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.831+0000 7fbfc560d700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfa00539b0 0x7fbfa0055e70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbfb001c000 tx=0x7fbfb0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.835+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfbc042320 con 0x7fbfc00b05f0 2022-07-12T08:48:18.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.963+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbfc0057460 con 0x7fbfc00b05f0 2022-07-12T08:48:18.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.963+0000 7fbfaeffd700 1 -- 172.21.15.61:0/2912913718 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbfbc018020 con 0x7fbfc00b05f0 2022-07-12T08:48:18.969 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:18.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfa00539b0 msgr2=0x7fbfa0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfa00539b0 0x7fbfa0055e70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fbfb001c000 tx=0x7fbfb0015000).stop 2022-07-12T08:48:18.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 msgr2=0x7fbfc0149040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:18.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc0149040 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 rx=0x7fbfc0066150 tx=0x7fbfbc007ca0).stop 2022-07-12T08:48:18.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 shutdown_connections 2022-07-12T08:48:18.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfc00b5480 0x7fbfc0142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbfa00539b0 0x7fbfa0055e70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfc00b3730 0x7fbfc0141f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 --2- 172.21.15.61:0/2912913718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfc00b05f0 0x7fbfc0149040 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:18.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 >> 172.21.15.61:0/2912913718 conn(0x7fbfc00b96b0 msgr2=0x7fbfc005c640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:18.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 shutdown_connections 2022-07-12T08:48:18.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:18.967+0000 7fbfc8072700 1 -- 172.21.15.61:0/2912913718 wait complete. 2022-07-12T08:48:18.982 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:48:19.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 Processor -- start 2022-07-12T08:48:19.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 -- start start 2022-07-12T08:48:19.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca8c0b05f0 0x7fca8c0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca8c0b3730 0x7fca8c142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8c0b3560 con 0x7fca8c0b5480 2022-07-12T08:48:19.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c0b5b80 con 0x7fca8c0b3730 2022-07-12T08:48:19.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca92643700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca8c0b5e20 con 0x7fca8c0b05f0 2022-07-12T08:48:19.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fca8c0b5480 0x7fca8c0afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46426/0 (socket says 172.21.15.61:46426) 2022-07-12T08:48:19.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca90be0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca8c0b3730 0x7fca8c142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8bfff700 1 -- 172.21.15.61:0/1719243627 learned_addr learned my addr 172.21.15.61:0/1719243627 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:19.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8b7fe700 1 --2- 172.21.15.61:0/1719243627 >> v2:172.21.15.111:3300/0 conn(0x7fca8c0b05f0 0x7fca8c0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca90be0700 1 -- 172.21.15.61:0/1719243627 >> v2:172.21.15.111:3300/0 conn(0x7fca8c0b05f0 msgr2=0x7fca8c0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca90be0700 1 --2- 172.21.15.61:0/1719243627 >> v2:172.21.15.111:3300/0 conn(0x7fca8c0b05f0 0x7fca8c0b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca90be0700 1 -- 172.21.15.61:0/1719243627 >> v1:172.21.15.61:6789/0 conn(0x7fca8c0b5480 legacy=0x7fca8c0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8affd700 1 -- 172.21.15.61:0/1719243627 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457476774 0 0) 0x7fca8c0b3560 con 0x7fca8c0b5480 2022-07-12T08:48:19.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca90be0700 1 -- 172.21.15.61:0/1719243627 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c0b5860 con 0x7fca8c0b3730 2022-07-12T08:48:19.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca90be0700 1 --2- 172.21.15.61:0/1719243627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b3730 0x7fca8c142f90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7fca840032f0 tx=0x7fca840091d0).ready entity=mon.0 client_cookie=8980ddd030c3e6b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8affd700 1 -- 172.21.15.61:0/1719243627 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca8400b680 con 0x7fca8c0b3730 2022-07-12T08:48:19.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8affd700 1 -- 172.21.15.61:0/1719243627 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8400b7e0 con 0x7fca8c0b3730 2022-07-12T08:48:19.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.107+0000 7fca8affd700 1 -- 172.21.15.61:0/1719243627 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca8400b950 con 0x7fca8c0b3730 2022-07-12T08:48:19.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- 172.21.15.61:0/1719243627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b3730 msgr2=0x7fca8c142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 --2- 172.21.15.61:0/1719243627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b3730 0x7fca8c142f90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7fca840032f0 tx=0x7fca840091d0).stop 2022-07-12T08:48:19.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- 172.21.15.61:0/1719243627 shutdown_connections 2022-07-12T08:48:19.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 --2- 172.21.15.61:0/1719243627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b3730 0x7fca8c142f90 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 --2- 172.21.15.61:0/1719243627 >> v2:172.21.15.111:3300/0 conn(0x7fca8c0b05f0 0x7fca8c0b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- 172.21.15.61:0/1719243627 >> 172.21.15.61:0/1719243627 conn(0x7fca8c0b96b0 msgr2=0x7fca8c0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:19.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- 172.21.15.61:0/1719243627 shutdown_connections 2022-07-12T08:48:19.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- 172.21.15.61:0/1719243627 wait complete. 2022-07-12T08:48:19.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 Processor -- start 2022-07-12T08:48:19.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- start start 2022-07-12T08:48:19.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0b05f0 0x7fca8c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca8c0b3730 0x7fca8c152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 0x7fca8c152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c140830 con 0x7fca8c0b5480 2022-07-12T08:48:19.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c140ad0 con 0x7fca8c0b05f0 2022-07-12T08:48:19.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca92643700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c140d70 con 0x7fca8c0b3730 2022-07-12T08:48:19.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 0x7fca8c152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 0x7fca8c152c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58218/0 (socket says 172.21.15.61:58218) 2022-07-12T08:48:19.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 -- 172.21.15.61:0/857777948 learned_addr learned my addr 172.21.15.61:0/857777948 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:19.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca8bfff700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0b05f0 0x7fca8c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca8b7fe700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca8c0b3730 0x7fca8c152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 -- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca8c0b3730 msgr2=0x7fca8c152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca8c0b3730 0x7fca8c152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 -- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0b05f0 msgr2=0x7fca8c0acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0b05f0 0x7fca8c0acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 -- 172.21.15.61:0/857777948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84009470 con 0x7fca8c0b5480 2022-07-12T08:48:19.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca8bfff700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0b05f0 0x7fca8c0acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:19.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.111+0000 7fca90be0700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 0x7fca8c152c10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fca840032f0 tx=0x7fca84016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca8400b680 con 0x7fca8c0b5480 2022-07-12T08:48:19.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca840107a0 con 0x7fca8c0b5480 2022-07-12T08:48:19.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c153100 con 0x7fca8c0b5480 2022-07-12T08:48:19.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca84010b00 con 0x7fca8c0b5480 2022-07-12T08:48:19.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c140440 con 0x7fca8c0b5480 2022-07-12T08:48:19.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c057460 con 0x7fca8c0b5480 2022-07-12T08:48:19.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fca84027020 con 0x7fca8c0b5480 2022-07-12T08:48:19.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca897fa700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca700539b0 0x7fca70055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fca8406d380 con 0x7fca8c0b5480 2022-07-12T08:48:19.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.115+0000 7fca8bfff700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca700539b0 0x7fca70055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.119+0000 7fca8bfff700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca700539b0 0x7fca70055e70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fca7c00b000 tx=0x7fca7c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.119+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca840442e0 con 0x7fca8c0b5480 2022-07-12T08:48:19.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.247+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca8c0ba5d0 con 0x7fca8c0b5480 2022-07-12T08:48:19.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.247+0000 7fca897fa700 1 -- 172.21.15.61:0/857777948 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7fca84047900 con 0x7fca8c0b5480 2022-07-12T08:48:19.252 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:19.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca700539b0 msgr2=0x7fca70055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca700539b0 0x7fca70055e70 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fca7c00b000 tx=0x7fca7c009040).stop 2022-07-12T08:48:19.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 msgr2=0x7fca8c152c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 0x7fca8c152c10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 rx=0x7fca840032f0 tx=0x7fca84016000).stop 2022-07-12T08:48:19.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 shutdown_connections 2022-07-12T08:48:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca8c0b5480 0x7fca8c152c10 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca8c0b3730 0x7fca8c152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca700539b0 0x7fca70055e70 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 --2- 172.21.15.61:0/857777948 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0b05f0 0x7fca8c0acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 >> 172.21.15.61:0/857777948 conn(0x7fca8c0b96b0 msgr2=0x7fca8c0b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:19.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 shutdown_connections 2022-07-12T08:48:19.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.251+0000 7fca92643700 1 -- 172.21.15.61:0/857777948 wait complete. 2022-07-12T08:48:19.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:48:19.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6d0fa4700 1 Processor -- start 2022-07-12T08:48:19.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6d0fa4700 1 -- start start 2022-07-12T08:48:19.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6d0fa4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 0x7fa6cc060c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6d0fa4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc140a40 con 0x7fa6cc060860 2022-07-12T08:48:19.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6d0fa4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc0ad370 con 0x7fa6cc0b02d0 2022-07-12T08:48:19.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6ca59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa6cc060860 0x7fa6cc141750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46440/0 (socket says 172.21.15.61:46440) 2022-07-12T08:48:19.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6d0fa4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc0ad8e0 con 0x7fa6cc0613d0 2022-07-12T08:48:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6ca59c700 1 -- 172.21.15.61:0/2887717489 learned_addr learned my addr 172.21.15.61:0/2887717489 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c9d9b700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 0x7fa6cc060c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864301324 0 0) 0x7fa6cc140a40 con 0x7fa6cc060860 2022-07-12T08:48:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0003260 con 0x7fa6cc060860 2022-07-12T08:48:19.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1818778610 0 0) 0x7fa6b0003260 con 0x7fa6cc060860 2022-07-12T08:48:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 msgr2=0x7fa6cc060c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 0x7fa6cc060c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 >> v1:172.21.15.111:6789/0 conn(0x7fa6cc0b02d0 legacy=0x7fa6cc13f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc061130 con 0x7fa6cc060860 2022-07-12T08:48:19.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fa6bc001660 con 0x7fa6cc060860 2022-07-12T08:48:19.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 >> v1:172.21.15.61:6789/0 conn(0x7fa6cc060860 legacy=0x7fa6cc141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 0x7fa6b0003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0003cb0 0x7fa6b0004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6b00049f0 0x7fa6b0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000b800 con 0x7fa6b0003cb0 2022-07-12T08:48:19.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000b940 con 0x7fa6b00049f0 2022-07-12T08:48:19.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000bb00 con 0x7fa6cc0613d0 2022-07-12T08:48:19.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6c9d9b700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0003cb0 0x7fa6b0004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.375+0000 7fa6ca59c700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 0x7fa6b0003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c9d9b700 1 -- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 msgr2=0x7fa6b0003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c9d9b700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc0613d0 0x7fa6b0003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c9d9b700 1 -- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6b00049f0 msgr2=0x7fa6b0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c9d9b700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6b00049f0 0x7fa6b0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-12T08:48:19.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c9d9b700 1 -- 172.21.15.61:0/2887717489 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0003260 con 0x7fa6b0003cb0 2022-07-12T08:48:19.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c9d9b700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0003cb0 0x7fa6b0004500 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fa6c000c000 tx=0x7fa6c0006df0).ready entity=mon.0 client_cookie=ea92032285d35238 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c000e830 con 0x7fa6b0003cb0 2022-07-12T08:48:19.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c000e990 con 0x7fa6b0003cb0 2022-07-12T08:48:19.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6c959a700 1 -- 172.21.15.61:0/2887717489 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c000ecf0 con 0x7fa6b0003cb0 2022-07-12T08:48:19.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/2887717489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0003cb0 msgr2=0x7fa6b0004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/2887717489 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0003cb0 0x7fa6b0004500 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7fa6c000c000 tx=0x7fa6c0006df0).stop 2022-07-12T08:48:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6cad9d700 1 -- 172.21.15.61:0/2887717489 reap_dead start 2022-07-12T08:48:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/2887717489 shutdown_connections 2022-07-12T08:48:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/2887717489 >> 172.21.15.61:0/2887717489 conn(0x7fa6cc0b23d0 msgr2=0x7fa6cc0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/2887717489 shutdown_connections 2022-07-12T08:48:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/2887717489 wait complete. 2022-07-12T08:48:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 Processor -- start 2022-07-12T08:48:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.379+0000 7fa6d0fa4700 1 -- start start 2022-07-12T08:48:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6cc063140 0x7fa6cc063520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 0x7fa6cc0652d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 0x7fa6cc1464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc064970 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc139e70 con 0x7fa6cc063140 2022-07-12T08:48:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc064b90 con 0x7fa6cc066bd0 2022-07-12T08:48:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 0x7fa6cc1464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6c9d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 0x7fa6cc0652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6ca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6cc063140 0x7fa6cc063520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6c9d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 0x7fa6cc0652d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40432/0 (socket says 172.21.15.61:40432) 2022-07-12T08:48:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 0x7fa6cc1464b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58238/0 (socket says 172.21.15.61:58238) 2022-07-12T08:48:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6c9d9b700 1 -- 172.21.15.61:0/1251112894 learned_addr learned my addr 172.21.15.61:0/1251112894 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 -- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 msgr2=0x7fa6cc0652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 0x7fa6cc0652d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 -- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6cc063140 msgr2=0x7fa6cc063520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6cc063140 0x7fa6cc063520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 -- 172.21.15.61:0/1251112894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc064e30 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6c9d9b700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 0x7fa6cc0652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6cad9d700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 0x7fa6cc1464b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fa6cc064ab0 tx=0x7fa6c4007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c400cdc0 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4005880 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc13ad40 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c4005be0 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.383+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc146bc0 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.387+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc064fc0 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.387+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa6c4010070 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.387+0000 7fa6bb7fe700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a8053ad0 0x7fa6a8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.387+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa6c406b3a0 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.387+0000 7fa6ca59c700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a8053ad0 0x7fa6a8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.387+0000 7fa6ca59c700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a8053ad0 0x7fa6a8055f90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa6bc005000 tx=0x7fa6bc001e10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.395+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa6c40421f0 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.523+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa6cc057460 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.523+0000 7fa6bb7fe700 1 -- 172.21.15.61:0/1251112894 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa6c4045810 con 0x7fa6cc13a8e0 2022-07-12T08:48:19.528 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:19.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.523+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a8053ad0 msgr2=0x7fa6a8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.523+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a8053ad0 0x7fa6a8055f90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fa6bc005000 tx=0x7fa6bc001e10).stop 2022-07-12T08:48:19.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.523+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 msgr2=0x7fa6cc1464b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.523+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 0x7fa6cc1464b0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 rx=0x7fa6cc064ab0 tx=0x7fa6c4007d40).stop 2022-07-12T08:48:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 shutdown_connections 2022-07-12T08:48:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6cc13a8e0 0x7fa6cc1464b0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6cc066bd0 0x7fa6cc0652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a8053ad0 0x7fa6a8055f90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 --2- 172.21.15.61:0/1251112894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6cc063140 0x7fa6cc063520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 >> 172.21.15.61:0/1251112894 conn(0x7fa6cc0b23d0 msgr2=0x7fa6cc141650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 shutdown_connections 2022-07-12T08:48:19.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.527+0000 7fa6d0fa4700 1 -- 172.21.15.61:0/1251112894 wait complete. 2022-07-12T08:48:19.540 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:48:19.541 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:19.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.651+0000 7f1f8bfff700 1 Processor -- start 2022-07-12T08:48:19.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.651+0000 7f1f8bfff700 1 -- start start 2022-07-12T08:48:19.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 0x7f1f8c143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0afac0 con 0x7f1f8c0af6e0 2022-07-12T08:48:19.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0611b0 con 0x7f1f8c13cb80 2022-07-12T08:48:19.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c143970 con 0x7f1f8c060c30 2022-07-12T08:48:19.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1f8c0af6e0 0x7f1f8c13c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46460/0 (socket says 172.21.15.61:46460) 2022-07-12T08:48:19.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 -- 172.21.15.61:0/1599785580 learned_addr learned my addr 172.21.15.61:0/1599785580 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:19.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8b7fe700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 0x7f1f8c143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369656106 0 0) 0x7f1f8c0afac0 con 0x7f1f8c0af6e0 2022-07-12T08:48:19.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f70003260 con 0x7f1f8c0af6e0 2022-07-12T08:48:19.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2829836122 0 0) 0x7f1f70003260 con 0x7f1f8c0af6e0 2022-07-12T08:48:19.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 msgr2=0x7f1f8c143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 0x7f1f8c143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 >> v1:172.21.15.111:6789/0 conn(0x7f1f8c13cb80 legacy=0x7f1f8c0b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c061450 con 0x7f1f8c0af6e0 2022-07-12T08:48:19.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f1f7c001680 con 0x7f1f8c0af6e0 2022-07-12T08:48:19.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 >> v1:172.21.15.61:6789/0 conn(0x7f1f8c0af6e0 legacy=0x7f1f8c13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f70003780 0x7f1f70003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70004500 0x7f1f70004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 0x7f1f70004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c143970 con 0x7f1f70003780 2022-07-12T08:48:19.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7000b750 con 0x7f1f70004500 2022-07-12T08:48:19.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7000b8c0 con 0x7f1f8c060c30 2022-07-12T08:48:19.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f70003780 0x7f1f70003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8a7fc700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70004500 0x7f1f70004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8b7fe700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 0x7f1f70004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 -- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 msgr2=0x7f1f70004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c060c30 0x7f1f70004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 -- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70004500 msgr2=0x7f1f70004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:19.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70004500 0x7f1f70004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 -- 172.21.15.61:0/1599785580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f70003260 con 0x7f1f70003780 2022-07-12T08:48:19.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8affd700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f70003780 0x7f1f70003b60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7f1f7c001390 tx=0x7f1f7c001e50).ready entity=mon.0 client_cookie=a68426f1c2238e4d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7c007170 con 0x7f1f70003780 2022-07-12T08:48:19.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0072d0 con 0x7f1f70003780 2022-07-12T08:48:19.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f89ffb700 1 -- 172.21.15.61:0/1599785580 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f7c00c890 con 0x7f1f70003780 2022-07-12T08:48:19.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 -- 172.21.15.61:0/1599785580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f70003780 msgr2=0x7f1f70003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/1599785580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f70003780 0x7f1f70003b60 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7f1f7c001390 tx=0x7f1f7c001e50).stop 2022-07-12T08:48:19.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8b7fe700 1 -- 172.21.15.61:0/1599785580 reap_dead start 2022-07-12T08:48:19.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.655+0000 7f1f8bfff700 1 -- 172.21.15.61:0/1599785580 shutdown_connections 2022-07-12T08:48:19.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- 172.21.15.61:0/1599785580 >> 172.21.15.61:0/1599785580 conn(0x7f1f8c0b5780 msgr2=0x7f1f8c0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:19.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- 172.21.15.61:0/1599785580 shutdown_connections 2022-07-12T08:48:19.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- 172.21.15.61:0/1599785580 wait complete. 2022-07-12T08:48:19.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 Processor -- start 2022-07-12T08:48:19.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- start start 2022-07-12T08:48:19.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70003780 0x7f1f8c065b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 0x7f1f8c1456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c065020 0x7f1f8c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c1464b0 con 0x7f1f8c146a50 2022-07-12T08:48:19.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c146750 con 0x7f1f70003780 2022-07-12T08:48:19.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c146190 con 0x7f1f8c065020 2022-07-12T08:48:19.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 0x7f1f8c1456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 0x7f1f8c1456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58256/0 (socket says 172.21.15.61:58256) 2022-07-12T08:48:19.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 -- 172.21.15.61:0/2282377586 learned_addr learned my addr 172.21.15.61:0/2282377586 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:19.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8affd700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70003780 0x7f1f8c065b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8b7fe700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c065020 0x7f1f8c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 -- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c065020 msgr2=0x7f1f8c1522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c065020 0x7f1f8c1522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 -- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70003780 msgr2=0x7f1f8c065b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.659+0000 7f1f8a7fc700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70003780 0x7f1f8c065b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8a7fc700 1 -- 172.21.15.61:0/2282377586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c064280 con 0x7f1f8c146a50 2022-07-12T08:48:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8b7fe700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c065020 0x7f1f8c1522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8affd700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70003780 0x7f1f8c065b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8a7fc700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 0x7f1f8c1456f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f1f80000f50 tx=0x7f1f800079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f8000fac0 con 0x7f1f8c146a50 2022-07-12T08:48:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8000fc20 con 0x7f1f8c146a50 2022-07-12T08:48:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c145be0 con 0x7f1f8c146a50 2022-07-12T08:48:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f80014450 con 0x7f1f8c146a50 2022-07-12T08:48:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c152970 con 0x7f1f8c146a50 2022-07-12T08:48:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c057460 con 0x7f1f8c146a50 2022-07-12T08:48:19.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1f8001b030 con 0x7f1f8c146a50 2022-07-12T08:48:19.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f7bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f68053a40 0x7f1f68055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:19.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1f8006c070 con 0x7f1f8c146a50 2022-07-12T08:48:19.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8affd700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f68053a40 0x7f1f68055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:19.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.663+0000 7f1f8affd700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f68053a40 0x7f1f68055f00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f1f7c019000 tx=0x7f1f7c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:19.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.675+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1f80042fd0 con 0x7f1f8c146a50 2022-07-12T08:48:19.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.843+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f8c068400 con 0x7f1f8c146a50 2022-07-12T08:48:19.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.843+0000 7f1f7bfff700 1 -- 172.21.15.61:0/2282377586 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f1f800465f0 con 0x7f1f8c146a50 2022-07-12T08:48:19.851 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:19.851 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:13.437914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3665156048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3665156048},{"type":"v1","addr":"172.21.15.61:6833","nonce":3665156048}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:19.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f68053a40 msgr2=0x7f1f68055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f68053a40 0x7f1f68055f00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f1f7c019000 tx=0x7f1f7c015000).stop 2022-07-12T08:48:19.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 msgr2=0x7f1f8c1456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:19.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 0x7f1f8c1456f0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 rx=0x7f1f80000f50 tx=0x7f1f800079e0).stop 2022-07-12T08:48:19.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 shutdown_connections 2022-07-12T08:48:19.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1f8c065020 0x7f1f8c1522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1f8c146a50 0x7f1f8c1456f0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1f68053a40 0x7f1f68055f00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 --2- 172.21.15.61:0/2282377586 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1f70003780 0x7f1f8c065b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:19.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 >> 172.21.15.61:0/2282377586 conn(0x7f1f8c0b5780 msgr2=0x7f1f8c0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:19.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 shutdown_connections 2022-07-12T08:48:19.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:19.847+0000 7f1f8bfff700 1 -- 172.21.15.61:0/2282377586 wait complete. 2022-07-12T08:48:19.860 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 44 2022-07-12T08:48:20.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:12.433872+0000', 'modified': '2022-07-12T08:48:13.437914+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5981}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5981': {'gid': 5981, 'name': 'a', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3665156048', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3665156048}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3665156048}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:48:20.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:48:20.071 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:48:20.197 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":5981,"want_state":"up:active","state":"up:active","rank_uptime":7.7476252629999998,"mdsmap_epoch":44,"osdmap_epoch":81,"osdmap_epoch_barrier":80,"uptime":12.089038917} 2022-07-12T08:48:20.198 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 5981, "mdsmap_epoch": 44, "osdmap_epoch": 81, "osdmap_epoch_barrier": 80, "rank_uptime": 7.747625263, "state": "up:active", "uptime": 12.089038917, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:48:20.199 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:48:20.199 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:48:20.200 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:48:20.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.206 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:20.216 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:48:20.239 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:20.240 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:20.299 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:20.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.301 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:48:20.318 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:48:20.319 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:20.328 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:48:20.355 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:20.356 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:20.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:48:20.395+0000 7f9eca909080 -1 init, newargv = 0x55ee23fc5810 newargc=9 2022-07-12T08:48:20.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21236]: starting ceph client 2022-07-12T08:48:20.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21236]: starting fuse 2022-07-12T08:48:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:20.432 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:20.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.433 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:48:20.484 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:48:20.485 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:48:20.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.486 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:48:20.486 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:48:20.486 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:48:20.487 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:48:20.491 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:48:20.491 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:20.492 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:48:20.492 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:48:20.493 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:48:20.496 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:20.497 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:48:20.497 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:48:20.498 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:48:20.499 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:48:20.500 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:20.503 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:48:20.504 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:48:20.505 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:48:20.505 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:48:20.506 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:48:20.507 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:20.508 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:48:20.508 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:48:20.557 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21236.asok 2022-07-12T08:48:20.562 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21236.asok 2022-07-12T08:48:20.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.563 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21236.asok status 2022-07-12T08:48:20.701 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:48:20.701 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:48:20.702 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:48:20.702 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:48:20.703 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:48:20.703 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:48:20.704 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:48:20.704 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21236", 2022-07-12T08:48:20.705 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:48:20.705 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:20.706 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:48:20.706 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:48:20.707 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 6784, 2022-07-12T08:48:20.707 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:48:20.707 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:48:20.708 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:48:20.708 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 6784 2022-07-12T08:48:20.709 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:20.709 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:48:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:48:20.711 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:48:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 4046384285 2022-07-12T08:48:20.712 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:48:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:48:20.713 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:48:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:48:20.714 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 4046384285 2022-07-12T08:48:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:20.715 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.6784 172.21.15.130:0/4046384285", 2022-07-12T08:48:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/4046384285", 2022-07-12T08:48:20.716 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:48:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 44, 2022-07-12T08:48:20.717 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 81, 2022-07-12T08:48:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:48:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:48:20.718 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:48:20.719 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:48:20.721 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.722 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:20.722 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:20.730 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:48:20.732 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:20.733 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:20.734 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:20.794 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:20.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 Processor -- start 2022-07-12T08:48:20.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 -- start start 2022-07-12T08:48:20.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe65c0b0150 0x7fe65c09a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:20.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe65c05f140 0x7fe65c09a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:20.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c067eb0 con 0x7fe65c09ad40 2022-07-12T08:48:20.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c05f650 con 0x7fe65c05f140 2022-07-12T08:48:20.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe660a4b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c0b0530 con 0x7fe65c0b0150 2022-07-12T08:48:20.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe65c05f140 0x7fe65c09a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:20.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe65c05f140 0x7fe65c09a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58262/0 (socket says 172.21.15.61:58262) 2022-07-12T08:48:20.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe65b59e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe65c09ad40 0x7fe65c144060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46484/0 (socket says 172.21.15.61:46484) 2022-07-12T08:48:20.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 -- 172.21.15.61:0/4242180337 learned_addr learned my addr 172.21.15.61:0/4242180337 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:20.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe65ad9d700 1 --2- 172.21.15.61:0/4242180337 >> v2:172.21.15.111:3300/0 conn(0x7fe65c0b0150 0x7fe65c09a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 -- 172.21.15.61:0/4242180337 >> v2:172.21.15.111:3300/0 conn(0x7fe65c0b0150 msgr2=0x7fe65c09a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 --2- 172.21.15.61:0/4242180337 >> v2:172.21.15.111:3300/0 conn(0x7fe65c0b0150 0x7fe65c09a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 -- 172.21.15.61:0/4242180337 >> v1:172.21.15.61:6789/0 conn(0x7fe65c09ad40 legacy=0x7fe65c144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 -- 172.21.15.61:0/4242180337 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c067c10 con 0x7fe65c05f140 2022-07-12T08:48:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe65a59c700 1 -- 172.21.15.61:0/4242180337 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480307350 0 0) 0x7fe65c067eb0 con 0x7fe65c09ad40 2022-07-12T08:48:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.907+0000 7fe653fff700 1 --2- 172.21.15.61:0/4242180337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c09a850 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7fe644002f70 tx=0x7fe644006eb0).ready entity=mon.0 client_cookie=3072449ffa266b3e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe65a59c700 1 -- 172.21.15.61:0/4242180337 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64400d610 con 0x7fe65c05f140 2022-07-12T08:48:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe65a59c700 1 -- 172.21.15.61:0/4242180337 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe644000e90 con 0x7fe65c05f140 2022-07-12T08:48:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe65a59c700 1 -- 172.21.15.61:0/4242180337 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64400d960 con 0x7fe65c05f140 2022-07-12T08:48:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 -- 172.21.15.61:0/4242180337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 msgr2=0x7fe65c09a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 --2- 172.21.15.61:0/4242180337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c09a850 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 rx=0x7fe644002f70 tx=0x7fe644006eb0).stop 2022-07-12T08:48:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 -- 172.21.15.61:0/4242180337 shutdown_connections 2022-07-12T08:48:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 --2- 172.21.15.61:0/4242180337 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c09a850 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 --2- 172.21.15.61:0/4242180337 >> v2:172.21.15.111:3300/0 conn(0x7fe65c0b0150 0x7fe65c09a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 -- 172.21.15.61:0/4242180337 >> 172.21.15.61:0/4242180337 conn(0x7fe65c0b1950 msgr2=0x7fe65c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 -- 172.21.15.61:0/4242180337 shutdown_connections 2022-07-12T08:48:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 -- 172.21.15.61:0/4242180337 wait complete. 2022-07-12T08:48:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 Processor -- start 2022-07-12T08:48:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.911+0000 7fe660a4b700 1 -- start start 2022-07-12T08:48:20.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c0983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:20.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 0x7fe65c0961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:20.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe65c0b0150 0x7fe65c144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:20.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c095d80 con 0x7fe65c05f140 2022-07-12T08:48:20.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c147e40 con 0x7fe65c0b0150 2022-07-12T08:48:20.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c147ba0 con 0x7fe65c09ad40 2022-07-12T08:48:20.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c0983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:20.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe653fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 0x7fe65c0961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:20.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe653fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 0x7fe65c0961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40464/0 (socket says 172.21.15.61:40464) 2022-07-12T08:48:20.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe653fff700 1 -- 172.21.15.61:0/30579749 learned_addr learned my addr 172.21.15.61:0/30579749 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:20.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c0983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58268/0 (socket says 172.21.15.61:58268) 2022-07-12T08:48:20.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65b59e700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe65c0b0150 0x7fe65c144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:20.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 -- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 msgr2=0x7fe65c0961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:20.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 0x7fe65c0961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:20.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 -- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe65c0b0150 msgr2=0x7fe65c144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:20.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe65c0b0150 0x7fe65c144400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:20.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 -- 172.21.15.61:0/30579749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64400b1e0 con 0x7fe65c05f140 2022-07-12T08:48:20.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe653fff700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 0x7fe65c0961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:20.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe65ad9d700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c0983b0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fe64c00a000 tx=0x7fe64c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:20.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64c0106e0 con 0x7fe65c05f140 2022-07-12T08:48:20.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c010840 con 0x7fe65c05f140 2022-07-12T08:48:20.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c0942e0 con 0x7fe65c05f140 2022-07-12T08:48:20.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe64c010ba0 con 0x7fe65c05f140 2022-07-12T08:48:20.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c1448f0 con 0x7fe65c05f140 2022-07-12T08:48:20.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe64c01a030 con 0x7fe65c05f140 2022-07-12T08:48:20.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.915+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c057460 con 0x7fe65c05f140 2022-07-12T08:48:20.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.919+0000 7fe658d99700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe640053a40 0x7fe640055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:20.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.919+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe64c06b9f0 con 0x7fe65c05f140 2022-07-12T08:48:20.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.919+0000 7fe653fff700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe640053a40 0x7fe640055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:20.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.919+0000 7fe653fff700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe640053a40 0x7fe640055f00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe64401c000 tx=0x7fe644015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:20.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:20.923+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe64c06de60 con 0x7fe65c05f140 2022-07-12T08:48:21.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.087+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe65c06b2b0 con 0x7fe65c05f140 2022-07-12T08:48:21.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe658d99700 1 -- 172.21.15.61:0/30579749 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fe64c042950 con 0x7fe65c05f140 2022-07-12T08:48:21.095 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:21.096 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:13.437914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3665156048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3665156048},{"type":"v1","addr":"172.21.15.61:6833","nonce":3665156048}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:21.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe640053a40 msgr2=0x7fe640055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:21.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe640053a40 0x7fe640055f00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe64401c000 tx=0x7fe644015000).stop 2022-07-12T08:48:21.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 msgr2=0x7fe65c0983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:21.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c0983b0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fe64c00a000 tx=0x7fe64c008040).stop 2022-07-12T08:48:21.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 shutdown_connections 2022-07-12T08:48:21.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe640053a40 0x7fe640055f00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe65c0b0150 0x7fe65c144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe65c09ad40 0x7fe65c0961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 --2- 172.21.15.61:0/30579749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe65c05f140 0x7fe65c0983b0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 >> 172.21.15.61:0/30579749 conn(0x7fe65c0b1950 msgr2=0x7fe65c0ab070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:21.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 shutdown_connections 2022-07-12T08:48:21.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.091+0000 7fe660a4b700 1 -- 172.21.15.61:0/30579749 wait complete. 2022-07-12T08:48:21.106 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 44 2022-07-12T08:48:21.109 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump inode 1 2022-07-12T08:48:21.219 INFO:teuthology.orchestra.run.smithi061.stdout:{"path":"/","ino":1,"rdev":0,"ctime":"2022-07-12T08:48:18.144262+0000","btime":"2022-07-12T08:48:12.440934+0000","mode":17407,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":16,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":0,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-12T08:48:12.440934+0000","atime":"0.000000","time_warp_seq":0,"change_attr":1,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-12T08:48:18.144262+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-12T08:48:12.440934+0000"},"version":3,"file_data_version":0,"xattr_version":1,"backtrace_version":3,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":3,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"authpin":1},"nref":9,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"xlockdone","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{"gather_set":[],"state":"sync","is_leased":false,"num_rdlocks":1,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent"],"client_caps":[{"client_id":6784,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2},{"client_id":6840,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[]} 2022-07-12T08:48:21.221 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "2022-07-12T08:48:12.440934+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "0.000000", "auth_pins": 2, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "xlockdone", "xlock_by": {} }, "backtrace_version": 3, "btime": "2022-07-12T08:48:12.440934+0000", "change_attr": 1, "client_caps": [ { "client_id": 6784, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" }, { "client_id": 6840, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-12T08:48:18.144262+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 16, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 17407, "mtime": "2022-07-12T08:48:12.440934+0000", "nestlock": {}, "nlink": 1, "nref": 9, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/", "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 3, "request": 1 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2022-07-12T08:48:18.144262+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": { "gather_set": [], "is_leased": false, "num_rdlocks": 1, "num_wrlocks": 0, "num_xlocks": 0, "state": "sync", "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 0, "truncate_size": 18446744073709551615, "uid": 0, "version": 3, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } 2022-07-12T08:48:21.221 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:48:21.222 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:21.244 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:21.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:21.292 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:21.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:21.337 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:48:21.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:21.338 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:21.338 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:21.354 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:48:21.354 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:21.355 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:48:21.356 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:48:21.356 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:21.372 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21764]: fuse finished with error 0 and tester_r 0 2022-07-12T08:48:21.374 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:21.375 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:21.375 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:21.380 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:48:21.382 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:48:21.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:48:21.453 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:21.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:21.466 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:21.466 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:21.479 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:48:21.480 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:21.481 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:48:21.482 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:48:21.482 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:21.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21236]: fuse finished with error 0 and tester_r 0 2022-07-12T08:48:21.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:21.540 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:21.541 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:21.549 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:48:21.552 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:48:21.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:48:21.598 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:21.646 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:21.647 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:21.647 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:21.657 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:21.659 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:48:21.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:48:21.660 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:21.671 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:21.671 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:48:21.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 Processor -- start 2022-07-12T08:48:21.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 -- start start 2022-07-12T08:48:21.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f56d40b5920 0x7f56d40b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40afa80 0x7f56d40b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f56d40b0520 0x7f56d40b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d4098510 con 0x7f56d40b0520 2022-07-12T08:48:21.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d40b5d00 con 0x7f56d40b5920 2022-07-12T08:48:21.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56db2bd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4098270 con 0x7f56d40afa80 2022-07-12T08:48:21.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f56d40b0520 0x7f56d40b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f56d40b0520 0x7f56d40b09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58280/0 (socket says 172.21.15.61:58280) 2022-07-12T08:48:21.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 -- 172.21.15.61:0/3441173854 learned_addr learned my addr 172.21.15.61:0/3441173854 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:21.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56d9aba700 1 --2- 172.21.15.61:0/3441173854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40afa80 0x7f56d40b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56da2bb700 1 --2- 172.21.15.61:0/3441173854 >> v2:172.21.15.111:3300/0 conn(0x7f56d40b5920 0x7f56d40b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 -- 172.21.15.61:0/3441173854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40afa80 msgr2=0x7f56d40b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:21.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 --2- 172.21.15.61:0/3441173854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40afa80 0x7f56d40b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 -- 172.21.15.61:0/3441173854 >> v2:172.21.15.111:3300/0 conn(0x7f56d40b5920 msgr2=0x7f56d40b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:21.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 --2- 172.21.15.61:0/3441173854 >> v2:172.21.15.111:3300/0 conn(0x7f56d40b5920 0x7f56d40b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.783+0000 7f56daabc700 1 -- 172.21.15.61:0/3441173854 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d40aa330 con 0x7f56d40b0520 2022-07-12T08:48:21.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56daabc700 1 --2- 172.21.15.61:0/3441173854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d40b09b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f56cc0032f0 tx=0x7f56cc009290).ready entity=mon.0 client_cookie=d47090ae82968857 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:21.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56d92b9700 1 -- 172.21.15.61:0/3441173854 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56cc00b680 con 0x7f56d40b0520 2022-07-12T08:48:21.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56d92b9700 1 -- 172.21.15.61:0/3441173854 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc00b7e0 con 0x7f56d40b0520 2022-07-12T08:48:21.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56d92b9700 1 -- 172.21.15.61:0/3441173854 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56cc00b950 con 0x7f56d40b0520 2022-07-12T08:48:21.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 -- 172.21.15.61:0/3441173854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 msgr2=0x7f56d40b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:21.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 --2- 172.21.15.61:0/3441173854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d40b09b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7f56cc0032f0 tx=0x7f56cc009290).stop 2022-07-12T08:48:21.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 -- 172.21.15.61:0/3441173854 shutdown_connections 2022-07-12T08:48:21.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 --2- 172.21.15.61:0/3441173854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40afa80 0x7f56d40b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 --2- 172.21.15.61:0/3441173854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d40b09b0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 --2- 172.21.15.61:0/3441173854 >> v2:172.21.15.111:3300/0 conn(0x7f56d40b5920 0x7f56d40b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 -- 172.21.15.61:0/3441173854 >> 172.21.15.61:0/3441173854 conn(0x7f56d4063380 msgr2=0x7f56d40b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:21.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 -- 172.21.15.61:0/3441173854 shutdown_connections 2022-07-12T08:48:21.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.787+0000 7f56db2bd700 1 -- 172.21.15.61:0/3441173854 wait complete. 2022-07-12T08:48:21.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 Processor -- start 2022-07-12T08:48:21.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 -- start start 2022-07-12T08:48:21.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d40afa80 0x7f56d40ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d4136500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40b5920 0x7f56d4142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4135e70 con 0x7f56d40b0520 2022-07-12T08:48:21.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4097a40 con 0x7f56d40afa80 2022-07-12T08:48:21.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56db2bd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4137e80 con 0x7f56d40b5920 2022-07-12T08:48:21.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d4136500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d4136500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58284/0 (socket says 172.21.15.61:58284) 2022-07-12T08:48:21.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 -- 172.21.15.61:0/267879288 learned_addr learned my addr 172.21.15.61:0/267879288 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:21.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56da2bb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d40afa80 0x7f56d40ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56daabc700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40b5920 0x7f56d4142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 -- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40b5920 msgr2=0x7f56d4142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:21.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40b5920 0x7f56d4142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 -- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d40afa80 msgr2=0x7f56d40ae260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:21.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d40afa80 0x7f56d40ae260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 -- 172.21.15.61:0/267879288 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc009530 con 0x7f56d40b0520 2022-07-12T08:48:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56d9aba700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d4136500 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f56c400c000 tx=0x7f56c4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.791+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56c4010780 con 0x7f56d40b0520 2022-07-12T08:48:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40108e0 con 0x7f56d40b0520 2022-07-12T08:48:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d40adb20 con 0x7f56d40b0520 2022-07-12T08:48:21.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56c4010c40 con 0x7f56d40b0520 2022-07-12T08:48:21.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d41381a0 con 0x7f56d40b0520 2022-07-12T08:48:21.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d4057460 con 0x7f56d40b0520 2022-07-12T08:48:21.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f56c401c460 con 0x7f56d40b0520 2022-07-12T08:48:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56c37fe700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56b40539d0 0x7f56b4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f56c406c340 con 0x7f56d40b0520 2022-07-12T08:48:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56da2bb700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56b40539d0 0x7f56b4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.795+0000 7f56da2bb700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56b40539d0 0x7f56b4055e90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f56d000b000 tx=0x7f56d0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:21.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.803+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56c4043130 con 0x7f56d40b0520 2022-07-12T08:48:21.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.931+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f56d40b1320 con 0x7f56d40b0520 2022-07-12T08:48:21.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.931+0000 7f56c37fe700 1 -- 172.21.15.61:0/267879288 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v81) v1 ==== 81+0+33696 (secure 0 0 0) 0x7f56c4046750 con 0x7f56d40b0520 2022-07-12T08:48:21.936 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:21.936 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:21.937 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 81, 2022-07-12T08:48:21.937 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:48:21.937 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:48:21.938 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:48:13.437499+0000", 2022-07-12T08:48:21.938 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:21.939 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:21.939 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:48:21.939 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:48:21.939 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:48:21.940 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:48:21.940 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:48:21.940 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:48:21.940 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:48:21.941 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:21.941 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:48:21.942 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:48:21.942 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:48:21.943 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:48:21.943 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:48:21.943 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 16, 2022-07-12T08:48:21.943 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:48:21.944 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:48:21.944 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:48:21.944 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:48:21.944 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:48:21.944 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:21.945 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:48:21.945 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:48:21.945 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:48:21.946 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:21.946 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:21.947 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:21.947 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:21.948 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:21.948 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:21.949 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:21.949 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:21.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:48:21.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:48:21.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:48:21.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:48:21.951 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:48:21.951 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:21.951 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:21.951 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:21.951 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:21.952 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:21.952 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:21.952 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:21.952 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.953 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:48:21.953 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:21.953 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:21.953 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:21.954 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:21.954 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:21.954 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:21.954 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:21.954 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:21.955 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:21.955 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:21.955 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:21.956 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:21.956 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:21.957 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:21.957 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:21.957 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:21.958 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:21.958 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:21.958 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:21.959 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:21.959 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:21.960 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:21.960 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:21.961 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:21.961 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:21.961 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:21.961 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.962 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:21.962 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:21.962 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:21.962 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:21.963 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:21.963 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:21.963 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:21.963 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:21.964 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:21.964 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:21.964 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:21.964 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:21.965 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:48:21.965 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.965 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:21.965 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:48:21.966 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:21.966 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.966 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:21.967 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:48:21.967 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:48:21.967 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:48:21.967 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:21.968 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:21.968 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:21.968 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:21.968 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:21.969 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:21.969 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:21.969 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:21.969 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:21.970 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:21.970 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:21.970 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:21.971 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:21.971 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:21.972 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:21.972 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:21.973 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:21.973 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:21.974 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:21.974 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:21.975 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.975 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:48:21.976 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:21.976 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:21.976 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:21.977 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:21.977 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:21.978 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:21.978 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:21.979 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:21.979 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:21.980 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:21.980 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:21.980 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:21.981 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:21.981 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:21.981 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:21.981 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:21.982 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:21.982 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:21.982 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:21.982 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:21.983 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:21.983 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:21.983 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:21.983 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:21.984 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:21.984 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:21.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.984 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:21.985 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:21.985 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:21.985 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:21.985 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:21.986 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:21.986 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:21.986 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:21.986 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:21.987 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:21.987 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:21.987 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:21.987 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:21.987 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:48:21.988 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:21.988 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.988 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:21.988 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 15, 2022-07-12T08:48:21.988 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:48:21.989 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:48:11.002774+0000", 2022-07-12T08:48:21.989 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:21.989 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:21.989 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:21.990 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:21.990 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:21.990 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:21.990 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:21.990 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:21.991 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:21.991 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:21.991 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:21.991 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:21.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:21.992 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:21.992 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:21.992 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:21.993 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:21.993 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:21.993 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:21.993 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:21.993 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:21.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "80", 2022-07-12T08:48:21.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:21.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:21.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:21.995 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:21.995 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:21.995 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:21.996 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:21.996 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:21.996 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:21.996 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:21.996 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:21.997 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:21.997 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:21.997 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:21.997 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:21.998 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:21.998 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:21.998 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:21.998 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:21.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:21.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:21.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:21.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:22.000 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:22.000 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:22.000 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:22.000 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.000 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:22.001 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:22.001 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:22.001 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:22.001 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:22.002 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:22.002 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:22.002 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:22.002 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:22.002 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:22.003 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:22.003 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:22.003 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:48:22.003 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:48:22.004 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:48:22.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.004 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:22.004 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:48:22.005 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:48:22.005 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.005 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.005 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.005 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.006 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 16, 2022-07-12T08:48:22.006 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:48:22.006 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:48:11.580821+0000", 2022-07-12T08:48:22.006 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:22.007 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:22.007 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:22.007 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:22.007 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:22.008 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:22.008 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:22.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:22.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:22.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:22.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:22.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:22.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:22.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:22.010 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:22.010 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:22.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:22.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:22.010 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:22.011 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:22.011 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.011 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "81", 2022-07-12T08:48:22.011 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:22.012 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:22.012 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:22.012 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:22.012 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:22.012 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:22.013 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:22.013 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:22.013 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:22.013 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:22.013 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:22.014 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:22.014 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:22.014 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:22.014 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:22.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:22.015 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:22.015 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:22.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:22.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:22.016 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:22.016 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:22.016 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:22.016 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:22.017 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:22.017 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:22.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.017 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:22.018 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:22.018 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:22.018 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:22.018 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:22.019 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:22.019 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:22.019 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:22.019 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:22.019 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:22.020 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:22.020 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:22.020 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:22.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:48:22.021 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:48:22.021 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.021 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.021 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.022 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:22.022 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:48:22.022 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.022 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:22.023 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:48:22.023 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.023 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.023 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.023 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.024 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.024 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.024 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.025 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:22.025 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.025 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.025 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.026 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.026 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:48:22.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.027 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.027 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:48:22.028 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.028 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.028 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.028 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.029 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.029 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.029 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.029 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.030 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:48:22.030 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.030 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.031 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.031 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.032 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:48:22.032 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.033 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.033 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.033 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.034 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.035 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:48:22.036 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.037 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:48:22.038 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.039 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.039 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.040 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.041 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.041 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.041 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:48:22.042 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.043 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:48:22.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:22.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.045 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.046 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:48:22.047 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:48:22.047 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:48:22.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:48:22.048 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.048 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.049 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.049 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.050 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:22.051 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:48:22.051 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.052 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.052 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.053 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.053 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.053 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.054 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.054 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 78, 2022-07-12T08:48:22.055 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.055 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.055 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:48:22.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:48:22.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.061 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.062 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:48:22.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.065 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:48:22.066 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.066 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.066 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.067 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.067 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.068 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:48:22.068 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:48:22.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.070 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.070 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.070 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.071 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:48:22.071 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.072 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:48:22.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:22.073 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.073 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.073 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.073 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:48:22.074 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:48:22.074 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:48:22.074 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:48:22.074 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.075 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.075 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.075 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.075 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.075 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.076 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:22.076 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:48:22.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.076 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.076 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.077 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.077 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.077 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.077 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.078 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:22.078 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.078 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.078 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.079 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.079 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:48:22.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.080 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.080 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.080 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:48:22.080 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.081 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.081 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.081 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.082 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:48:22.083 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.083 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.083 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:48:22.084 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.084 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.084 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.085 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.085 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:48:22.086 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.086 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.086 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.086 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:48:22.087 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.087 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.087 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.088 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.088 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.088 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:48:22.089 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.089 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.090 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:48:22.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:22.090 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.090 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.091 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:48:22.091 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:48:22.091 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:48:22.091 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:48:22.092 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.092 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.092 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.092 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.093 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:22.093 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:48:22.094 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.094 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.094 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.095 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.095 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.095 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.095 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.096 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:22.096 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.096 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.096 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.096 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:48:22.097 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.098 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.098 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.098 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.098 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:48:22.098 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.099 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.099 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.099 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:48:22.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.101 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.102 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:48:22.102 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.102 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.102 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.103 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.103 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.103 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.103 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.104 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:48:22.104 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.104 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.105 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.105 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:48:22.105 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.105 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.105 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.106 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.106 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.107 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:48:22.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.107 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.108 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:48:22.108 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:22.108 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.109 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.109 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.109 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:48:22.109 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:48:22.110 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:48:22.110 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:48:22.110 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.110 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.110 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.111 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.111 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:22.112 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:48:22.112 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.112 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.112 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.112 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.113 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.113 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.113 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.113 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:22.114 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.114 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.114 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.114 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.115 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.115 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:48:22.115 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.115 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.116 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.116 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:48:22.116 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.116 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.117 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.117 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.117 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.118 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:48:22.118 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.118 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.119 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:48:22.119 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.120 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.120 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.120 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.120 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.121 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.121 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:48:22.121 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.122 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.122 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.122 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:48:22.122 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.123 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.123 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.123 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:48:22.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.125 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.125 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.126 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:48:22.126 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:22.126 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.126 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.127 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.127 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:48:22.127 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:48:22.127 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:48:22.128 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:48:22.128 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.128 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.128 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.128 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.129 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:22.129 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:48:22.130 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.130 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.130 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.130 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.131 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.131 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.131 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.131 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:22.131 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.132 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.132 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.132 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.133 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:48:22.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.133 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.134 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.134 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:48:22.134 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.134 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.135 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.135 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:48:22.136 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.137 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:48:22.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.138 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.138 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.138 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.138 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:48:22.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.140 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:48:22.141 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.141 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.142 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.142 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.143 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:48:22.143 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:48:22.144 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:22.144 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.145 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.145 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:48:22.145 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:48:22.146 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:48:22.146 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:48:22.146 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.146 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.147 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.147 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.147 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.147 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:22.148 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:48:22.148 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.148 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.148 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.149 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.149 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.149 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:22.150 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.150 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.150 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.150 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.151 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:48:22.151 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.151 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.152 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:48:22.152 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.153 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.153 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.154 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.154 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.154 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:48:22.155 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.155 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.155 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.156 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:48:22.156 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.156 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.156 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.157 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.157 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.157 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.158 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:48:22.158 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.158 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.159 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.159 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.159 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:48:22.159 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.159 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.160 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.160 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.160 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.161 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.161 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:48:22.161 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.161 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.162 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.162 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:48:22.162 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:22.163 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.163 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.163 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:48:22.163 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:48:22.164 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:48:22.164 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:48:22.164 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.164 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.165 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.165 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.165 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.166 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:22.166 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:48:22.166 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:22.166 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:22.166 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:22.167 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:22.167 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:22.167 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:22.167 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:22.168 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-12T08:48:22.168 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:22.168 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:22.169 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:22.169 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.170 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.170 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:48:22.171 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.172 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:48:22.174 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.175 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.175 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.175 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.176 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:22.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.178 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:48:22.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:48:22.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.182 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.182 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.183 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:22.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.184 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.185 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:48:22.186 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.186 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.187 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.188 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:48:22.190 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.190 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.191 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.191 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:22.193 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:22.193 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.194 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:22.194 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:48:22.195 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.196 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.196 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.196 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:22.197 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:48:22.197 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:22.198 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.198 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.199 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:48:22.199 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:48:22.200 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:48:22.200 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:48:22.201 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:22.201 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:22.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:22.202 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:22.202 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.203 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:22.203 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:48:22.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.204 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:22.205 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.205 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.206 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.206 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.207 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.207 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:48:22.208 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.208 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.209 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:22.209 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.210 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.210 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.211 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.211 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.211 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:48:22.212 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.212 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.212 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:22.213 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.213 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.213 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.213 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.214 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.214 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:48:22.214 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.214 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.215 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:22.215 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.215 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.216 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.216 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.216 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:48:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.218 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.218 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.218 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.218 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.219 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:48:22.219 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.219 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.219 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:22.220 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.220 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.220 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.220 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.220 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:48:22.221 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.221 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.221 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.222 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:22.222 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.222 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.222 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.223 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.223 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.223 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:48:22.223 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:22.224 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:22.224 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:22.225 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:22.225 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:22.225 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:22.225 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:22.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:48:22.226 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:22.226 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.226 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:22.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:48:22.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:48:22.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:48:22.227 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:48:22.227 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:48:22.228 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:48:22.228 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:48:22.228 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:48:22.228 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:48:22.229 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:48:22.229 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:48:22.229 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:48:22.229 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:48:22.230 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:22.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:22.230 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:48:22.230 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:48:22.230 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:48:22.231 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:48:22.231 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:48:22.231 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:22.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56b40539d0 msgr2=0x7f56b4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56b40539d0 0x7f56b4055e90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f56d000b000 tx=0x7f56d0006df0).stop 2022-07-12T08:48:22.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 msgr2=0x7f56d4136500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d4136500 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 rx=0x7f56c400c000 tx=0x7f56c4006df0).stop 2022-07-12T08:48:22.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 shutdown_connections 2022-07-12T08:48:22.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f56d40b5920 0x7f56d4142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f56d40b0520 0x7f56d4136500 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f56b40539d0 0x7f56b4055e90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 --2- 172.21.15.61:0/267879288 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f56d40afa80 0x7f56d40ae260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 >> 172.21.15.61:0/267879288 conn(0x7f56d4063380 msgr2=0x7f56d4098650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:22.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 shutdown_connections 2022-07-12T08:48:22.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:21.935+0000 7f56db2bd700 1 -- 172.21.15.61:0/267879288 wait complete. 2022-07-12T08:48:22.237 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:22.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.343+0000 7ff9d6563700 1 Processor -- start 2022-07-12T08:48:22.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.343+0000 7ff9d6563700 1 -- start start 2022-07-12T08:48:22.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d00aa7f0 0x7ff9d0098200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d005f650 con 0x7ff9d00b1450 2022-07-12T08:48:22.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d00aabd0 con 0x7ff9d005f140 2022-07-12T08:48:22.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0098e00 con 0x7ff9d00aa7f0 2022-07-12T08:48:22.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d4d60700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff9d00b1450 0x7ff9d00986f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46510/0 (socket says 172.21.15.61:46510) 2022-07-12T08:48:22.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d4d60700 1 -- 172.21.15.61:0/1177615119 learned_addr learned my addr 172.21.15.61:0/1177615119 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:22.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d5561700 1 --2- 172.21.15.61:0/1177615119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d00aa7f0 0x7ff9d0098200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142637326 0 0) 0x7ff9d00aabd0 con 0x7ff9d005f140 2022-07-12T08:48:22.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b8003260 con 0x7ff9d005f140 2022-07-12T08:48:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811609816 0 0) 0x7ff9d005f650 con 0x7ff9d00b1450 2022-07-12T08:48:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9d00aabd0 con 0x7ff9d00b1450 2022-07-12T08:48:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d5561700 1 -- 172.21.15.61:0/1177615119 >> v1:172.21.15.111:6789/0 conn(0x7ff9d005f140 legacy=0x7ff9d0141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d5561700 1 -- 172.21.15.61:0/1177615119 >> v1:172.21.15.61:6789/0 conn(0x7ff9d00b1450 legacy=0x7ff9d00986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d5561700 1 -- 172.21.15.61:0/1177615119 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d0067c80 con 0x7ff9d00aa7f0 2022-07-12T08:48:22.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d5561700 1 --2- 172.21.15.61:0/1177615119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d00aa7f0 0x7ff9d0098200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff9cc002f70 tx=0x7ff9cc006df0).ready entity=mon.2 client_cookie=90fc93e5a8b3911a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9cc00d5c0 con 0x7ff9d00aa7f0 2022-07-12T08:48:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc000e90 con 0x7ff9d00aa7f0 2022-07-12T08:48:22.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9bffff700 1 -- 172.21.15.61:0/1177615119 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9cc00d910 con 0x7ff9d00aa7f0 2022-07-12T08:48:22.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- 172.21.15.61:0/1177615119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d00aa7f0 msgr2=0x7ff9d0098200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 --2- 172.21.15.61:0/1177615119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d00aa7f0 0x7ff9d0098200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff9cc002f70 tx=0x7ff9cc006df0).stop 2022-07-12T08:48:22.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- 172.21.15.61:0/1177615119 shutdown_connections 2022-07-12T08:48:22.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 --2- 172.21.15.61:0/1177615119 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d00aa7f0 0x7ff9d0098200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- 172.21.15.61:0/1177615119 >> 172.21.15.61:0/1177615119 conn(0x7ff9d00b1950 msgr2=0x7ff9d00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:22.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- 172.21.15.61:0/1177615119 shutdown_connections 2022-07-12T08:48:22.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.347+0000 7ff9d6563700 1 -- 172.21.15.61:0/1177615119 wait complete. 2022-07-12T08:48:22.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 Processor -- start 2022-07-12T08:48:22.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- start start 2022-07-12T08:48:22.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d005f140 0x7ff9d0140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9d00aa7f0 0x7ff9d00ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 0x7ff9d0150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00ae760 con 0x7ff9d00b1450 2022-07-12T08:48:22.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00aea00 con 0x7ff9d00aa7f0 2022-07-12T08:48:22.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0067680 con 0x7ff9d005f140 2022-07-12T08:48:22.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 0x7ff9d0150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 0x7ff9d0150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58300/0 (socket says 172.21.15.61:58300) 2022-07-12T08:48:22.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 -- 172.21.15.61:0/2632152638 learned_addr learned my addr 172.21.15.61:0/2632152638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:22.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d4d60700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9d00aa7f0 0x7ff9d00ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5561700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d005f140 0x7ff9d0140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 -- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d005f140 msgr2=0x7ff9d0140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d005f140 0x7ff9d0140090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 -- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9d00aa7f0 msgr2=0x7ff9d00ad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9d00aa7f0 0x7ff9d00ad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 -- 172.21.15.61:0/2632152638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc00b180 con 0x7ff9d00b1450 2022-07-12T08:48:22.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d4d60700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9d00aa7f0 0x7ff9d00ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:22.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d5d62700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 0x7ff9d0150b70 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff9c8001490 tx=0x7ff9c8007df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:22.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9c8002310 con 0x7ff9d00b1450 2022-07-12T08:48:22.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8002470 con 0x7ff9d00b1450 2022-07-12T08:48:22.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d00ae330 con 0x7ff9d00b1450 2022-07-12T08:48:22.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9c8001840 con 0x7ff9d00b1450 2022-07-12T08:48:22.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d0151200 con 0x7ff9d00b1450 2022-07-12T08:48:22.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.351+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d0057460 con 0x7ff9d00b1450 2022-07-12T08:48:22.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.355+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff9c8019030 con 0x7ff9d00b1450 2022-07-12T08:48:22.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.355+0000 7ff9be7fc700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9b0053920 0x7ff9b0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.355+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff9c806b6b0 con 0x7ff9d00b1450 2022-07-12T08:48:22.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.355+0000 7ff9d5561700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9b0053920 0x7ff9b0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.355+0000 7ff9d5561700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9b0053920 0x7ff9b0055de0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ff9cc018000 tx=0x7ff9cc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:22.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.363+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff9c8042610 con 0x7ff9d00b1450 2022-07-12T08:48:22.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.523+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9d006b2b0 con 0x7ff9d00b1450 2022-07-12T08:48:22.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.523+0000 7ff9be7fc700 1 -- 172.21.15.61:0/2632152638 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+2143 (secure 0 0 0) 0x7ff9c8045c30 con 0x7ff9d00b1450 2022-07-12T08:48:22.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:22.530 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":44,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:13.437914+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5981},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5981":{"gid":5981,"name":"a","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3665156048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3665156048},{"type":"v1","addr":"172.21.15.61:6833","nonce":3665156048}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:22.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9b0053920 msgr2=0x7ff9b0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9b0053920 0x7ff9b0055de0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ff9cc018000 tx=0x7ff9cc014000).stop 2022-07-12T08:48:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 msgr2=0x7ff9d0150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 0x7ff9d0150b70 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 rx=0x7ff9c8001490 tx=0x7ff9c8007df0).stop 2022-07-12T08:48:22.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 shutdown_connections 2022-07-12T08:48:22.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9d00b1450 0x7ff9d0150b70 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9b0053920 0x7ff9b0055de0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9d00aa7f0 0x7ff9d00ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 --2- 172.21.15.61:0/2632152638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9d005f140 0x7ff9d0140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 >> 172.21.15.61:0/2632152638 conn(0x7ff9d00b1950 msgr2=0x7ff9d0098120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:22.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 shutdown_connections 2022-07-12T08:48:22.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.527+0000 7ff9d6563700 1 -- 172.21.15.61:0/2632152638 wait complete. 2022-07-12T08:48:22.540 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 44 2022-07-12T08:48:22.543 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:48:22.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 Processor -- start 2022-07-12T08:48:22.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- start start 2022-07-12T08:48:22.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f35e00ab670 0x7f35e005f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f35e00afa60 0x7f35e00afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00abc70 con 0x7f35e00ab290 2022-07-12T08:48:22.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00aba90 con 0x7f35e00ab670 2022-07-12T08:48:22.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e005c720 con 0x7f35e00afa60 2022-07-12T08:48:22.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e4cd6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f35e00ab670 0x7f35e005f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e54d7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f35e00ab290 0x7f35e005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46522/0 (socket says 172.21.15.61:46522) 2022-07-12T08:48:22.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e54d7700 1 -- 172.21.15.61:0/3181948370 learned_addr learned my addr 172.21.15.61:0/3181948370 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:22.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e5cd8700 1 --2- 172.21.15.61:0/3181948370 >> v2:172.21.15.111:3300/0 conn(0x7f35e00afa60 0x7f35e00afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e4cd6700 1 -- 172.21.15.61:0/3181948370 >> v2:172.21.15.111:3300/0 conn(0x7f35e00afa60 msgr2=0x7f35e00afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:22.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e4cd6700 1 --2- 172.21.15.61:0/3181948370 >> v2:172.21.15.111:3300/0 conn(0x7f35e00afa60 0x7f35e00afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e4cd6700 1 -- 172.21.15.61:0/3181948370 >> v1:172.21.15.61:6789/0 conn(0x7f35e00ab290 legacy=0x7f35e005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e4cd6700 1 -- 172.21.15.61:0/3181948370 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e005c9c0 con 0x7f35e00ab670 2022-07-12T08:48:22.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e4cd6700 1 --2- 172.21.15.61:0/3181948370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab670 0x7f35e005f140 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f35d80032f0 tx=0x7f35d80091d0).ready entity=mon.0 client_cookie=45d68bb5f983d2d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:22.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35d7fff700 1 -- 172.21.15.61:0/3181948370 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d8009ea0 con 0x7f35e00ab670 2022-07-12T08:48:22.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35d7fff700 1 -- 172.21.15.61:0/3181948370 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d800b870 con 0x7f35e00ab670 2022-07-12T08:48:22.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35d7fff700 1 -- 172.21.15.61:0/3181948370 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d800b9e0 con 0x7f35e00ab670 2022-07-12T08:48:22.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- 172.21.15.61:0/3181948370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab670 msgr2=0x7f35e005f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 --2- 172.21.15.61:0/3181948370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab670 0x7f35e005f140 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 rx=0x7f35d80032f0 tx=0x7f35d80091d0).stop 2022-07-12T08:48:22.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- 172.21.15.61:0/3181948370 shutdown_connections 2022-07-12T08:48:22.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 --2- 172.21.15.61:0/3181948370 >> v2:172.21.15.111:3300/0 conn(0x7f35e00afa60 0x7f35e00afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 --2- 172.21.15.61:0/3181948370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab670 0x7f35e005f140 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- 172.21.15.61:0/3181948370 >> 172.21.15.61:0/3181948370 conn(0x7f35e00b1950 msgr2=0x7f35e00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:22.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- 172.21.15.61:0/3181948370 shutdown_connections 2022-07-12T08:48:22.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.647+0000 7f35e64d9700 1 -- 172.21.15.61:0/3181948370 wait complete. 2022-07-12T08:48:22.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 Processor -- start 2022-07-12T08:48:22.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 -- start start 2022-07-12T08:48:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 0x7f35e00673f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e00ab670 0x7f35e0099450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35e00afa60 0x7f35e0065a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e0064210 con 0x7f35e00ab290 2022-07-12T08:48:22.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00644b0 con 0x7f35e00ab670 2022-07-12T08:48:22.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00adf00 con 0x7f35e00afa60 2022-07-12T08:48:22.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 0x7f35e00673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 0x7f35e00673f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58310/0 (socket says 172.21.15.61:58310) 2022-07-12T08:48:22.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 -- 172.21.15.61:0/1644451878 learned_addr learned my addr 172.21.15.61:0/1644451878 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:22.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e5cd8700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35e00afa60 0x7f35e0065a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e4cd6700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e00ab670 0x7f35e0099450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 -- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35e00afa60 msgr2=0x7f35e0065a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35e00afa60 0x7f35e0065a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 -- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e00ab670 msgr2=0x7f35e0099450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:22.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e00ab670 0x7f35e0099450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:22.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 -- 172.21.15.61:0/1644451878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8009470 con 0x7f35e00ab290 2022-07-12T08:48:22.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e4cd6700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e00ab670 0x7f35e0099450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:22.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e5cd8700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35e00afa60 0x7f35e0065a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:22.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e54d7700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 0x7f35e00673f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f35e00615d0 tx=0x7f35dc007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35dc00ce30 con 0x7f35e00ab290 2022-07-12T08:48:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc0072e0 con 0x7f35e00ab290 2022-07-12T08:48:22.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e00ae3a0 con 0x7f35e00ab290 2022-07-12T08:48:22.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35dc005800 con 0x7f35e00ab290 2022-07-12T08:48:22.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.651+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e00af110 con 0x7f35e00ab290 2022-07-12T08:48:22.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.655+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0057460 con 0x7f35e00ab290 2022-07-12T08:48:22.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.655+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f35dc010070 con 0x7f35e00ab290 2022-07-12T08:48:22.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.655+0000 7f35d67fc700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35c0053ad0 0x7f35c0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:22.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.655+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f35dc06b580 con 0x7f35e00ab290 2022-07-12T08:48:22.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.655+0000 7f35e4cd6700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35c0053ad0 0x7f35c0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:22.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.655+0000 7f35e4cd6700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35c0053ad0 0x7f35c0055f90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f35d8019000 tx=0x7f35d8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:22.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.663+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35dc0423d0 con 0x7f35e00ab290 2022-07-12T08:48:22.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:22.827+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f35e006b2b0 con 0x7f35e00ab290 2022-07-12T08:48:23.301 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:48:23.299+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:48:23.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.311+0000 7f35d67fc700 1 -- 172.21.15.61:0/1644451878 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) v1 ==== 161+0+0 (secure 0 0 0) 0x7f35dc018070 con 0x7f35e00ab290 2022-07-12T08:48:23.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35c0053ad0 msgr2=0x7f35c0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35c0053ad0 0x7f35c0055f90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f35d8019000 tx=0x7f35d8016000).stop 2022-07-12T08:48:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 msgr2=0x7f35e00673f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 0x7f35e00673f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f35e00615d0 tx=0x7f35dc007c00).stop 2022-07-12T08:48:23.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 shutdown_connections 2022-07-12T08:48:23.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f35e00afa60 0x7f35e0065a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35c0053ad0 0x7f35c0055f90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35e00ab670 0x7f35e0099450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 --2- 172.21.15.61:0/1644451878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e00ab290 0x7f35e00673f0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 >> 172.21.15.61:0/1644451878 conn(0x7f35e00b1950 msgr2=0x7f35e00b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:23.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 shutdown_connections 2022-07-12T08:48:23.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.315+0000 7f35e64d9700 1 -- 172.21.15.61:0/1644451878 wait complete. 2022-07-12T08:48:23.328 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:48:23.337 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:23.348 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:48:23.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 Processor -- start 2022-07-12T08:48:23.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 -- start start 2022-07-12T08:48:23.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e813ce60 0x7f90e80b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e80b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e813d240 con 0x7f90e8142d50 2022-07-12T08:48:23.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e80b24c0 con 0x7f90e813ce60 2022-07-12T08:48:23.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90ee154700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e81431f0 con 0x7f90e80b47f0 2022-07-12T08:48:23.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e813ce60 0x7f90e80b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e813ce60 0x7f90e80b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58332/0 (socket says 172.21.15.61:58332) 2022-07-12T08:48:23.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e80b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 -- 172.21.15.61:0/2366900202 learned_addr learned my addr 172.21.15.61:0/2366900202 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:23.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e6ffd700 1 --1- 172.21.15.61:0/2366900202 >> v1:172.21.15.111:6789/0 conn(0x7f90e8142d50 0x7f90e80b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:60150/0 (socket says 172.21.15.61:60150) 2022-07-12T08:48:23.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 -- 172.21.15.61:0/2366900202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 msgr2=0x7f90e80b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:23.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2366900202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e80b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 -- 172.21.15.61:0/2366900202 >> v1:172.21.15.111:6789/0 conn(0x7f90e8142d50 legacy=0x7f90e80b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 -- 172.21.15.61:0/2366900202 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e8143490 con 0x7f90e813ce60 2022-07-12T08:48:23.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.447+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2366900202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e813ce60 0x7f90e80b15a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f90d8002f70 tx=0x7f90d8006eb0).ready entity=mon.0 client_cookie=7bfc01bccbbdff0e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:23.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90e67fc700 1 -- 172.21.15.61:0/2366900202 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d800d570 con 0x7f90e813ce60 2022-07-12T08:48:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90e67fc700 1 -- 172.21.15.61:0/2366900202 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8000e90 con 0x7f90e813ce60 2022-07-12T08:48:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90e67fc700 1 -- 172.21.15.61:0/2366900202 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d800d900 con 0x7f90e813ce60 2022-07-12T08:48:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- 172.21.15.61:0/2366900202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e813ce60 msgr2=0x7f90e80b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 --2- 172.21.15.61:0/2366900202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e813ce60 0x7f90e80b15a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f90d8002f70 tx=0x7f90d8006eb0).stop 2022-07-12T08:48:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- 172.21.15.61:0/2366900202 shutdown_connections 2022-07-12T08:48:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 --2- 172.21.15.61:0/2366900202 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e80b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 --2- 172.21.15.61:0/2366900202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e813ce60 0x7f90e80b15a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- 172.21.15.61:0/2366900202 >> 172.21.15.61:0/2366900202 conn(0x7f90e805f9f0 msgr2=0x7f90e8066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- 172.21.15.61:0/2366900202 shutdown_connections 2022-07-12T08:48:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- 172.21.15.61:0/2366900202 wait complete. 2022-07-12T08:48:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 Processor -- start 2022-07-12T08:48:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- start start 2022-07-12T08:48:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e814a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e813ce60 0x7f90e813ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 0x7f90e813b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e813e260 con 0x7f90e8142d50 2022-07-12T08:48:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e813e500 con 0x7f90e813ce60 2022-07-12T08:48:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90ee154700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e813e7a0 con 0x7f90e80b47f0 2022-07-12T08:48:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.451+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 0x7f90e813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e813ce60 0x7f90e813ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e77fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e814a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 0x7f90e813b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58338/0 (socket says 172.21.15.61:58338) 2022-07-12T08:48:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 -- 172.21.15.61:0/2485121168 learned_addr learned my addr 172.21.15.61:0/2485121168 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 -- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 msgr2=0x7f90e814a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e814a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 -- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e813ce60 msgr2=0x7f90e813ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e813ce60 0x7f90e813ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 -- 172.21.15.61:0/2485121168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d800b1e0 con 0x7f90e8142d50 2022-07-12T08:48:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e7fff700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 0x7f90e813b120 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f90e0006e00 tx=0x7f90e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e000c640 con 0x7f90e8142d50 2022-07-12T08:48:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e000c7a0 con 0x7f90e8142d50 2022-07-12T08:48:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e000cb00 con 0x7f90e8142d50 2022-07-12T08:48:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e813ec40 con 0x7f90e8142d50 2022-07-12T08:48:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e80afa40 con 0x7f90e8142d50 2022-07-12T08:48:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e80b7ee0 con 0x7f90e8142d50 2022-07-12T08:48:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f90e000cc60 con 0x7f90e8142d50 2022-07-12T08:48:23.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e4ff9700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f90c80539b0 0x7f90c8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.455+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f90e006c1b0 con 0x7f90e8142d50 2022-07-12T08:48:23.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.459+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f90c80539b0 0x7f90c8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.459+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f90c80539b0 0x7f90c8055e70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f90d801c000 tx=0x7f90d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:23.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.463+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90e00430c0 con 0x7f90e8142d50 2022-07-12T08:48:23.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.631+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90e813e960 con 0x7f90e8142d50 2022-07-12T08:48:23.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.631+0000 7f90e4ff9700 1 -- 172.21.15.61:0/2485121168 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f90e00466e0 con 0x7f90e8142d50 2022-07-12T08:48:23.636 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:23.637 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41}],"filesystems":[{"mdsmap":{"epoch":45,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:12.433872+0000","modified":"2022-07-12T08:48:23.301065+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-12T08:48:23.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f90c80539b0 msgr2=0x7f90c8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f90c80539b0 0x7f90c8055e70 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f90d801c000 tx=0x7f90d8015000).stop 2022-07-12T08:48:23.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 msgr2=0x7f90e813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 0x7f90e813b120 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 rx=0x7f90e0006e00 tx=0x7f90e00091d0).stop 2022-07-12T08:48:23.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 shutdown_connections 2022-07-12T08:48:23.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e8142d50 0x7f90e813b120 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f90c80539b0 0x7f90c8055e70 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90e813ce60 0x7f90e813ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 --2- 172.21.15.61:0/2485121168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90e80b47f0 0x7f90e814a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 >> 172.21.15.61:0/2485121168 conn(0x7f90e805f9f0 msgr2=0x7f90e80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:23.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 shutdown_connections 2022-07-12T08:48:23.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.635+0000 7f90ee154700 1 -- 172.21.15.61:0/2485121168 wait complete. 2022-07-12T08:48:23.643 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-12T08:48:23.649 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:48:23.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.799+0000 7fc11bfff700 1 Processor -- start 2022-07-12T08:48:23.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.799+0000 7fc11bfff700 1 -- start start 2022-07-12T08:48:23.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc11c05cd20 0x7fc11c05d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c060e80 0x7fc11c14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c14a560 con 0x7fc11c05f860 2022-07-12T08:48:23.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c097d40 con 0x7fc11c05cd20 2022-07-12T08:48:23.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0aafa0 con 0x7fc11c060e80 2022-07-12T08:48:23.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc11c05f860 0x7fc11c05fcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46564/0 (socket says 172.21.15.61:46564) 2022-07-12T08:48:23.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11b7fe700 1 -- 172.21.15.61:0/2684392172 learned_addr learned my addr 172.21.15.61:0/2684392172 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:23.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11affd700 1 --2- 172.21.15.61:0/2684392172 >> v2:172.21.15.61:3300/0 conn(0x7fc11c05cd20 0x7fc11c05d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11a7fc700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c060e80 0x7fc11c14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc119ffb700 1 -- 172.21.15.61:0/2684392172 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466639113 0 0) 0x7fc11c14a560 con 0x7fc11c05f860 2022-07-12T08:48:23.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc119ffb700 1 -- 172.21.15.61:0/2684392172 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0fc003260 con 0x7fc11c05f860 2022-07-12T08:48:23.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11affd700 1 -- 172.21.15.61:0/2684392172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c060e80 msgr2=0x7fc11c14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:23.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11affd700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c060e80 0x7fc11c14ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc119ffb700 1 -- 172.21.15.61:0/2684392172 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1817019725 0 0) 0x7fc0fc003260 con 0x7fc11c05f860 2022-07-12T08:48:23.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11affd700 1 -- 172.21.15.61:0/2684392172 >> v1:172.21.15.61:6789/0 conn(0x7fc11c05f860 legacy=0x7fc11c05fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11affd700 1 -- 172.21.15.61:0/2684392172 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c14a790 con 0x7fc11c05cd20 2022-07-12T08:48:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11a7fc700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c060e80 0x7fc11c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11affd700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c05cd20 0x7fc11c05d100 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7fc114001800 tx=0x7fc104006df0).ready entity=mon.0 client_cookie=d23e39829d6e111 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:23.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc119ffb700 1 -- 172.21.15.61:0/2684392172 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc114001660 con 0x7fc11c05cd20 2022-07-12T08:48:23.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc119ffb700 1 -- 172.21.15.61:0/2684392172 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc104000e90 con 0x7fc11c05cd20 2022-07-12T08:48:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc119ffb700 1 -- 172.21.15.61:0/2684392172 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc104012460 con 0x7fc11c05cd20 2022-07-12T08:48:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- 172.21.15.61:0/2684392172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c05cd20 msgr2=0x7fc11c05d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c05cd20 0x7fc11c05d100 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7fc114001800 tx=0x7fc104006df0).stop 2022-07-12T08:48:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- 172.21.15.61:0/2684392172 shutdown_connections 2022-07-12T08:48:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c060e80 0x7fc11c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 --2- 172.21.15.61:0/2684392172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c05cd20 0x7fc11c05d100 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- 172.21.15.61:0/2684392172 >> 172.21.15.61:0/2684392172 conn(0x7fc11c0b1a50 msgr2=0x7fc11c0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:23.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- 172.21.15.61:0/2684392172 shutdown_connections 2022-07-12T08:48:23.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.803+0000 7fc11bfff700 1 -- 172.21.15.61:0/2684392172 wait complete. 2022-07-12T08:48:23.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 Processor -- start 2022-07-12T08:48:23.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 -- start start 2022-07-12T08:48:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c05f860 0x7fc11c09aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 0x7fc11c0638f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc11c063de0 0x7fc11c1465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0acb50 con 0x7fc11c060e80 2022-07-12T08:48:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c0acdf0 con 0x7fc11c063de0 2022-07-12T08:48:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c09a5f0 con 0x7fc11c05f860 2022-07-12T08:48:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 0x7fc11c0638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 0x7fc11c0638f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58352/0 (socket says 172.21.15.61:58352) 2022-07-12T08:48:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 -- 172.21.15.61:0/4293815743 learned_addr learned my addr 172.21.15.61:0/4293815743 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11affd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c05f860 0x7fc11c09aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11b7fe700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc11c063de0 0x7fc11c1465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 -- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c05f860 msgr2=0x7fc11c09aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c05f860 0x7fc11c09aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 -- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc11c063de0 msgr2=0x7fc11c1465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:23.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc11c063de0 0x7fc11c1465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:23.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 -- 172.21.15.61:0/4293815743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c009470 con 0x7fc11c060e80 2022-07-12T08:48:23.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11a7fc700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 0x7fc11c0638f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7fc10c00c000 tx=0x7fc10c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:23.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc10c00c850 con 0x7fc11c060e80 2022-07-12T08:48:23.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c00c9b0 con 0x7fc11c060e80 2022-07-12T08:48:23.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc10400b180 con 0x7fc11c060e80 2022-07-12T08:48:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.807+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc10c00cd10 con 0x7fc11c060e80 2022-07-12T08:48:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.811+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c067850 con 0x7fc11c060e80 2022-07-12T08:48:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.811+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c09af80 con 0x7fc11c060e80 2022-07-12T08:48:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.811+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc10c022070 con 0x7fc11c060e80 2022-07-12T08:48:23.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.811+0000 7fc113fff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc100053ad0 0x7fc100055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:23.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.811+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc10c06c5d0 con 0x7fc11c060e80 2022-07-12T08:48:23.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.811+0000 7fc11affd700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc100053ad0 0x7fc100055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:23.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.815+0000 7fc11affd700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc100053ad0 0x7fc100055f90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fc10401a000 tx=0x7fc104014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:23.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.819+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc10c0434e0 con 0x7fc11c060e80 2022-07-12T08:48:23.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:23.979+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc11c09a7b0 con 0x7fc11c060e80 2022-07-12T08:48:24.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.315+0000 7fc113fff700 1 -- 172.21.15.61:0/4293815743 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc10c046b00 con 0x7fc11c060e80 2022-07-12T08:48:24.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc100053ad0 msgr2=0x7fc100055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:24.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc100053ad0 0x7fc100055f90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fc10401a000 tx=0x7fc104014000).stop 2022-07-12T08:48:24.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 msgr2=0x7fc11c0638f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:24.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 0x7fc11c0638f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 rx=0x7fc10c00c000 tx=0x7fc10c0091d0).stop 2022-07-12T08:48:24.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 shutdown_connections 2022-07-12T08:48:24.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc100053ad0 0x7fc100055f90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc11c063de0 0x7fc11c1465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc11c060e80 0x7fc11c0638f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 --2- 172.21.15.61:0/4293815743 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc11c05f860 0x7fc11c09aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 >> 172.21.15.61:0/4293815743 conn(0x7fc11c0b1a50 msgr2=0x7fc11c05fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:24.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 shutdown_connections 2022-07-12T08:48:24.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.319+0000 7fc11bfff700 1 -- 172.21.15.61:0/4293815743 wait complete. 2022-07-12T08:48:24.339 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:48:24.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.455+0000 7f0c804ec700 1 Processor -- start 2022-07-12T08:48:24.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.455+0000 7f0c804ec700 1 -- start start 2022-07-12T08:48:24.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b3180 0x7f0c780b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0c780b2630 0x7f0c781420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c780ba290 con 0x7f0c780b9eb0 2022-07-12T08:48:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c780b1410 con 0x7f0c780b2630 2022-07-12T08:48:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780b4010 con 0x7f0c780b3180 2022-07-12T08:48:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7e288700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0c780b9eb0 0x7f0c780b0a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46578/0 (socket says 172.21.15.61:46578) 2022-07-12T08:48:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7e288700 1 -- 172.21.15.61:0/1064845223 learned_addr learned my addr 172.21.15.61:0/1064845223 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:24.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/1064845223 >> v2:172.21.15.111:3300/0 conn(0x7f0c780b2630 0x7f0c781420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:24.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7da87700 1 --2- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b3180 0x7f0c780b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:24.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7d286700 1 -- 172.21.15.61:0/1064845223 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839857094 0 0) 0x7f0c780ba290 con 0x7f0c780b9eb0 2022-07-12T08:48:24.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7d286700 1 -- 172.21.15.61:0/1064845223 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60003260 con 0x7f0c780b9eb0 2022-07-12T08:48:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7ea89700 1 -- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b3180 msgr2=0x7f0c780b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b3180 0x7f0c780b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7ea89700 1 -- 172.21.15.61:0/1064845223 >> v1:172.21.15.61:6789/0 conn(0x7f0c780b9eb0 legacy=0x7f0c780b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:24.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7d286700 1 -- 172.21.15.61:0/1064845223 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 315700825 0 0) 0x7f0c60003260 con 0x7f0c780b9eb0 2022-07-12T08:48:24.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7ea89700 1 -- 172.21.15.61:0/1064845223 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c780b42b0 con 0x7f0c780b2630 2022-07-12T08:48:24.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b2630 0x7f0c781420b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0c74001980 tx=0x7f0c700091d0).ready entity=mon.1 client_cookie=d9f51316699f7810 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:24.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7d286700 1 -- 172.21.15.61:0/1064845223 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c740017e0 con 0x7f0c780b2630 2022-07-12T08:48:24.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7d286700 1 -- 172.21.15.61:0/1064845223 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7000f510 con 0x7f0c780b2630 2022-07-12T08:48:24.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c7d286700 1 -- 172.21.15.61:0/1064845223 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c7000f680 con 0x7f0c780b2630 2022-07-12T08:48:24.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b2630 msgr2=0x7f0c781420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:24.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 --2- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b2630 0x7f0c781420b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0c74001980 tx=0x7f0c700091d0).stop 2022-07-12T08:48:24.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- 172.21.15.61:0/1064845223 shutdown_connections 2022-07-12T08:48:24.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 --2- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b2630 0x7f0c781420b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 --2- 172.21.15.61:0/1064845223 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b3180 0x7f0c780b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- 172.21.15.61:0/1064845223 >> 172.21.15.61:0/1064845223 conn(0x7f0c780678c0 msgr2=0x7f0c780b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:24.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- 172.21.15.61:0/1064845223 shutdown_connections 2022-07-12T08:48:24.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.459+0000 7f0c804ec700 1 -- 172.21.15.61:0/1064845223 wait complete. 2022-07-12T08:48:24.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 Processor -- start 2022-07-12T08:48:24.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 -- start start 2022-07-12T08:48:24.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b2630 0x7f0c78147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:24.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 0x7f0c780b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:24.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b9eb0 0x7f0c7813dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:24.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c78156360 con 0x7f0c780b3180 2022-07-12T08:48:24.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780af320 con 0x7f0c780b9eb0 2022-07-12T08:48:24.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.463+0000 7f0c804ec700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c780af5c0 con 0x7f0c780b2630 2022-07-12T08:48:24.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7da87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 0x7f0c780b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7e288700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b2630 0x7f0c78147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7da87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 0x7f0c780b6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58366/0 (socket says 172.21.15.61:58366) 2022-07-12T08:48:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7da87700 1 -- 172.21.15.61:0/2125767907 learned_addr learned my addr 172.21.15.61:0/2125767907 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:24.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b9eb0 0x7f0c7813dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:24.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 -- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b2630 msgr2=0x7f0c78147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:24.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b2630 0x7f0c78147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 -- 172.21.15.61:0/2125767907 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 msgr2=0x7f0c780b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:24.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 0x7f0c780b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:24.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 -- 172.21.15.61:0/2125767907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70009470 con 0x7f0c780b9eb0 2022-07-12T08:48:24.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7da87700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 0x7f0c780b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:24.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c7ea89700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b9eb0 0x7f0c7813dc60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0c74001980 tx=0x7f0c70016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:24.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c7000fe80 con 0x7f0c780b9eb0 2022-07-12T08:48:24.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c700032d0 con 0x7f0c780b9eb0 2022-07-12T08:48:24.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c780af860 con 0x7f0c780b9eb0 2022-07-12T08:48:24.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c7000f680 con 0x7f0c780b9eb0 2022-07-12T08:48:24.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c78155410 con 0x7f0c780b9eb0 2022-07-12T08:48:24.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.467+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c78057460 con 0x7f0c780b9eb0 2022-07-12T08:48:24.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.471+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0c70026030 con 0x7f0c780b9eb0 2022-07-12T08:48:24.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.471+0000 7f0c677fe700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c58053a40 0x7f0c58055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:24.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.471+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0c7006c580 con 0x7f0c780b9eb0 2022-07-12T08:48:24.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.471+0000 7f0c7e288700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c58053a40 0x7f0c58055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:24.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.471+0000 7f0c7e288700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c58053a40 0x7f0c58055f00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f0c7400d000 tx=0x7f0c74002ae0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:24.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.479+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c70043320 con 0x7f0c780b9eb0 2022-07-12T08:48:24.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:24.611+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0c78066150 con 0x7f0c780b9eb0 2022-07-12T08:48:25.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.330+0000 7f0c677fe700 1 -- 172.21.15.61:0/2125767907 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 185+0+0 (secure 0 0 0) 0x7f0c700150e0 con 0x7f0c780b9eb0 2022-07-12T08:48:25.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c58053a40 msgr2=0x7f0c58055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:25.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c58053a40 0x7f0c58055f00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f0c7400d000 tx=0x7f0c74002ae0).stop 2022-07-12T08:48:25.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b9eb0 msgr2=0x7f0c7813dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:25.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b9eb0 0x7f0c7813dc60 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f0c74001980 tx=0x7f0c70016000).stop 2022-07-12T08:48:25.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 shutdown_connections 2022-07-12T08:48:25.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c58053a40 0x7f0c58055f00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c780b9eb0 0x7f0c7813dc60 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c780b3180 0x7f0c780b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 --2- 172.21.15.61:0/2125767907 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c780b2630 0x7f0c78147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 >> 172.21.15.61:0/2125767907 conn(0x7f0c780678c0 msgr2=0x7f0c780afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:25.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 shutdown_connections 2022-07-12T08:48:25.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.334+0000 7f0c804ec700 1 -- 172.21.15.61:0/2125767907 wait complete. 2022-07-12T08:48:25.341 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:48:25.352 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:48:25.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.462+0000 7f63fffff700 1 Processor -- start 2022-07-12T08:48:25.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.462+0000 7f63fffff700 1 -- start start 2022-07-12T08:48:25.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f64000b0120 0x7f64000b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:25.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64000aeb30 0x7f640013ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:25.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64000afbd0 con 0x7f64000b1180 2022-07-12T08:48:25.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64000aefa0 con 0x7f64000b0120 2022-07-12T08:48:25.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64000acb60 con 0x7f64000aeb30 2022-07-12T08:48:25.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f64000b0120 0x7f64000b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:25.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f64000b0120 0x7f64000b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58378/0 (socket says 172.21.15.61:58378) 2022-07-12T08:48:25.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 -- 172.21.15.61:0/2442956226 learned_addr learned my addr 172.21.15.61:0/2442956226 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:25.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63ff7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64000aeb30 0x7f640013ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:25.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 -- 172.21.15.61:0/2442956226 >> v2:172.21.15.111:3300/0 conn(0x7f64000aeb30 msgr2=0x7f640013ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:25.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 --2- 172.21.15.61:0/2442956226 >> v2:172.21.15.111:3300/0 conn(0x7f64000aeb30 0x7f640013ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fe7fc700 1 -- 172.21.15.61:0/2442956226 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18783900 0 0) 0x7f64000afbd0 con 0x7f64000b1180 2022-07-12T08:48:25.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 -- 172.21.15.61:0/2442956226 >> v1:172.21.15.111:6789/0 conn(0x7f64000b1180 legacy=0x7f64000ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:25.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 -- 172.21.15.61:0/2442956226 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64000ad340 con 0x7f64000b0120 2022-07-12T08:48:25.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63f7fff700 1 --2- 172.21.15.61:0/2442956226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b0120 0x7f64000b1560 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f63e8002f70 tx=0x7f63e8006f40).ready entity=mon.0 client_cookie=3fc0cb4c15c3ae4f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:25.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fe7fc700 1 -- 172.21.15.61:0/2442956226 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e800da80 con 0x7f64000b0120 2022-07-12T08:48:25.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fe7fc700 1 -- 172.21.15.61:0/2442956226 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8000e90 con 0x7f64000b0120 2022-07-12T08:48:25.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fe7fc700 1 -- 172.21.15.61:0/2442956226 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e800ddd0 con 0x7f64000b0120 2022-07-12T08:48:25.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 -- 172.21.15.61:0/2442956226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b0120 msgr2=0x7f64000b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:25.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 --2- 172.21.15.61:0/2442956226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b0120 0x7f64000b1560 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f63e8002f70 tx=0x7f63e8006f40).stop 2022-07-12T08:48:25.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 -- 172.21.15.61:0/2442956226 shutdown_connections 2022-07-12T08:48:25.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 --2- 172.21.15.61:0/2442956226 >> v2:172.21.15.111:3300/0 conn(0x7f64000aeb30 0x7f640013ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 --2- 172.21.15.61:0/2442956226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b0120 0x7f64000b1560 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.466+0000 7f63fffff700 1 -- 172.21.15.61:0/2442956226 >> 172.21.15.61:0/2442956226 conn(0x7f64000b5780 msgr2=0x7f64000b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:25.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.470+0000 7f63fffff700 1 -- 172.21.15.61:0/2442956226 shutdown_connections 2022-07-12T08:48:25.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.470+0000 7f63fffff700 1 -- 172.21.15.61:0/2442956226 wait complete. 2022-07-12T08:48:25.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.470+0000 7f63fffff700 1 Processor -- start 2022-07-12T08:48:25.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.470+0000 7f63fffff700 1 -- start start 2022-07-12T08:48:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64000aeb30 0x7f6400064510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000b0120 0x7f64000659e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b1180 0x7f6400065ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400146a40 con 0x7f64000b1180 2022-07-12T08:48:25.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64000669f0 con 0x7f64000b0120 2022-07-12T08:48:25.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6400066be0 con 0x7f64000aeb30 2022-07-12T08:48:25.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b1180 0x7f6400065ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:25.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b1180 0x7f6400065ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58384/0 (socket says 172.21.15.61:58384) 2022-07-12T08:48:25.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000b0120 0x7f64000659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:25.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63ff7fe700 1 -- 172.21.15.61:0/4113522683 learned_addr learned my addr 172.21.15.61:0/4113522683 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:25.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63feffd700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64000aeb30 0x7f6400064510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 -- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64000aeb30 msgr2=0x7f6400064510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64000aeb30 0x7f6400064510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 -- 172.21.15.61:0/4113522683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b1180 msgr2=0x7f6400065ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b1180 0x7f6400065ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 -- 172.21.15.61:0/4113522683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e800b460 con 0x7f64000b0120 2022-07-12T08:48:25.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63f7fff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000b0120 0x7f64000659e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f63e8002f70 tx=0x7f63e8006f40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:25.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e801d070 con 0x7f64000b0120 2022-07-12T08:48:25.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e80185f0 con 0x7f64000b0120 2022-07-12T08:48:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6400066e80 con 0x7f64000b0120 2022-07-12T08:48:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63e8018950 con 0x7f64000b0120 2022-07-12T08:48:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.474+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6400144da0 con 0x7f64000b0120 2022-07-12T08:48:25.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.478+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6400067010 con 0x7f64000b0120 2022-07-12T08:48:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.478+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f63e80068b0 con 0x7f64000b0120 2022-07-12T08:48:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.478+0000 7f63fcff9700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63e4053a40 0x7f63e4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.478+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f63e806c9c0 con 0x7f64000b0120 2022-07-12T08:48:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.478+0000 7f63feffd700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63e4053a40 0x7f63e4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:25.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.478+0000 7f63feffd700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63e4053a40 0x7f63e4055f00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f63f000d000 tx=0x7f63f0007b30).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:25.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.486+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63e8043a80 con 0x7f64000b0120 2022-07-12T08:48:25.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:25.614+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6400057460 con 0x7f64000b0120 2022-07-12T08:48:26.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.338+0000 7f63fcff9700 1 -- 172.21.15.61:0/4113522683 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 173+0+0 (secure 0 0 0) 0x7f63e80470a0 con 0x7f64000b0120 2022-07-12T08:48:26.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63e4053a40 msgr2=0x7f63e4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63e4053a40 0x7f63e4055f00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f63f000d000 tx=0x7f63f0007b30).stop 2022-07-12T08:48:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000b0120 msgr2=0x7f64000659e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000b0120 0x7f64000659e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f63e8002f70 tx=0x7f63e8006f40).stop 2022-07-12T08:48:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 shutdown_connections 2022-07-12T08:48:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64000b1180 0x7f6400065ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f63e4053a40 0x7f63e4055f00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64000b0120 0x7f64000659e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 --2- 172.21.15.61:0/4113522683 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64000aeb30 0x7f6400064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 >> 172.21.15.61:0/4113522683 conn(0x7f64000b5780 msgr2=0x7f64000b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 shutdown_connections 2022-07-12T08:48:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.342+0000 7f63fffff700 1 -- 172.21.15.61:0/4113522683 wait complete. 2022-07-12T08:48:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:48:26.362 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:48:26.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 Processor -- start 2022-07-12T08:48:26.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 -- start start 2022-07-12T08:48:26.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 0x7f85a405d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 0x7f85a4146a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a414a560 con 0x7f85a4060e80 2022-07-12T08:48:26.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a4097d40 con 0x7f85a41465a0 2022-07-12T08:48:26.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a8cfe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a414b340 con 0x7f85a405cd20 2022-07-12T08:48:26.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 0x7f85a405d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 0x7f85a4146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 0x7f85a405d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40584/0 (socket says 172.21.15.61:40584) 2022-07-12T08:48:26.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 0x7f85a4146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:36966/0 (socket says 172.21.15.61:36966) 2022-07-12T08:48:26.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 -- 172.21.15.61:0/1277494059 learned_addr learned my addr 172.21.15.61:0/1277494059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:26.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 -- 172.21.15.61:0/1277494059 >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 msgr2=0x7f85a4146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:26.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 --2- 172.21.15.61:0/1277494059 >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 0x7f85a4146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 -- 172.21.15.61:0/1277494059 >> v1:172.21.15.111:6789/0 conn(0x7f85a4060e80 legacy=0x7f85a414ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 -- 172.21.15.61:0/1277494059 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a40aafa0 con 0x7f85a405cd20 2022-07-12T08:48:26.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a3fff700 1 --2- 172.21.15.61:0/1277494059 >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 0x7f85a4146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:26.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a37fe700 1 --2- 172.21.15.61:0/1277494059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 0x7f85a405d100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f858c002f70 tx=0x7f858c006eb0).ready entity=mon.2 client_cookie=d82b316896c82fee server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:26.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a27fc700 1 -- 172.21.15.61:0/1277494059 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f858c00d570 con 0x7f85a405cd20 2022-07-12T08:48:26.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a27fc700 1 -- 172.21.15.61:0/1277494059 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858c000e90 con 0x7f85a405cd20 2022-07-12T08:48:26.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.466+0000 7f85a27fc700 1 -- 172.21.15.61:0/1277494059 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f858c00d900 con 0x7f85a405cd20 2022-07-12T08:48:26.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1277494059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 msgr2=0x7f85a405d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1277494059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 0x7f85a405d100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f858c002f70 tx=0x7f858c006eb0).stop 2022-07-12T08:48:26.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1277494059 shutdown_connections 2022-07-12T08:48:26.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1277494059 >> v2:172.21.15.111:3300/0 conn(0x7f85a41465a0 0x7f85a4146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1277494059 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a405cd20 0x7f85a405d100 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1277494059 >> 172.21.15.61:0/1277494059 conn(0x7f85a40b1a50 msgr2=0x7f85a40b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:26.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1277494059 shutdown_connections 2022-07-12T08:48:26.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1277494059 wait complete. 2022-07-12T08:48:26.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 Processor -- start 2022-07-12T08:48:26.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- start start 2022-07-12T08:48:26.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a405cd20 0x7f85a41418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 0x7f85a4155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a41465a0 0x7f85a4156470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4159510 con 0x7f85a4060e80 2022-07-12T08:48:26.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a4159740 con 0x7f85a405cd20 2022-07-12T08:48:26.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a8cfe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a41599e0 con 0x7f85a41465a0 2022-07-12T08:48:26.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 0x7f85a4155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 0x7f85a4155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58396/0 (socket says 172.21.15.61:58396) 2022-07-12T08:48:26.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 -- 172.21.15.61:0/1115196535 learned_addr learned my addr 172.21.15.61:0/1115196535 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:26.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a37fe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a405cd20 0x7f85a41418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a3fff700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a41465a0 0x7f85a4156470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 -- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a41465a0 msgr2=0x7f85a4156470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a41465a0 0x7f85a4156470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 -- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a405cd20 msgr2=0x7f85a41418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a405cd20 0x7f85a41418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a2ffd700 1 -- 172.21.15.61:0/1115196535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c00b180 con 0x7f85a4060e80 2022-07-12T08:48:26.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.470+0000 7f85a37fe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a405cd20 0x7f85a41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:26.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a2ffd700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 0x7f85a4155f80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f85a4062600 tx=0x7f85940014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:26.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f859400e4a0 con 0x7f85a4060e80 2022-07-12T08:48:26.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859400e600 con 0x7f85a4060e80 2022-07-12T08:48:26.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c00b1e0 con 0x7f85a4060e80 2022-07-12T08:48:26.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f859400e960 con 0x7f85a4060e80 2022-07-12T08:48:26.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a415a120 con 0x7f85a4060e80 2022-07-12T08:48:26.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a4057460 con 0x7f85a4060e80 2022-07-12T08:48:26.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.474+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8594019030 con 0x7f85a4060e80 2022-07-12T08:48:26.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.478+0000 7f85a0ff9700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8588053af0 0x7f8588055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.478+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f859406b300 con 0x7f85a4060e80 2022-07-12T08:48:26.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.478+0000 7f85a37fe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8588053af0 0x7f8588055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.478+0000 7f85a37fe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8588053af0 0x7f8588055fb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f858c01c000 tx=0x7f858c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:26.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.482+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8594042430 con 0x7f85a4060e80 2022-07-12T08:48:26.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.614+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f85a406b2b0 con 0x7f85a4060e80 2022-07-12T08:48:26.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.618+0000 7f85a0ff9700 1 -- 172.21.15.61:0/1115196535 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8594045a50 con 0x7f85a4060e80 2022-07-12T08:48:26.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8588053af0 msgr2=0x7f8588055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8588053af0 0x7f8588055fb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f858c01c000 tx=0x7f858c015000).stop 2022-07-12T08:48:26.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 msgr2=0x7f85a4155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 0x7f85a4155f80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 rx=0x7f85a4062600 tx=0x7f85940014a0).stop 2022-07-12T08:48:26.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 shutdown_connections 2022-07-12T08:48:26.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85a41465a0 0x7f85a4156470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a4060e80 0x7f85a4155f80 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8588053af0 0x7f8588055fb0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 --2- 172.21.15.61:0/1115196535 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85a405cd20 0x7f85a41418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 >> 172.21.15.61:0/1115196535 conn(0x7f85a40b1a50 msgr2=0x7f85a4146a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:26.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 shutdown_connections 2022-07-12T08:48:26.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.622+0000 7f85a8cfe700 1 -- 172.21.15.61:0/1115196535 wait complete. 2022-07-12T08:48:26.640 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-12T08:48:26.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.746+0000 7f5d79d26700 1 Processor -- start 2022-07-12T08:48:26.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.746+0000 7f5d79d26700 1 -- start start 2022-07-12T08:48:26.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5d740ab670 0x7f5d7405f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740afa60 0x7f5d740afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d740abc70 con 0x7f5d740ab290 2022-07-12T08:48:26.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d740aba90 con 0x7f5d740ab670 2022-07-12T08:48:26.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7405c720 con 0x7f5d740afa60 2022-07-12T08:48:26.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5d740ab670 0x7f5d7405f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5d740ab670 0x7f5d7405f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58402/0 (socket says 172.21.15.61:58402) 2022-07-12T08:48:26.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 -- 172.21.15.61:0/2791766189 learned_addr learned my addr 172.21.15.61:0/2791766189 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:26.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79525700 1 --2- 172.21.15.61:0/2791766189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740afa60 0x7f5d740afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 -- 172.21.15.61:0/2791766189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740afa60 msgr2=0x7f5d740afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:26.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 --2- 172.21.15.61:0/2791766189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740afa60 0x7f5d740afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 -- 172.21.15.61:0/2791766189 >> v1:172.21.15.111:6789/0 conn(0x7f5d740ab290 legacy=0x7f5d7405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 -- 172.21.15.61:0/2791766189 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7405c9c0 con 0x7f5d740ab670 2022-07-12T08:48:26.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d73fff700 1 --2- 172.21.15.61:0/2791766189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740ab670 0x7f5d7405f140 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f5d64002f70 tx=0x7f5d64006eb0).ready entity=mon.0 client_cookie=7ce88e161bf05e60 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:26.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d737fe700 1 -- 172.21.15.61:0/2791766189 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6400d5c0 con 0x7f5d740ab670 2022-07-12T08:48:26.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d737fe700 1 -- 172.21.15.61:0/2791766189 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d64000e90 con 0x7f5d740ab670 2022-07-12T08:48:26.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d737fe700 1 -- 172.21.15.61:0/2791766189 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6400d950 con 0x7f5d740ab670 2022-07-12T08:48:26.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- 172.21.15.61:0/2791766189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740ab670 msgr2=0x7f5d7405f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 --2- 172.21.15.61:0/2791766189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740ab670 0x7f5d7405f140 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f5d64002f70 tx=0x7f5d64006eb0).stop 2022-07-12T08:48:26.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- 172.21.15.61:0/2791766189 shutdown_connections 2022-07-12T08:48:26.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 --2- 172.21.15.61:0/2791766189 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740afa60 0x7f5d740afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 --2- 172.21.15.61:0/2791766189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740ab670 0x7f5d7405f140 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- 172.21.15.61:0/2791766189 >> 172.21.15.61:0/2791766189 conn(0x7f5d740b1950 msgr2=0x7f5d740b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:26.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- 172.21.15.61:0/2791766189 shutdown_connections 2022-07-12T08:48:26.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.750+0000 7f5d79d26700 1 -- 172.21.15.61:0/2791766189 wait complete. 2022-07-12T08:48:26.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 Processor -- start 2022-07-12T08:48:26.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 -- start start 2022-07-12T08:48:26.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740ab290 0x7f5d74099db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d740ab670 0x7f5d74067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 0x7f5d741594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740ae250 con 0x7f5d740afa60 2022-07-12T08:48:26.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740aec50 con 0x7f5d740ab670 2022-07-12T08:48:26.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d740aeef0 con 0x7f5d740ab290 2022-07-12T08:48:26.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 0x7f5d741594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 0x7f5d741594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58412/0 (socket says 172.21.15.61:58412) 2022-07-12T08:48:26.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 -- 172.21.15.61:0/1000145556 learned_addr learned my addr 172.21.15.61:0/1000145556 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:26.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d73fff700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d740ab670 0x7f5d74067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d78d24700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740ab290 0x7f5d74099db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 -- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740ab290 msgr2=0x7f5d74099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740ab290 0x7f5d74099db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 -- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d740ab670 msgr2=0x7f5d74067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:26.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d740ab670 0x7f5d74067130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:26.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 -- 172.21.15.61:0/1000145556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d6400b1e0 con 0x7f5d740afa60 2022-07-12T08:48:26.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79525700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 0x7f5d741594a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f5d6c00a000 tx=0x7f5d6c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:26.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6c0127e0 con 0x7f5d740afa60 2022-07-12T08:48:26.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6c012940 con 0x7f5d740afa60 2022-07-12T08:48:26.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d6c012ca0 con 0x7f5d740afa60 2022-07-12T08:48:26.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d740af190 con 0x7f5d740afa60 2022-07-12T08:48:26.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.754+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d740ae490 con 0x7f5d740afa60 2022-07-12T08:48:26.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.758+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d74057460 con 0x7f5d740afa60 2022-07-12T08:48:26.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.758+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5d6c013480 con 0x7f5d740afa60 2022-07-12T08:48:26.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.758+0000 7f5d71ffb700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d54053ad0 0x7f5d54055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:26.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.758+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5d6c06bfc0 con 0x7f5d740afa60 2022-07-12T08:48:26.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.758+0000 7f5d78d24700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d54053ad0 0x7f5d54055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:26.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.758+0000 7f5d78d24700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d54053ad0 0x7f5d54055f90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f5d6800d000 tx=0x7f5d68007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:26.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.766+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d6c043090 con 0x7f5d740afa60 2022-07-12T08:48:26.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:26.938+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7f5d74068350 con 0x7f5d740afa60 2022-07-12T08:48:27.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.342+0000 7f5d71ffb700 1 -- 172.21.15.61:0/1000145556 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v323) v1 ==== 126+0+0 (secure 0 0 0) 0x7f5d6c012e00 con 0x7f5d740afa60 2022-07-12T08:48:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d54053ad0 msgr2=0x7f5d54055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d54053ad0 0x7f5d54055f90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f5d6800d000 tx=0x7f5d68007b90).stop 2022-07-12T08:48:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 msgr2=0x7f5d741594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 0x7f5d741594a0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 rx=0x7f5d6c00a000 tx=0x7f5d6c008040).stop 2022-07-12T08:48:27.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 shutdown_connections 2022-07-12T08:48:27.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d740afa60 0x7f5d741594a0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5d54053ad0 0x7f5d54055f90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5d740ab670 0x7f5d74067130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 --2- 172.21.15.61:0/1000145556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5d740ab290 0x7f5d74099db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 >> 172.21.15.61:0/1000145556 conn(0x7f5d740b1950 msgr2=0x7f5d740b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:27.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 shutdown_connections 2022-07-12T08:48:27.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.346+0000 7f5d79d26700 1 -- 172.21.15.61:0/1000145556 wait complete. 2022-07-12T08:48:27.365 INFO:tasks.cephfs_test_runner:test_dump_inode (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:48:27.366 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:48:27.366 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-12T08:48:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.474+0000 7fa41bfff700 1 Processor -- start 2022-07-12T08:48:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.474+0000 7fa41bfff700 1 -- start start 2022-07-12T08:48:27.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa41c05f140 0x7fa41c05ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 0x7fa41c0afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c05f580 con 0x7fa41c0afa60 2022-07-12T08:48:27.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa41c0abc70 con 0x7fa41c05f140 2022-07-12T08:48:27.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c05c720 con 0x7fa41c0ab290 2022-07-12T08:48:27.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 0x7fa41c0afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 0x7fa41c0afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58418/0 (socket says 172.21.15.61:58418) 2022-07-12T08:48:27.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41b7fe700 1 -- 172.21.15.61:0/25578451 learned_addr learned my addr 172.21.15.61:0/25578451 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:27.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41a7fc700 1 --2- 172.21.15.61:0/25578451 >> v2:172.21.15.111:3300/0 conn(0x7fa41c05f140 0x7fa41c05ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 --2- 172.21.15.61:0/25578451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 -- 172.21.15.61:0/25578451 >> v2:172.21.15.111:3300/0 conn(0x7fa41c05f140 msgr2=0x7fa41c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:27.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 --2- 172.21.15.61:0/25578451 >> v2:172.21.15.111:3300/0 conn(0x7fa41c05f140 0x7fa41c05ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 -- 172.21.15.61:0/25578451 >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 msgr2=0x7fa41c0afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:27.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 --2- 172.21.15.61:0/25578451 >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 0x7fa41c0afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 -- 172.21.15.61:0/25578451 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c05c9c0 con 0x7fa41c0ab290 2022-07-12T08:48:27.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41b7fe700 1 --2- 172.21.15.61:0/25578451 >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 0x7fa41c0afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:27.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41affd700 1 --2- 172.21.15.61:0/25578451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0ab670 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fa40c0032f0 tx=0x7fa40c0091d0).ready entity=mon.2 client_cookie=885dc1df6554511 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:27.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa419ffb700 1 -- 172.21.15.61:0/25578451 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa40c00b600 con 0x7fa41c0ab290 2022-07-12T08:48:27.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa419ffb700 1 -- 172.21.15.61:0/25578451 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa40c00b760 con 0x7fa41c0ab290 2022-07-12T08:48:27.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa419ffb700 1 -- 172.21.15.61:0/25578451 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa40c00b8d0 con 0x7fa41c0ab290 2022-07-12T08:48:27.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- 172.21.15.61:0/25578451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 msgr2=0x7fa41c0ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:27.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- 172.21.15.61:0/25578451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0ab670 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fa40c0032f0 tx=0x7fa40c0091d0).stop 2022-07-12T08:48:27.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- 172.21.15.61:0/25578451 shutdown_connections 2022-07-12T08:48:27.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- 172.21.15.61:0/25578451 >> v2:172.21.15.61:3300/0 conn(0x7fa41c0afa60 0x7fa41c0afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- 172.21.15.61:0/25578451 >> v2:172.21.15.111:3300/0 conn(0x7fa41c05f140 0x7fa41c05ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 --2- 172.21.15.61:0/25578451 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0ab670 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- 172.21.15.61:0/25578451 >> 172.21.15.61:0/25578451 conn(0x7fa41c0b1950 msgr2=0x7fa41c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:27.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- 172.21.15.61:0/25578451 shutdown_connections 2022-07-12T08:48:27.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.478+0000 7fa41bfff700 1 -- 172.21.15.61:0/25578451 wait complete. 2022-07-12T08:48:27.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 Processor -- start 2022-07-12T08:48:27.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 -- start start 2022-07-12T08:48:27.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c05f140 0x7fa41c156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0671e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 0x7fa41c064300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c1489c0 con 0x7fa41c0afa60 2022-07-12T08:48:27.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c063b30 con 0x7fa41c05f140 2022-07-12T08:48:27.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.482+0000 7fa41bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa41c148720 con 0x7fa41c0ab290 2022-07-12T08:48:27.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 0x7fa41c064300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 0x7fa41c064300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58426/0 (socket says 172.21.15.61:58426) 2022-07-12T08:48:27.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 -- 172.21.15.61:0/4098809647 learned_addr learned my addr 172.21.15.61:0/4098809647 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:27.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41affd700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c05f140 0x7fa41c156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41a7fc700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 -- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 msgr2=0x7fa41c0671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:27.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 -- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c05f140 msgr2=0x7fa41c156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:27.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c05f140 0x7fa41c156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:27.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 -- 172.21.15.61:0/4098809647 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41400b1e0 con 0x7fa41c0afa60 2022-07-12T08:48:27.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41b7fe700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 0x7fa41c064300 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7fa41400c000 tx=0x7fa414006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:27.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa414012600 con 0x7fa41c0afa60 2022-07-12T08:48:27.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa414012760 con 0x7fa41c0afa60 2022-07-12T08:48:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40c009470 con 0x7fa41c0afa60 2022-07-12T08:48:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa414012ac0 con 0x7fa41c0afa60 2022-07-12T08:48:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c0aefb0 con 0x7fa41c0afa60 2022-07-12T08:48:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c057460 con 0x7fa41c0afa60 2022-07-12T08:48:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa414022070 con 0x7fa41c0afa60 2022-07-12T08:48:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.486+0000 7fa413fff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa400053920 0x7fa400055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.490+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa41406bf70 con 0x7fa41c0afa60 2022-07-12T08:48:27.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.490+0000 7fa41affd700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa400053920 0x7fa400055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:27.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.490+0000 7fa41affd700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa400053920 0x7fa400055de0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa40c01b000 tx=0x7fa40c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:27.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.494+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa414043260 con 0x7fa41c0afa60 2022-07-12T08:48:27.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:27.666+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7fa41c06b2b0 con 0x7fa41c0afa60 2022-07-12T08:48:28.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.342+0000 7fa413fff700 1 -- 172.21.15.61:0/4098809647 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v324) v1 ==== 141+0+0 (secure 0 0 0) 0x7fa414016020 con 0x7fa41c0afa60 2022-07-12T08:48:28.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa400053920 msgr2=0x7fa400055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa400053920 0x7fa400055de0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa40c01b000 tx=0x7fa40c016000).stop 2022-07-12T08:48:28.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 msgr2=0x7fa41c064300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 0x7fa41c064300 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7fa41400c000 tx=0x7fa414006eb0).stop 2022-07-12T08:48:28.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 shutdown_connections 2022-07-12T08:48:28.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa41c0afa60 0x7fa41c064300 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa41c0ab290 0x7fa41c0671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa400053920 0x7fa400055de0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 --2- 172.21.15.61:0/4098809647 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa41c05f140 0x7fa41c156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 >> 172.21.15.61:0/4098809647 conn(0x7fa41c0b1950 msgr2=0x7fa41c098580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:28.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 shutdown_connections 2022-07-12T08:48:28.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.346+0000 7fa41bfff700 1 -- 172.21.15.61:0/4098809647 wait complete. 2022-07-12T08:48:28.370 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:48:28.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 Processor -- start 2022-07-12T08:48:28.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 -- start start 2022-07-12T08:48:28.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff52814a650 0x7ff528146460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5280b5e80 con 0x7ff528146950 2022-07-12T08:48:28.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52805d2d0 con 0x7ff5280b5aa0 2022-07-12T08:48:28.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff52805c540 con 0x7ff52814a650 2022-07-12T08:48:28.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52d5d9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff52814a650 0x7ff528146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52e5db700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff528146950 0x7ff5280ac9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46652/0 (socket says 172.21.15.61:46652) 2022-07-12T08:48:28.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52e5db700 1 -- 172.21.15.61:0/2850615810 learned_addr learned my addr 172.21.15.61:0/2850615810 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:28.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2850615810 >> v1:172.21.15.111:6789/0 conn(0x7ff5280b5aa0 legacy=0x7ff52805be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2850615810 >> v1:172.21.15.61:6789/0 conn(0x7ff528146950 legacy=0x7ff5280ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2850615810 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5280b05a0 con 0x7ff52814a650 2022-07-12T08:48:28.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52d5d9700 1 --2- 172.21.15.61:0/2850615810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff52814a650 0x7ff528146460 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524006df0).ready entity=mon.0 client_cookie=3573d34c389bf3be server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:28.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52cdd8700 1 -- 172.21.15.61:0/2850615810 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff52400d610 con 0x7ff52814a650 2022-07-12T08:48:28.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52cdd8700 1 -- 172.21.15.61:0/2850615810 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff524000e90 con 0x7ff52814a650 2022-07-12T08:48:28.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff52cdd8700 1 -- 172.21.15.61:0/2850615810 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff52400d960 con 0x7ff52814a650 2022-07-12T08:48:28.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 -- 172.21.15.61:0/2850615810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff52814a650 msgr2=0x7ff528146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.482+0000 7ff53003e700 1 --2- 172.21.15.61:0/2850615810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff52814a650 0x7ff528146460 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524006df0).stop 2022-07-12T08:48:28.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- 172.21.15.61:0/2850615810 shutdown_connections 2022-07-12T08:48:28.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 --2- 172.21.15.61:0/2850615810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff52814a650 0x7ff528146460 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- 172.21.15.61:0/2850615810 >> 172.21.15.61:0/2850615810 conn(0x7ff5280634d0 msgr2=0x7ff5280b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:28.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- 172.21.15.61:0/2850615810 shutdown_connections 2022-07-12T08:48:28.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- 172.21.15.61:0/2850615810 wait complete. 2022-07-12T08:48:28.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 Processor -- start 2022-07-12T08:48:28.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- start start 2022-07-12T08:48:28.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5280b5aa0 0x7ff528149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 0x7ff528064f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff52814a650 0x7ff528065420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5280b3550 con 0x7ff528146950 2022-07-12T08:48:28.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52809e3b0 con 0x7ff5280b5aa0 2022-07-12T08:48:28.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff53003e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52809e650 con 0x7ff52814a650 2022-07-12T08:48:28.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 0x7ff528064f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 0x7ff528064f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58438/0 (socket says 172.21.15.61:58438) 2022-07-12T08:48:28.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2175942967 learned_addr learned my addr 172.21.15.61:0/2175942967 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:28.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52e5db700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff52814a650 0x7ff528065420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52ddda700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5280b5aa0 0x7ff528149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff52814a650 msgr2=0x7ff528065420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff52814a650 0x7ff528065420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5280b5aa0 msgr2=0x7ff528149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5280b5aa0 0x7ff528149a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.486+0000 7ff52d5d9700 1 -- 172.21.15.61:0/2175942967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52400b180 con 0x7ff528146950 2022-07-12T08:48:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff52d5d9700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 0x7ff528064f30 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff52400de70 con 0x7ff528146950 2022-07-12T08:48:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff524011b40 con 0x7ff528146950 2022-07-12T08:48:28.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff524011ea0 con 0x7ff528146950 2022-07-12T08:48:28.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52809e8f0 con 0x7ff528146950 2022-07-12T08:48:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5280649f0 con 0x7ff528146950 2022-07-12T08:48:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff528057460 con 0x7ff528146950 2022-07-12T08:48:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff52401a040 con 0x7ff528146950 2022-07-12T08:48:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff51effd700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff508053af0 0x7ff508055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff52406ca90 con 0x7ff528146950 2022-07-12T08:48:28.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.490+0000 7ff52ddda700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff508053af0 0x7ff508055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.494+0000 7ff52ddda700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff508053af0 0x7ff508055fb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7ff51800d000 tx=0x7ff518007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:28.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.498+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff524043cd0 con 0x7ff528146950 2022-07-12T08:48:28.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.622+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff5280ac020 con 0x7ff528146950 2022-07-12T08:48:28.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.630+0000 7ff51effd700 1 -- 172.21.15.61:0/2175942967 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff524023020 con 0x7ff528146950 2022-07-12T08:48:28.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff508053af0 msgr2=0x7ff508055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff508053af0 0x7ff508055fb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7ff51800d000 tx=0x7ff518007b40).stop 2022-07-12T08:48:28.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 msgr2=0x7ff528064f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 0x7ff528064f30 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 rx=0x7ff524002f70 tx=0x7ff524015000).stop 2022-07-12T08:48:28.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 shutdown_connections 2022-07-12T08:48:28.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff52814a650 0x7ff528065420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff528146950 0x7ff528064f30 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff508053af0 0x7ff508055fb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 --2- 172.21.15.61:0/2175942967 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff5280b5aa0 0x7ff528149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 >> 172.21.15.61:0/2175942967 conn(0x7ff5280634d0 msgr2=0x7ff5280aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:28.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 shutdown_connections 2022-07-12T08:48:28.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.634+0000 7ff53003e700 1 -- 172.21.15.61:0/2175942967 wait complete. 2022-07-12T08:48:28.653 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:48:28.654 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:28.675 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:28.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:28.720 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:28.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:28.767 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:48:28.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:28.768 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:28.768 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:28.776 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:48:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:28.779 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:28.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:28.781 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:28.781 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:28.793 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:48:28.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:28.795 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:28.796 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:28.797 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:28.798 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:28.806 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:28.810 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:48:28.811 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:48:28.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 Processor -- start 2022-07-12T08:48:28.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 -- start start 2022-07-12T08:48:28.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8da80b2ac0 0x7f8da80b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8da8060860 0x7f8da8140480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da8140970 con 0x7f8da8065710 2022-07-12T08:48:28.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da813faa0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8dacbbe700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da813fd40 con 0x7f8da8060860 2022-07-12T08:48:28.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8da80b2ac0 0x7f8da80b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8da80b2ac0 0x7f8da80b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58446/0 (socket says 172.21.15.61:58446) 2022-07-12T08:48:28.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 -- 172.21.15.61:0/2181086794 learned_addr learned my addr 172.21.15.61:0/2181086794 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:28.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8d9ffff700 1 --2- 172.21.15.61:0/2181086794 >> v2:172.21.15.111:3300/0 conn(0x7f8da8060860 0x7f8da8140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 -- 172.21.15.61:0/2181086794 >> v2:172.21.15.111:3300/0 conn(0x7f8da8060860 msgr2=0x7f8da8140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:28.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 --2- 172.21.15.61:0/2181086794 >> v2:172.21.15.111:3300/0 conn(0x7f8da8060860 0x7f8da8140480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 -- 172.21.15.61:0/2181086794 >> v1:172.21.15.111:6789/0 conn(0x7f8da8065710 legacy=0x7f8da8139660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 -- 172.21.15.61:0/2181086794 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da813ffe0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8d9ffff700 1 --2- 172.21.15.61:0/2181086794 >> v2:172.21.15.111:3300/0 conn(0x7f8da8060860 0x7f8da8140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da659c700 1 --2- 172.21.15.61:0/2181086794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da80b2ea0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f8da80b3f10 tx=0x7f8d98006f40).ready entity=mon.0 client_cookie=20122a7d90ff24f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da5d9b700 1 -- 172.21.15.61:0/2181086794 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d9800da40 con 0x7f8da80b2ac0 2022-07-12T08:48:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da5d9b700 1 -- 172.21.15.61:0/2181086794 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d98000e90 con 0x7f8da80b2ac0 2022-07-12T08:48:28.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.922+0000 7f8da5d9b700 1 -- 172.21.15.61:0/2181086794 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d9800ddd0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 -- 172.21.15.61:0/2181086794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 msgr2=0x7f8da80b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/2181086794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da80b2ea0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7f8da80b3f10 tx=0x7f8d98006f40).stop 2022-07-12T08:48:28.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 -- 172.21.15.61:0/2181086794 shutdown_connections 2022-07-12T08:48:28.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/2181086794 >> v2:172.21.15.111:3300/0 conn(0x7f8da8060860 0x7f8da8140480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/2181086794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da80b2ea0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 -- 172.21.15.61:0/2181086794 >> 172.21.15.61:0/2181086794 conn(0x7f8da8063af0 msgr2=0x7f8da8063ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:28.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 -- 172.21.15.61:0/2181086794 shutdown_connections 2022-07-12T08:48:28.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 -- 172.21.15.61:0/2181086794 wait complete. 2022-07-12T08:48:28.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 Processor -- start 2022-07-12T08:48:28.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.926+0000 7f8dacbbe700 1 -- start start 2022-07-12T08:48:28.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8da8060860 0x7f8da8141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8da8065710 0x7f8da813b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da813ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da81431f0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da8142830 con 0x7f8da8065710 2022-07-12T08:48:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da80afd60 con 0x7f8da8060860 2022-07-12T08:48:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da813ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da659c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8da8060860 0x7f8da8141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da813ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58454/0 (socket says 172.21.15.61:58454) 2022-07-12T08:48:28.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 -- 172.21.15.61:0/915826204 learned_addr learned my addr 172.21.15.61:0/915826204 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:28.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8d9ffff700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8da8065710 0x7f8da813b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 -- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8da8060860 msgr2=0x7f8da8141bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8da8060860 0x7f8da8141bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 -- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8da8065710 msgr2=0x7f8da813b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:28.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8da8065710 0x7f8da813b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:28.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 -- 172.21.15.61:0/915826204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9000b180 con 0x7f8da80b2ac0 2022-07-12T08:48:28.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da659c700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8da8060860 0x7f8da8141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:28.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8da6d9d700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da813ba20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f8da813f9c0 tx=0x7f8da0007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:28.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da000f470 con 0x7f8da80b2ac0 2022-07-12T08:48:28.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da000f5d0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9800b460 con 0x7f8da80b2ac0 2022-07-12T08:48:28.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8da000f930 con 0x7f8da80b2ac0 2022-07-12T08:48:28.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.930+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da81420e0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.934+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da8057460 con 0x7f8da80b2ac0 2022-07-12T08:48:28.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.934+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8da0019030 con 0x7f8da80b2ac0 2022-07-12T08:48:28.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.934+0000 7f8d9f7fe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8d8c053a40 0x7f8d8c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:28.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.934+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8da006b2d0 con 0x7f8da80b2ac0 2022-07-12T08:48:28.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.934+0000 7f8da659c700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8d8c053a40 0x7f8d8c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:28.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.934+0000 7f8da659c700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8d8c053a40 0x7f8d8c055f00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f8d9801e000 tx=0x7f8da8140170).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:28.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:28.942+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8da0042470 con 0x7f8da80b2ac0 2022-07-12T08:48:29.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.070+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8da8142590 con 0x7f8da80b2ac0 2022-07-12T08:48:29.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8d9f7fe700 1 -- 172.21.15.61:0/915826204 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v84) v1 ==== 81+0+28876 (secure 0 0 0) 0x7f8da0017020 con 0x7f8da80b2ac0 2022-07-12T08:48:29.079 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:29.079 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:29.080 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 84, 2022-07-12T08:48:29.080 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:48:29.080 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:48:29.080 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:48:26.327808+0000", 2022-07-12T08:48:29.081 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:29.081 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:29.081 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:48:29.081 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:48:29.082 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:48:29.082 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:48:29.082 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:48:29.082 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:48:29.083 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:48:29.083 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:29.083 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:48:29.083 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:48:29.083 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:48:29.084 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:48:29.084 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:48:29.084 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 16, 2022-07-12T08:48:29.084 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:48:29.085 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:48:29.085 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:48:29.085 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:48:29.085 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:48:29.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.086 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:48:29.086 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:48:29.086 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:48:29.086 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:29.087 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:29.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:29.088 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:29.088 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:29.089 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:29.089 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:29.089 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:29.090 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:48:29.090 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:48:29.090 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:48:29.090 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:48:29.091 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:48:29.091 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:29.091 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:29.091 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:29.091 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:29.092 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:29.092 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:29.092 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:29.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.093 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:48:29.093 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:29.093 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:29.093 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:29.093 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:29.094 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:29.094 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:29.094 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:29.094 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:29.095 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:29.095 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:29.095 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:29.095 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:29.095 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:29.096 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:29.096 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:29.096 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:29.096 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:29.096 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:29.097 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:29.097 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:29.097 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:29.098 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:29.098 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:29.098 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:29.098 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:29.099 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:29.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.099 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:29.099 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:29.100 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:29.100 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:29.100 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:29.100 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:29.100 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:29.101 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:29.101 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:29.101 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:29.102 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:29.102 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:29.102 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:48:29.103 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.103 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:29.103 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:48:29.104 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.105 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:48:29.106 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:48:29.106 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:48:29.107 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:29.107 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:29.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:29.108 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:29.109 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:29.109 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:29.110 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:29.110 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:29.110 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:29.111 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:29.111 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:29.112 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:29.112 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:29.112 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:29.112 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:29.113 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:29.113 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:29.113 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:29.113 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:29.114 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:29.114 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.114 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:48:29.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:29.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:29.116 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:29.117 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:29.117 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:29.117 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:29.118 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:29.118 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:29.119 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:29.119 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:29.120 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:29.120 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:29.120 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:29.121 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:29.121 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:29.122 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:29.122 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:29.122 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:29.122 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:29.123 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:29.123 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:29.123 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:29.123 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:29.124 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:29.124 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:29.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:29.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.125 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:29.125 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:29.125 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:29.126 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:29.126 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:29.126 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:29.126 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:29.127 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:29.127 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:29.127 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:29.127 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:29.128 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:29.128 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:29.128 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:48:29.129 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.129 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.129 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:29.130 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:48:29.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.130 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:29.131 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:48:29.131 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.132 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.132 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.132 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.134 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.134 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:29.135 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.135 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.135 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.137 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:48:29.138 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.138 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:48:29.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.141 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.142 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.142 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.143 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:48:29.143 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:48:29.144 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.145 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.145 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.146 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.146 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.147 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:48:29.147 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.149 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:48:29.149 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.150 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.150 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.151 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.151 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.152 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.153 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:48:29.153 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.154 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.154 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:48:29.155 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:29.156 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.156 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.156 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.157 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:48:29.157 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:48:29.158 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:48:29.158 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:48:29.159 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.159 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.159 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.160 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.161 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:29.162 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:48:29.162 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.163 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.163 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.164 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.164 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 78, 2022-07-12T08:48:29.166 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.166 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.167 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.167 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.168 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:48:29.168 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.168 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.169 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.169 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:48:29.169 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.170 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.170 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.170 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:48:29.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.172 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.173 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:48:29.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.174 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.174 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.175 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:48:29.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.177 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:48:29.178 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.179 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.179 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.180 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.181 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:48:29.182 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.183 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.184 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:48:29.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:29.185 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.185 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.186 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.186 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:48:29.187 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:48:29.187 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:48:29.188 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:48:29.189 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.191 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.192 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.192 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.193 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.193 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.193 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:29.194 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:48:29.194 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.194 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.194 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.195 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.195 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.195 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.195 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.196 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:29.196 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.196 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.196 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.197 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.197 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:48:29.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.198 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.199 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.199 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.200 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:48:29.201 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.201 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.202 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.202 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.203 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.203 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.204 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.205 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:48:29.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.206 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.206 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.207 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:48:29.207 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.208 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.208 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.209 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:48:29.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.212 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.213 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.213 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:48:29.214 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.214 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.215 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.216 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:48:29.218 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:48:29.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:29.221 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.221 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.222 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:48:29.223 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:48:29.223 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:48:29.223 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:48:29.224 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.224 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.225 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.225 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.226 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.226 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.226 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:29.227 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:48:29.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.228 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.228 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.229 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.229 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.229 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.230 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.230 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:29.231 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.231 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.232 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.233 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.233 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:48:29.234 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:48:29.235 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.236 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.236 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.236 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:48:29.237 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.238 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.238 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.238 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:48:29.239 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.239 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.239 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.240 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.240 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:48:29.241 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.241 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.242 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:48:29.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.243 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.243 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.243 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:48:29.245 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:48:29.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:29.246 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.246 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.247 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:48:29.247 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:48:29.247 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:48:29.248 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:48:29.248 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.248 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.248 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.249 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.249 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:29.250 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:48:29.250 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.250 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.250 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.251 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.251 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.251 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:29.252 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.252 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.253 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:48:29.255 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.255 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.256 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.256 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:48:29.257 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.258 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.258 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.258 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.260 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:48:29.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.261 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:48:29.262 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.262 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.263 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.263 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:48:29.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.265 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.265 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:48:29.266 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.266 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.267 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.268 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.268 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.269 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.269 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.270 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:48:29.270 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.270 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.271 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.271 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.272 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:48:29.272 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:29.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.273 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.273 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.274 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:48:29.274 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:48:29.274 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:48:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:48:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.275 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.276 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.276 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:29.276 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:48:29.277 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.277 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.277 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.277 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.277 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.278 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.278 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.278 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:29.279 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.279 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.279 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.280 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:48:29.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:48:29.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.283 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.284 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:48:29.286 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:48:29.288 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.288 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.288 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.290 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.291 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.291 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.291 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:48:29.292 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.293 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.293 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.294 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:48:29.294 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.295 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.295 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.295 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.295 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.296 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.296 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:48:29.297 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.298 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:48:29.299 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:29.299 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.299 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.300 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:48:29.300 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:48:29.300 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:48:29.300 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:48:29.301 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.301 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.301 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.301 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.302 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.302 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:29.302 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:48:29.302 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.303 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.303 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.303 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.303 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.304 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.304 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:29.304 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.305 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.305 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.306 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:48:29.306 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.306 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.306 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.307 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:48:29.307 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.307 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.308 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.308 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.309 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.309 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:48:29.309 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.310 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.310 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.310 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:48:29.310 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.311 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.311 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.311 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.311 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.312 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.312 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.312 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:48:29.313 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.313 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.313 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.313 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:48:29.314 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.314 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.314 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.314 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.315 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.315 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.315 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:48:29.316 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.317 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.317 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:48:29.317 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:29.318 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.318 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.318 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:48:29.319 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:48:29.319 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:48:29.319 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:48:29.319 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.320 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.320 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.320 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.321 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.321 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:29.321 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:48:29.321 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:29.322 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:29.322 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:29.322 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:29.323 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:29.323 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:29.323 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:29.323 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-12T08:48:29.324 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:29.324 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:29.324 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:29.324 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.325 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:48:29.326 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.326 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.326 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.327 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.327 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:48:29.327 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.327 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.328 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.328 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:29.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:48:29.330 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.330 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.331 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.331 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:48:29.331 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.331 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.331 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.332 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:29.332 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.332 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.333 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:48:29.333 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.333 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.334 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:48:29.335 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.335 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.335 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.335 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.336 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:29.336 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:29.336 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.336 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:29.336 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:48:29.337 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.337 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.337 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:29.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:48:29.338 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:29.338 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.338 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.339 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.339 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:48:29.339 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:48:29.339 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:48:29.340 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:48:29.340 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:29.340 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:29.340 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:29.341 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:29.341 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.341 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:29.341 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:48:29.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.342 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:29.342 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.342 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.343 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.343 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.343 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.343 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:48:29.344 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.344 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.346 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:29.346 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.346 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.346 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.347 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.347 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.347 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:48:29.347 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.348 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.348 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:29.348 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.349 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.349 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.349 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.349 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.350 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:48:29.350 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.350 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.351 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.351 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:29.352 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.352 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.352 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.353 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.353 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.354 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:48:29.354 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.355 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.356 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:29.356 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.357 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.357 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.357 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.358 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.358 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:48:29.358 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.359 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.359 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.360 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:29.360 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.360 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.360 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.360 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.361 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.361 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:48:29.361 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.362 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.362 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:29.362 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.363 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.363 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.364 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.364 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.364 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:48:29.364 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.365 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:29.366 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:29.366 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:29.367 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:29.367 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:29.368 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:29.368 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:29.368 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:48:29.369 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:29.369 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.370 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:29.370 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:48:29.371 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:48:29.371 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:48:29.372 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:48:29.372 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:48:29.373 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/3665156048": "2022-07-13T08:48:22.831453+0000", 2022-07-12T08:48:29.373 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/3665156048": "2022-07-13T08:48:22.831453+0000" 2022-07-12T08:48:29.373 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.374 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:48:29.374 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:48:29.375 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:48:29.375 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:48:29.375 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:48:29.376 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:48:29.376 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:48:29.376 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:48:29.377 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:29.377 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:29.377 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:48:29.378 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:48:29.378 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:48:29.379 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:48:29.379 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:48:29.379 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:29.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8d8c053a40 msgr2=0x7f8d8c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8d8c053a40 0x7f8d8c055f00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f8d9801e000 tx=0x7f8da8140170).stop 2022-07-12T08:48:29.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 msgr2=0x7f8da813ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da813ba20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7f8da813f9c0 tx=0x7f8da0007ed0).stop 2022-07-12T08:48:29.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 shutdown_connections 2022-07-12T08:48:29.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8da80b2ac0 0x7f8da813ba20 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8d8c053a40 0x7f8d8c055f00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8da8065710 0x7f8da813b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.074+0000 7f8dacbbe700 1 --2- 172.21.15.61:0/915826204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8da8060860 0x7f8da8141bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.078+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 >> 172.21.15.61:0/915826204 conn(0x7f8da8063af0 msgr2=0x7f8da805f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.078+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 shutdown_connections 2022-07-12T08:48:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.078+0000 7f8dacbbe700 1 -- 172.21.15.61:0/915826204 wait complete. 2022-07-12T08:48:29.390 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:29.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.498+0000 7fb10a1e0700 1 Processor -- start 2022-07-12T08:48:29.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.498+0000 7fb10a1e0700 1 -- start start 2022-07-12T08:48:29.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb1040b15a0 0x7fb1040b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10413d240 con 0x7fb10413ce60 2022-07-12T08:48:29.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1041431f0 con 0x7fb104142d50 2022-07-12T08:48:29.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1040b24c0 con 0x7fb1040b15a0 2022-07-12T08:48:29.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1037fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb10413ce60 0x7fb1040b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46678/0 (socket says 172.21.15.61:46678) 2022-07-12T08:48:29.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1037fe700 1 -- 172.21.15.61:0/2777477768 learned_addr learned my addr 172.21.15.61:0/2777477768 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:29.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb103fff700 1 --2- 172.21.15.61:0/2777477768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb1040b15a0 0x7fb1040b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826484650 0 0) 0x7fb10413d240 con 0x7fb10413ce60 2022-07-12T08:48:29.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e4003260 con 0x7fb10413ce60 2022-07-12T08:48:29.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109371813 0 0) 0x7fb1041431f0 con 0x7fb104142d50 2022-07-12T08:48:29.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10413d240 con 0x7fb104142d50 2022-07-12T08:48:29.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb103fff700 1 -- 172.21.15.61:0/2777477768 >> v1:172.21.15.111:6789/0 conn(0x7fb104142d50 legacy=0x7fb1040b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb103fff700 1 -- 172.21.15.61:0/2777477768 >> v1:172.21.15.61:6789/0 conn(0x7fb10413ce60 legacy=0x7fb1040b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb103fff700 1 -- 172.21.15.61:0/2777477768 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb104143490 con 0x7fb1040b15a0 2022-07-12T08:48:29.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb103fff700 1 --2- 172.21.15.61:0/2777477768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb1040b15a0 0x7fb1040b4f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb0fc002f70 tx=0x7fb0fc006df0).ready entity=mon.2 client_cookie=bd68c2ed5c0ab0a1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:29.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc00d5c0 con 0x7fb1040b15a0 2022-07-12T08:48:29.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc000e90 con 0x7fb1040b15a0 2022-07-12T08:48:29.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb1027fc700 1 -- 172.21.15.61:0/2777477768 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0fc00d910 con 0x7fb1040b15a0 2022-07-12T08:48:29.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- 172.21.15.61:0/2777477768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb1040b15a0 msgr2=0x7fb1040b4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/2777477768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb1040b15a0 0x7fb1040b4f00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb0fc002f70 tx=0x7fb0fc006df0).stop 2022-07-12T08:48:29.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- 172.21.15.61:0/2777477768 shutdown_connections 2022-07-12T08:48:29.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/2777477768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb1040b15a0 0x7fb1040b4f00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- 172.21.15.61:0/2777477768 >> 172.21.15.61:0/2777477768 conn(0x7fb10405f9f0 msgr2=0x7fb104066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:29.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- 172.21.15.61:0/2777477768 shutdown_connections 2022-07-12T08:48:29.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.502+0000 7fb10a1e0700 1 -- 172.21.15.61:0/2777477768 wait complete. 2022-07-12T08:48:29.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 Processor -- start 2022-07-12T08:48:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 -- start start 2022-07-12T08:48:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1040b15a0 0x7fb1040b9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 0x7fb10413d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb104142d50 0x7fb10413bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10413ee30 con 0x7fb10413ce60 2022-07-12T08:48:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10413f0d0 con 0x7fb1040b15a0 2022-07-12T08:48:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10413f370 con 0x7fb104142d50 2022-07-12T08:48:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 0x7fb10413d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 0x7fb10413d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58464/0 (socket says 172.21.15.61:58464) 2022-07-12T08:48:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 -- 172.21.15.61:0/1576981371 learned_addr learned my addr 172.21.15.61:0/1576981371 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb1037fe700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1040b15a0 0x7fb1040b9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb103fff700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb104142d50 0x7fb10413bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 -- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb104142d50 msgr2=0x7fb10413bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb104142d50 0x7fb10413bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 -- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1040b15a0 msgr2=0x7fb1040b9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1040b15a0 0x7fb1040b9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 -- 172.21.15.61:0/1576981371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc00b180 con 0x7fb10413ce60 2022-07-12T08:48:29.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb102ffd700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 0x7fb10413d7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7fb0ec001490 tx=0x7fb0ec007ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:29.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0ec00f470 con 0x7fb10413ce60 2022-07-12T08:48:29.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0ec00f5d0 con 0x7fb10413ce60 2022-07-12T08:48:29.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10413f810 con 0x7fb10413ce60 2022-07-12T08:48:29.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0ec00f930 con 0x7fb10413ce60 2022-07-12T08:48:29.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.506+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1040b9300 con 0x7fb10413ce60 2022-07-12T08:48:29.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.510+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb104057460 con 0x7fb10413ce60 2022-07-12T08:48:29.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.510+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb0ec019030 con 0x7fb10413ce60 2022-07-12T08:48:29.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.510+0000 7fb100ff9700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0e8053920 0x7fb0e8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.510+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb0ec06b350 con 0x7fb10413ce60 2022-07-12T08:48:29.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.510+0000 7fb1037fe700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0e8053920 0x7fb0e8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.510+0000 7fb1037fe700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0e8053920 0x7fb0e8055de0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fb0f400d000 tx=0x7fb0f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:29.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.518+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0ec0424f0 con 0x7fb10413ce60 2022-07-12T08:48:29.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.682+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1040b3e20 con 0x7fb10413ce60 2022-07-12T08:48:29.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.682+0000 7fb100ff9700 1 -- 172.21.15.61:0/1576981371 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+1209 (secure 0 0 0) 0x7fb0ec045b10 con 0x7fb10413ce60 2022-07-12T08:48:29.691 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:29.691 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6140,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/265074027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":265074027},{"type":"v1","addr":"172.21.15.61:6833","nonce":265074027}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[]} 2022-07-12T08:48:29.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0e8053920 msgr2=0x7fb0e8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0e8053920 0x7fb0e8055de0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fb0f400d000 tx=0x7fb0f4009040).stop 2022-07-12T08:48:29.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 msgr2=0x7fb10413d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 0x7fb10413d7a0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7fb0ec001490 tx=0x7fb0ec007ea0).stop 2022-07-12T08:48:29.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 shutdown_connections 2022-07-12T08:48:29.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb104142d50 0x7fb10413bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb10413ce60 0x7fb10413d7a0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0e8053920 0x7fb0e8055de0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 --2- 172.21.15.61:0/1576981371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1040b15a0 0x7fb1040b9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 >> 172.21.15.61:0/1576981371 conn(0x7fb10405f9f0 msgr2=0x7fb1040b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:29.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 shutdown_connections 2022-07-12T08:48:29.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.686+0000 7fb10a1e0700 1 -- 172.21.15.61:0/1576981371 wait complete. 2022-07-12T08:48:29.697 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 46 2022-07-12T08:48:29.702 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:29.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.798+0000 7f0ad7fff700 1 Processor -- start 2022-07-12T08:48:29.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad7fff700 1 -- start start 2022-07-12T08:48:29.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ad80af6e0 0x7f0ad8143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0ad8060c30 0x7f0ad813c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad80afac0 con 0x7f0ad813c960 2022-07-12T08:48:29.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad8143970 con 0x7f0ad80af6e0 2022-07-12T08:48:29.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad7fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad80611b0 con 0x7f0ad8060c30 2022-07-12T08:48:29.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ad80af6e0 0x7f0ad8143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ad80af6e0 0x7f0ad8143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58472/0 (socket says 172.21.15.61:58472) 2022-07-12T08:48:29.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 -- 172.21.15.61:0/1810748358 learned_addr learned my addr 172.21.15.61:0/1810748358 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:29.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0acffff700 1 --2- 172.21.15.61:0/1810748358 >> v2:172.21.15.111:3300/0 conn(0x7f0ad8060c30 0x7f0ad813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad67fc700 1 -- 172.21.15.61:0/1810748358 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974619617 0 0) 0x7f0ad80afac0 con 0x7f0ad813c960 2022-07-12T08:48:29.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad67fc700 1 -- 172.21.15.61:0/1810748358 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab8003260 con 0x7f0ad813c960 2022-07-12T08:48:29.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 -- 172.21.15.61:0/1810748358 >> v2:172.21.15.111:3300/0 conn(0x7f0ad8060c30 msgr2=0x7f0ad813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:29.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 --2- 172.21.15.61:0/1810748358 >> v2:172.21.15.111:3300/0 conn(0x7f0ad8060c30 0x7f0ad813c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 -- 172.21.15.61:0/1810748358 >> v1:172.21.15.111:6789/0 conn(0x7f0ad813c960 legacy=0x7f0ad8145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 -- 172.21.15.61:0/1810748358 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad8061450 con 0x7f0ad80af6e0 2022-07-12T08:48:29.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad6ffd700 1 --2- 172.21.15.61:0/1810748358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad80af6e0 0x7f0ad8143480 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f0ac8002f70 tx=0x7f0ac8006eb0).ready entity=mon.0 client_cookie=92fd909a3e4e452 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:29.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad67fc700 1 -- 172.21.15.61:0/1810748358 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ac800d610 con 0x7f0ad80af6e0 2022-07-12T08:48:29.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.802+0000 7f0ad67fc700 1 -- 172.21.15.61:0/1810748358 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac8000e90 con 0x7f0ad80af6e0 2022-07-12T08:48:29.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad67fc700 1 -- 172.21.15.61:0/1810748358 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ac800d9a0 con 0x7f0ad80af6e0 2022-07-12T08:48:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 -- 172.21.15.61:0/1810748358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad80af6e0 msgr2=0x7f0ad8143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/1810748358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad80af6e0 0x7f0ad8143480 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 rx=0x7f0ac8002f70 tx=0x7f0ac8006eb0).stop 2022-07-12T08:48:29.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 -- 172.21.15.61:0/1810748358 shutdown_connections 2022-07-12T08:48:29.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/1810748358 >> v2:172.21.15.111:3300/0 conn(0x7f0ad8060c30 0x7f0ad813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/1810748358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad80af6e0 0x7f0ad8143480 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 -- 172.21.15.61:0/1810748358 >> 172.21.15.61:0/1810748358 conn(0x7f0ad80b5780 msgr2=0x7f0ad80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:29.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 -- 172.21.15.61:0/1810748358 shutdown_connections 2022-07-12T08:48:29.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 -- 172.21.15.61:0/1810748358 wait complete. 2022-07-12T08:48:29.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 Processor -- start 2022-07-12T08:48:29.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.806+0000 7f0ad7fff700 1 -- start start 2022-07-12T08:48:29.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad8060c30 0x7f0ad8142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad80af6e0 0x7f0ad8066c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 0x7f0ad8154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad81466e0 con 0x7f0ad813c960 2022-07-12T08:48:29.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8146980 con 0x7f0ad8060c30 2022-07-12T08:48:29.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad8065440 con 0x7f0ad80af6e0 2022-07-12T08:48:29.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 0x7f0ad8154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 0x7f0ad8154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58482/0 (socket says 172.21.15.61:58482) 2022-07-12T08:48:29.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad8060c30 0x7f0ad8142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 -- 172.21.15.61:0/27078080 learned_addr learned my addr 172.21.15.61:0/27078080 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:29.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0acffff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad80af6e0 0x7f0ad8066c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 -- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad80af6e0 msgr2=0x7f0ad8066c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad80af6e0 0x7f0ad8066c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 -- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad8060c30 msgr2=0x7f0ad8142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:29.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad8060c30 0x7f0ad8142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:29.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 -- 172.21.15.61:0/27078080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac800b1e0 con 0x7f0ad813c960 2022-07-12T08:48:29.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad77fe700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 0x7f0ad8154e00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f0ad0001490 tx=0x7f0ad0007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ad0002310 con 0x7f0ad813c960 2022-07-12T08:48:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ad0002470 con 0x7f0ad813c960 2022-07-12T08:48:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ad00017a0 con 0x7f0ad813c960 2022-07-12T08:48:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad8145ee0 con 0x7f0ad813c960 2022-07-12T08:48:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.810+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad81461c0 con 0x7f0ad813c960 2022-07-12T08:48:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.814+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad8146b40 con 0x7f0ad813c960 2022-07-12T08:48:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.814+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0ad0019020 con 0x7f0ad813c960 2022-07-12T08:48:29.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.814+0000 7f0ad4ff9700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0abc053a40 0x7f0abc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:29.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.814+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0ad006bac0 con 0x7f0ad813c960 2022-07-12T08:48:29.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.814+0000 7f0ad6ffd700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0abc053a40 0x7f0abc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:29.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.814+0000 7f0ad6ffd700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0abc053a40 0x7f0abc055f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0ac801c000 tx=0x7f0ac8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:29.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.822+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0ad0042c60 con 0x7f0ad813c960 2022-07-12T08:48:29.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.990+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ad8057460 con 0x7f0ad813c960 2022-07-12T08:48:29.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.990+0000 7f0ad4ff9700 1 -- 172.21.15.61:0/27078080 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f0ad0046280 con 0x7f0ad813c960 2022-07-12T08:48:29.997 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:29.997 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":46,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5978,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1990159009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1990159009},{"type":"v1","addr":"172.21.15.111:6835","nonce":1990159009}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":41},{"gid":6140,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/265074027","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":265074027},{"type":"v1","addr":"172.21.15.61:6833","nonce":265074027}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":46}],"filesystems":[]} 2022-07-12T08:48:30.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0abc053a40 msgr2=0x7f0abc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:30.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0abc053a40 0x7f0abc055f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0ac801c000 tx=0x7f0ac8015000).stop 2022-07-12T08:48:30.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 msgr2=0x7f0ad8154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:30.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 0x7f0ad8154e00 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f0ad0001490 tx=0x7f0ad0007e60).stop 2022-07-12T08:48:30.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 shutdown_connections 2022-07-12T08:48:30.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad813c960 0x7f0ad8154e00 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad80af6e0 0x7f0ad8066c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0abc053a40 0x7f0abc055f00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 --2- 172.21.15.61:0/27078080 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad8060c30 0x7f0ad8142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 >> 172.21.15.61:0/27078080 conn(0x7f0ad80b5780 msgr2=0x7f0ad80b46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:30.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 shutdown_connections 2022-07-12T08:48:30.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:29.994+0000 7f0ad7fff700 1 -- 172.21.15.61:0/27078080 wait complete. 2022-07-12T08:48:30.007 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 46 2022-07-12T08:48:30.012 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:48:30.012 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:48:30.013 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:48:30.014 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:48:30.014 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:48:30.015 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:48:30.016 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:48:30.016 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:48:30.029 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:48:30.030 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:48:30.033 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:48:30.033 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:48:30.039 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:48:30.040 DEBUG:teuthology.parallel:result is None 2022-07-12T08:48:30.079 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:48:30.079 DEBUG:teuthology.parallel:result is None 2022-07-12T08:48:30.081 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:48:30.106 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:48:30.148 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:48:30.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 Processor -- start 2022-07-12T08:48:30.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 -- start start 2022-07-12T08:48:30.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3bc0af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb3bc0b3f60 0x7fb3bc0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3bc0aecb0 con 0x7fb3bc0af280 2022-07-12T08:48:30.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc143b00 con 0x7fb3bc0b3f60 2022-07-12T08:48:30.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3c0eac700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1435c0 con 0x7fb3bc09bce0 2022-07-12T08:48:30.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3bb7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb3bc0af280 0x7fb3bc060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46716/0 (socket says 172.21.15.61:46716) 2022-07-12T08:48:30.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3bb7fe700 1 -- 172.21.15.61:0/1909778952 learned_addr learned my addr 172.21.15.61:0/1909778952 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:30.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3baffd700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3bc0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473204910 0 0) 0x7fb3bc0aecb0 con 0x7fb3bc0af280 2022-07-12T08:48:30.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3bbfff700 1 --2- 172.21.15.61:0/1909778952 >> v2:172.21.15.111:3300/0 conn(0x7fb3bc0b3f60 0x7fb3bc0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a8003260 con 0x7fb3bc0af280 2022-07-12T08:48:30.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754966848 0 0) 0x7fb3a8003260 con 0x7fb3bc0af280 2022-07-12T08:48:30.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 msgr2=0x7fb3bc0af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:30.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3bc0af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 >> v2:172.21.15.111:3300/0 conn(0x7fb3bc0b3f60 msgr2=0x7fb3bc0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:30.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 --2- 172.21.15.61:0/1909778952 >> v2:172.21.15.111:3300/0 conn(0x7fb3bc0b3f60 0x7fb3bc0b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc143860 con 0x7fb3bc0af280 2022-07-12T08:48:30.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fb3b4001b60 con 0x7fb3bc0af280 2022-07-12T08:48:30.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3baffd700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3bc0af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:30.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 >> v1:172.21.15.61:6789/0 conn(0x7fb3bc0af280 legacy=0x7fb3bc060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:30.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a8003750 0x7fb3a8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3a8004500 0x7fb3a8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3a8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1435c0 con 0x7fb3a8004500 2022-07-12T08:48:30.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a800b750 con 0x7fb3a8003750 2022-07-12T08:48:30.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a800b8c0 con 0x7fb3bc09bce0 2022-07-12T08:48:30.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3baffd700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3a8004500 0x7fb3a8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.186+0000 7fb3bb7fe700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a8003750 0x7fb3a8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bbfff700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3a8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 -- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 msgr2=0x7fb3a8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:30.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09bce0 0x7fb3a8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 -- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a8003750 msgr2=0x7fb3a8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:30.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3a8003750 0x7fb3a8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 -- 172.21.15.61:0/1909778952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8003260 con 0x7fb3a8004500 2022-07-12T08:48:30.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3a8004500 0x7fb3a8004980 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fb3b000a000 tx=0x7fb3b0008040).ready entity=mon.0 client_cookie=998d3ebaa9f03c1b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:30.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b0012610 con 0x7fb3a8004500 2022-07-12T08:48:30.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b0012770 con 0x7fb3a8004500 2022-07-12T08:48:30.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3ba7fc700 1 -- 172.21.15.61:0/1909778952 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b0012ad0 con 0x7fb3a8004500 2022-07-12T08:48:30.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/1909778952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3a8004500 msgr2=0x7fb3a8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:30.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/1909778952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3a8004500 0x7fb3a8004980 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fb3b000a000 tx=0x7fb3b0008040).stop 2022-07-12T08:48:30.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bbfff700 1 -- 172.21.15.61:0/1909778952 reap_dead start 2022-07-12T08:48:30.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/1909778952 shutdown_connections 2022-07-12T08:48:30.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/1909778952 >> 172.21.15.61:0/1909778952 conn(0x7fb3bc0b92c0 msgr2=0x7fb3bc0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:30.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/1909778952 shutdown_connections 2022-07-12T08:48:30.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/1909778952 wait complete. 2022-07-12T08:48:30.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 Processor -- start 2022-07-12T08:48:30.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- start start 2022-07-12T08:48:30.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 0x7fb3bc151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3bc09c860 0x7fb3bc155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09c350 0x7fb3bc155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc155630 con 0x7fb3bc0b7560 2022-07-12T08:48:30.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc152450 con 0x7fb3bc09c860 2022-07-12T08:48:30.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3c0eac700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc156470 con 0x7fb3bc09c350 2022-07-12T08:48:30.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 0x7fb3bc151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 0x7fb3bc151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58508/0 (socket says 172.21.15.61:58508) 2022-07-12T08:48:30.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 -- 172.21.15.61:0/124587654 learned_addr learned my addr 172.21.15.61:0/124587654 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:30.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bbfff700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09c350 0x7fb3bc155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3bc09c860 0x7fb3bc155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 -- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09c350 msgr2=0x7fb3bc155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:30.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09c350 0x7fb3bc155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 -- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3bc09c860 msgr2=0x7fb3bc155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:30.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3bc09c860 0x7fb3bc155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bb7fe700 1 -- 172.21.15.61:0/124587654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0b7940 con 0x7fb3bc0b7560 2022-07-12T08:48:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3bbfff700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09c350 0x7fb3bc155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.190+0000 7fb3baffd700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3bc09c860 0x7fb3bc155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3bb7fe700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 0x7fb3bc151f60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fb3b400d000 tx=0x7fb3b4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:30.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b4005c60 con 0x7fb3bc0b7560 2022-07-12T08:48:30.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b4005dc0 con 0x7fb3bc0b7560 2022-07-12T08:48:30.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc0b0e70 con 0x7fb3bc0b7560 2022-07-12T08:48:30.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3b40072c0 con 0x7fb3bc0b7560 2022-07-12T08:48:30.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc142400 con 0x7fb3bc0b7560 2022-07-12T08:48:30.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc1557f0 con 0x7fb3bc0b7560 2022-07-12T08:48:30.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb3b401a030 con 0x7fb3bc0b7560 2022-07-12T08:48:30.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3b8ff9700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb39c053ad0 0x7fb39c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:30.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb3b406bca0 con 0x7fb3bc0b7560 2022-07-12T08:48:30.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.194+0000 7fb3baffd700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb39c053ad0 0x7fb39c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:30.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.198+0000 7fb3baffd700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb39c053ad0 0x7fb39c055f90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb3b000a000 tx=0x7fb3b0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:30.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.202+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3bc1557f0 con 0x7fb3bc0b7560 2022-07-12T08:48:30.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:30.338+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fb3bc057460 con 0x7fb3bc0b7560 2022-07-12T08:48:31.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.186+0000 7fb3b8ff9700 1 -- 172.21.15.61:0/124587654 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v86) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb3bc057460 con 0x7fb3bc0b7560 2022-07-12T08:48:31.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb39c053ad0 msgr2=0x7fb39c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb39c053ad0 0x7fb39c055f90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fb3b000a000 tx=0x7fb3b0008040).stop 2022-07-12T08:48:31.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 msgr2=0x7fb3bc151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 0x7fb3bc151f60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fb3b400d000 tx=0x7fb3b4009040).stop 2022-07-12T08:48:31.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 shutdown_connections 2022-07-12T08:48:31.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3bc09c350 0x7fb3bc155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb39c053ad0 0x7fb39c055f90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3bc09c860 0x7fb3bc155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 --2- 172.21.15.61:0/124587654 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3bc0b7560 0x7fb3bc151f60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 >> 172.21.15.61:0/124587654 conn(0x7fb3bc0b92c0 msgr2=0x7fb3bc05bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:31.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 shutdown_connections 2022-07-12T08:48:31.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.190+0000 7fb3c0eac700 1 -- 172.21.15.61:0/124587654 wait complete. 2022-07-12T08:48:31.198 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:48:31.208 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:48:31.209 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:48:31.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 Processor -- start 2022-07-12T08:48:31.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 -- start start 2022-07-12T08:48:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc0b4390 0x7f30bc140570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f30bc1431e0 0x7f30bc143670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30bc0b3e80 con 0x7f30bc0b6250 2022-07-12T08:48:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30bc147390 con 0x7f30bc1431e0 2022-07-12T08:48:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c3c92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0b6630 con 0x7f30bc0b4390 2022-07-12T08:48:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c1a2e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f30bc0b6250 0x7f30bc13fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46736/0 (socket says 172.21.15.61:46736) 2022-07-12T08:48:31.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c1a2e700 1 -- 172.21.15.61:0/4206657769 learned_addr learned my addr 172.21.15.61:0/4206657769 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:31.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c122d700 1 --2- 172.21.15.61:0/4206657769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc0b4390 0x7f30bc140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c222f700 1 --2- 172.21.15.61:0/4206657769 >> v2:172.21.15.111:3300/0 conn(0x7f30bc1431e0 0x7f30bc143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c0a2c700 1 -- 172.21.15.61:0/4206657769 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404292799 0 0) 0x7f30bc0b3e80 con 0x7f30bc0b6250 2022-07-12T08:48:31.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c0a2c700 1 -- 172.21.15.61:0/4206657769 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a4003260 con 0x7f30bc0b6250 2022-07-12T08:48:31.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c122d700 1 -- 172.21.15.61:0/4206657769 >> v2:172.21.15.111:3300/0 conn(0x7f30bc1431e0 msgr2=0x7f30bc143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c122d700 1 --2- 172.21.15.61:0/4206657769 >> v2:172.21.15.111:3300/0 conn(0x7f30bc1431e0 0x7f30bc143670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c122d700 1 -- 172.21.15.61:0/4206657769 >> v1:172.21.15.61:6789/0 conn(0x7f30bc0b6250 legacy=0x7f30bc13fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c122d700 1 -- 172.21.15.61:0/4206657769 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc1467b0 con 0x7f30bc0b4390 2022-07-12T08:48:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c222f700 1 --2- 172.21.15.61:0/4206657769 >> v2:172.21.15.111:3300/0 conn(0x7f30bc1431e0 0x7f30bc143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.314+0000 7f30c122d700 1 --2- 172.21.15.61:0/4206657769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc0b4390 0x7f30bc140570 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f30ac002f70 tx=0x7f30ac006df0).ready entity=mon.2 client_cookie=6fc88100e9fa8832 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c0a2c700 1 -- 172.21.15.61:0/4206657769 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac00d4a0 con 0x7f30bc0b4390 2022-07-12T08:48:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c0a2c700 1 -- 172.21.15.61:0/4206657769 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac000e90 con 0x7f30bc0b4390 2022-07-12T08:48:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c0a2c700 1 -- 172.21.15.61:0/4206657769 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac00d7f0 con 0x7f30bc0b4390 2022-07-12T08:48:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- 172.21.15.61:0/4206657769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc0b4390 msgr2=0x7f30bc140570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 --2- 172.21.15.61:0/4206657769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc0b4390 0x7f30bc140570 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f30ac002f70 tx=0x7f30ac006df0).stop 2022-07-12T08:48:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- 172.21.15.61:0/4206657769 shutdown_connections 2022-07-12T08:48:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 --2- 172.21.15.61:0/4206657769 >> v2:172.21.15.111:3300/0 conn(0x7f30bc1431e0 0x7f30bc143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 --2- 172.21.15.61:0/4206657769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc0b4390 0x7f30bc140570 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- 172.21.15.61:0/4206657769 >> 172.21.15.61:0/4206657769 conn(0x7f30bc066f00 msgr2=0x7f30bc0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- 172.21.15.61:0/4206657769 shutdown_connections 2022-07-12T08:48:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- 172.21.15.61:0/4206657769 wait complete. 2022-07-12T08:48:31.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 Processor -- start 2022-07-12T08:48:31.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- start start 2022-07-12T08:48:31.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30bc0b4390 0x7f30bc0b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 0x7f30bc13e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc1431e0 0x7f30bc144370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc151ec0 con 0x7f30bc0b6250 2022-07-12T08:48:31.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc0aeda0 con 0x7f30bc0b4390 2022-07-12T08:48:31.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c3c92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30bc13eeb0 con 0x7f30bc1431e0 2022-07-12T08:48:31.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c122d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 0x7f30bc13e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c122d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 0x7f30bc13e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58522/0 (socket says 172.21.15.61:58522) 2022-07-12T08:48:31.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.318+0000 7f30c122d700 1 -- 172.21.15.61:0/3907989690 learned_addr learned my addr 172.21.15.61:0/3907989690 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:31.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c222f700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc1431e0 0x7f30bc144370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c1a2e700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30bc0b4390 0x7f30bc0b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c122d700 1 -- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc1431e0 msgr2=0x7f30bc144370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c122d700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc1431e0 0x7f30bc144370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c122d700 1 -- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30bc0b4390 msgr2=0x7f30bc0b7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c122d700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30bc0b4390 0x7f30bc0b7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c122d700 1 -- 172.21.15.61:0/3907989690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4009470 con 0x7f30bc0b6250 2022-07-12T08:48:31.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c122d700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 0x7f30bc13e0c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f30ac002f70 tx=0x7f30ac014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac016070 con 0x7f30bc0b6250 2022-07-12T08:48:31.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac013930 con 0x7f30bc0b6250 2022-07-12T08:48:31.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac00b180 con 0x7f30bc0b6250 2022-07-12T08:48:31.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30ac013c90 con 0x7f30bc0b6250 2022-07-12T08:48:31.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30bc144860 con 0x7f30bc0b6250 2022-07-12T08:48:31.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30bc145f30 con 0x7f30bc0b6250 2022-07-12T08:48:31.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f30ac025030 con 0x7f30bc0b6250 2022-07-12T08:48:31.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30aaffd700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f309c053a40 0x7f309c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f30ac06cb00 con 0x7f30bc0b6250 2022-07-12T08:48:31.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.322+0000 7f30c1a2e700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f309c053a40 0x7f309c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.326+0000 7f30c1a2e700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f309c053a40 0x7f309c055f00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f30b800b000 tx=0x7f30b8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.330+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f30ac043d90 con 0x7f30bc0b6250 2022-07-12T08:48:31.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.502+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f30bc142dc0 con 0x7f30bc0b6250 2022-07-12T08:48:31.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30aaffd700 1 -- 172.21.15.61:0/3907989690 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f30ac023020 con 0x7f30bc0b6250 2022-07-12T08:48:31.511 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:31.511 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:31.512 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:48:31.512 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.512 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:48:31.513 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:48:31.513 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.514 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:31.514 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.515 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.515 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.516 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.516 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.516 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.517 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:48:31.518 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:48:31.518 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.519 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:31.519 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.519 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.520 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.520 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.521 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.521 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.521 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:48:31.522 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:48:31.522 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.523 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.523 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.523 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.523 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.524 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.524 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.524 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:48:31.524 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:48:31.525 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.525 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.525 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.525 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.525 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.526 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.526 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.526 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:48:31.527 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:48:31.527 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.527 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.528 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.528 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.528 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.528 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.528 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.529 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:48:31.529 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:48:31.529 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.529 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.530 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.530 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.530 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.530 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.531 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.531 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:48:31.531 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:48:31.532 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.533 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.533 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.533 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.533 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:48:31.534 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:48:31.534 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.534 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.534 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.535 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.535 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.535 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.535 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.536 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:48:31.536 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:48:31.536 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.537 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.537 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.537 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.538 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.538 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.538 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.538 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:48:31.539 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:48:31.539 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.540 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.540 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.540 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.540 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.540 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:48:31.541 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:48:31.541 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.541 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:31.542 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:31.542 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:31.542 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.543 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.543 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.544 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:48:31.544 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:48:31.545 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.545 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:31.546 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:31.546 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:31.547 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.548 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:48:31.549 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:48:31.549 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:31.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:31.551 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:31.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.552 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.552 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.553 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:48:31.553 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:48:31.553 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.554 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:31.554 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:31.555 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:31.555 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.555 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.556 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.556 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.557 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:48:31.557 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:48:31.557 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.558 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:31.558 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:48:31.559 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.559 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.560 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:31.560 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:31.561 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:48:31.561 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:48:31.561 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:31.562 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:31.562 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:48:31.563 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:31.563 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.564 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:31.564 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:31.564 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:31.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f309c053a40 msgr2=0x7f309c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f309c053a40 0x7f309c055f00 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f30b800b000 tx=0x7f30b8009040).stop 2022-07-12T08:48:31.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 msgr2=0x7f30bc13e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 0x7f30bc13e0c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f30ac002f70 tx=0x7f30ac014000).stop 2022-07-12T08:48:31.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 shutdown_connections 2022-07-12T08:48:31.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30bc1431e0 0x7f30bc144370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30bc0b6250 0x7f30bc13e0c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f309c053a40 0x7f309c055f00 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 --2- 172.21.15.61:0/3907989690 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f30bc0b4390 0x7f30bc0b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 >> 172.21.15.61:0/3907989690 conn(0x7f30bc066f00 msgr2=0x7f30bc0b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 shutdown_connections 2022-07-12T08:48:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.506+0000 7f30c3c92700 1 -- 172.21.15.61:0/3907989690 wait complete. 2022-07-12T08:48:31.574 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:31.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 Processor -- start 2022-07-12T08:48:31.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 -- start start 2022-07-12T08:48:31.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fecd8143720 0x7fecd8143b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fecd8060b00 0x7fecd80981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd8061030 con 0x7fecd81442b0 2022-07-12T08:48:31.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd80b1d20 con 0x7fecd8143720 2022-07-12T08:48:31.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd80612d0 con 0x7fecd8060b00 2022-07-12T08:48:31.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fecd8143720 0x7fecd8143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fecd8143720 0x7fecd8143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58534/0 (socket says 172.21.15.61:58534) 2022-07-12T08:48:31.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 -- 172.21.15.61:0/227463146 learned_addr learned my addr 172.21.15.61:0/227463146 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:31.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecdc9c7700 1 --2- 172.21.15.61:0/227463146 >> v2:172.21.15.111:3300/0 conn(0x7fecd8060b00 0x7fecd80981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd6ffd700 1 -- 172.21.15.61:0/227463146 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175291143 0 0) 0x7fecd8061030 con 0x7fecd81442b0 2022-07-12T08:48:31.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 -- 172.21.15.61:0/227463146 >> v2:172.21.15.111:3300/0 conn(0x7fecd8060b00 msgr2=0x7fecd80981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:31.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 --2- 172.21.15.61:0/227463146 >> v2:172.21.15.111:3300/0 conn(0x7fecd8060b00 0x7fecd80981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 -- 172.21.15.61:0/227463146 >> v1:172.21.15.111:6789/0 conn(0x7fecd81442b0 legacy=0x7fecd80603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 -- 172.21.15.61:0/227463146 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd8140f40 con 0x7fecd8143720 2022-07-12T08:48:31.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd7fff700 1 --2- 172.21.15.61:0/227463146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd8143720 0x7fecd8143b00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7feccc002f70 tx=0x7feccc006eb0).ready entity=mon.0 client_cookie=f1e7216432e99d2c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd6ffd700 1 -- 172.21.15.61:0/227463146 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feccc00d5c0 con 0x7fecd8143720 2022-07-12T08:48:31.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd6ffd700 1 -- 172.21.15.61:0/227463146 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc000e90 con 0x7fecd8143720 2022-07-12T08:48:31.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecd6ffd700 1 -- 172.21.15.61:0/227463146 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feccc00d910 con 0x7fecd8143720 2022-07-12T08:48:31.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 -- 172.21.15.61:0/227463146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd8143720 msgr2=0x7fecd8143b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 --2- 172.21.15.61:0/227463146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd8143720 0x7fecd8143b00 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7feccc002f70 tx=0x7feccc006eb0).stop 2022-07-12T08:48:31.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 -- 172.21.15.61:0/227463146 shutdown_connections 2022-07-12T08:48:31.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 --2- 172.21.15.61:0/227463146 >> v2:172.21.15.111:3300/0 conn(0x7fecd8060b00 0x7fecd80981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.682+0000 7fecde42a700 1 --2- 172.21.15.61:0/227463146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd8143720 0x7fecd8143b00 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- 172.21.15.61:0/227463146 >> 172.21.15.61:0/227463146 conn(0x7fecd8064720 msgr2=0x7fecd80b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:31.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- 172.21.15.61:0/227463146 shutdown_connections 2022-07-12T08:48:31.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- 172.21.15.61:0/227463146 wait complete. 2022-07-12T08:48:31.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 Processor -- start 2022-07-12T08:48:31.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- start start 2022-07-12T08:48:31.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd8060b00 0x7fecd8146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd8143720 0x7fecd809a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 0x7fecd813f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd80675f0 con 0x7fecd81442b0 2022-07-12T08:48:31.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd809aa40 con 0x7fecd8060b00 2022-07-12T08:48:31.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecde42a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd80670d0 con 0x7fecd8143720 2022-07-12T08:48:31.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecdc9c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 0x7fecd813f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecdc9c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 0x7fecd813f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58540/0 (socket says 172.21.15.61:58540) 2022-07-12T08:48:31.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecdc9c7700 1 -- 172.21.15.61:0/450145312 learned_addr learned my addr 172.21.15.61:0/450145312 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:31.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecd7fff700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd8060b00 0x7fecd8146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.686+0000 7fecd77fe700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd8143720 0x7fecd809a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecdc9c7700 1 -- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd8143720 msgr2=0x7fecd809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecdc9c7700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd8143720 0x7fecd809a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecdc9c7700 1 -- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd8060b00 msgr2=0x7fecd8146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecdc9c7700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd8060b00 0x7fecd8146d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecdc9c7700 1 -- 172.21.15.61:0/450145312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc00b1e0 con 0x7fecd81442b0 2022-07-12T08:48:31.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd77fe700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd8143720 0x7fecd809a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:31.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecdc9c7700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 0x7fecd813f170 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7fecd809c090 tx=0x7fecc8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc8010780 con 0x7fecd81442b0 2022-07-12T08:48:31.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc80108e0 con 0x7fecd81442b0 2022-07-12T08:48:31.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd809c1f0 con 0x7fecd81442b0 2022-07-12T08:48:31.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc8010c40 con 0x7fecd81442b0 2022-07-12T08:48:31.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd8067350 con 0x7fecd81442b0 2022-07-12T08:48:31.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd8057460 con 0x7fecd81442b0 2022-07-12T08:48:31.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fecc801c460 con 0x7fecd81442b0 2022-07-12T08:48:31.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd57fa700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539d0 0x7fecb8055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fecc806be40 con 0x7fecd81442b0 2022-07-12T08:48:31.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.690+0000 7fecd7fff700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539d0 0x7fecb8055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.694+0000 7fecd7fff700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539d0 0x7fecb8055e90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7feccc01c000 tx=0x7feccc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.698+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fecc8043070 con 0x7fecd81442b0 2022-07-12T08:48:31.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.822+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecd809abe0 con 0x7fecd81442b0 2022-07-12T08:48:31.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.822+0000 7fecd57fa700 1 -- 172.21.15.61:0/450145312 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13230 (secure 0 0 0) 0x7fecc8046690 con 0x7fecd81442b0 2022-07-12T08:48:31.827 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:31.828 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":86,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:31.181207+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:31.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539d0 msgr2=0x7fecb8055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539d0 0x7fecb8055e90 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7feccc01c000 tx=0x7feccc015000).stop 2022-07-12T08:48:31.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 msgr2=0x7fecd813f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 0x7fecd813f170 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 rx=0x7fecd809c090 tx=0x7fecc8006df0).stop 2022-07-12T08:48:31.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 shutdown_connections 2022-07-12T08:48:31.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd81442b0 0x7fecd813f170 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd8143720 0x7fecd809a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539d0 0x7fecb8055e90 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 --2- 172.21.15.61:0/450145312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd8060b00 0x7fecd8146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 >> 172.21.15.61:0/450145312 conn(0x7fecd8064720 msgr2=0x7fecd8097c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:31.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 shutdown_connections 2022-07-12T08:48:31.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.826+0000 7fecde42a700 1 -- 172.21.15.61:0/450145312 wait complete. 2022-07-12T08:48:31.841 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:48:31.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:31.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 Processor -- start 2022-07-12T08:48:31.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 -- start start 2022-07-12T08:48:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 0x7f257c0b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f257c13e950 0x7f257c13ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c0b15d0 con 0x7f257c0ae800 2022-07-12T08:48:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c0ac820 con 0x7f257c13e950 2022-07-12T08:48:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2582218700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b1800 con 0x7f257c0acda0 2022-07-12T08:48:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2580a15700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f257c0ae800 0x7f257c0aec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46762/0 (socket says 172.21.15.61:46762) 2022-07-12T08:48:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2580a15700 1 -- 172.21.15.61:0/3974183930 learned_addr learned my addr 172.21.15.61:0/3974183930 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2581216700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 0x7f257c0b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2581a17700 1 --2- 172.21.15.61:0/3974183930 >> v2:172.21.15.111:3300/0 conn(0x7f257c13e950 0x7f257c13ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894143790 0 0) 0x7f257c0b15d0 con 0x7f257c0ae800 2022-07-12T08:48:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.950+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2560003260 con 0x7f257c0ae800 2022-07-12T08:48:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2565059796 0 0) 0x7f2560003260 con 0x7f257c0ae800 2022-07-12T08:48:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 msgr2=0x7f257c0b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 0x7f257c0b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 >> v2:172.21.15.111:3300/0 conn(0x7f257c13e950 msgr2=0x7f257c13ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 --2- 172.21.15.61:0/3974183930 >> v2:172.21.15.111:3300/0 conn(0x7f257c13e950 0x7f257c13ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c067430 con 0x7f257c0ae800 2022-07-12T08:48:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f256c001660 con 0x7f257c0ae800 2022-07-12T08:48:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 >> v1:172.21.15.61:6789/0 conn(0x7f257c0ae800 legacy=0x7f257c0aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 0x7f2560003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2560003c80 0x7f2560004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25600049f0 0x7f2560004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0ac820 con 0x7f2560003c80 2022-07-12T08:48:31.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0b1800 con 0x7f25600049f0 2022-07-12T08:48:31.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256000b750 con 0x7f257c0acda0 2022-07-12T08:48:31.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2560003c80 0x7f2560004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2581216700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 0x7f2560003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2581a17700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25600049f0 0x7f2560004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 -- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 msgr2=0x7f2560003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:31.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0acda0 0x7f2560003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 -- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25600049f0 msgr2=0x7f2560004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:31.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25600049f0 0x7f2560004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 -- 172.21.15.61:0/3974183930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560003260 con 0x7f2560003c80 2022-07-12T08:48:31.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2580a15700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2560003c80 0x7f2560004500 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f25600054d0 tx=0x7f256c007630).ready entity=mon.0 client_cookie=d237c2a3ccbb9787 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f256c00eb90 con 0x7f2560003c80 2022-07-12T08:48:31.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c00ecf0 con 0x7f2560003c80 2022-07-12T08:48:31.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2573fff700 1 -- 172.21.15.61:0/3974183930 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f256c0072c0 con 0x7f2560003c80 2022-07-12T08:48:31.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2582218700 1 -- 172.21.15.61:0/3974183930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2560003c80 msgr2=0x7f2560004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2582218700 1 --2- 172.21.15.61:0/3974183930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2560003c80 0x7f2560004500 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 rx=0x7f25600054d0 tx=0x7f256c007630).stop 2022-07-12T08:48:31.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2581a17700 1 -- 172.21.15.61:0/3974183930 reap_dead start 2022-07-12T08:48:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2582218700 1 -- 172.21.15.61:0/3974183930 shutdown_connections 2022-07-12T08:48:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2582218700 1 -- 172.21.15.61:0/3974183930 >> 172.21.15.61:0/3974183930 conn(0x7f257c05f140 msgr2=0x7f257c05f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2582218700 1 -- 172.21.15.61:0/3974183930 shutdown_connections 2022-07-12T08:48:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.954+0000 7f2582218700 1 -- 172.21.15.61:0/3974183930 wait complete. 2022-07-12T08:48:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 Processor -- start 2022-07-12T08:48:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 -- start start 2022-07-12T08:48:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2560003c80 0x7f257c145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 0x7f257c064ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0ab450 0x7f257c0ab8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0649f0 con 0x7f257c09fb40 2022-07-12T08:48:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0671f0 con 0x7f2560003c80 2022-07-12T08:48:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c064c20 con 0x7f257c0ab450 2022-07-12T08:48:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 0x7f257c064ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 0x7f257c064ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58558/0 (socket says 172.21.15.61:58558) 2022-07-12T08:48:31.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 -- 172.21.15.61:0/3488731158 learned_addr learned my addr 172.21.15.61:0/3488731158 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2581216700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2560003c80 0x7f257c145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2581a17700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0ab450 0x7f257c0ab8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 -- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0ab450 msgr2=0x7f257c0ab8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0ab450 0x7f257c0ab8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 -- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2560003c80 msgr2=0x7f257c145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:31.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2560003c80 0x7f257c145760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:31.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 -- 172.21.15.61:0/3488731158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c0653b0 con 0x7f257c09fb40 2022-07-12T08:48:31.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2580a15700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 0x7f257c064ec0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f25600054d0 tx=0x7f256c007630).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f256c00eb90 con 0x7f257c09fb40 2022-07-12T08:48:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c00e8e0 con 0x7f257c09fb40 2022-07-12T08:48:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c1511e0 con 0x7f257c09fb40 2022-07-12T08:48:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f256c011660 con 0x7f257c09fb40 2022-07-12T08:48:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.958+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c0abfe0 con 0x7f257c09fb40 2022-07-12T08:48:31.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.962+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c065860 con 0x7f257c09fb40 2022-07-12T08:48:31.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.962+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f256c0117c0 con 0x7f257c09fb40 2022-07-12T08:48:31.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.962+0000 7f25727fc700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25640539d0 0x7f2564055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:31.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.962+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f256c070090 con 0x7f257c09fb40 2022-07-12T08:48:31.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.962+0000 7f2581216700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25640539d0 0x7f2564055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:31.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.966+0000 7f2581216700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25640539d0 0x7f2564055e90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f257800b000 tx=0x7f2578008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:31.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:31.966+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f256c047360 con 0x7f257c09fb40 2022-07-12T08:48:32.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.094+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f257c1513d0 con 0x7f257c09fb40 2022-07-12T08:48:32.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.094+0000 7f25727fc700 1 -- 172.21.15.61:0/3488731158 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v86) v1 ==== 110+0+10 (secure 0 0 0) 0x7f256c04a940 con 0x7f257c09fb40 2022-07-12T08:48:32.102 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:32.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25640539d0 msgr2=0x7f2564055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25640539d0 0x7f2564055e90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f257800b000 tx=0x7f2578008040).stop 2022-07-12T08:48:32.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 msgr2=0x7f257c064ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 0x7f257c064ec0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f25600054d0 tx=0x7f256c007630).stop 2022-07-12T08:48:32.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 shutdown_connections 2022-07-12T08:48:32.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f257c0ab450 0x7f257c0ab8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f257c09fb40 0x7f257c064ec0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25640539d0 0x7f2564055e90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 --2- 172.21.15.61:0/3488731158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2560003c80 0x7f257c145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 >> 172.21.15.61:0/3488731158 conn(0x7f257c05f140 msgr2=0x7f257c0b0500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:32.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 shutdown_connections 2022-07-12T08:48:32.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.098+0000 7f2582218700 1 -- 172.21.15.61:0/3488731158 wait complete. 2022-07-12T08:48:32.114 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:32.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.222+0000 7f52d7a5b700 1 Processor -- start 2022-07-12T08:48:32.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- start start 2022-07-12T08:48:32.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52d00b0120 0x7f52d0144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d014aad0 0x7f52d014af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d00b0500 con 0x7f52d00ab730 2022-07-12T08:48:32.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d00b1040 con 0x7f52d00b0120 2022-07-12T08:48:32.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0144d50 con 0x7f52d014aad0 2022-07-12T08:48:32.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52d00b0120 0x7f52d0144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d5ff8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d014aad0 0x7f52d014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d4ff6700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f52d00ab730 0x7f52d0144640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:60386/0 (socket says 172.21.15.61:60386) 2022-07-12T08:48:32.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f52d00b0120 0x7f52d0144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:37138/0 (socket says 172.21.15.61:37138) 2022-07-12T08:48:32.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d4ff6700 1 -- 172.21.15.61:0/792450758 learned_addr learned my addr 172.21.15.61:0/792450758 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:32.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 -- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d014aad0 msgr2=0x7f52d014af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:32.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 --2- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d014aad0 0x7f52d014af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 -- 172.21.15.61:0/792450758 >> v1:172.21.15.111:6789/0 conn(0x7f52d00ab730 legacy=0x7f52d0144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 -- 172.21.15.61:0/792450758 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d005d0b0 con 0x7f52d00b0120 2022-07-12T08:48:32.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d57f7700 1 --2- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00b0120 0x7f52d0144150 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f52c00032f0 tx=0x7f52c00091d0).ready entity=mon.1 client_cookie=1460f4788120aae3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52cffff700 1 -- 172.21.15.61:0/792450758 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c000b6c0 con 0x7f52d00b0120 2022-07-12T08:48:32.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52cffff700 1 -- 172.21.15.61:0/792450758 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c000b820 con 0x7f52d00b0120 2022-07-12T08:48:32.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52cffff700 1 -- 172.21.15.61:0/792450758 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c000b990 con 0x7f52d00b0120 2022-07-12T08:48:32.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00b0120 msgr2=0x7f52d0144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00b0120 0x7f52d0144150 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f52c00032f0 tx=0x7f52c00091d0).stop 2022-07-12T08:48:32.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- 172.21.15.61:0/792450758 shutdown_connections 2022-07-12T08:48:32.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d014aad0 0x7f52d014af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/792450758 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00b0120 0x7f52d0144150 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- 172.21.15.61:0/792450758 >> 172.21.15.61:0/792450758 conn(0x7f52d00b5290 msgr2=0x7f52d00b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:32.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.226+0000 7f52d7a5b700 1 -- 172.21.15.61:0/792450758 shutdown_connections 2022-07-12T08:48:32.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 -- 172.21.15.61:0/792450758 wait complete. 2022-07-12T08:48:32.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 Processor -- start 2022-07-12T08:48:32.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 -- start start 2022-07-12T08:48:32.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00ab730 0x7f52d0143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d00b0120 0x7f52d0159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 0x7f52d009e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d009e220 con 0x7f52d014aad0 2022-07-12T08:48:32.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ae420 con 0x7f52d00ab730 2022-07-12T08:48:32.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d7a5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ace10 con 0x7f52d00b0120 2022-07-12T08:48:32.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 0x7f52d009e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d57f7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00ab730 0x7f52d0143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d4ff6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d00b0120 0x7f52d0159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 0x7f52d009e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58574/0 (socket says 172.21.15.61:58574) 2022-07-12T08:48:32.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 -- 172.21.15.61:0/3137480639 learned_addr learned my addr 172.21.15.61:0/3137480639 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:32.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 -- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d00b0120 msgr2=0x7f52d0159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d00b0120 0x7f52d0159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 -- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00ab730 msgr2=0x7f52d0143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00ab730 0x7f52d0143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 -- 172.21.15.61:0/3137480639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c0009470 con 0x7f52d014aad0 2022-07-12T08:48:32.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52d5ff8700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 0x7f52d009e650 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f52d00ae5b0 tx=0x7f52c8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.230+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c8011750 con 0x7f52d014aad0 2022-07-12T08:48:32.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c80118b0 con 0x7f52d014aad0 2022-07-12T08:48:32.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d00ae690 con 0x7f52d014aad0 2022-07-12T08:48:32.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52c8011c10 con 0x7f52d014aad0 2022-07-12T08:48:32.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d00ac870 con 0x7f52d014aad0 2022-07-12T08:48:32.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d00ad870 con 0x7f52d014aad0 2022-07-12T08:48:32.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f52c8011d70 con 0x7f52d014aad0 2022-07-12T08:48:32.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52ce7fc700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f52b0053a40 0x7f52b0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f52c806be20 con 0x7f52d014aad0 2022-07-12T08:48:32.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52d57f7700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f52b0053a40 0x7f52b0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.234+0000 7f52d57f7700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f52b0053a40 0x7f52b0055f00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f52c001b000 tx=0x7f52c0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.238+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f52d00ad870 con 0x7f52d014aad0 2022-07-12T08:48:32.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.366+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f52d00ad870 con 0x7f52d014aad0 2022-07-12T08:48:32.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.366+0000 7f52ce7fc700 1 -- 172.21.15.61:0/3137480639 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v86) v1 ==== 92+0+10 (secure 0 0 0) 0x7f52d00ad870 con 0x7f52d014aad0 2022-07-12T08:48:32.372 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:32.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f52b0053a40 msgr2=0x7f52b0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f52b0053a40 0x7f52b0055f00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f52c001b000 tx=0x7f52c0016000).stop 2022-07-12T08:48:32.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 msgr2=0x7f52d009e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 0x7f52d009e650 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7f52d00ae5b0 tx=0x7f52c8006df0).stop 2022-07-12T08:48:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 shutdown_connections 2022-07-12T08:48:32.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52d014aad0 0x7f52d009e650 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f52d00b0120 0x7f52d0159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f52b0053a40 0x7f52b0055f00 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 --2- 172.21.15.61:0/3137480639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52d00ab730 0x7f52d0143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 >> 172.21.15.61:0/3137480639 conn(0x7f52d00b5290 msgr2=0x7f52d014af10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:32.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 shutdown_connections 2022-07-12T08:48:32.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.370+0000 7f52d7a5b700 1 -- 172.21.15.61:0/3137480639 wait complete. 2022-07-12T08:48:32.385 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:48:32.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 Processor -- start 2022-07-12T08:48:32.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- start start 2022-07-12T08:48:32.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf280b01f0 0x7fcf280ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcf2814a8f0 0x7fcf281443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf280ad2f0 con 0x7fcf2805c050 2022-07-12T08:48:32.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf280b0a40 con 0x7fcf2814a8f0 2022-07-12T08:48:32.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2805c430 con 0x7fcf280b01f0 2022-07-12T08:48:32.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcf2814a8f0 0x7fcf281443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcf2814a8f0 0x7fcf281443c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58582/0 (socket says 172.21.15.61:58582) 2022-07-12T08:48:32.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 -- 172.21.15.61:0/864599869 learned_addr learned my addr 172.21.15.61:0/864599869 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:32.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2d844700 1 --2- 172.21.15.61:0/864599869 >> v2:172.21.15.111:3300/0 conn(0x7fcf280b01f0 0x7fcf280ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 -- 172.21.15.61:0/864599869 >> v2:172.21.15.111:3300/0 conn(0x7fcf280b01f0 msgr2=0x7fcf280ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:32.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 --2- 172.21.15.61:0/864599869 >> v2:172.21.15.111:3300/0 conn(0x7fcf280b01f0 0x7fcf280ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 -- 172.21.15.61:0/864599869 >> v1:172.21.15.111:6789/0 conn(0x7fcf2805c050 legacy=0x7fcf2805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:32.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 -- 172.21.15.61:0/864599869 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf280b2420 con 0x7fcf2814a8f0 2022-07-12T08:48:32.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2d844700 1 --2- 172.21.15.61:0/864599869 >> v2:172.21.15.111:3300/0 conn(0x7fcf280b01f0 0x7fcf280ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:32.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2e846700 1 --2- 172.21.15.61:0/864599869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2814a8f0 0x7fcf281443c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7fcf24002f70 tx=0x7fcf24006eb0).ready entity=mon.0 client_cookie=85acd79c964dbd66 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2d043700 1 -- 172.21.15.61:0/864599869 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf2400d610 con 0x7fcf2814a8f0 2022-07-12T08:48:32.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2d043700 1 -- 172.21.15.61:0/864599869 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf24000e90 con 0x7fcf2814a8f0 2022-07-12T08:48:32.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf2d043700 1 -- 172.21.15.61:0/864599869 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf2400d9a0 con 0x7fcf2814a8f0 2022-07-12T08:48:32.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- 172.21.15.61:0/864599869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2814a8f0 msgr2=0x7fcf281443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 --2- 172.21.15.61:0/864599869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2814a8f0 0x7fcf281443c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 rx=0x7fcf24002f70 tx=0x7fcf24006eb0).stop 2022-07-12T08:48:32.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- 172.21.15.61:0/864599869 shutdown_connections 2022-07-12T08:48:32.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 --2- 172.21.15.61:0/864599869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2814a8f0 0x7fcf281443c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 --2- 172.21.15.61:0/864599869 >> v2:172.21.15.111:3300/0 conn(0x7fcf280b01f0 0x7fcf280ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- 172.21.15.61:0/864599869 >> 172.21.15.61:0/864599869 conn(0x7fcf28062c00 msgr2=0x7fcf28062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:32.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- 172.21.15.61:0/864599869 shutdown_connections 2022-07-12T08:48:32.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.482+0000 7fcf302a9700 1 -- 172.21.15.61:0/864599869 wait complete. 2022-07-12T08:48:32.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 Processor -- start 2022-07-12T08:48:32.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 -- start start 2022-07-12T08:48:32.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 0x7fcf28141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf280b01f0 0x7fcf28141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf2814a8f0 0x7fcf28143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2815a930 con 0x7fcf2805c050 2022-07-12T08:48:32.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2815a690 con 0x7fcf2814a8f0 2022-07-12T08:48:32.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf28144310 con 0x7fcf280b01f0 2022-07-12T08:48:32.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 0x7fcf28141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 0x7fcf28141390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58584/0 (socket says 172.21.15.61:58584) 2022-07-12T08:48:32.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 -- 172.21.15.61:0/2258930731 learned_addr learned my addr 172.21.15.61:0/2258930731 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:32.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2d844700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf280b01f0 0x7fcf28141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e846700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf2814a8f0 0x7fcf28143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 -- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf280b01f0 msgr2=0x7fcf28141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf280b01f0 0x7fcf28141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 -- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf2814a8f0 msgr2=0x7fcf28143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf2814a8f0 0x7fcf28143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 -- 172.21.15.61:0/2258930731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2400b1e0 con 0x7fcf2805c050 2022-07-12T08:48:32.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf2e045700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 0x7fcf28141390 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fcf280b62e0 tx=0x7fcf1c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf1c012400 con 0x7fcf2805c050 2022-07-12T08:48:32.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c012560 con 0x7fcf2805c050 2022-07-12T08:48:32.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf1c0128c0 con 0x7fcf2805c050 2022-07-12T08:48:32.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf281447b0 con 0x7fcf2805c050 2022-07-12T08:48:32.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.486+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf280ae710 con 0x7fcf2805c050 2022-07-12T08:48:32.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.490+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2815a4a0 con 0x7fcf2805c050 2022-07-12T08:48:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.490+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcf1c019030 con 0x7fcf2805c050 2022-07-12T08:48:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.490+0000 7fcf1b7fe700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf08053ad0 0x7fcf08055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.490+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcf1c06ae90 con 0x7fcf2805c050 2022-07-12T08:48:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.490+0000 7fcf2d844700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf08053ad0 0x7fcf08055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.490+0000 7fcf2d844700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf08053ad0 0x7fcf08055f90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fcf2000d000 tx=0x7fcf20008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.498+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf1c0420c0 con 0x7fcf2805c050 2022-07-12T08:48:32.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcf28057460 con 0x7fcf2805c050 2022-07-12T08:48:32.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf1b7fe700 1 -- 172.21.15.61:0/2258930731 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) v1 ==== 84+0+143 (secure 0 0 0) 0x7fcf1c0456e0 con 0x7fcf2805c050 2022-07-12T08:48:32.633 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:32.633 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:48:32.634 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:32.635 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:48:32.635 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:48:32.636 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:32.636 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:32.637 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:48:32.637 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:48:32.638 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:32.639 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:48:32.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf08053ad0 msgr2=0x7fcf08055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf08053ad0 0x7fcf08055f90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fcf2000d000 tx=0x7fcf20008040).stop 2022-07-12T08:48:32.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 msgr2=0x7fcf28141390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 0x7fcf28141390 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7fcf280b62e0 tx=0x7fcf1c007c50).stop 2022-07-12T08:48:32.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 shutdown_connections 2022-07-12T08:48:32.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf08053ad0 0x7fcf08055f90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf2814a8f0 0x7fcf28143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf280b01f0 0x7fcf28141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 --2- 172.21.15.61:0/2258930731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf2805c050 0x7fcf28141390 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.626+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 >> 172.21.15.61:0/2258930731 conn(0x7fcf28062c00 msgr2=0x7fcf28146130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:32.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.630+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 shutdown_connections 2022-07-12T08:48:32.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.630+0000 7fcf302a9700 1 -- 172.21.15.61:0/2258930731 wait complete. 2022-07-12T08:48:32.651 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:48:32.652 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:48:32.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 Processor -- start 2022-07-12T08:48:32.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 -- start start 2022-07-12T08:48:32.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac13fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1aac1388d0 0x7f1aac138d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aac0b04e0 con 0x7f1aac061ac0 2022-07-12T08:48:32.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aac13f2c0 con 0x7f1aac1388d0 2022-07-12T08:48:32.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab2ebc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac0616e0 con 0x7f1aac061300 2022-07-12T08:48:32.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab1459700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1aac1388d0 0x7f1aac138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac13fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac13fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40788/0 (socket says 172.21.15.61:40788) 2022-07-12T08:48:32.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 -- 172.21.15.61:0/3657823654 learned_addr learned my addr 172.21.15.61:0/3657823654 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:32.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 -- 172.21.15.61:0/3657823654 >> v2:172.21.15.61:3300/0 conn(0x7f1aac1388d0 msgr2=0x7f1aac138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:32.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 --2- 172.21.15.61:0/3657823654 >> v2:172.21.15.61:3300/0 conn(0x7f1aac1388d0 0x7f1aac138d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 -- 172.21.15.61:0/3657823654 >> v1:172.21.15.111:6789/0 conn(0x7f1aac061ac0 legacy=0x7f1aac0b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab0c58700 1 -- 172.21.15.61:0/3657823654 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac13f4b0 con 0x7f1aac061300 2022-07-12T08:48:32.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.778+0000 7f1ab1459700 1 --2- 172.21.15.61:0/3657823654 >> v2:172.21.15.61:3300/0 conn(0x7f1aac1388d0 0x7f1aac138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:32.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab0c58700 1 --2- 172.21.15.61:0/3657823654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac13fbf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1a9c0032f0 tx=0x7f1a9c009290).ready entity=mon.2 client_cookie=a23b24eb290023e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1aab7fe700 1 -- 172.21.15.61:0/3657823654 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a9c00b8a0 con 0x7f1aac061300 2022-07-12T08:48:32.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1aab7fe700 1 -- 172.21.15.61:0/3657823654 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9c00ba00 con 0x7f1aac061300 2022-07-12T08:48:32.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1aab7fe700 1 -- 172.21.15.61:0/3657823654 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a9c00bb70 con 0x7f1aac061300 2022-07-12T08:48:32.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/3657823654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 msgr2=0x7f1aac13fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/3657823654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac13fbf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1a9c0032f0 tx=0x7f1a9c009290).stop 2022-07-12T08:48:32.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/3657823654 shutdown_connections 2022-07-12T08:48:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/3657823654 >> v2:172.21.15.61:3300/0 conn(0x7f1aac1388d0 0x7f1aac138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/3657823654 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac13fbf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/3657823654 >> 172.21.15.61:0/3657823654 conn(0x7f1aac0b5a40 msgr2=0x7f1aac0b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/3657823654 shutdown_connections 2022-07-12T08:48:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/3657823654 wait complete. 2022-07-12T08:48:32.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 Processor -- start 2022-07-12T08:48:32.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.782+0000 7f1ab2ebc700 1 -- start start 2022-07-12T08:48:32.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac0b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac061ac0 0x7f1aac0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 0x7f1aac0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac0b25d0 con 0x7f1aac1388d0 2022-07-12T08:48:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac0a9ca0 con 0x7f1aac061ac0 2022-07-12T08:48:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac0a9f40 con 0x7f1aac061300 2022-07-12T08:48:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 0x7f1aac0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 0x7f1aac0b20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58602/0 (socket says 172.21.15.61:58602) 2022-07-12T08:48:32.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab0c58700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac0b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 -- 172.21.15.61:0/1124419409 learned_addr learned my addr 172.21.15.61:0/1124419409 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:32.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1aabfff700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac061ac0 0x7f1aac0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 -- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 msgr2=0x7f1aac0b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac0b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 -- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac061ac0 msgr2=0x7f1aac0b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:32.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac061ac0 0x7f1aac0b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:32.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 -- 172.21.15.61:0/1124419409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa400b180 con 0x7f1aac1388d0 2022-07-12T08:48:32.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab1459700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 0x7f1aac0b20e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f1aa400c000 tx=0x7f1aa4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa4010780 con 0x7f1aac1388d0 2022-07-12T08:48:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c009530 con 0x7f1aac1388d0 2022-07-12T08:48:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa40108e0 con 0x7f1aac1388d0 2022-07-12T08:48:32.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa4010c40 con 0x7f1aac1388d0 2022-07-12T08:48:32.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac0ac0b0 con 0x7f1aac1388d0 2022-07-12T08:48:32.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.786+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac057460 con 0x7f1aac1388d0 2022-07-12T08:48:32.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.790+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1aa401c460 con 0x7f1aac1388d0 2022-07-12T08:48:32.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.790+0000 7f1aa9ffb700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a90053ad0 0x7f1a90055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:32.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.790+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f1aa406c080 con 0x7f1aac1388d0 2022-07-12T08:48:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.790+0000 7f1ab0c58700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a90053ad0 0x7f1a90055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.790+0000 7f1ab0c58700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a90053ad0 0x7f1a90055f90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1a9c01b000 tx=0x7f1a9c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.794+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1aa4043210 con 0x7f1aac1388d0 2022-07-12T08:48:32.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:32.918+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f1aac06ac10 con 0x7f1aac1388d0 2022-07-12T08:48:33.228 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:33.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:33.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:33.238 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:33.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:48:33.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.374+0000 7f1aa9ffb700 1 -- 172.21.15.61:0/1124419409 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v87) v1 ==== 133+0+0 (secure 0 0 0) 0x7f1aa4046830 con 0x7f1aac1388d0 2022-07-12T08:48:33.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a90053ad0 msgr2=0x7f1a90055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:33.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a90053ad0 0x7f1a90055f90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1a9c01b000 tx=0x7f1a9c016000).stop 2022-07-12T08:48:33.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 msgr2=0x7f1aac0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:33.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 0x7f1aac0b20e0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7f1aa400c000 tx=0x7f1aa4006df0).stop 2022-07-12T08:48:33.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 shutdown_connections 2022-07-12T08:48:33.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aac1388d0 0x7f1aac0b20e0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a90053ad0 0x7f1a90055f90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac061ac0 0x7f1aac0b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 --2- 172.21.15.61:0/1124419409 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1aac061300 0x7f1aac0b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.378+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 >> 172.21.15.61:0/1124419409 conn(0x7f1aac0b5a40 msgr2=0x7f1aac13fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:33.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.382+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 shutdown_connections 2022-07-12T08:48:33.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.382+0000 7f1ab2ebc700 1 -- 172.21.15.61:0/1124419409 wait complete. 2022-07-12T08:48:33.388 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:48:33.401 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:48:33.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 Processor -- start 2022-07-12T08:48:33.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 -- start start 2022-07-12T08:48:33.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1540b7b50 0x7fa154066290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:33.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa154142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:33.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa154065b90 con 0x7fa1540b1eb0 2022-07-12T08:48:33.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1540b7f30 con 0x7fa1540b7b50 2022-07-12T08:48:33.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa153fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa154066780 con 0x7fa15405bd10 2022-07-12T08:48:33.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa152ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1540b7b50 0x7fa154066290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:33.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa1527fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa1540b1eb0 0x7fa154145f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46824/0 (socket says 172.21.15.61:46824) 2022-07-12T08:48:33.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa1527fc700 1 -- 172.21.15.61:0/3116270226 learned_addr learned my addr 172.21.15.61:0/3116270226 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:33.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa1537fe700 1 --2- 172.21.15.61:0/3116270226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa154142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa152ffd700 1 -- 172.21.15.61:0/3116270226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 msgr2=0x7fa154142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa152ffd700 1 --2- 172.21.15.61:0/3116270226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa154142ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa152ffd700 1 -- 172.21.15.61:0/3116270226 >> v1:172.21.15.61:6789/0 conn(0x7fa1540b1eb0 legacy=0x7fa154145f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa152ffd700 1 -- 172.21.15.61:0/3116270226 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15405d1c0 con 0x7fa1540b7b50 2022-07-12T08:48:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa152ffd700 1 --2- 172.21.15.61:0/3116270226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b7b50 0x7fa154066290 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7fa13c0032f0 tx=0x7fa13c0091d0).ready entity=mon.0 client_cookie=3d35be48b1e91e30 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.514+0000 7fa151ffb700 1 -- 172.21.15.61:0/3116270226 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa13c00b6c0 con 0x7fa1540b7b50 2022-07-12T08:48:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa151ffb700 1 -- 172.21.15.61:0/3116270226 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13c00b820 con 0x7fa1540b7b50 2022-07-12T08:48:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa151ffb700 1 -- 172.21.15.61:0/3116270226 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa13c00b990 con 0x7fa1540b7b50 2022-07-12T08:48:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- 172.21.15.61:0/3116270226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b7b50 msgr2=0x7fa154066290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 --2- 172.21.15.61:0/3116270226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b7b50 0x7fa154066290 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 rx=0x7fa13c0032f0 tx=0x7fa13c0091d0).stop 2022-07-12T08:48:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- 172.21.15.61:0/3116270226 shutdown_connections 2022-07-12T08:48:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 --2- 172.21.15.61:0/3116270226 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa154142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 --2- 172.21.15.61:0/3116270226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b7b50 0x7fa154066290 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- 172.21.15.61:0/3116270226 >> 172.21.15.61:0/3116270226 conn(0x7fa154065650 msgr2=0x7fa1540b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:33.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- 172.21.15.61:0/3116270226 shutdown_connections 2022-07-12T08:48:33.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- 172.21.15.61:0/3116270226 wait complete. 2022-07-12T08:48:33.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 Processor -- start 2022-07-12T08:48:33.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- start start 2022-07-12T08:48:33.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa1541556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:33.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 0x7fa1540acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:33.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1540b7b50 0x7fa1541149a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1540b05b0 con 0x7fa1540b1eb0 2022-07-12T08:48:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa154146730 con 0x7fa1540b7b50 2022-07-12T08:48:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa153fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1541469d0 con 0x7fa15405bd10 2022-07-12T08:48:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa1527fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 0x7fa1540acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa152ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa1541556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa1527fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 0x7fa1540acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58616/0 (socket says 172.21.15.61:58616) 2022-07-12T08:48:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa1527fc700 1 -- 172.21.15.61:0/1877505787 learned_addr learned my addr 172.21.15.61:0/1877505787 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.518+0000 7fa1537fe700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1540b7b50 0x7fa1541149a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa1527fc700 1 -- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 msgr2=0x7fa1541556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa1527fc700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa1541556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa1527fc700 1 -- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1540b7b50 msgr2=0x7fa1541149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa1527fc700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1540b7b50 0x7fa1541149a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa1527fc700 1 -- 172.21.15.61:0/1877505787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13c009470 con 0x7fa1540b1eb0 2022-07-12T08:48:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa1527fc700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 0x7fa1540acea0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7fa1540648a0 tx=0x7fa144007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1440070d0 con 0x7fa1540b1eb0 2022-07-12T08:48:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa144007230 con 0x7fa1540b1eb0 2022-07-12T08:48:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa154146c70 con 0x7fa1540b1eb0 2022-07-12T08:48:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1440055d0 con 0x7fa1540b1eb0 2022-07-12T08:48:33.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1540ae240 con 0x7fa1540b1eb0 2022-07-12T08:48:33.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1540affa0 con 0x7fa1540b1eb0 2022-07-12T08:48:33.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa144010070 con 0x7fa1540b1eb0 2022-07-12T08:48:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa14bfff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa138053a60 0x7fa138055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fa14406b1b0 con 0x7fa1540b1eb0 2022-07-12T08:48:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.522+0000 7fa152ffd700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa138053a60 0x7fa138055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:33.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.526+0000 7fa152ffd700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa138053a60 0x7fa138055f20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fa13c01b000 tx=0x7fa13c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:33.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.530+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1440422e0 con 0x7fa1540b1eb0 2022-07-12T08:48:33.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:33.658+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fa154057460 con 0x7fa1540b1eb0 2022-07-12T08:48:34.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.162+0000 7fa14bfff700 1 -- 172.21.15.61:0/1877505787 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v88) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa144018070 con 0x7fa1540b1eb0 2022-07-12T08:48:34.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa138053a60 msgr2=0x7fa138055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa138053a60 0x7fa138055f20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fa13c01b000 tx=0x7fa13c016000).stop 2022-07-12T08:48:34.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 msgr2=0x7fa1540acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 0x7fa1540acea0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7fa1540648a0 tx=0x7fa144007c30).stop 2022-07-12T08:48:34.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 shutdown_connections 2022-07-12T08:48:34.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa138053a60 0x7fa138055f20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa1540b7b50 0x7fa1541149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1540b1eb0 0x7fa1540acea0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 --2- 172.21.15.61:0/1877505787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa15405bd10 0x7fa1541556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 >> 172.21.15.61:0/1877505787 conn(0x7fa154065650 msgr2=0x7fa154066950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:34.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 shutdown_connections 2022-07-12T08:48:34.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.170+0000 7fa153fff700 1 -- 172.21.15.61:0/1877505787 wait complete. 2022-07-12T08:48:34.180 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:48:34.187 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:48:34.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 Processor -- start 2022-07-12T08:48:34.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 -- start start 2022-07-12T08:48:34.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90240b34c0 0x7f9024061a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9024143710 0x7f9024141a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90240b38a0 con 0x7f90240b2300 2022-07-12T08:48:34.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90240b2930 con 0x7f90240b34c0 2022-07-12T08:48:34.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902b316700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90240b2bd0 con 0x7f9024143710 2022-07-12T08:48:34.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90240b34c0 0x7f9024061a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90240b34c0 0x7f9024061a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58620/0 (socket says 172.21.15.61:58620) 2022-07-12T08:48:34.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 -- 172.21.15.61:0/1757748215 learned_addr learned my addr 172.21.15.61:0/1757748215 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:34.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902ab15700 1 --2- 172.21.15.61:0/1757748215 >> v2:172.21.15.111:3300/0 conn(0x7f9024143710 0x7f9024141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 -- 172.21.15.61:0/1757748215 >> v2:172.21.15.111:3300/0 conn(0x7f9024143710 msgr2=0x7f9024141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:34.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 --2- 172.21.15.61:0/1757748215 >> v2:172.21.15.111:3300/0 conn(0x7f9024143710 0x7f9024141a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 -- 172.21.15.61:0/1757748215 >> v1:172.21.15.111:6789/0 conn(0x7f90240b2300 legacy=0x7f9024143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 -- 172.21.15.61:0/1757748215 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024143ca0 con 0x7f90240b34c0 2022-07-12T08:48:34.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f902a314700 1 --2- 172.21.15.61:0/1757748215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90240b34c0 0x7f9024061a70 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f9020002f70 tx=0x7f9020006eb0).ready entity=mon.0 client_cookie=2786310e97e54588 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f9029312700 1 -- 172.21.15.61:0/1757748215 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f902000d5c0 con 0x7f90240b34c0 2022-07-12T08:48:34.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f9029312700 1 -- 172.21.15.61:0/1757748215 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9020000e90 con 0x7f90240b34c0 2022-07-12T08:48:34.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.294+0000 7f9029312700 1 -- 172.21.15.61:0/1757748215 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f902000d950 con 0x7f90240b34c0 2022-07-12T08:48:34.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- 172.21.15.61:0/1757748215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90240b34c0 msgr2=0x7f9024061a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 --2- 172.21.15.61:0/1757748215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90240b34c0 0x7f9024061a70 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f9020002f70 tx=0x7f9020006eb0).stop 2022-07-12T08:48:34.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- 172.21.15.61:0/1757748215 shutdown_connections 2022-07-12T08:48:34.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 --2- 172.21.15.61:0/1757748215 >> v2:172.21.15.111:3300/0 conn(0x7f9024143710 0x7f9024141a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 --2- 172.21.15.61:0/1757748215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90240b34c0 0x7f9024061a70 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- 172.21.15.61:0/1757748215 >> 172.21.15.61:0/1757748215 conn(0x7f90240b46d0 msgr2=0x7f902405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:34.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- 172.21.15.61:0/1757748215 shutdown_connections 2022-07-12T08:48:34.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- 172.21.15.61:0/1757748215 wait complete. 2022-07-12T08:48:34.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 Processor -- start 2022-07-12T08:48:34.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- start start 2022-07-12T08:48:34.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90240b2300 0x7f9024145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 0x7f90241526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 0x7f9024152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902413b040 con 0x7f9024143710 2022-07-12T08:48:34.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90240aeac0 con 0x7f90240b34c0 2022-07-12T08:48:34.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902b316700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90240aed60 con 0x7f90240b2300 2022-07-12T08:48:34.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.298+0000 7f902ab15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 0x7f9024152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902a314700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90240b2300 0x7f9024145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 0x7f90241526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902ab15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 0x7f9024152bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58630/0 (socket says 172.21.15.61:58630) 2022-07-12T08:48:34.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902ab15700 1 -- 172.21.15.61:0/1764391668 learned_addr learned my addr 172.21.15.61:0/1764391668 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:34.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902a314700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90240b2300 0x7f9024145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40822/0 (socket says 172.21.15.61:40822) 2022-07-12T08:48:34.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 0x7f90241526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:37202/0 (socket says 172.21.15.61:37202) 2022-07-12T08:48:34.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 -- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90240b2300 msgr2=0x7f9024145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90240b2300 0x7f9024145760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 -- 172.21.15.61:0/1764391668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 msgr2=0x7f9024152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 0x7f9024152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 -- 172.21.15.61:0/1764391668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902000b1e0 con 0x7f90240b34c0 2022-07-12T08:48:34.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902ab15700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 0x7f9024152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:34.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f9029b13700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 0x7f90241526e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9024061020 tx=0x7f9014007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9014012400 con 0x7f90240b34c0 2022-07-12T08:48:34.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9014012560 con 0x7f90240b34c0 2022-07-12T08:48:34.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90140128c0 con 0x7f90240b34c0 2022-07-12T08:48:34.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90240af000 con 0x7f90240b34c0 2022-07-12T08:48:34.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90240ae620 con 0x7f90240b34c0 2022-07-12T08:48:34.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90240ae930 con 0x7f90240b34c0 2022-07-12T08:48:34.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.302+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9014019030 con 0x7f90240b34c0 2022-07-12T08:48:34.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.306+0000 7f90137fe700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9004053920 0x7f9004055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.306+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f901406b120 con 0x7f90240b34c0 2022-07-12T08:48:34.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.306+0000 7f902a314700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9004053920 0x7f9004055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.306+0000 7f902a314700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9004053920 0x7f9004055de0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f902001c000 tx=0x7f9020015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.314+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9014042050 con 0x7f90240b34c0 2022-07-12T08:48:34.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.482+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f90240b5a60 con 0x7f90240b34c0 2022-07-12T08:48:34.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f90137fe700 1 -- 172.21.15.61:0/1764391668 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 17 and data pool 18 v49) v1 ==== 189+0+0 (secure 0 0 0) 0x7f9014045670 con 0x7f90240b34c0 2022-07-12T08:48:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9004053920 msgr2=0x7f9004055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9004053920 0x7f9004055de0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f902001c000 tx=0x7f9020015000).stop 2022-07-12T08:48:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 msgr2=0x7f90241526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 0x7f90241526e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f9024061020 tx=0x7f9014007b90).stop 2022-07-12T08:48:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 shutdown_connections 2022-07-12T08:48:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9024143710 0x7f9024152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9004053920 0x7f9004055de0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f90240b34c0 0x7f90241526e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 --2- 172.21.15.61:0/1764391668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f90240b2300 0x7f9024145760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 >> 172.21.15.61:0/1764391668 conn(0x7f90240b46d0 msgr2=0x7f90240b5c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 shutdown_connections 2022-07-12T08:48:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.498+0000 7f902b316700 1 -- 172.21.15.61:0/1764391668 wait complete. 2022-07-12T08:48:34.510 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 17 and data pool 18 2022-07-12T08:48:34.519 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:34.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 Processor -- start 2022-07-12T08:48:34.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 -- start start 2022-07-12T08:48:34.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f83300ab290 0x7f83300ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f833005f140 0x7f833005ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83300abc70 con 0x7f83300afa60 2022-07-12T08:48:34.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833005f580 con 0x7f83300ab290 2022-07-12T08:48:34.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833005c720 con 0x7f833005f140 2022-07-12T08:48:34.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f83300ab290 0x7f83300ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f83300afa60 0x7f833014a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46856/0 (socket says 172.21.15.61:46856) 2022-07-12T08:48:34.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f83300ab290 0x7f83300ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58636/0 (socket says 172.21.15.61:58636) 2022-07-12T08:48:34.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832f7fe700 1 -- 172.21.15.61:0/4279026160 learned_addr learned my addr 172.21.15.61:0/4279026160 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:34.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832e7fc700 1 --2- 172.21.15.61:0/4279026160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f833005f140 0x7f833005ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 -- 172.21.15.61:0/4279026160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f833005f140 msgr2=0x7f833005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:34.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 --2- 172.21.15.61:0/4279026160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f833005f140 0x7f833005ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 -- 172.21.15.61:0/4279026160 >> v1:172.21.15.61:6789/0 conn(0x7f83300afa60 legacy=0x7f833014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832dffb700 1 -- 172.21.15.61:0/4279026160 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348577741 0 0) 0x7f83300abc70 con 0x7f83300afa60 2022-07-12T08:48:34.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 -- 172.21.15.61:0/4279026160 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833005c9c0 con 0x7f83300ab290 2022-07-12T08:48:34.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832effd700 1 --2- 172.21.15.61:0/4279026160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83300ab290 0x7f83300ab670 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f8324002f70 tx=0x7f8324006df0).ready entity=mon.0 client_cookie=a521070aa2291f1b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832dffb700 1 -- 172.21.15.61:0/4279026160 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832400d570 con 0x7f83300ab290 2022-07-12T08:48:34.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832dffb700 1 -- 172.21.15.61:0/4279026160 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324000e90 con 0x7f83300ab290 2022-07-12T08:48:34.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832dffb700 1 -- 172.21.15.61:0/4279026160 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f832400d8c0 con 0x7f83300ab290 2022-07-12T08:48:34.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 -- 172.21.15.61:0/4279026160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83300ab290 msgr2=0x7f83300ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.614+0000 7f832ffff700 1 --2- 172.21.15.61:0/4279026160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83300ab290 0x7f83300ab670 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7f8324002f70 tx=0x7f8324006df0).stop 2022-07-12T08:48:34.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- 172.21.15.61:0/4279026160 shutdown_connections 2022-07-12T08:48:34.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 --2- 172.21.15.61:0/4279026160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f833005f140 0x7f833005ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 --2- 172.21.15.61:0/4279026160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83300ab290 0x7f83300ab670 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- 172.21.15.61:0/4279026160 >> 172.21.15.61:0/4279026160 conn(0x7f83300b1950 msgr2=0x7f83300b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:34.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- 172.21.15.61:0/4279026160 shutdown_connections 2022-07-12T08:48:34.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- 172.21.15.61:0/4279026160 wait complete. 2022-07-12T08:48:34.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 Processor -- start 2022-07-12T08:48:34.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- start start 2022-07-12T08:48:34.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 0x7f83301488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83300ab290 0x7f8330099900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83300afa60 0x7f8330155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301592b0 con 0x7f833005f140 2022-07-12T08:48:34.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8330066a50 con 0x7f83300afa60 2022-07-12T08:48:34.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8330066cf0 con 0x7f83300ab290 2022-07-12T08:48:34.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 0x7f83301488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 0x7f83301488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58642/0 (socket says 172.21.15.61:58642) 2022-07-12T08:48:34.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 -- 172.21.15.61:0/468346102 learned_addr learned my addr 172.21.15.61:0/468346102 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:34.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832e7fc700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83300ab290 0x7f8330099900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832f7fe700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83300afa60 0x7f8330155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 -- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83300ab290 msgr2=0x7f8330099900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83300ab290 0x7f8330099900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 -- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83300afa60 msgr2=0x7f8330155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83300afa60 0x7f8330155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 -- 172.21.15.61:0/468346102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832400b180 con 0x7f833005f140 2022-07-12T08:48:34.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832e7fc700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83300ab290 0x7f8330099900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:34.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.618+0000 7f832effd700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 0x7f83301488c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f8324002f70 tx=0x7f8324014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8324016070 con 0x7f833005f140 2022-07-12T08:48:34.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324013a70 con 0x7f833005f140 2022-07-12T08:48:34.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83300666d0 con 0x7f833005f140 2022-07-12T08:48:34.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8324013dd0 con 0x7f833005f140 2022-07-12T08:48:34.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8330067410 con 0x7f833005f140 2022-07-12T08:48:34.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83301593f0 con 0x7f833005f140 2022-07-12T08:48:34.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8324025030 con 0x7f833005f140 2022-07-12T08:48:34.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f831ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f830c0539b0 0x7f830c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f832406cf20 con 0x7f833005f140 2022-07-12T08:48:34.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.622+0000 7f832e7fc700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f830c0539b0 0x7f830c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.626+0000 7f832e7fc700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f830c0539b0 0x7f830c055e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f832800b000 tx=0x7f8328008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.630+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8324043e80 con 0x7f833005f140 2022-07-12T08:48:34.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.754+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f833006b2b0 con 0x7f833005f140 2022-07-12T08:48:34.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.754+0000 7f831ffff700 1 -- 172.21.15.61:0/468346102 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f8324023070 con 0x7f833005f140 2022-07-12T08:48:34.759 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:34.760 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":89,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:34.488995+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:32.975784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:33.715444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:34.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.754+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f830c0539b0 msgr2=0x7f830c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.754+0000 7f832ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f830c0539b0 0x7f830c055e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f832800b000 tx=0x7f8328008040).stop 2022-07-12T08:48:34.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.754+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 msgr2=0x7f83301488c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.754+0000 7f832ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 0x7f83301488c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7f8324002f70 tx=0x7f8324014000).stop 2022-07-12T08:48:34.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 shutdown_connections 2022-07-12T08:48:34.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f830c0539b0 0x7f830c055e70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83300afa60 0x7f8330155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83300ab290 0x7f8330099900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 --2- 172.21.15.61:0/468346102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f833005f140 0x7f83301488c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 >> 172.21.15.61:0/468346102 conn(0x7f83300b1950 msgr2=0x7f83300b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:34.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 shutdown_connections 2022-07-12T08:48:34.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.758+0000 7f832ffff700 1 -- 172.21.15.61:0/468346102 wait complete. 2022-07-12T08:48:34.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:34.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.870+0000 7f854858a700 1 Processor -- start 2022-07-12T08:48:34.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.870+0000 7f854858a700 1 -- start start 2022-07-12T08:48:34.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f854014b370 0x7f85400afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854005fb90 con 0x7f85400b2af0 2022-07-12T08:48:34.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8540060060 con 0x7f85400b0e90 2022-07-12T08:48:34.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85400b1930 con 0x7f854014b370 2022-07-12T08:48:34.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546326700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f85400b2af0 0x7f8540060250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46868/0 (socket says 172.21.15.61:46868) 2022-07-12T08:48:34.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546326700 1 -- 172.21.15.61:0/264003902 learned_addr learned my addr 172.21.15.61:0/264003902 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:34.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546b27700 1 --2- 172.21.15.61:0/264003902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f854014b370 0x7f85400afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8545324700 1 -- 172.21.15.61:0/264003902 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655167796 0 0) 0x7f854005fb90 con 0x7f85400b2af0 2022-07-12T08:48:34.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8545324700 1 -- 172.21.15.61:0/264003902 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8528003260 con 0x7f85400b2af0 2022-07-12T08:48:34.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546b27700 1 -- 172.21.15.61:0/264003902 >> v1:172.21.15.111:6789/0 conn(0x7f85400b0e90 legacy=0x7f85400af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546b27700 1 -- 172.21.15.61:0/264003902 >> v1:172.21.15.61:6789/0 conn(0x7f85400b2af0 legacy=0x7f8540060250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546b27700 1 -- 172.21.15.61:0/264003902 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854005fdc0 con 0x7f854014b370 2022-07-12T08:48:34.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8546b27700 1 --2- 172.21.15.61:0/264003902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f854014b370 0x7f85400afd20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f85380032f0 tx=0x7f85380091d0).ready entity=mon.2 client_cookie=96cf1ab11288b066 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8545324700 1 -- 172.21.15.61:0/264003902 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853800b6c0 con 0x7f854014b370 2022-07-12T08:48:34.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8545324700 1 -- 172.21.15.61:0/264003902 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853800b820 con 0x7f854014b370 2022-07-12T08:48:34.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f8545324700 1 -- 172.21.15.61:0/264003902 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853800b990 con 0x7f854014b370 2022-07-12T08:48:34.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- 172.21.15.61:0/264003902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f854014b370 msgr2=0x7f85400afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 --2- 172.21.15.61:0/264003902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f854014b370 0x7f85400afd20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f85380032f0 tx=0x7f85380091d0).stop 2022-07-12T08:48:34.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- 172.21.15.61:0/264003902 shutdown_connections 2022-07-12T08:48:34.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 --2- 172.21.15.61:0/264003902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f854014b370 0x7f85400afd20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- 172.21.15.61:0/264003902 >> 172.21.15.61:0/264003902 conn(0x7f8540063bd0 msgr2=0x7f8540063fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:34.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- 172.21.15.61:0/264003902 shutdown_connections 2022-07-12T08:48:34.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.874+0000 7f854858a700 1 -- 172.21.15.61:0/264003902 wait complete. 2022-07-12T08:48:34.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 Processor -- start 2022-07-12T08:48:34.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 -- start start 2022-07-12T08:48:34.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 0x7f8540065ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85400b2af0 0x7f8540156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f854014b370 0x7f85400b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85400b4c50 con 0x7f85400b0e90 2022-07-12T08:48:34.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85401562d0 con 0x7f854014b370 2022-07-12T08:48:34.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f854858a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8540156570 con 0x7f85400b2af0 2022-07-12T08:48:34.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 0x7f8540065ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 0x7f8540065ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58656/0 (socket says 172.21.15.61:58656) 2022-07-12T08:48:34.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 -- 172.21.15.61:0/3454413834 learned_addr learned my addr 172.21.15.61:0/3454413834 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:34.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8545b25700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85400b2af0 0x7f8540156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546b27700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f854014b370 0x7f85400b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 -- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85400b2af0 msgr2=0x7f8540156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85400b2af0 0x7f8540156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 -- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f854014b370 msgr2=0x7f85400b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:34.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f854014b370 0x7f85400b5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:34.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 -- 172.21.15.61:0/3454413834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538009470 con 0x7f85400b0e90 2022-07-12T08:48:34.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.878+0000 7f8546326700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 0x7f8540065ad0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f853c001290 tx=0x7f853c002950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853c0070d0 con 0x7f85400b0e90 2022-07-12T08:48:34.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c007230 con 0x7f85400b0e90 2022-07-12T08:48:34.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8540156810 con 0x7f85400b0e90 2022-07-12T08:48:34.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853c0073a0 con 0x7f85400b0e90 2022-07-12T08:48:34.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85400b46c0 con 0x7f85400b0e90 2022-07-12T08:48:34.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8540057460 con 0x7f85400b0e90 2022-07-12T08:48:34.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f853c010070 con 0x7f85400b0e90 2022-07-12T08:48:34.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f852f7fe700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f85200539b0 0x7f8520055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:34.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f853c06b630 con 0x7f85400b0e90 2022-07-12T08:48:34.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.882+0000 7f8545b25700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f85200539b0 0x7f8520055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:34.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.886+0000 7f8545b25700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f85200539b0 0x7f8520055e70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f853000d000 tx=0x7f8530007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:34.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:34.886+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f853c042420 con 0x7f85400b0e90 2022-07-12T08:48:35.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.014+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8540068470 con 0x7f85400b0e90 2022-07-12T08:48:35.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.018+0000 7f852f7fe700 1 -- 172.21.15.61:0/3454413834 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f853c045a40 con 0x7f85400b0e90 2022-07-12T08:48:35.026 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:35.026 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":89,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:34.488995+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:32.975784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:33.715444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:35.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f85200539b0 msgr2=0x7f8520055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f85200539b0 0x7f8520055e70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f853000d000 tx=0x7f8530007b90).stop 2022-07-12T08:48:35.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 msgr2=0x7f8540065ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 0x7f8540065ad0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f853c001290 tx=0x7f853c002950).stop 2022-07-12T08:48:35.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 shutdown_connections 2022-07-12T08:48:35.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f85200539b0 0x7f8520055e70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f854014b370 0x7f85400b5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85400b2af0 0x7f8540156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 --2- 172.21.15.61:0/3454413834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85400b0e90 0x7f8540065ad0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 >> 172.21.15.61:0/3454413834 conn(0x7f8540063bd0 msgr2=0x7f85401474f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:35.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 shutdown_connections 2022-07-12T08:48:35.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.022+0000 7f854858a700 1 -- 172.21.15.61:0/3454413834 wait complete. 2022-07-12T08:48:35.041 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:48:35.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 Processor -- start 2022-07-12T08:48:35.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 -- start start 2022-07-12T08:48:35.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c0b6a90 0x7f861c14a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f861c0b1290 0x7f861c146320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0b6e70 con 0x7f861c05fd40 2022-07-12T08:48:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c0b25b0 con 0x7f861c0b1290 2022-07-12T08:48:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8622672700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c0ad6b0 con 0x7f861c0b6a90 2022-07-12T08:48:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621e71700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f861c0b1290 0x7f861c146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8620e6f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f861c05fd40 0x7f861c0adb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46882/0 (socket says 172.21.15.61:46882) 2022-07-12T08:48:35.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8620e6f700 1 -- 172.21.15.61:0/2949001214 learned_addr learned my addr 172.21.15.61:0/2949001214 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:35.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621670700 1 --2- 172.21.15.61:0/2949001214 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c0b6a90 0x7f861c14a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621e71700 1 -- 172.21.15.61:0/2949001214 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c0b6a90 msgr2=0x7f861c14a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621e71700 1 --2- 172.21.15.61:0/2949001214 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c0b6a90 0x7f861c14a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621e71700 1 -- 172.21.15.61:0/2949001214 >> v1:172.21.15.61:6789/0 conn(0x7f861c05fd40 legacy=0x7f861c0adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621e71700 1 -- 172.21.15.61:0/2949001214 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0b2850 con 0x7f861c0b1290 2022-07-12T08:48:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8621e71700 1 --2- 172.21.15.61:0/2949001214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c146320 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f86140032f0 tx=0x7f86140091d0).ready entity=mon.0 client_cookie=2759839bae39491c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8613fff700 1 -- 172.21.15.61:0/2949001214 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8614009ea0 con 0x7f861c0b1290 2022-07-12T08:48:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8613fff700 1 -- 172.21.15.61:0/2949001214 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861400b8a0 con 0x7f861c0b1290 2022-07-12T08:48:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.146+0000 7f8613fff700 1 -- 172.21.15.61:0/2949001214 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f861400ba10 con 0x7f861c0b1290 2022-07-12T08:48:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 -- 172.21.15.61:0/2949001214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 msgr2=0x7f861c146320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 --2- 172.21.15.61:0/2949001214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c146320 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7f86140032f0 tx=0x7f86140091d0).stop 2022-07-12T08:48:35.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 -- 172.21.15.61:0/2949001214 shutdown_connections 2022-07-12T08:48:35.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 --2- 172.21.15.61:0/2949001214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c146320 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 --2- 172.21.15.61:0/2949001214 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c0b6a90 0x7f861c14a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 -- 172.21.15.61:0/2949001214 >> 172.21.15.61:0/2949001214 conn(0x7f861c0645a0 msgr2=0x7f861c0b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:35.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 -- 172.21.15.61:0/2949001214 shutdown_connections 2022-07-12T08:48:35.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 -- 172.21.15.61:0/2949001214 wait complete. 2022-07-12T08:48:35.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 Processor -- start 2022-07-12T08:48:35.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.150+0000 7f8622672700 1 -- start start 2022-07-12T08:48:35.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c05fd40 0x7f861c159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c067040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f861c0b6a90 0x7f861c067530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c067a20 con 0x7f861c0b1290 2022-07-12T08:48:35.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c1431f0 con 0x7f861c0b6a90 2022-07-12T08:48:35.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c15a470 con 0x7f861c05fd40 2022-07-12T08:48:35.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c067040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c067040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58670/0 (socket says 172.21.15.61:58670) 2022-07-12T08:48:35.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 -- 172.21.15.61:0/362445583 learned_addr learned my addr 172.21.15.61:0/362445583 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:35.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8621e71700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f861c0b6a90 0x7f861c067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8621670700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c05fd40 0x7f861c159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 -- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c05fd40 msgr2=0x7f861c159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c05fd40 0x7f861c159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 -- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f861c0b6a90 msgr2=0x7f861c067530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f861c0b6a90 0x7f861c067530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 -- 172.21.15.61:0/362445583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8614009470 con 0x7f861c0b1290 2022-07-12T08:48:35.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8620e6f700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c067040 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f861c0b1ae0 tx=0x7f860c007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f860c001d70 con 0x7f861c0b1290 2022-07-12T08:48:35.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c007110 con 0x7f861c0b1290 2022-07-12T08:48:35.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c15a710 con 0x7f861c0b1290 2022-07-12T08:48:35.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f860c007470 con 0x7f861c0b1290 2022-07-12T08:48:35.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c14c580 con 0x7f861c0b1290 2022-07-12T08:48:35.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.154+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c057460 con 0x7f861c0b1290 2022-07-12T08:48:35.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.158+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f860c01b070 con 0x7f861c0b1290 2022-07-12T08:48:35.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.158+0000 7f86127fc700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8604053ad0 0x7f8604055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.158+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f860c06b1b0 con 0x7f861c0b1290 2022-07-12T08:48:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.158+0000 7f8621670700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8604053ad0 0x7f8604055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.158+0000 7f8621670700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8604053ad0 0x7f8604055f90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f861800c000 tx=0x7f8618008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.162+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f860c042110 con 0x7f861c0b1290 2022-07-12T08:48:35.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.326+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f861c0b0be0 con 0x7f861c0b1290 2022-07-12T08:48:35.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.506+0000 7f86127fc700 1 -- 172.21.15.61:0/362445583 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v50) v1 ==== 118+0+0 (secure 0 0 0) 0x7f860c045730 con 0x7f861c0b1290 2022-07-12T08:48:35.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8604053ad0 msgr2=0x7f8604055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8604053ad0 0x7f8604055f90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f861800c000 tx=0x7f8618008040).stop 2022-07-12T08:48:35.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 msgr2=0x7f861c067040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c067040 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 rx=0x7f861c0b1ae0 tx=0x7f860c007c30).stop 2022-07-12T08:48:35.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 shutdown_connections 2022-07-12T08:48:35.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8604053ad0 0x7f8604055f90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f861c0b6a90 0x7f861c067530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f861c0b1290 0x7f861c067040 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 --2- 172.21.15.61:0/362445583 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f861c05fd40 0x7f861c159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 >> 172.21.15.61:0/362445583 conn(0x7f861c0645a0 msgr2=0x7f861c147400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 shutdown_connections 2022-07-12T08:48:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.510+0000 7f8622672700 1 -- 172.21.15.61:0/362445583 wait complete. 2022-07-12T08:48:35.531 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:35.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 Processor -- start 2022-07-12T08:48:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- start start 2022-07-12T08:48:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff0440b52e0 0x7ff0440b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440b56c0 con 0x7ff044141e50 2022-07-12T08:48:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0440b4bd0 con 0x7ff0440b4050 2022-07-12T08:48:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff044142f00 con 0x7ff0440b52e0 2022-07-12T08:48:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff0440b52e0 0x7ff0440b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff0440b52e0 0x7ff0440b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58676/0 (socket says 172.21.15.61:58676) 2022-07-12T08:48:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 -- 172.21.15.61:0/557329100 learned_addr learned my addr 172.21.15.61:0/557329100 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff042d9d700 1 --1- 172.21.15.61:0/557329100 >> v1:172.21.15.61:6789/0 conn(0x7ff044141e50 0x7ff0441422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46898/0 (socket says 172.21.15.61:46898) 2022-07-12T08:48:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 -- 172.21.15.61:0/557329100 >> v1:172.21.15.111:6789/0 conn(0x7ff0440b4050 legacy=0x7ff0440b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 -- 172.21.15.61:0/557329100 >> v1:172.21.15.61:6789/0 conn(0x7ff044141e50 legacy=0x7ff0441422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 -- 172.21.15.61:0/557329100 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0440b0e80 con 0x7ff0440b52e0 2022-07-12T08:48:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04259c700 1 --2- 172.21.15.61:0/557329100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0440b21b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7ff02c002f70 tx=0x7ff02c006eb0).ready entity=mon.0 client_cookie=57aa8a3b1c098693 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04159a700 1 -- 172.21.15.61:0/557329100 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff02c00d660 con 0x7ff0440b52e0 2022-07-12T08:48:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04159a700 1 -- 172.21.15.61:0/557329100 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c000e90 con 0x7ff0440b52e0 2022-07-12T08:48:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff04159a700 1 -- 172.21.15.61:0/557329100 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff02c00d9f0 con 0x7ff0440b52e0 2022-07-12T08:48:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- 172.21.15.61:0/557329100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 msgr2=0x7ff0440b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 --2- 172.21.15.61:0/557329100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0440b21b0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7ff02c002f70 tx=0x7ff02c006eb0).stop 2022-07-12T08:48:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- 172.21.15.61:0/557329100 shutdown_connections 2022-07-12T08:48:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 --2- 172.21.15.61:0/557329100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0440b21b0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- 172.21.15.61:0/557329100 >> 172.21.15.61:0/557329100 conn(0x7ff044067020 msgr2=0x7ff044067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- 172.21.15.61:0/557329100 shutdown_connections 2022-07-12T08:48:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.634+0000 7ff048f7c700 1 -- 172.21.15.61:0/557329100 wait complete. 2022-07-12T08:48:35.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.638+0000 7ff048f7c700 1 Processor -- start 2022-07-12T08:48:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.638+0000 7ff048f7c700 1 -- start start 2022-07-12T08:48:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0440b4050 0x7ff044146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0441471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff044141e50 0x7ff044155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff044114850 con 0x7ff0440b52e0 2022-07-12T08:48:35.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441557e0 con 0x7ff0440b4050 2022-07-12T08:48:35.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff044155a80 con 0x7ff044141e50 2022-07-12T08:48:35.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0441471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0441471d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58684/0 (socket says 172.21.15.61:58684) 2022-07-12T08:48:35.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 -- 172.21.15.61:0/1279241804 learned_addr learned my addr 172.21.15.61:0/1279241804 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:35.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff04259c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0440b4050 0x7ff044146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff042d9d700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff044141e50 0x7ff044155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 -- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff044141e50 msgr2=0x7ff044155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff044141e50 0x7ff044155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 -- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0440b4050 msgr2=0x7ff044146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0440b4050 0x7ff044146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 -- 172.21.15.61:0/1279241804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02c00b1e0 con 0x7ff0440b52e0 2022-07-12T08:48:35.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff042d9d700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff044141e50 0x7ff044155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:35.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff04259c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0440b4050 0x7ff044146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:35.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff041d9b700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0441471d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7ff0440ba7b0 tx=0x7ff0340014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff034010510 con 0x7ff0440b52e0 2022-07-12T08:48:35.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff034010670 con 0x7ff0440b52e0 2022-07-12T08:48:35.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0441476c0 con 0x7ff0440b52e0 2022-07-12T08:48:35.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0340107e0 con 0x7ff0440b52e0 2022-07-12T08:48:35.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0440b8890 con 0x7ff0440b52e0 2022-07-12T08:48:35.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff044155c40 con 0x7ff0440b52e0 2022-07-12T08:48:35.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff034010070 con 0x7ff0440b52e0 2022-07-12T08:48:35.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.642+0000 7ff03b7fe700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff01c053af0 0x7ff01c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.646+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff03406ba50 con 0x7ff0440b52e0 2022-07-12T08:48:35.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.646+0000 7ff04259c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff01c053af0 0x7ff01c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.646+0000 7ff04259c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff01c053af0 0x7ff01c055fb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7ff02c01a000 tx=0x7ff02c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.650+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0340429b0 con 0x7ff0440b52e0 2022-07-12T08:48:35.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.814+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff044057460 con 0x7ff0440b52e0 2022-07-12T08:48:35.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.814+0000 7ff03b7fe700 1 -- 172.21.15.61:0/1279241804 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2137 (secure 0 0 0) 0x7ff034045fd0 con 0x7ff0440b52e0 2022-07-12T08:48:35.822 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:35.822 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":50,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:35.492783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6901":{"gid":6901,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/167461784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":167461784},{"type":"v1","addr":"172.21.15.111:6835","nonce":167461784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff01c053af0 msgr2=0x7ff01c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff01c053af0 0x7ff01c055fb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7ff02c01a000 tx=0x7ff02c015000).stop 2022-07-12T08:48:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 msgr2=0x7ff0441471d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0441471d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 rx=0x7ff0440ba7b0 tx=0x7ff0340014a0).stop 2022-07-12T08:48:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 shutdown_connections 2022-07-12T08:48:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff044141e50 0x7ff044155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0440b52e0 0x7ff0441471d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff01c053af0 0x7ff01c055fb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 --2- 172.21.15.61:0/1279241804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0440b4050 0x7ff044146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 >> 172.21.15.61:0/1279241804 conn(0x7ff044067020 msgr2=0x7ff0441484b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 shutdown_connections 2022-07-12T08:48:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.818+0000 7ff048f7c700 1 -- 172.21.15.61:0/1279241804 wait complete. 2022-07-12T08:48:35.831 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 50 2022-07-12T08:48:35.838 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:35.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 Processor -- start 2022-07-12T08:48:35.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 -- start start 2022-07-12T08:48:35.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c05c050 0x7fc07c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc07c0b01f0 0x7fc07c14a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c0ad2f0 con 0x7fc07c05cb00 2022-07-12T08:48:35.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07c0b0a40 con 0x7fc07c0b01f0 2022-07-12T08:48:35.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc082d58700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c05c430 con 0x7fc07c05c050 2022-07-12T08:48:35.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc0812f5700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc07c05cb00 0x7fc07c1443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46912/0 (socket says 172.21.15.61:46912) 2022-07-12T08:48:35.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc0812f5700 1 -- 172.21.15.61:0/1992094088 learned_addr learned my addr 172.21.15.61:0/1992094088 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:35.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc080af4700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c05c050 0x7fc07c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07bfff700 1 --2- 172.21.15.61:0/1992094088 >> v2:172.21.15.111:3300/0 conn(0x7fc07c0b01f0 0x7fc07c14a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07b7fe700 1 -- 172.21.15.61:0/1992094088 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676211745 0 0) 0x7fc07c0ad2f0 con 0x7fc07c05cb00 2022-07-12T08:48:35.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07b7fe700 1 -- 172.21.15.61:0/1992094088 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c003260 con 0x7fc07c05cb00 2022-07-12T08:48:35.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07bfff700 1 -- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c05c050 msgr2=0x7fc07c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:35.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07bfff700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c05c050 0x7fc07c0ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07bfff700 1 -- 172.21.15.61:0/1992094088 >> v1:172.21.15.61:6789/0 conn(0x7fc07c05cb00 legacy=0x7fc07c1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07bfff700 1 -- 172.21.15.61:0/1992094088 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c0b2420 con 0x7fc07c0b01f0 2022-07-12T08:48:35.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc080af4700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c05c050 0x7fc07c0ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:35.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.938+0000 7fc07bfff700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c0b01f0 0x7fc07c14a8f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc064002f70 tx=0x7fc064006df0).ready entity=mon.1 client_cookie=4636b5424a9486f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc07b7fe700 1 -- 172.21.15.61:0/1992094088 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06400d4a0 con 0x7fc07c0b01f0 2022-07-12T08:48:35.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc07b7fe700 1 -- 172.21.15.61:0/1992094088 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc064000e90 con 0x7fc07c0b01f0 2022-07-12T08:48:35.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc07b7fe700 1 -- 172.21.15.61:0/1992094088 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06400d7f0 con 0x7fc07c0b01f0 2022-07-12T08:48:35.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c0b01f0 msgr2=0x7fc07c14a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c0b01f0 0x7fc07c14a8f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc064002f70 tx=0x7fc064006df0).stop 2022-07-12T08:48:35.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- 172.21.15.61:0/1992094088 shutdown_connections 2022-07-12T08:48:35.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c0b01f0 0x7fc07c14a8f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 --2- 172.21.15.61:0/1992094088 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c05c050 0x7fc07c0ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- 172.21.15.61:0/1992094088 >> 172.21.15.61:0/1992094088 conn(0x7fc07c062c00 msgr2=0x7fc07c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:35.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- 172.21.15.61:0/1992094088 shutdown_connections 2022-07-12T08:48:35.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- 172.21.15.61:0/1992094088 wait complete. 2022-07-12T08:48:35.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 Processor -- start 2022-07-12T08:48:35.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- start start 2022-07-12T08:48:35.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 0x7fc07c067540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c05cb00 0x7fc07c145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c0b01f0 0x7fc07c146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c064210 con 0x7fc07c05c050 2022-07-12T08:48:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c063d70 con 0x7fc07c05cb00 2022-07-12T08:48:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc082d58700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c065d10 con 0x7fc07c0b01f0 2022-07-12T08:48:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.942+0000 7fc080af4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 0x7fc07c067540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 0x7fc07c067540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58696/0 (socket says 172.21.15.61:58696) 2022-07-12T08:48:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc0812f5700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c0b01f0 0x7fc07c146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 -- 172.21.15.61:0/3639663169 learned_addr learned my addr 172.21.15.61:0/3639663169 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:35.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc07bfff700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c05cb00 0x7fc07c145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 -- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c0b01f0 msgr2=0x7fc07c146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c0b01f0 0x7fc07c146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 -- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c05cb00 msgr2=0x7fc07c145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c05cb00 0x7fc07c145b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 -- 172.21.15.61:0/3639663169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c009470 con 0x7fc07c05c050 2022-07-12T08:48:35.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc0812f5700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c0b01f0 0x7fc07c146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:35.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc080af4700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 0x7fc07c067540 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fc06c00c000 tx=0x7fc06c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06c00c850 con 0x7fc07c05c050 2022-07-12T08:48:35.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06400b180 con 0x7fc07c05c050 2022-07-12T08:48:35.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c00c9b0 con 0x7fc07c05c050 2022-07-12T08:48:35.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06c00cd10 con 0x7fc07c05c050 2022-07-12T08:48:35.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c064f00 con 0x7fc07c05c050 2022-07-12T08:48:35.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.946+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c057460 con 0x7fc07c05c050 2022-07-12T08:48:35.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.950+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc06c01b400 con 0x7fc07c05c050 2022-07-12T08:48:35.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.950+0000 7fc079ffb700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:35.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.950+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc06c06c8f0 con 0x7fc07c05c050 2022-07-12T08:48:35.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.950+0000 7fc07bfff700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:35.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.950+0000 7fc07bfff700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fc06401a000 tx=0x7fc064014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:35.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:35.954+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc06c043750 con 0x7fc07c05c050 2022-07-12T08:48:36.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.078+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc07c0447a0 con 0x7fc07c05c050 2022-07-12T08:48:36.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.082+0000 7fc079ffb700 1 -- 172.21.15.61:0/3639663169 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16089 (secure 0 0 0) 0x7fc06c046d70 con 0x7fc07c05c050 2022-07-12T08:48:36.087 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:36.088 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":90,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:35.492446+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:32.975784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:33.715444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:36.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 msgr2=0x7fc060055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7fc06401a000 tx=0x7fc064014000).stop 2022-07-12T08:48:36.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 msgr2=0x7fc07c067540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 0x7fc07c067540 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7fc06c00c000 tx=0x7fc06c0091d0).stop 2022-07-12T08:48:36.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 shutdown_connections 2022-07-12T08:48:36.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc07c0b01f0 0x7fc07c146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc07c05cb00 0x7fc07c145b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 --2- 172.21.15.61:0/3639663169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc07c05c050 0x7fc07c067540 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 >> 172.21.15.61:0/3639663169 conn(0x7fc07c062c00 msgr2=0x7fc07c147b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:36.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 shutdown_connections 2022-07-12T08:48:36.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.086+0000 7fc082d58700 1 -- 172.21.15.61:0/3639663169 wait complete. 2022-07-12T08:48:36.106 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:48:36.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 Processor -- start 2022-07-12T08:48:36.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 -- start start 2022-07-12T08:48:36.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1eb00afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1eb01465a0 0x7f1eb00b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb0142f00 con 0x7f1eb00ba140 2022-07-12T08:48:36.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00b5a70 con 0x7f1eb01465a0 2022-07-12T08:48:36.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb869b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00b1470 con 0x7f1eb00b4800 2022-07-12T08:48:36.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6437700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1eb00ba140 0x7f1eb00b0d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46922/0 (socket says 172.21.15.61:46922) 2022-07-12T08:48:36.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6437700 1 -- 172.21.15.61:0/1662599564 learned_addr learned my addr 172.21.15.61:0/1662599564 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:36.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6c38700 1 --2- 172.21.15.61:0/1662599564 >> v2:172.21.15.111:3300/0 conn(0x7f1eb01465a0 0x7f1eb00b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1eb00afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492278143 0 0) 0x7f1eb0142f00 con 0x7f1eb00ba140 2022-07-12T08:48:36.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c003260 con 0x7f1eb00ba140 2022-07-12T08:48:36.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280926296 0 0) 0x7f1e9c003260 con 0x7f1eb00ba140 2022-07-12T08:48:36.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 msgr2=0x7f1eb00afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1eb00afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 >> v2:172.21.15.111:3300/0 conn(0x7f1eb01465a0 msgr2=0x7f1eb00b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 --2- 172.21.15.61:0/1662599564 >> v2:172.21.15.111:3300/0 conn(0x7f1eb01465a0 0x7f1eb00b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00b3430 con 0x7f1eb00ba140 2022-07-12T08:48:36.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f1ea8001a00 con 0x7f1eb00ba140 2022-07-12T08:48:36.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 >> v1:172.21.15.61:6789/0 conn(0x7f1eb00ba140 legacy=0x7f1eb00b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9c003750 0x7f1e9c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1e9c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1e9c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00b5a70 con 0x7f1e9c004500 2022-07-12T08:48:36.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00b1470 con 0x7f1e9c003750 2022-07-12T08:48:36.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9c00b750 con 0x7f1eb00b4800 2022-07-12T08:48:36.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1e9c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6437700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9c003750 0x7f1e9c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6c38700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1e9c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 -- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 msgr2=0x7f1e9c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1e9c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 -- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9c003750 msgr2=0x7f1e9c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9c003750 0x7f1e9c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 -- 172.21.15.61:0/1662599564 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c003260 con 0x7f1e9c004500 2022-07-12T08:48:36.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6c38700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b4800 0x7f1e9c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:36.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb6437700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1e9c003750 0x7f1e9c003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:36.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.222+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1e9c004980 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f1ea400c000 tx=0x7f1ea4006df0).ready entity=mon.0 client_cookie=356135ae2c9289ff server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea400e830 con 0x7f1e9c004500 2022-07-12T08:48:36.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea400e990 con 0x7f1e9c004500 2022-07-12T08:48:36.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb5435700 1 -- 172.21.15.61:0/1662599564 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea400ecf0 con 0x7f1e9c004500 2022-07-12T08:48:36.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 -- 172.21.15.61:0/1662599564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 msgr2=0x7f1e9c004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 --2- 172.21.15.61:0/1662599564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1e9c004980 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f1ea400c000 tx=0x7f1ea4006df0).stop 2022-07-12T08:48:36.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb6c38700 1 -- 172.21.15.61:0/1662599564 reap_dead start 2022-07-12T08:48:36.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 -- 172.21.15.61:0/1662599564 shutdown_connections 2022-07-12T08:48:36.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 -- 172.21.15.61:0/1662599564 >> 172.21.15.61:0/1662599564 conn(0x7f1eb0067a00 msgr2=0x7f1eb00b96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:36.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 -- 172.21.15.61:0/1662599564 shutdown_connections 2022-07-12T08:48:36.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 -- 172.21.15.61:0/1662599564 wait complete. 2022-07-12T08:48:36.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 Processor -- start 2022-07-12T08:48:36.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.226+0000 7f1eb869b700 1 -- start start 2022-07-12T08:48:36.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1eb00ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb0113390 0x7f1eb00ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb0155330 0x7f1eb01557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0114230 con 0x7f1e9c004500 2022-07-12T08:48:36.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00adf70 con 0x7f1eb0113390 2022-07-12T08:48:36.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00ae210 con 0x7f1eb0155330 2022-07-12T08:48:36.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1eb00ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1eb00ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58716/0 (socket says 172.21.15.61:58716) 2022-07-12T08:48:36.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 -- 172.21.15.61:0/3659118514 learned_addr learned my addr 172.21.15.61:0/3659118514 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:36.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6c38700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb0155330 0x7f1eb01557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb0113390 0x7f1eb00ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 -- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb0155330 msgr2=0x7f1eb01557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb0155330 0x7f1eb01557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 -- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb0113390 msgr2=0x7f1eb00ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb0113390 0x7f1eb00ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 -- 172.21.15.61:0/3659118514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00ae4b0 con 0x7f1e9c004500 2022-07-12T08:48:36.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb0113390 0x7f1eb00ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:36.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6c38700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb0155330 0x7f1eb01557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:36.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb6437700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1eb00ae950 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f1ea800b000 tx=0x7f1ea8001860).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea800ca50 con 0x7f1e9c004500 2022-07-12T08:48:36.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea800cbb0 con 0x7f1e9c004500 2022-07-12T08:48:36.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb0145f50 con 0x7f1e9c004500 2022-07-12T08:48:36.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea8014430 con 0x7f1e9c004500 2022-07-12T08:48:36.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.230+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb00acfa0 con 0x7f1e9c004500 2022-07-12T08:48:36.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.234+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb00ae640 con 0x7f1e9c004500 2022-07-12T08:48:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.234+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1ea8014590 con 0x7f1e9c004500 2022-07-12T08:48:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.234+0000 7f1eaf7fe700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e90053a40 0x7f1e90055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.234+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ea806c370 con 0x7f1e9c004500 2022-07-12T08:48:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.234+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e90053a40 0x7f1e90055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.234+0000 7f1eb5c36700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e90053a40 0x7f1e90055f00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f1ea401a000 tx=0x7f1ea4019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.238+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ea8043160 con 0x7f1e9c004500 2022-07-12T08:48:36.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.410+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1eb0057460 con 0x7f1e9c004500 2022-07-12T08:48:36.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.414+0000 7f1eaf7fe700 1 -- 172.21.15.61:0/3659118514 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1ea8046780 con 0x7f1e9c004500 2022-07-12T08:48:36.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e90053a40 msgr2=0x7f1e90055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e90053a40 0x7f1e90055f00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f1ea401a000 tx=0x7f1ea4019000).stop 2022-07-12T08:48:36.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 msgr2=0x7f1eb00ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1eb00ae950 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f1ea800b000 tx=0x7f1ea8001860).stop 2022-07-12T08:48:36.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 shutdown_connections 2022-07-12T08:48:36.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb0155330 0x7f1eb01557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e90053a40 0x7f1e90055f00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb0113390 0x7f1eb00ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 --2- 172.21.15.61:0/3659118514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e9c004500 0x7f1eb00ae950 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 >> 172.21.15.61:0/3659118514 conn(0x7f1eb0067a00 msgr2=0x7f1eb00b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 shutdown_connections 2022-07-12T08:48:36.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.418+0000 7f1eb869b700 1 -- 172.21.15.61:0/3659118514 wait complete. 2022-07-12T08:48:36.426 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:48:36.435 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:48:36.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 Processor -- start 2022-07-12T08:48:36.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 -- start start 2022-07-12T08:48:36.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4400af490 0x7fc440146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc44005d100 0x7fc4401431c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc440146200 con 0x7fc44005cd20 2022-07-12T08:48:36.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4401464a0 con 0x7fc4400af490 2022-07-12T08:48:36.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc445949700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc440146740 con 0x7fc44005d100 2022-07-12T08:48:36.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc43ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4400af490 0x7fc440146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc444947700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc44005cd20 0x7fc44005f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46942/0 (socket says 172.21.15.61:46942) 2022-07-12T08:48:36.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.546+0000 7fc444947700 1 -- 172.21.15.61:0/681635112 learned_addr learned my addr 172.21.15.61:0/681635112 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:36.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445148700 1 --2- 172.21.15.61:0/681635112 >> v2:172.21.15.111:3300/0 conn(0x7fc44005d100 0x7fc4401431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43ffff700 1 -- 172.21.15.61:0/681635112 >> v2:172.21.15.111:3300/0 conn(0x7fc44005d100 msgr2=0x7fc4401431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43ffff700 1 --2- 172.21.15.61:0/681635112 >> v2:172.21.15.111:3300/0 conn(0x7fc44005d100 0x7fc4401431c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43ffff700 1 -- 172.21.15.61:0/681635112 >> v1:172.21.15.61:6789/0 conn(0x7fc44005cd20 legacy=0x7fc44005f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43ffff700 1 -- 172.21.15.61:0/681635112 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc440060870 con 0x7fc4400af490 2022-07-12T08:48:36.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445148700 1 --2- 172.21.15.61:0/681635112 >> v2:172.21.15.111:3300/0 conn(0x7fc44005d100 0x7fc4401431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:36.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43ffff700 1 --2- 172.21.15.61:0/681635112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc440146be0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7fc428002f70 tx=0x7fc428006df0).ready entity=mon.0 client_cookie=6d6f43ab91cc0a92 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43f7fe700 1 -- 172.21.15.61:0/681635112 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42800d5c0 con 0x7fc4400af490 2022-07-12T08:48:36.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43f7fe700 1 -- 172.21.15.61:0/681635112 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc428000e90 con 0x7fc4400af490 2022-07-12T08:48:36.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43f7fe700 1 -- 172.21.15.61:0/681635112 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42800d910 con 0x7fc4400af490 2022-07-12T08:48:36.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- 172.21.15.61:0/681635112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 msgr2=0x7fc440146be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 --2- 172.21.15.61:0/681635112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc440146be0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7fc428002f70 tx=0x7fc428006df0).stop 2022-07-12T08:48:36.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- 172.21.15.61:0/681635112 shutdown_connections 2022-07-12T08:48:36.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 --2- 172.21.15.61:0/681635112 >> v2:172.21.15.111:3300/0 conn(0x7fc44005d100 0x7fc4401431c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 --2- 172.21.15.61:0/681635112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc440146be0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- 172.21.15.61:0/681635112 >> 172.21.15.61:0/681635112 conn(0x7fc4400b9e20 msgr2=0x7fc4400b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:36.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- 172.21.15.61:0/681635112 shutdown_connections 2022-07-12T08:48:36.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- 172.21.15.61:0/681635112 wait complete. 2022-07-12T08:48:36.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 Processor -- start 2022-07-12T08:48:36.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- start start 2022-07-12T08:48:36.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc44005cd20 0x7fc4400b8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc44005d100 0x7fc4400b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc4401404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44009c0d0 con 0x7fc4400af490 2022-07-12T08:48:36.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4400b2d40 con 0x7fc44005d100 2022-07-12T08:48:36.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445949700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4400b2f70 con 0x7fc44005cd20 2022-07-12T08:48:36.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445148700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc4401404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445148700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc4401404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58734/0 (socket says 172.21.15.61:58734) 2022-07-12T08:48:36.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc445148700 1 -- 172.21.15.61:0/4055685285 learned_addr learned my addr 172.21.15.61:0/4055685285 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:36.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc43ffff700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc44005d100 0x7fc4400b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.550+0000 7fc444947700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc44005cd20 0x7fc4400b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445148700 1 -- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc44005cd20 msgr2=0x7fc4400b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445148700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc44005cd20 0x7fc4400b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445148700 1 -- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc44005d100 msgr2=0x7fc4400b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445148700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc44005d100 0x7fc4400b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445148700 1 -- 172.21.15.61:0/4055685285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42800b180 con 0x7fc4400af490 2022-07-12T08:48:36.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445148700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc4401404f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fc438006cc0 tx=0x7fc438008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc43800f710 con 0x7fc4400af490 2022-07-12T08:48:36.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43800f870 con 0x7fc4400af490 2022-07-12T08:48:36.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc43800fbd0 con 0x7fc4400af490 2022-07-12T08:48:36.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4400b3210 con 0x7fc4400af490 2022-07-12T08:48:36.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4400b3950 con 0x7fc4400af490 2022-07-12T08:48:36.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc440057460 con 0x7fc4400af490 2022-07-12T08:48:36.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc43801a030 con 0x7fc4400af490 2022-07-12T08:48:36.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc43dffb700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc424053ad0 0x7fc424055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.554+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc43806bec0 con 0x7fc4400af490 2022-07-12T08:48:36.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.558+0000 7fc444947700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc424053ad0 0x7fc424055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.558+0000 7fc444947700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc424053ad0 0x7fc424055f90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fc43000d000 tx=0x7fc430009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.562+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc438042d10 con 0x7fc4400af490 2022-07-12T08:48:36.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.730+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc4400b04a0 con 0x7fc4400af490 2022-07-12T08:48:36.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc43dffb700 1 -- 172.21.15.61:0/4055685285 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc438046330 con 0x7fc4400af490 2022-07-12T08:48:36.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc424053ad0 msgr2=0x7fc424055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc424053ad0 0x7fc424055f90 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7fc43000d000 tx=0x7fc430009040).stop 2022-07-12T08:48:36.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 msgr2=0x7fc4401404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc4401404f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fc438006cc0 tx=0x7fc438008040).stop 2022-07-12T08:48:36.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 shutdown_connections 2022-07-12T08:48:36.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4400af490 0x7fc4401404f0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc424053ad0 0x7fc424055f90 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc44005d100 0x7fc4400b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 --2- 172.21.15.61:0/4055685285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc44005cd20 0x7fc4400b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 >> 172.21.15.61:0/4055685285 conn(0x7fc4400b9e20 msgr2=0x7fc4400b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:36.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 shutdown_connections 2022-07-12T08:48:36.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.742+0000 7fc445949700 1 -- 172.21.15.61:0/4055685285 wait complete. 2022-07-12T08:48:36.751 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:48:36.760 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:48:36.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 Processor -- start 2022-07-12T08:48:36.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 -- start start 2022-07-12T08:48:36.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005cd20 0x7fe830146be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe8300af490 0x7fe83005d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe83005f140 0x7fe83005f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe830146200 con 0x7fe83005f140 2022-07-12T08:48:36.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8301464a0 con 0x7fe8300af490 2022-07-12T08:48:36.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe836726700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe830146740 con 0x7fe83005cd20 2022-07-12T08:48:36.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe83005f140 0x7fe83005f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe83005f140 0x7fe83005f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58742/0 (socket says 172.21.15.61:58742) 2022-07-12T08:48:36.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835724700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005cd20 0x7fe830146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 -- 172.21.15.61:0/1544365970 learned_addr learned my addr 172.21.15.61:0/1544365970 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:36.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe834f23700 1 --2- 172.21.15.61:0/1544365970 >> v2:172.21.15.111:3300/0 conn(0x7fe8300af490 0x7fe83005d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 -- 172.21.15.61:0/1544365970 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005cd20 msgr2=0x7fe830146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 --2- 172.21.15.61:0/1544365970 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005cd20 0x7fe830146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 -- 172.21.15.61:0/1544365970 >> v2:172.21.15.111:3300/0 conn(0x7fe8300af490 msgr2=0x7fe83005d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:36.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 --2- 172.21.15.61:0/1544365970 >> v2:172.21.15.111:3300/0 conn(0x7fe8300af490 0x7fe83005d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 -- 172.21.15.61:0/1544365970 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe830060870 con 0x7fe83005f140 2022-07-12T08:48:36.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.870+0000 7fe835f25700 1 --2- 172.21.15.61:0/1544365970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005f140 0x7fe83005f5f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fe8280032f0 tx=0x7fe828009290).ready entity=mon.0 client_cookie=ed6b70199b6f398a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe827fff700 1 -- 172.21.15.61:0/1544365970 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe82800b680 con 0x7fe83005f140 2022-07-12T08:48:36.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe827fff700 1 -- 172.21.15.61:0/1544365970 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82800b7e0 con 0x7fe83005f140 2022-07-12T08:48:36.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe827fff700 1 -- 172.21.15.61:0/1544365970 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe82800b950 con 0x7fe83005f140 2022-07-12T08:48:36.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 -- 172.21.15.61:0/1544365970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005f140 msgr2=0x7fe83005f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 --2- 172.21.15.61:0/1544365970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005f140 0x7fe83005f5f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 rx=0x7fe8280032f0 tx=0x7fe828009290).stop 2022-07-12T08:48:36.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 -- 172.21.15.61:0/1544365970 shutdown_connections 2022-07-12T08:48:36.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 --2- 172.21.15.61:0/1544365970 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005f140 0x7fe83005f5f0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 --2- 172.21.15.61:0/1544365970 >> v2:172.21.15.111:3300/0 conn(0x7fe8300af490 0x7fe83005d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 --2- 172.21.15.61:0/1544365970 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005cd20 0x7fe830146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 -- 172.21.15.61:0/1544365970 >> 172.21.15.61:0/1544365970 conn(0x7fe8300b9e20 msgr2=0x7fe8300b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:36.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 -- 172.21.15.61:0/1544365970 shutdown_connections 2022-07-12T08:48:36.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 -- 172.21.15.61:0/1544365970 wait complete. 2022-07-12T08:48:36.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 Processor -- start 2022-07-12T08:48:36.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.874+0000 7fe836726700 1 -- start start 2022-07-12T08:48:36.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 0x7fe8300b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005f140 0x7fe8300b7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8300af490 0x7fe83009e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830155ac0 con 0x7fe83005cd20 2022-07-12T08:48:36.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe830155cd0 con 0x7fe8300af490 2022-07-12T08:48:36.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe830155f70 con 0x7fe83005f140 2022-07-12T08:48:36.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 0x7fe8300b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 0x7fe8300b3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58744/0 (socket says 172.21.15.61:58744) 2022-07-12T08:48:36.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 -- 172.21.15.61:0/2227448371 learned_addr learned my addr 172.21.15.61:0/2227448371 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:36.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe834f23700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005f140 0x7fe8300b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835f25700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8300af490 0x7fe83009e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 -- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005f140 msgr2=0x7fe8300b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005f140 0x7fe8300b7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 -- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8300af490 msgr2=0x7fe83009e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:36.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8300af490 0x7fe83009e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 -- 172.21.15.61:0/2227448371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe828009530 con 0x7fe83005cd20 2022-07-12T08:48:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835f25700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8300af490 0x7fe83009e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe835724700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 0x7fe8300b3540 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fe82000c000 tx=0x7fe820006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820010780 con 0x7fe83005cd20 2022-07-12T08:48:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8200108e0 con 0x7fe83005cd20 2022-07-12T08:48:36.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83009ee70 con 0x7fe83005cd20 2022-07-12T08:48:36.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe820010c40 con 0x7fe83005cd20 2022-07-12T08:48:36.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83009ebf0 con 0x7fe83005cd20 2022-07-12T08:48:36.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.878+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83009bd90 con 0x7fe83005cd20 2022-07-12T08:48:36.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.882+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe82001c460 con 0x7fe83005cd20 2022-07-12T08:48:36.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.882+0000 7fe8267fc700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe818053920 0x7fe818055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:36.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.882+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe82006c3a0 con 0x7fe83005cd20 2022-07-12T08:48:36.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.882+0000 7fe834f23700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe818053920 0x7fe818055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:36.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.882+0000 7fe834f23700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe818053920 0x7fe818055de0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fe82c00b000 tx=0x7fe82c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:36.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:36.886+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe820043200 con 0x7fe83005cd20 2022-07-12T08:48:37.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.054+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe830057460 con 0x7fe83005cd20 2022-07-12T08:48:37.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.062+0000 7fe8267fc700 1 -- 172.21.15.61:0/2227448371 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v31) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe820046820 con 0x7fe83005cd20 2022-07-12T08:48:37.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe818053920 msgr2=0x7fe818055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe818053920 0x7fe818055de0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7fe82c00b000 tx=0x7fe82c006df0).stop 2022-07-12T08:48:37.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 msgr2=0x7fe8300b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 0x7fe8300b3540 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fe82000c000 tx=0x7fe820006df0).stop 2022-07-12T08:48:37.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 shutdown_connections 2022-07-12T08:48:37.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe818053920 0x7fe818055de0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8300af490 0x7fe83009e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe83005f140 0x7fe8300b7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 --2- 172.21.15.61:0/2227448371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe83005cd20 0x7fe8300b3540 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 >> 172.21.15.61:0/2227448371 conn(0x7fe8300b9e20 msgr2=0x7fe8300b8420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:37.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 shutdown_connections 2022-07-12T08:48:37.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.066+0000 7fe836726700 1 -- 172.21.15.61:0/2227448371 wait complete. 2022-07-12T08:48:37.075 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:48:37.083 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:37.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 Processor -- start 2022-07-12T08:48:37.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- start start 2022-07-12T08:48:37.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03c805c050 0x7f03c80ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c814a8f0 0x7f03c81443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80ad2f0 con 0x7f03c80b01f0 2022-07-12T08:48:37.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c80b0a40 con 0x7f03c805c050 2022-07-12T08:48:37.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c805c430 con 0x7f03c814a8f0 2022-07-12T08:48:37.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cccac700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f03c805c050 0x7f03c80ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03c7fff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f03c80b01f0 0x7f03c805cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:60574/0 (socket says 172.21.15.61:60574) 2022-07-12T08:48:37.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03c7fff700 1 -- 172.21.15.61:0/2153642377 learned_addr learned my addr 172.21.15.61:0/2153642377 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:37.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cd4ad700 1 --2- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c814a8f0 0x7f03c81443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cccac700 1 -- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c814a8f0 msgr2=0x7f03c81443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:37.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cccac700 1 --2- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c814a8f0 0x7f03c81443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cccac700 1 -- 172.21.15.61:0/2153642377 >> v1:172.21.15.111:6789/0 conn(0x7f03c80b01f0 legacy=0x7f03c805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cccac700 1 -- 172.21.15.61:0/2153642377 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80b2420 con 0x7f03c805c050 2022-07-12T08:48:37.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cccac700 1 --2- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c805c050 0x7f03c80ac710 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f03b0002f70 tx=0x7f03b0006df0).ready entity=mon.1 client_cookie=9bfd76194ab59b6c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:37.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03c77fe700 1 -- 172.21.15.61:0/2153642377 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03b000d570 con 0x7f03c805c050 2022-07-12T08:48:37.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03c77fe700 1 -- 172.21.15.61:0/2153642377 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b0000e90 con 0x7f03c805c050 2022-07-12T08:48:37.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03c77fe700 1 -- 172.21.15.61:0/2153642377 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03b000d8c0 con 0x7f03c805c050 2022-07-12T08:48:37.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c805c050 msgr2=0x7f03c80ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 --2- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c805c050 0x7f03c80ac710 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f03b0002f70 tx=0x7f03b0006df0).stop 2022-07-12T08:48:37.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- 172.21.15.61:0/2153642377 shutdown_connections 2022-07-12T08:48:37.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 --2- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c814a8f0 0x7f03c81443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 --2- 172.21.15.61:0/2153642377 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c805c050 0x7f03c80ac710 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- 172.21.15.61:0/2153642377 >> 172.21.15.61:0/2153642377 conn(0x7f03c8062c00 msgr2=0x7f03c8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:37.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- 172.21.15.61:0/2153642377 shutdown_connections 2022-07-12T08:48:37.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.182+0000 7f03cef10700 1 -- 172.21.15.61:0/2153642377 wait complete. 2022-07-12T08:48:37.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 Processor -- start 2022-07-12T08:48:37.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 -- start start 2022-07-12T08:48:37.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c805c050 0x7f03c8149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 0x7f03c81448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c814a8f0 0x7f03c8144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c81452d0 con 0x7f03c80b01f0 2022-07-12T08:48:37.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8066fb0 con 0x7f03c814a8f0 2022-07-12T08:48:37.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cef10700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c814e1b0 con 0x7f03c805c050 2022-07-12T08:48:37.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 0x7f03c81448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cccac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c805c050 0x7f03c8149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 0x7f03c81448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58760/0 (socket says 172.21.15.61:58760) 2022-07-12T08:48:37.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 -- 172.21.15.61:0/3982086830 learned_addr learned my addr 172.21.15.61:0/3982086830 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:37.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cccac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c805c050 0x7f03c8149d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40954/0 (socket says 172.21.15.61:40954) 2022-07-12T08:48:37.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03cd4ad700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c814a8f0 0x7f03c8144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 -- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c805c050 msgr2=0x7f03c8149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c805c050 0x7f03c8149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 -- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c814a8f0 msgr2=0x7f03c8144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c814a8f0 0x7f03c8144de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 -- 172.21.15.61:0/3982086830 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b000b180 con 0x7f03c80b01f0 2022-07-12T08:48:37.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.186+0000 7f03c7fff700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 0x7f03c81448f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f03c80b62e0 tx=0x7f03b80014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:37.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03b8005ba0 con 0x7f03c80b01f0 2022-07-12T08:48:37.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8005d00 con 0x7f03c80b01f0 2022-07-12T08:48:37.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c814df10 con 0x7f03c80b01f0 2022-07-12T08:48:37.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03b80072c0 con 0x7f03c80b01f0 2022-07-12T08:48:37.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8067910 con 0x7f03c80b01f0 2022-07-12T08:48:37.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c814ec20 con 0x7f03c80b01f0 2022-07-12T08:48:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f03b8010070 con 0x7f03c80b01f0 2022-07-12T08:48:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03c5ffb700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03a00539b0 0x7f03a0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f03b806b750 con 0x7f03c80b01f0 2022-07-12T08:48:37.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.190+0000 7f03cccac700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03a00539b0 0x7f03a0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.194+0000 7f03cccac700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03a00539b0 0x7f03a0055e70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f03b001a000 tx=0x7f03b0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:37.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.198+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03b80426b0 con 0x7f03c80b01f0 2022-07-12T08:48:37.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c8057460 con 0x7f03c80b01f0 2022-07-12T08:48:37.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03c5ffb700 1 -- 172.21.15.61:0/3982086830 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f03b8018070 con 0x7f03c80b01f0 2022-07-12T08:48:37.368 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:37.368 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":50,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:35.492783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6901":{"gid":6901,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/167461784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":167461784},{"type":"v1","addr":"172.21.15.111:6835","nonce":167461784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:37.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03a00539b0 msgr2=0x7f03a0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03a00539b0 0x7f03a0055e70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f03b001a000 tx=0x7f03b0014000).stop 2022-07-12T08:48:37.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 msgr2=0x7f03c81448f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 0x7f03c81448f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 rx=0x7f03c80b62e0 tx=0x7f03b80014a0).stop 2022-07-12T08:48:37.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 shutdown_connections 2022-07-12T08:48:37.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03a00539b0 0x7f03a0055e70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03c814a8f0 0x7f03c8144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03c80b01f0 0x7f03c81448f0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 --2- 172.21.15.61:0/3982086830 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f03c805c050 0x7f03c8149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.362+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 >> 172.21.15.61:0/3982086830 conn(0x7f03c8062c00 msgr2=0x7f03c8145d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:37.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.366+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 shutdown_connections 2022-07-12T08:48:37.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.366+0000 7f03cef10700 1 -- 172.21.15.61:0/3982086830 wait complete. 2022-07-12T08:48:37.374 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 50 2022-07-12T08:48:37.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:34.489352+0000', 'modified': '2022-07-12T08:48:35.492783+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6901}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6901': {'gid': 6901, 'name': 'b', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/167461784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 167461784}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 167461784}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:48:37.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:48:37.584 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:48:37.701 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6901,"want_state":"up:active","state":"up:active","rank_uptime":3.1972697960000001,"mdsmap_epoch":50,"osdmap_epoch":90,"osdmap_epoch_barrier":89,"uptime":7.5845100319999998} 2022-07-12T08:48:37.702 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6901, "mdsmap_epoch": 50, "osdmap_epoch": 90, "osdmap_epoch_barrier": 89, "rank_uptime": 3.197269796, "state": "up:active", "uptime": 7.584510032, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:48:37.702 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:37.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.810+0000 7f7692690700 1 Processor -- start 2022-07-12T08:48:37.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- start start 2022-07-12T08:48:37.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f768c14a8f0 0x7f768c1443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0ad2f0 con 0x7f768c0b01f0 2022-07-12T08:48:37.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0b0a40 con 0x7f768c14a8f0 2022-07-12T08:48:37.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c05c430 con 0x7f768c05c050 2022-07-12T08:48:37.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f768c0b01f0 0x7f768c05cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46984/0 (socket says 172.21.15.61:46984) 2022-07-12T08:48:37.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768b7fe700 1 -- 172.21.15.61:0/2532829230 learned_addr learned my addr 172.21.15.61:0/2532829230 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:37.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7690c2d700 1 --2- 172.21.15.61:0/2532829230 >> v2:172.21.15.61:3300/0 conn(0x7f768c14a8f0 0x7f768c1443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768bfff700 1 --2- 172.21.15.61:0/2532829230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768affd700 1 -- 172.21.15.61:0/2532829230 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719439783 0 0) 0x7f768c0ad2f0 con 0x7f768c0b01f0 2022-07-12T08:48:37.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768affd700 1 -- 172.21.15.61:0/2532829230 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7678003260 con 0x7f768c0b01f0 2022-07-12T08:48:37.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7690c2d700 1 -- 172.21.15.61:0/2532829230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 msgr2=0x7f768c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:37.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7690c2d700 1 --2- 172.21.15.61:0/2532829230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c0ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7690c2d700 1 -- 172.21.15.61:0/2532829230 >> v1:172.21.15.61:6789/0 conn(0x7f768c0b01f0 legacy=0x7f768c05cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768affd700 1 -- 172.21.15.61:0/2532829230 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556707242 0 0) 0x7f7678003260 con 0x7f768c0b01f0 2022-07-12T08:48:37.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7690c2d700 1 -- 172.21.15.61:0/2532829230 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0b2420 con 0x7f768c14a8f0 2022-07-12T08:48:37.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7690c2d700 1 --2- 172.21.15.61:0/2532829230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c1443c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f767c001800 tx=0x7f76840091d0).ready entity=mon.0 client_cookie=ae2b9f7d2b3bd0fa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:37.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768affd700 1 -- 172.21.15.61:0/2532829230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f767c001660 con 0x7f768c14a8f0 2022-07-12T08:48:37.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768affd700 1 -- 172.21.15.61:0/2532829230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768400f510 con 0x7f768c14a8f0 2022-07-12T08:48:37.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f768affd700 1 -- 172.21.15.61:0/2532829230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f768400f680 con 0x7f768c14a8f0 2022-07-12T08:48:37.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- 172.21.15.61:0/2532829230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 msgr2=0x7f768c1443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 --2- 172.21.15.61:0/2532829230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c1443c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f767c001800 tx=0x7f76840091d0).stop 2022-07-12T08:48:37.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- 172.21.15.61:0/2532829230 shutdown_connections 2022-07-12T08:48:37.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 --2- 172.21.15.61:0/2532829230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c1443c0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 --2- 172.21.15.61:0/2532829230 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c0ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- 172.21.15.61:0/2532829230 >> 172.21.15.61:0/2532829230 conn(0x7f768c062c00 msgr2=0x7f768c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:37.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- 172.21.15.61:0/2532829230 shutdown_connections 2022-07-12T08:48:37.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.814+0000 7f7692690700 1 -- 172.21.15.61:0/2532829230 wait complete. 2022-07-12T08:48:37.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 Processor -- start 2022-07-12T08:48:37.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 -- start start 2022-07-12T08:48:37.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c14e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768c0b01f0 0x7f768c1563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c141290 con 0x7f768c14a8f0 2022-07-12T08:48:37.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c159ab0 con 0x7f768c0b01f0 2022-07-12T08:48:37.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7692690700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0ad830 con 0x7f768c05c050 2022-07-12T08:48:37.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7690c2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f768bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c14e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f768b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768c0b01f0 0x7f768c1563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f768bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c14e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:40968/0 (socket says 172.21.15.61:40968) 2022-07-12T08:48:37.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f768bfff700 1 -- 172.21.15.61:0/1230127622 learned_addr learned my addr 172.21.15.61:0/1230127622 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:37.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.818+0000 7f7690c2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58776/0 (socket says 172.21.15.61:58776) 2022-07-12T08:48:37.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768bfff700 1 -- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768c0b01f0 msgr2=0x7f768c1563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768bfff700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768c0b01f0 0x7f768c1563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768bfff700 1 -- 172.21.15.61:0/1230127622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 msgr2=0x7f768c159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768bfff700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768bfff700 1 -- 172.21.15.61:0/1230127622 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7684009470 con 0x7f768c05c050 2022-07-12T08:48:37.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768b7fe700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768c0b01f0 0x7f768c1563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:37.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f768bfff700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c14e160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f768000c000 tx=0x7f7680006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:37.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7680010780 con 0x7f768c05c050 2022-07-12T08:48:37.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76800108e0 con 0x7f768c05c050 2022-07-12T08:48:37.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7680010c40 con 0x7f768c05c050 2022-07-12T08:48:37.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c1568d0 con 0x7f768c05c050 2022-07-12T08:48:37.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c143550 con 0x7f768c05c050 2022-07-12T08:48:37.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f768001c460 con 0x7f768c05c050 2022-07-12T08:48:37.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c057460 con 0x7f768c05c050 2022-07-12T08:48:37.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.822+0000 7f76897fa700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f766c0539b0 0x7f766c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:37.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.826+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f768006c370 con 0x7f768c05c050 2022-07-12T08:48:37.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.826+0000 7f768b7fe700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f766c0539b0 0x7f766c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:37.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.826+0000 7f768b7fe700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f766c0539b0 0x7f766c055e70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f767c00d000 tx=0x7f767c001ab0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:37.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.834+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76800431d0 con 0x7f768c05c050 2022-07-12T08:48:37.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.958+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f768c0447a0 con 0x7f768c05c050 2022-07-12T08:48:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.958+0000 7f76897fa700 1 -- 172.21.15.61:0/1230127622 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f76800467f0 con 0x7f768c05c050 2022-07-12T08:48:37.966 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:37.967 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":90,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:35.492446+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:32.975784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:33.715444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:37.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f766c0539b0 msgr2=0x7f766c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f766c0539b0 0x7f766c055e70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f767c00d000 tx=0x7f767c001ab0).stop 2022-07-12T08:48:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 msgr2=0x7f768c14e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c14e160 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f768000c000 tx=0x7f7680006df0).stop 2022-07-12T08:48:37.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 shutdown_connections 2022-07-12T08:48:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768c14a8f0 0x7f768c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f766c0539b0 0x7f766c055e70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768c0b01f0 0x7f768c1563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 --2- 172.21.15.61:0/1230127622 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768c05c050 0x7f768c14e160 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 >> 172.21.15.61:0/1230127622 conn(0x7f768c062c00 msgr2=0x7f768c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 shutdown_connections 2022-07-12T08:48:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:37.962+0000 7f7692690700 1 -- 172.21.15.61:0/1230127622 wait complete. 2022-07-12T08:48:37.983 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:48:32.975784+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:48:33.715444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:48:37.984 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 Processor -- start 2022-07-12T08:48:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 -- start start 2022-07-12T08:48:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c143a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa98c067350 0x7fa98c066310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c13f880 con 0x7fa98c0b6c20 2022-07-12T08:48:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0638e0 con 0x7fa98c067350 2022-07-12T08:48:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa990ffc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b7000 con 0x7fa98c0b0b70 2022-07-12T08:48:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98a59c700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fa98c0b6c20 0x7fa98c143380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:60600/0 (socket says 172.21.15.61:60600) 2022-07-12T08:48:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98a59c700 1 -- 172.21.15.61:0/2222087150 learned_addr learned my addr 172.21.15.61:0/2222087150 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98ad9d700 1 --2- 172.21.15.61:0/2222087150 >> v2:172.21.15.111:3300/0 conn(0x7fa98c067350 0x7fa98c066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa989d9b700 1 --2- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98ad9d700 1 -- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 msgr2=0x7fa98c143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98ad9d700 1 --2- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98ad9d700 1 -- 172.21.15.61:0/2222087150 >> v1:172.21.15.111:6789/0 conn(0x7fa98c0b6c20 legacy=0x7fa98c143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98ad9d700 1 -- 172.21.15.61:0/2222087150 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c142ef0 con 0x7fa98c067350 2022-07-12T08:48:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98ad9d700 1 --2- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c067350 0x7fa98c066310 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa9840032f0 tx=0x7fa98400b470).ready entity=mon.1 client_cookie=7998ba034d57547c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98959a700 1 -- 172.21.15.61:0/2222087150 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa98400dd70 con 0x7fa98c067350 2022-07-12T08:48:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98959a700 1 -- 172.21.15.61:0/2222087150 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa984010470 con 0x7fa98c067350 2022-07-12T08:48:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.090+0000 7fa98959a700 1 -- 172.21.15.61:0/2222087150 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9840107d0 con 0x7fa98c067350 2022-07-12T08:48:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c067350 msgr2=0x7fa98c066310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c067350 0x7fa98c066310 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fa9840032f0 tx=0x7fa98400b470).stop 2022-07-12T08:48:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- 172.21.15.61:0/2222087150 shutdown_connections 2022-07-12T08:48:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c067350 0x7fa98c066310 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2222087150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- 172.21.15.61:0/2222087150 >> 172.21.15.61:0/2222087150 conn(0x7fa98c0646a0 msgr2=0x7fa98c0b61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- 172.21.15.61:0/2222087150 shutdown_connections 2022-07-12T08:48:38.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- 172.21.15.61:0/2222087150 wait complete. 2022-07-12T08:48:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 Processor -- start 2022-07-12T08:48:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- start start 2022-07-12T08:48:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 0x7fa98c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c0acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c0b6c20 0x7fa98c10fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c110880 con 0x7fa98c067350 2022-07-12T08:48:38.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b0200 con 0x7fa98c0b6c20 2022-07-12T08:48:38.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa990ffc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b04a0 con 0x7fa98c0b0b70 2022-07-12T08:48:38.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 0x7fa98c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 0x7fa98c14d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58786/0 (socket says 172.21.15.61:58786) 2022-07-12T08:48:38.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 -- 172.21.15.61:0/2424607833 learned_addr learned my addr 172.21.15.61:0/2424607833 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:38.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98ad9d700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c0b6c20 0x7fa98c10fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa989d9b700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c0acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 -- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 msgr2=0x7fa98c0acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c0acea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 -- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c0b6c20 msgr2=0x7fa98c10fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c0b6c20 0x7fa98c10fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 -- 172.21.15.61:0/2424607833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98400b710 con 0x7fa98c067350 2022-07-12T08:48:38.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98ad9d700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c0b6c20 0x7fa98c10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:38.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa98a59c700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 0x7fa98c14d9a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7fa98c06ab20 tx=0x7fa97c007c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa97c0070d0 con 0x7fa98c067350 2022-07-12T08:48:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.094+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c007230 con 0x7fa98c067350 2022-07-12T08:48:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c10f3c0 con 0x7fa98c067350 2022-07-12T08:48:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa97c005660 con 0x7fa98c067350 2022-07-12T08:48:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c0ad860 con 0x7fa98c067350 2022-07-12T08:48:38.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c0b0660 con 0x7fa98c067350 2022-07-12T08:48:38.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa97c01a060 con 0x7fa98c067350 2022-07-12T08:48:38.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa9837fe700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa970053a40 0x7fa970055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa97c06b600 con 0x7fa98c067350 2022-07-12T08:48:38.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa989d9b700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa970053a40 0x7fa970055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.098+0000 7fa989d9b700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa970053a40 0x7fa970055f00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fa97400b000 tx=0x7fa974008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.106+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa97c042560 con 0x7fa98c067350 2022-07-12T08:48:38.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa98c057460 con 0x7fa98c067350 2022-07-12T08:48:38.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa9837fe700 1 -- 172.21.15.61:0/2424607833 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v90) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa97c018070 con 0x7fa98c067350 2022-07-12T08:48:38.238 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:38.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa970053a40 msgr2=0x7fa970055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa970053a40 0x7fa970055f00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7fa97400b000 tx=0x7fa974008040).stop 2022-07-12T08:48:38.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 msgr2=0x7fa98c14d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 0x7fa98c14d9a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 rx=0x7fa98c06ab20 tx=0x7fa97c007c90).stop 2022-07-12T08:48:38.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 shutdown_connections 2022-07-12T08:48:38.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa970053a40 0x7fa970055f00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa98c0b6c20 0x7fa98c10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa98c0b0b70 0x7fa98c0acea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 --2- 172.21.15.61:0/2424607833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa98c067350 0x7fa98c14d9a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 >> 172.21.15.61:0/2424607833 conn(0x7fa98c0646a0 msgr2=0x7fa98c05f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 shutdown_connections 2022-07-12T08:48:38.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.234+0000 7fa990ffc700 1 -- 172.21.15.61:0/2424607833 wait complete. 2022-07-12T08:48:38.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:38.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.358+0000 7fe76359e700 1 Processor -- start 2022-07-12T08:48:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.358+0000 7fe76359e700 1 -- start start 2022-07-12T08:48:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe76359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe764064580 0x7fe7641460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe76359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7640b1170 con 0x7fe764060590 2022-07-12T08:48:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe76359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76405f1e0 con 0x7fe7640b0d90 2022-07-12T08:48:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe76359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7640b31b0 con 0x7fe764064580 2022-07-12T08:48:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe762d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe764064580 0x7fe7641460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe761d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe764060590 0x7fe764141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47014/0 (socket says 172.21.15.61:47014) 2022-07-12T08:48:38.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe761d9b700 1 -- 172.21.15.61:0/3889537321 learned_addr learned my addr 172.21.15.61:0/3889537321 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:38.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe76159a700 1 -- 172.21.15.61:0/3889537321 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056774009 0 0) 0x7fe76405f1e0 con 0x7fe7640b0d90 2022-07-12T08:48:38.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe762d9d700 1 -- 172.21.15.61:0/3889537321 >> v1:172.21.15.111:6789/0 conn(0x7fe7640b0d90 legacy=0x7fe7640ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe762d9d700 1 -- 172.21.15.61:0/3889537321 >> v1:172.21.15.61:6789/0 conn(0x7fe764060590 legacy=0x7fe764141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe762d9d700 1 -- 172.21.15.61:0/3889537321 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7640adce0 con 0x7fe764064580 2022-07-12T08:48:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe762d9d700 1 --2- 172.21.15.61:0/3889537321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe7641460c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe754001110 tx=0x7fe75c0091d0).ready entity=mon.0 client_cookie=e2074884103a3c24 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.362+0000 7fe76159a700 1 -- 172.21.15.61:0/3889537321 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe75c00b760 con 0x7fe764064580 2022-07-12T08:48:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76159a700 1 -- 172.21.15.61:0/3889537321 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c00b8c0 con 0x7fe764064580 2022-07-12T08:48:38.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76159a700 1 -- 172.21.15.61:0/3889537321 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe75c00ba30 con 0x7fe764064580 2022-07-12T08:48:38.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 -- 172.21.15.61:0/3889537321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 msgr2=0x7fe7641460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 --2- 172.21.15.61:0/3889537321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe7641460c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7fe754001110 tx=0x7fe75c0091d0).stop 2022-07-12T08:48:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 -- 172.21.15.61:0/3889537321 shutdown_connections 2022-07-12T08:48:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 --2- 172.21.15.61:0/3889537321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe7641460c0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 -- 172.21.15.61:0/3889537321 >> 172.21.15.61:0/3889537321 conn(0x7fe764061f50 msgr2=0x7fe764063a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 -- 172.21.15.61:0/3889537321 shutdown_connections 2022-07-12T08:48:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 -- 172.21.15.61:0/3889537321 wait complete. 2022-07-12T08:48:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 Processor -- start 2022-07-12T08:48:38.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.366+0000 7fe76359e700 1 -- start start 2022-07-12T08:48:38.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe764060590 0x7fe764144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe764065a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7640b0d90 0x7fe764065f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764066470 con 0x7fe764064580 2022-07-12T08:48:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764140330 con 0x7fe7640b0d90 2022-07-12T08:48:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640653a0 con 0x7fe764060590 2022-07-12T08:48:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe764065a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe764065a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58800/0 (socket says 172.21.15.61:58800) 2022-07-12T08:48:38.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 -- 172.21.15.61:0/3390809680 learned_addr learned my addr 172.21.15.61:0/3390809680 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76259c700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe764060590 0x7fe764144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe762d9d700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7640b0d90 0x7fe764065f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 -- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe764060590 msgr2=0x7fe764144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe764060590 0x7fe764144820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 -- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7640b0d90 msgr2=0x7fe764065f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7640b0d90 0x7fe764065f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 -- 172.21.15.61:0/3390809680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c009470 con 0x7fe764064580 2022-07-12T08:48:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe761d9b700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe764065a90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fe7540011d0 tx=0x7fe754007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7540072e0 con 0x7fe764064580 2022-07-12T08:48:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe754007440 con 0x7fe764064580 2022-07-12T08:48:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe764064ae0 con 0x7fe764064580 2022-07-12T08:48:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe754005800 con 0x7fe764064580 2022-07-12T08:48:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe764144d90 con 0x7fe764064580 2022-07-12T08:48:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.370+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe764057460 con 0x7fe764064580 2022-07-12T08:48:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.374+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe754010070 con 0x7fe764064580 2022-07-12T08:48:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.374+0000 7fe7537fe700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe740053a40 0x7fe740055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.374+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe75406b620 con 0x7fe764064580 2022-07-12T08:48:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.374+0000 7fe76259c700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe740053a40 0x7fe740055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.374+0000 7fe76259c700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe740053a40 0x7fe740055f00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fe75800b000 tx=0x7fe758009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.378+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe754042580 con 0x7fe764064580 2022-07-12T08:48:38.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.506+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe7640657d0 con 0x7fe764064580 2022-07-12T08:48:38.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.506+0000 7fe7537fe700 1 -- 172.21.15.61:0/3390809680 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v90) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe754045ba0 con 0x7fe764064580 2022-07-12T08:48:38.512 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:38.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe740053a40 msgr2=0x7fe740055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe740053a40 0x7fe740055f00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7fe75800b000 tx=0x7fe758009040).stop 2022-07-12T08:48:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 msgr2=0x7fe764065a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe764065a90 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 rx=0x7fe7540011d0 tx=0x7fe754007b90).stop 2022-07-12T08:48:38.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 shutdown_connections 2022-07-12T08:48:38.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe740053a40 0x7fe740055f00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe7640b0d90 0x7fe764065f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe764064580 0x7fe764065a90 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 --2- 172.21.15.61:0/3390809680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe764060590 0x7fe764144820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 >> 172.21.15.61:0/3390809680 conn(0x7fe764061f50 msgr2=0x7fe7640b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 shutdown_connections 2022-07-12T08:48:38.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.510+0000 7fe76359e700 1 -- 172.21.15.61:0/3390809680 wait complete. 2022-07-12T08:48:38.526 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:48:38.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.630+0000 7f05c55a3700 1 Processor -- start 2022-07-12T08:48:38.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.630+0000 7f05c55a3700 1 -- start start 2022-07-12T08:48:38.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05c00b1f70 0x7f05c0141f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c0067ae0 0x7f05c0142430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05c0063f90 0x7f05c0142920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c0066770 con 0x7f05c00b1f70 2022-07-12T08:48:38.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c0142e10 con 0x7f05c0063f90 2022-07-12T08:48:38.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c005cc60 con 0x7f05c0067ae0 2022-07-12T08:48:38.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05c00b1f70 0x7f05c0141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bf7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c0067ae0 0x7f05c0142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05c00b1f70 0x7f05c0141f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58808/0 (socket says 172.21.15.61:58808) 2022-07-12T08:48:38.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 -- 172.21.15.61:0/712793257 learned_addr learned my addr 172.21.15.61:0/712793257 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:38.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c4da2700 1 --2- 172.21.15.61:0/712793257 >> v2:172.21.15.111:3300/0 conn(0x7f05c0063f90 0x7f05c0142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 -- 172.21.15.61:0/712793257 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c0067ae0 msgr2=0x7f05c0142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 --2- 172.21.15.61:0/712793257 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c0067ae0 0x7f05c0142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 -- 172.21.15.61:0/712793257 >> v2:172.21.15.111:3300/0 conn(0x7f05c0063f90 msgr2=0x7f05c0142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 --2- 172.21.15.61:0/712793257 >> v2:172.21.15.111:3300/0 conn(0x7f05c0063f90 0x7f05c0142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 -- 172.21.15.61:0/712793257 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c00b2350 con 0x7f05c00b1f70 2022-07-12T08:48:38.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c4da2700 1 --2- 172.21.15.61:0/712793257 >> v2:172.21.15.111:3300/0 conn(0x7f05c0063f90 0x7f05c0142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:38.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05bffff700 1 --2- 172.21.15.61:0/712793257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c00b1f70 0x7f05c0141f40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f05b4002f70 tx=0x7f05b4006eb0).ready entity=mon.0 client_cookie=f3245a3162ad3259 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05beffd700 1 -- 172.21.15.61:0/712793257 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b400d570 con 0x7f05c00b1f70 2022-07-12T08:48:38.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05beffd700 1 -- 172.21.15.61:0/712793257 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4000e90 con 0x7f05c00b1f70 2022-07-12T08:48:38.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05beffd700 1 -- 172.21.15.61:0/712793257 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b400d900 con 0x7f05c00b1f70 2022-07-12T08:48:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 -- 172.21.15.61:0/712793257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c00b1f70 msgr2=0x7f05c0141f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- 172.21.15.61:0/712793257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c00b1f70 0x7f05c0141f40 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f05b4002f70 tx=0x7f05b4006eb0).stop 2022-07-12T08:48:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 -- 172.21.15.61:0/712793257 shutdown_connections 2022-07-12T08:48:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- 172.21.15.61:0/712793257 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c0067ae0 0x7f05c0142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- 172.21.15.61:0/712793257 >> v2:172.21.15.111:3300/0 conn(0x7f05c0063f90 0x7f05c0142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 --2- 172.21.15.61:0/712793257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c00b1f70 0x7f05c0141f40 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.634+0000 7f05c55a3700 1 -- 172.21.15.61:0/712793257 >> 172.21.15.61:0/712793257 conn(0x7f05c0062d70 msgr2=0x7f05c0063150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 -- 172.21.15.61:0/712793257 shutdown_connections 2022-07-12T08:48:38.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 -- 172.21.15.61:0/712793257 wait complete. 2022-07-12T08:48:38.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 Processor -- start 2022-07-12T08:48:38.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 -- start start 2022-07-12T08:48:38.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 0x7f05c010fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c0067ae0 0x7f05c013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c00b1f70 0x7f05c013f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0110c70 con 0x7f05c0063f90 2022-07-12T08:48:38.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0110320 con 0x7f05c0067ae0 2022-07-12T08:48:38.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c55a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c010f220 con 0x7f05c00b1f70 2022-07-12T08:48:38.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 0x7f05c010fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 0x7f05c010fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58814/0 (socket says 172.21.15.61:58814) 2022-07-12T08:48:38.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 -- 172.21.15.61:0/2408938382 learned_addr learned my addr 172.21.15.61:0/2408938382 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:38.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05c4da2700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c00b1f70 0x7f05c013f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bf7fe700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c0067ae0 0x7f05c013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 -- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c00b1f70 msgr2=0x7f05c013f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c00b1f70 0x7f05c013f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 -- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c0067ae0 msgr2=0x7f05c013edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c0067ae0 0x7f05c013edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.638+0000 7f05bffff700 1 -- 172.21.15.61:0/2408938382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b800b180 con 0x7f05c0063f90 2022-07-12T08:48:38.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bffff700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 0x7f05c010fad0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f05b4002f70 tx=0x7f05b4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b4011760 con 0x7f05c0063f90 2022-07-12T08:48:38.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b40118c0 con 0x7f05c0063f90 2022-07-12T08:48:38.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b400b1e0 con 0x7f05c0063f90 2022-07-12T08:48:38.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05b4011c20 con 0x7f05c0063f90 2022-07-12T08:48:38.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c013f790 con 0x7f05c0063f90 2022-07-12T08:48:38.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c0112400 con 0x7f05c0063f90 2022-07-12T08:48:38.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f05b4025030 con 0x7f05c0063f90 2022-07-12T08:48:38.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bd7fa700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a00539b0 0x7f05a0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f05b406ce00 con 0x7f05c0063f90 2022-07-12T08:48:38.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.642+0000 7f05bf7fe700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a00539b0 0x7f05a0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.646+0000 7f05bf7fe700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a00539b0 0x7f05a0055e70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f05b000d000 tx=0x7f05b000b470).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.650+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05b4043c50 con 0x7f05c0063f90 2022-07-12T08:48:38.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.778+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05c010f3e0 con 0x7f05c0063f90 2022-07-12T08:48:38.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.778+0000 7f05bd7fa700 1 -- 172.21.15.61:0/2408938382 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+10 (secure 0 0 0) 0x7f05c010f3e0 con 0x7f05c0063f90 2022-07-12T08:48:38.785 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:38.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a00539b0 msgr2=0x7f05a0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a00539b0 0x7f05a0055e70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f05b000d000 tx=0x7f05b000b470).stop 2022-07-12T08:48:38.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 msgr2=0x7f05c010fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 0x7f05c010fad0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f05b4002f70 tx=0x7f05b4015000).stop 2022-07-12T08:48:38.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 shutdown_connections 2022-07-12T08:48:38.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05c00b1f70 0x7f05c013f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f05a00539b0 0x7f05a0055e70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05c0067ae0 0x7f05c013edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 --2- 172.21.15.61:0/2408938382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05c0063f90 0x7f05c010fad0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 >> 172.21.15.61:0/2408938382 conn(0x7f05c0062d70 msgr2=0x7f05c0126080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 shutdown_connections 2022-07-12T08:48:38.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.782+0000 7f05c55a3700 1 -- 172.21.15.61:0/2408938382 wait complete. 2022-07-12T08:48:38.797 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:48:38.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410f2c1700 1 Processor -- start 2022-07-12T08:48:38.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410f2c1700 1 -- start start 2022-07-12T08:48:38.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410f2c1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b1a30 0x7f41080b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410f2c1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410813f250 con 0x7f410809b9d0 2022-07-12T08:48:38.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410f2c1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410813f4f0 con 0x7f41080b21c0 2022-07-12T08:48:38.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410f2c1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41080ac490 con 0x7f41080b1a30 2022-07-12T08:48:38.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410dabe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f410809b9d0 0x7f41080b1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47042/0 (socket says 172.21.15.61:47042) 2022-07-12T08:48:38.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410dabe700 1 -- 172.21.15.61:0/1459708166 learned_addr learned my addr 172.21.15.61:0/1459708166 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:38.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410eac0700 1 --2- 172.21.15.61:0/1459708166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b1a30 0x7f41080b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410d2bd700 1 -- 172.21.15.61:0/1459708166 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483641039 0 0) 0x7f410813f250 con 0x7f410809b9d0 2022-07-12T08:48:38.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410d2bd700 1 -- 172.21.15.61:0/1459708166 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e8003260 con 0x7f410809b9d0 2022-07-12T08:48:38.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410eac0700 1 -- 172.21.15.61:0/1459708166 >> v1:172.21.15.111:6789/0 conn(0x7f41080b21c0 legacy=0x7f41080603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410eac0700 1 -- 172.21.15.61:0/1459708166 >> v1:172.21.15.61:6789/0 conn(0x7f410809b9d0 legacy=0x7f41080b1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410eac0700 1 -- 172.21.15.61:0/1459708166 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108140070 con 0x7f41080b1a30 2022-07-12T08:48:38.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410eac0700 1 --2- 172.21.15.61:0/1459708166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b1a30 0x7f41080b25a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4100002f70 tx=0x7f4100006df0).ready entity=mon.2 client_cookie=216ca642c27cbc4e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.902+0000 7f410d2bd700 1 -- 172.21.15.61:0/1459708166 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f410000d570 con 0x7f41080b1a30 2022-07-12T08:48:38.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410d2bd700 1 -- 172.21.15.61:0/1459708166 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4100000e90 con 0x7f41080b1a30 2022-07-12T08:48:38.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410d2bd700 1 -- 172.21.15.61:0/1459708166 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f410000d8c0 con 0x7f41080b1a30 2022-07-12T08:48:38.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 -- 172.21.15.61:0/1459708166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b1a30 msgr2=0x7f41080b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 --2- 172.21.15.61:0/1459708166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b1a30 0x7f41080b25a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4100002f70 tx=0x7f4100006df0).stop 2022-07-12T08:48:38.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 -- 172.21.15.61:0/1459708166 shutdown_connections 2022-07-12T08:48:38.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 --2- 172.21.15.61:0/1459708166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b1a30 0x7f41080b25a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 -- 172.21.15.61:0/1459708166 >> 172.21.15.61:0/1459708166 conn(0x7f41080b57c0 msgr2=0x7f41080b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:38.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 -- 172.21.15.61:0/1459708166 shutdown_connections 2022-07-12T08:48:38.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 -- 172.21.15.61:0/1459708166 wait complete. 2022-07-12T08:48:38.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 Processor -- start 2022-07-12T08:48:38.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.906+0000 7f410f2c1700 1 -- start start 2022-07-12T08:48:38.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 0x7f410808f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b21c0 0x7f41080648d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4108066330 0x7f4108064dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080af870 con 0x7f410809b9d0 2022-07-12T08:48:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080900c0 con 0x7f4108066330 2022-07-12T08:48:38.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410813b9e0 con 0x7f41080b21c0 2022-07-12T08:48:38.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 0x7f410808f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 0x7f410808f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58828/0 (socket says 172.21.15.61:58828) 2022-07-12T08:48:38.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 -- 172.21.15.61:0/2929098843 learned_addr learned my addr 172.21.15.61:0/2929098843 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:38.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410dabe700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b21c0 0x7f41080648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410eac0700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4108066330 0x7f4108064dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 -- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b21c0 msgr2=0x7f41080648d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b21c0 0x7f41080648d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 -- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4108066330 msgr2=0x7f4108064dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:38.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4108066330 0x7f4108064dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:38.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 -- 172.21.15.61:0/2929098843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410000b180 con 0x7f410809b9d0 2022-07-12T08:48:38.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410dabe700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b21c0 0x7f41080648d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:38.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410e2bf700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 0x7f410808f9d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f4108062bd0 tx=0x7f40f0007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40f000cea0 con 0x7f410809b9d0 2022-07-12T08:48:38.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0007030 con 0x7f410809b9d0 2022-07-12T08:48:38.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41080668b0 con 0x7f410809b9d0 2022-07-12T08:48:38.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40f0007390 con 0x7f410809b9d0 2022-07-12T08:48:38.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4108066df0 con 0x7f410809b9d0 2022-07-12T08:48:38.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f40f0010030 con 0x7f410809b9d0 2022-07-12T08:48:38.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.910+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4108057460 con 0x7f410809b9d0 2022-07-12T08:48:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.914+0000 7f41077fe700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40ec0539d0 0x7f40ec055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.914+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f40f006bb20 con 0x7f410809b9d0 2022-07-12T08:48:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.914+0000 7f410dabe700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40ec0539d0 0x7f40ec055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:38.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.914+0000 7f410dabe700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40ec0539d0 0x7f40ec055e90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f40f800b000 tx=0x7f40f8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:38.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:38.918+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f40f0042970 con 0x7f410809b9d0 2022-07-12T08:48:39.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.042+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41080abd60 con 0x7f410809b9d0 2022-07-12T08:48:39.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.042+0000 7f41077fe700 1 -- 172.21.15.61:0/2929098843 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+10 (secure 0 0 0) 0x7f40f0045f90 con 0x7f410809b9d0 2022-07-12T08:48:39.049 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:39.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40ec0539d0 msgr2=0x7f40ec055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40ec0539d0 0x7f40ec055e90 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f40f800b000 tx=0x7f40f8009040).stop 2022-07-12T08:48:39.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 msgr2=0x7f410808f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 0x7f410808f9d0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 rx=0x7f4108062bd0 tx=0x7f40f0007c00).stop 2022-07-12T08:48:39.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 shutdown_connections 2022-07-12T08:48:39.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40ec0539d0 0x7f40ec055e90 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4108066330 0x7f4108064dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f41080b21c0 0x7f41080648d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 --2- 172.21.15.61:0/2929098843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f410809b9d0 0x7f410808f9d0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 >> 172.21.15.61:0/2929098843 conn(0x7f41080b57c0 msgr2=0x7f4108142420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:39.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 shutdown_connections 2022-07-12T08:48:39.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.046+0000 7f410f2c1700 1 -- 172.21.15.61:0/2929098843 wait complete. 2022-07-12T08:48:39.062 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:48:39.063 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:39.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.162+0000 7f2aa9406700 1 Processor -- start 2022-07-12T08:48:39.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.162+0000 7f2aa9406700 1 -- start start 2022-07-12T08:48:39.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.162+0000 7f2aa9406700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2aa40aee00 0x7f2aa413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:39.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.162+0000 7f2aa9406700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa40ace30 con 0x7f2aa40aca50 2022-07-12T08:48:39.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.162+0000 7f2aa9406700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa40ae420 con 0x7f2aa413ebb0 2022-07-12T08:48:39.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.162+0000 7f2aa9406700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa40b1910 con 0x7f2aa40aee00 2022-07-12T08:48:39.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2aa40aee00 0x7f2aa413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:39.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2aa40aee00 0x7f2aa413e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58838/0 (socket says 172.21.15.61:58838) 2022-07-12T08:48:39.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 -- 172.21.15.61:0/2973206016 learned_addr learned my addr 172.21.15.61:0/2973206016 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:39.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 -- 172.21.15.61:0/2973206016 >> v1:172.21.15.111:6789/0 conn(0x7f2aa413ebb0 legacy=0x7f2aa413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 -- 172.21.15.61:0/2973206016 >> v1:172.21.15.61:6789/0 conn(0x7f2aa40aca50 legacy=0x7f2aa40b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 -- 172.21.15.61:0/2973206016 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa40ae6c0 con 0x7f2aa40aee00 2022-07-12T08:48:39.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa27fc700 1 --2- 172.21.15.61:0/2973206016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa413e6c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f2a98002f70 tx=0x7f2a98006eb0).ready entity=mon.0 client_cookie=2bb9a993a5722f49 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:39.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa1ffb700 1 -- 172.21.15.61:0/2973206016 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a9800d660 con 0x7f2aa40aee00 2022-07-12T08:48:39.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa1ffb700 1 -- 172.21.15.61:0/2973206016 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98000e90 con 0x7f2aa40aee00 2022-07-12T08:48:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa1ffb700 1 -- 172.21.15.61:0/2973206016 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a9800d9f0 con 0x7f2aa40aee00 2022-07-12T08:48:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 -- 172.21.15.61:0/2973206016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 msgr2=0x7f2aa413e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 --2- 172.21.15.61:0/2973206016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa413e6c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f2a98002f70 tx=0x7f2a98006eb0).stop 2022-07-12T08:48:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 -- 172.21.15.61:0/2973206016 shutdown_connections 2022-07-12T08:48:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 --2- 172.21.15.61:0/2973206016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa413e6c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 -- 172.21.15.61:0/2973206016 >> 172.21.15.61:0/2973206016 conn(0x7f2aa4062c00 msgr2=0x7f2aa4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 -- 172.21.15.61:0/2973206016 shutdown_connections 2022-07-12T08:48:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.166+0000 7f2aa9406700 1 -- 172.21.15.61:0/2973206016 wait complete. 2022-07-12T08:48:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 Processor -- start 2022-07-12T08:48:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 -- start start 2022-07-12T08:48:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2aa40aca50 0x7f2aa40670d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa4131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa413ebb0 0x7f2aa4131d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:39.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa413e050 con 0x7f2aa40aee00 2022-07-12T08:48:39.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4132240 con 0x7f2aa40aca50 2022-07-12T08:48:39.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4138eb0 con 0x7f2aa413ebb0 2022-07-12T08:48:39.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa4131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:39.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa4131860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58844/0 (socket says 172.21.15.61:58844) 2022-07-12T08:48:39.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 -- 172.21.15.61:0/725444220 learned_addr learned my addr 172.21.15.61:0/725444220 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:39.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa2ffd700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2aa40aca50 0x7f2aa40670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:39.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa37fe700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa413ebb0 0x7f2aa4131d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:39.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 -- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa413ebb0 msgr2=0x7f2aa4131d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa413ebb0 0x7f2aa4131d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 -- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2aa40aca50 msgr2=0x7f2aa40670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2aa40aca50 0x7f2aa40670d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 -- 172.21.15.61:0/725444220 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a9800b1e0 con 0x7f2aa40aee00 2022-07-12T08:48:39.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa2ffd700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2aa40aca50 0x7f2aa40670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:39.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa37fe700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa413ebb0 0x7f2aa4131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:39.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa27fc700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa4131860 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f2a98002f70 tx=0x7f2a98015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:39.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a98011c40 con 0x7f2aa40aee00 2022-07-12T08:48:39.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a98011da0 con 0x7f2aa40aee00 2022-07-12T08:48:39.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa4139150 con 0x7f2aa40aee00 2022-07-12T08:48:39.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a9801e650 con 0x7f2aa40aee00 2022-07-12T08:48:39.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.170+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa4138070 con 0x7f2aa40aee00 2022-07-12T08:48:39.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.174+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa41392e0 con 0x7f2aa40aee00 2022-07-12T08:48:39.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.174+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2a98025030 con 0x7f2aa40aee00 2022-07-12T08:48:39.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.174+0000 7f2a93fff700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a800539b0 0x7f2a80055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:39.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.174+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2a9806d300 con 0x7f2aa40aee00 2022-07-12T08:48:39.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.174+0000 7f2aa2ffd700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a800539b0 0x7f2a80055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:39.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.174+0000 7f2aa2ffd700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a800539b0 0x7f2a80055e70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f2a9400b000 tx=0x7f2a94009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:39.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.178+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2a98044260 con 0x7f2aa40aee00 2022-07-12T08:48:39.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aa4057460 con 0x7f2aa40aee00 2022-07-12T08:48:39.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2a93fff700 1 -- 172.21.15.61:0/725444220 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f2a98023020 con 0x7f2aa40aee00 2022-07-12T08:48:39.350 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:39.351 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":50,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:35.492783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6901":{"gid":6901,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/167461784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":167461784},{"type":"v1","addr":"172.21.15.111:6835","nonce":167461784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:39.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a800539b0 msgr2=0x7f2a80055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a800539b0 0x7f2a80055e70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f2a9400b000 tx=0x7f2a94009040).stop 2022-07-12T08:48:39.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 msgr2=0x7f2aa4131860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:39.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa4131860 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f2a98002f70 tx=0x7f2a98015000).stop 2022-07-12T08:48:39.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 shutdown_connections 2022-07-12T08:48:39.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2aa413ebb0 0x7f2aa4131d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2aa40aee00 0x7f2aa4131860 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2a800539b0 0x7f2a80055e70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 --2- 172.21.15.61:0/725444220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2aa40aca50 0x7f2aa40670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:39.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 >> 172.21.15.61:0/725444220 conn(0x7f2aa4062c00 msgr2=0x7f2aa405ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:39.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 shutdown_connections 2022-07-12T08:48:39.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:39.346+0000 7f2aa9406700 1 -- 172.21.15.61:0/725444220 wait complete. 2022-07-12T08:48:39.355 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 50 2022-07-12T08:48:39.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:34.489352+0000', 'modified': '2022-07-12T08:48:35.492783+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6901}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6901': {'gid': 6901, 'name': 'b', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/167461784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 167461784}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 167461784}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:48:39.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:48:39.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:48:39.684 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6901,"want_state":"up:active","state":"up:active","rank_uptime":5.1794978169999997,"mdsmap_epoch":50,"osdmap_epoch":90,"osdmap_epoch_barrier":89,"uptime":9.5667388570000007} 2022-07-12T08:48:39.685 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6901, "mdsmap_epoch": 50, "osdmap_epoch": 90, "osdmap_epoch_barrier": 89, "rank_uptime": 5.179497817, "state": "up:active", "uptime": 9.566738857, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:48:39.686 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:48:39.687 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:48:39.688 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:48:39.689 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:39.690 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:39.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:48:39.763 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:39.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:39.796 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:39.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:39.800 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:48:39.851 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:48:39.851 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:39.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:48:39.873 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:39.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:39.888 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:39.890 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:39.891 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:48:39.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:48:39.925+0000 7fb9e1278080 -1 init, newargv = 0x557c34294720 newargc=9ceph-fuse[21847]: starting ceph client 2022-07-12T08:48:39.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr: 2022-07-12T08:48:39.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21847]: starting fuse 2022-07-12T08:48:41.037 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:48:41.057 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:41.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:41.114 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:41.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:41.118 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:48:41.127 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:48:41.128 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:48:41.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:41.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:48:41.130 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:48:41.130 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:48:41.131 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:48:41.131 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:48:41.132 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:41.133 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:48:41.133 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:48:41.134 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:48:41.134 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:41.135 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:48:41.135 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:48:41.135 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:48:41.136 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:48:41.136 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:41.137 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:48:41.137 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:48:41.137 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:48:41.138 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:48:41.139 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:48:41.140 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:41.141 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:48:41.141 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:48:41.217 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21847.asok 2022-07-12T08:48:41.223 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21847.asok 2022-07-12T08:48:41.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:41.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21847.asok status 2022-07-12T08:48:41.322 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:48:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:48:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:48:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:48:41.323 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:48:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:48:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:48:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21847", 2022-07-12T08:48:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:48:41.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:48:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:48:41.325 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 7170, 2022-07-12T08:48:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:48:41.326 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:48:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:48:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7170 2022-07-12T08:48:41.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:41.328 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:48:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:48:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:48:41.329 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2280360847 2022-07-12T08:48:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:48:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:48:41.330 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:48:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:48:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2280360847 2022-07-12T08:48:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:48:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.7170 172.21.15.121:0/2280360847", 2022-07-12T08:48:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2280360847", 2022-07-12T08:48:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:48:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 50, 2022-07-12T08:48:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 90, 2022-07-12T08:48:41.332 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:48:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:48:41.333 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:48:41.333 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:48:41.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:41.338 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:41.338 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:48:41.348 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:41.349 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:41.350 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:41.410 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 Processor -- start 2022-07-12T08:48:41.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 -- start start 2022-07-12T08:48:41.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17d40af280 0x7f17d4060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d40b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d40aecb0 con 0x7f17d409bce0 2022-07-12T08:48:41.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d40afd70 con 0x7f17d40af280 2022-07-12T08:48:41.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d3fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4143b00 con 0x7f17d40b3f60 2022-07-12T08:48:41.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d37fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d40b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17d40af280 0x7f17d4060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f17d40af280 0x7f17d4060680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:37424/0 (socket says 172.21.15.61:37424) 2022-07-12T08:48:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 -- 172.21.15.61:0/2254108140 learned_addr learned my addr 172.21.15.61:0/2254108140 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d27fc700 1 --1- 172.21.15.61:0/2254108140 >> v1:172.21.15.111:6789/0 conn(0x7f17d409bce0 0x7f17d40af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:60670/0 (socket says 172.21.15.61:60670) 2022-07-12T08:48:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 -- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 msgr2=0x7f17d40b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 --2- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d40b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 -- 172.21.15.61:0/2254108140 >> v1:172.21.15.111:6789/0 conn(0x7f17d409bce0 legacy=0x7f17d40af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 -- 172.21.15.61:0/2254108140 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d41435c0 con 0x7f17d40af280 2022-07-12T08:48:41.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d2ffd700 1 --2- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d4060680 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f17c4002f70 tx=0x7f17c4006e50).ready entity=mon.1 client_cookie=eb32fcf81433cb14 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:41.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d1ffb700 1 -- 172.21.15.61:0/2254108140 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c400d5c0 con 0x7f17d40af280 2022-07-12T08:48:41.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d1ffb700 1 -- 172.21.15.61:0/2254108140 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4000e90 con 0x7f17d40af280 2022-07-12T08:48:41.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.518+0000 7f17d1ffb700 1 -- 172.21.15.61:0/2254108140 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c400d950 con 0x7f17d40af280 2022-07-12T08:48:41.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 msgr2=0x7f17d4060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 --2- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d4060680 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f17c4002f70 tx=0x7f17c4006e50).stop 2022-07-12T08:48:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- 172.21.15.61:0/2254108140 shutdown_connections 2022-07-12T08:48:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 --2- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d40b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 --2- 172.21.15.61:0/2254108140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d4060680 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- 172.21.15.61:0/2254108140 >> 172.21.15.61:0/2254108140 conn(0x7f17d40b92c0 msgr2=0x7f17d40b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:41.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- 172.21.15.61:0/2254108140 shutdown_connections 2022-07-12T08:48:41.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- 172.21.15.61:0/2254108140 wait complete. 2022-07-12T08:48:41.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 Processor -- start 2022-07-12T08:48:41.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- start start 2022-07-12T08:48:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 0x7f17d4145030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d413d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4146230 con 0x7f17d409bce0 2022-07-12T08:48:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d41464d0 con 0x7f17d40af280 2022-07-12T08:48:41.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.522+0000 7f17d3fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4146770 con 0x7f17d40b3f60 2022-07-12T08:48:41.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 0x7f17d4145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 0x7f17d4145030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58856/0 (socket says 172.21.15.61:58856) 2022-07-12T08:48:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 -- 172.21.15.61:0/638794598 learned_addr learned my addr 172.21.15.61:0/638794598 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d27fc700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d37fe700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d413d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 -- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 msgr2=0x7f17d413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d413d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 -- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 msgr2=0x7f17d40b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d40b62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 -- 172.21.15.61:0/638794598 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c400b1e0 con 0x7f17d409bce0 2022-07-12T08:48:41.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d2ffd700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 0x7f17d4145030 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f17c4002f70 tx=0x7f17c4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:41.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c4016070 con 0x7f17d409bce0 2022-07-12T08:48:41.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c40137d0 con 0x7f17d409bce0 2022-07-12T08:48:41.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17c4013b30 con 0x7f17d409bce0 2022-07-12T08:48:41.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d4145eb0 con 0x7f17d409bce0 2022-07-12T08:48:41.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d4146e50 con 0x7f17d409bce0 2022-07-12T08:48:41.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.526+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d4146930 con 0x7f17d409bce0 2022-07-12T08:48:41.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.530+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f17c4025030 con 0x7f17d409bce0 2022-07-12T08:48:41.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.530+0000 7f17cbfff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f17b8053a60 0x7f17b8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.530+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f17c406ce10 con 0x7f17d409bce0 2022-07-12T08:48:41.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.530+0000 7f17d27fc700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f17b8053a60 0x7f17b8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.530+0000 7f17d27fc700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f17b8053a60 0x7f17b8055f20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f17bc00d000 tx=0x7f17bc007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:41.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.534+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f17c4043d70 con 0x7f17d409bce0 2022-07-12T08:48:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.662+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17d406b2b0 con 0x7f17d409bce0 2022-07-12T08:48:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.662+0000 7f17cbfff700 1 -- 172.21.15.61:0/638794598 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f17c4023090 con 0x7f17d409bce0 2022-07-12T08:48:41.669 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:41.670 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":90,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:35.492446+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:32.975784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:33.715444+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:41.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f17b8053a60 msgr2=0x7f17b8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f17b8053a60 0x7f17b8055f20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f17bc00d000 tx=0x7f17bc007b80).stop 2022-07-12T08:48:41.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 msgr2=0x7f17d4145030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 0x7f17d4145030 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f17c4002f70 tx=0x7f17c4014000).stop 2022-07-12T08:48:41.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 shutdown_connections 2022-07-12T08:48:41.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f17d40b3f60 0x7f17d413d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f17b8053a60 0x7f17b8055f20 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f17d40af280 0x7f17d40b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 --2- 172.21.15.61:0/638794598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17d409bce0 0x7f17d4145030 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 >> 172.21.15.61:0/638794598 conn(0x7f17d40b92c0 msgr2=0x7f17d40af660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 shutdown_connections 2022-07-12T08:48:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.666+0000 7f17d3fff700 1 -- 172.21.15.61:0/638794598 wait complete. 2022-07-12T08:48:41.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:48:32.975784+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:48:33.715444+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:48:41.686 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:41.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.794+0000 7fb468841700 1 Processor -- start 2022-07-12T08:48:41.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.794+0000 7fb468841700 1 -- start start 2022-07-12T08:48:41.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb468841700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464060490 0x7fb4640acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb468841700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 0x7fb464146430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb468841700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb464146920 0x7fb46405c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb468841700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb464060870 con 0x7fb4640b0a90 2022-07-12T08:48:41.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb468841700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb464098930 con 0x7fb464146920 2022-07-12T08:48:41.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb468841700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb464098bd0 con 0x7fb464060490 2022-07-12T08:48:41.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb46259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 0x7fb464146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464060490 0x7fb4640acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb46359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb464146920 0x7fb46405c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb46259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 0x7fb464146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58868/0 (socket says 172.21.15.61:58868) 2022-07-12T08:48:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb46259c700 1 -- 172.21.15.61:0/1852403217 learned_addr learned my addr 172.21.15.61:0/1852403217 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 -- 172.21.15.61:0/1852403217 >> v2:172.21.15.111:3300/0 conn(0x7fb464146920 msgr2=0x7fb46405c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 --2- 172.21.15.61:0/1852403217 >> v2:172.21.15.111:3300/0 conn(0x7fb464146920 0x7fb46405c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 -- 172.21.15.61:0/1852403217 >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 msgr2=0x7fb464146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:41.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 --2- 172.21.15.61:0/1852403217 >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 0x7fb464146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 -- 172.21.15.61:0/1852403217 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb464063450 con 0x7fb464060490 2022-07-12T08:48:41.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb46259c700 1 --2- 172.21.15.61:0/1852403217 >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 0x7fb464146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:41.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb46359e700 1 --2- 172.21.15.61:0/1852403217 >> v2:172.21.15.111:3300/0 conn(0x7fb464146920 0x7fb46405c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:41.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb462d9d700 1 --2- 172.21.15.61:0/1852403217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464060490 0x7fb4640acd00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fb454002f70 tx=0x7fb454006df0).ready entity=mon.2 client_cookie=7ca269d3231b99ba server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:41.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb461d9b700 1 -- 172.21.15.61:0/1852403217 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb45400d4a0 con 0x7fb464060490 2022-07-12T08:48:41.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb461d9b700 1 -- 172.21.15.61:0/1852403217 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb454000e90 con 0x7fb464060490 2022-07-12T08:48:41.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.798+0000 7fb461d9b700 1 -- 172.21.15.61:0/1852403217 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb45400d7f0 con 0x7fb464060490 2022-07-12T08:48:41.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- 172.21.15.61:0/1852403217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464060490 msgr2=0x7fb4640acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- 172.21.15.61:0/1852403217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464060490 0x7fb4640acd00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fb454002f70 tx=0x7fb454006df0).stop 2022-07-12T08:48:41.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- 172.21.15.61:0/1852403217 shutdown_connections 2022-07-12T08:48:41.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- 172.21.15.61:0/1852403217 >> v2:172.21.15.111:3300/0 conn(0x7fb464146920 0x7fb46405c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- 172.21.15.61:0/1852403217 >> v2:172.21.15.61:3300/0 conn(0x7fb4640b0a90 0x7fb464146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- 172.21.15.61:0/1852403217 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464060490 0x7fb4640acd00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- 172.21.15.61:0/1852403217 >> 172.21.15.61:0/1852403217 conn(0x7fb4640b6b80 msgr2=0x7fb464060d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- 172.21.15.61:0/1852403217 shutdown_connections 2022-07-12T08:48:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- 172.21.15.61:0/1852403217 wait complete. 2022-07-12T08:48:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 Processor -- start 2022-07-12T08:48:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- start start 2022-07-12T08:48:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 0x7fb4640ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4640b0a90 0x7fb4640aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464146920 0x7fb46413c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46409c180 con 0x7fb464060490 2022-07-12T08:48:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4640af890 con 0x7fb4640b0a90 2022-07-12T08:48:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb468841700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46413e0d0 con 0x7fb464146920 2022-07-12T08:48:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 0x7fb4640ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 0x7fb4640ae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58870/0 (socket says 172.21.15.61:58870) 2022-07-12T08:48:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 -- 172.21.15.61:0/2753763862 learned_addr learned my addr 172.21.15.61:0/2753763862 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb46259c700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4640b0a90 0x7fb4640aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb46359e700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464146920 0x7fb46413c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 -- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464146920 msgr2=0x7fb46413c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464146920 0x7fb46413c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 -- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4640b0a90 msgr2=0x7fb4640aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.802+0000 7fb462d9d700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4640b0a90 0x7fb4640aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb462d9d700 1 -- 172.21.15.61:0/2753763862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45800b1e0 con 0x7fb464060490 2022-07-12T08:48:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb46259c700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4640b0a90 0x7fb4640aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb462d9d700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 0x7fb4640ae960 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7fb454002f70 tx=0x7fb454014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb454013d50 con 0x7fb464060490 2022-07-12T08:48:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45401e400 con 0x7fb464060490 2022-07-12T08:48:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb45401e760 con 0x7fb464060490 2022-07-12T08:48:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c009470 con 0x7fb464060490 2022-07-12T08:48:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45400b180 con 0x7fb464060490 2022-07-12T08:48:41.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4640ad8f0 con 0x7fb464060490 2022-07-12T08:48:41.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb45401c070 con 0x7fb464060490 2022-07-12T08:48:41.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb453fff700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb440053920 0x7fb440055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:41.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb45406c980 con 0x7fb464060490 2022-07-12T08:48:41.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.806+0000 7fb46259c700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb440053920 0x7fb440055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:41.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.810+0000 7fb46259c700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb440053920 0x7fb440055de0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb45800b000 tx=0x7fb458008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:41.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.814+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4640ad8f0 con 0x7fb464060490 2022-07-12T08:48:41.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.934+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb4640ad8f0 con 0x7fb464060490 2022-07-12T08:48:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.934+0000 7fb453fff700 1 -- 172.21.15.61:0/2753763862 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v90) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb4640ad8f0 con 0x7fb464060490 2022-07-12T08:48:41.942 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb440053920 msgr2=0x7fb440055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb440053920 0x7fb440055de0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb45800b000 tx=0x7fb458008040).stop 2022-07-12T08:48:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 msgr2=0x7fb4640ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 0x7fb4640ae960 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7fb454002f70 tx=0x7fb454014000).stop 2022-07-12T08:48:41.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 shutdown_connections 2022-07-12T08:48:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb464146920 0x7fb46413c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb440053920 0x7fb440055de0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4640b0a90 0x7fb4640aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 --2- 172.21.15.61:0/2753763862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb464060490 0x7fb4640ae960 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:41.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 >> 172.21.15.61:0/2753763862 conn(0x7fb4640b6b80 msgr2=0x7fb46405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:41.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 shutdown_connections 2022-07-12T08:48:41.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:41.938+0000 7fb468841700 1 -- 172.21.15.61:0/2753763862 wait complete. 2022-07-12T08:48:41.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:42.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 Processor -- start 2022-07-12T08:48:42.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 -- start start 2022-07-12T08:48:42.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2e340aea80 0x7f2e34064b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e340b0c20 0x7f2e34144310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e34140110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3405d240 con 0x7f2e340aea80 2022-07-12T08:48:42.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e34064400 con 0x7f2e340b0c20 2022-07-12T08:48:42.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e38dab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340646a0 con 0x7f2e34144800 2022-07-12T08:48:42.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2e340aea80 0x7f2e34064b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e31d9b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2e340b0c20 0x7f2e34144310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2e340aea80 0x7f2e34064b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58880/0 (socket says 172.21.15.61:58880) 2022-07-12T08:48:42.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 -- 172.21.15.61:0/3065388729 learned_addr learned my addr 172.21.15.61:0/3065388729 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e32d9d700 1 --2- 172.21.15.61:0/3065388729 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e34140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 -- 172.21.15.61:0/3065388729 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 msgr2=0x7f2e34140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 --2- 172.21.15.61:0/3065388729 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e34140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 -- 172.21.15.61:0/3065388729 >> v2:172.21.15.111:3300/0 conn(0x7f2e340b0c20 msgr2=0x7f2e34144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 --2- 172.21.15.61:0/3065388729 >> v2:172.21.15.111:3300/0 conn(0x7f2e340b0c20 0x7f2e34144310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 -- 172.21.15.61:0/3065388729 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340affd0 con 0x7f2e340aea80 2022-07-12T08:48:42.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e31d9b700 1 --2- 172.21.15.61:0/3065388729 >> v2:172.21.15.111:3300/0 conn(0x7f2e340b0c20 0x7f2e34144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:42.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.062+0000 7f2e3259c700 1 --2- 172.21.15.61:0/3065388729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064b40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f2e1c002f70 tx=0x7f2e1c006eb0).ready entity=mon.0 client_cookie=e97a254afe67ea4e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e3159a700 1 -- 172.21.15.61:0/3065388729 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e1c00d570 con 0x7f2e340aea80 2022-07-12T08:48:42.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e3159a700 1 -- 172.21.15.61:0/3065388729 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c000e90 con 0x7f2e340aea80 2022-07-12T08:48:42.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e3159a700 1 -- 172.21.15.61:0/3065388729 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e1c00d900 con 0x7f2e340aea80 2022-07-12T08:48:42.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 -- 172.21.15.61:0/3065388729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 msgr2=0x7f2e34064b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 --2- 172.21.15.61:0/3065388729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064b40 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f2e1c002f70 tx=0x7f2e1c006eb0).stop 2022-07-12T08:48:42.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 -- 172.21.15.61:0/3065388729 shutdown_connections 2022-07-12T08:48:42.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 --2- 172.21.15.61:0/3065388729 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e34140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 --2- 172.21.15.61:0/3065388729 >> v2:172.21.15.111:3300/0 conn(0x7f2e340b0c20 0x7f2e34144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 --2- 172.21.15.61:0/3065388729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064b40 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 -- 172.21.15.61:0/3065388729 >> 172.21.15.61:0/3065388729 conn(0x7f2e340b5290 msgr2=0x7f2e340b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 -- 172.21.15.61:0/3065388729 shutdown_connections 2022-07-12T08:48:42.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 -- 172.21.15.61:0/3065388729 wait complete. 2022-07-12T08:48:42.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 Processor -- start 2022-07-12T08:48:42.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.066+0000 7f2e38dab700 1 -- start start 2022-07-12T08:48:42.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e340b0c20 0x7f2e34065290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e340ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3413c200 con 0x7f2e340aea80 2022-07-12T08:48:42.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340b4b90 con 0x7f2e340b0c20 2022-07-12T08:48:42.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340b4e30 con 0x7f2e34144800 2022-07-12T08:48:42.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58884/0 (socket says 172.21.15.61:58884) 2022-07-12T08:48:42.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 -- 172.21.15.61:0/4213905007 learned_addr learned my addr 172.21.15.61:0/4213905007 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e31d9b700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e340b0c20 0x7f2e34065290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e32d9d700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e340ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 -- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 msgr2=0x7f2e340ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e340ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 -- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e340b0c20 msgr2=0x7f2e34065290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e340b0c20 0x7f2e34065290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 -- 172.21.15.61:0/4213905007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24009470 con 0x7f2e340aea80 2022-07-12T08:48:42.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e3259c700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064da0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f2e1c002f70 tx=0x7f2e1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e1c0117e0 con 0x7f2e340aea80 2022-07-12T08:48:42.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c011940 con 0x7f2e340aea80 2022-07-12T08:48:42.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c00b1e0 con 0x7f2e340aea80 2022-07-12T08:48:42.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e1c011ca0 con 0x7f2e340aea80 2022-07-12T08:48:42.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e340b3cb0 con 0x7f2e340aea80 2022-07-12T08:48:42.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.070+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2e1c025030 con 0x7f2e340aea80 2022-07-12T08:48:42.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.074+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e34057460 con 0x7f2e340aea80 2022-07-12T08:48:42.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.074+0000 7f2e2b7fe700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e18053920 0x7f2e18055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.074+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2e1c06cf10 con 0x7f2e340aea80 2022-07-12T08:48:42.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.074+0000 7f2e31d9b700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e18053920 0x7f2e18055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.074+0000 7f2e31d9b700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e18053920 0x7f2e18055de0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f2e2400b000 tx=0x7f2e240091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.078+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e1c043e70 con 0x7f2e340aea80 2022-07-12T08:48:42.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.206+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2e340447a0 con 0x7f2e340aea80 2022-07-12T08:48:42.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.206+0000 7f2e2b7fe700 1 -- 172.21.15.61:0/4213905007 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v90) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2e1c0230a0 con 0x7f2e340aea80 2022-07-12T08:48:42.223 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:42.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e18053920 msgr2=0x7f2e18055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e18053920 0x7f2e18055de0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f2e2400b000 tx=0x7f2e240091d0).stop 2022-07-12T08:48:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 msgr2=0x7f2e34064da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064da0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f2e1c002f70 tx=0x7f2e1c015000).stop 2022-07-12T08:48:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 shutdown_connections 2022-07-12T08:48:42.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e34144800 0x7f2e340ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e18053920 0x7f2e18055de0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e340b0c20 0x7f2e34065290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 --2- 172.21.15.61:0/4213905007 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e340aea80 0x7f2e34064da0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 >> 172.21.15.61:0/4213905007 conn(0x7f2e340b5290 msgr2=0x7f2e34142330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 shutdown_connections 2022-07-12T08:48:42.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.210+0000 7f2e38dab700 1 -- 172.21.15.61:0/4213905007 wait complete. 2022-07-12T08:48:42.232 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:48:42.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.338+0000 7f31dbfff700 1 Processor -- start 2022-07-12T08:48:42.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.338+0000 7f31dbfff700 1 -- start start 2022-07-12T08:48:42.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f31dc061840 0x7f31dc142f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0b1650 0x7f31dc0b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f31dc0af0d0 0x7f31dc0af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc13f900 con 0x7f31dc0af0d0 2022-07-12T08:48:42.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc0ac340 con 0x7f31dc061840 2022-07-12T08:48:42.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc061c20 con 0x7f31dc0b1650 2022-07-12T08:48:42.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f31dc0af0d0 0x7f31dc0af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f31dc0af0d0 0x7f31dc0af560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58894/0 (socket says 172.21.15.61:58894) 2022-07-12T08:48:42.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 -- 172.21.15.61:0/1169348231 learned_addr learned my addr 172.21.15.61:0/1169348231 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31d3fff700 1 --2- 172.21.15.61:0/1169348231 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0b1650 0x7f31dc0b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31daffd700 1 --2- 172.21.15.61:0/1169348231 >> v2:172.21.15.111:3300/0 conn(0x7f31dc061840 0x7f31dc142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 -- 172.21.15.61:0/1169348231 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0b1650 msgr2=0x7f31dc0b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 --2- 172.21.15.61:0/1169348231 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0b1650 0x7f31dc0b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 -- 172.21.15.61:0/1169348231 >> v2:172.21.15.111:3300/0 conn(0x7f31dc061840 msgr2=0x7f31dc142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 --2- 172.21.15.61:0/1169348231 >> v2:172.21.15.111:3300/0 conn(0x7f31dc061840 0x7f31dc142f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 -- 172.21.15.61:0/1169348231 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0b1f80 con 0x7f31dc0af0d0 2022-07-12T08:48:42.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31db7fe700 1 --2- 172.21.15.61:0/1169348231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0af0d0 0x7f31dc0af560 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f31d40032f0 tx=0x7f31d4009290).ready entity=mon.0 client_cookie=79d597070408023d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31da7fc700 1 -- 172.21.15.61:0/1169348231 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31d400b870 con 0x7f31dc0af0d0 2022-07-12T08:48:42.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31da7fc700 1 -- 172.21.15.61:0/1169348231 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d400b9d0 con 0x7f31dc0af0d0 2022-07-12T08:48:42.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31da7fc700 1 -- 172.21.15.61:0/1169348231 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31d400bb40 con 0x7f31dc0af0d0 2022-07-12T08:48:42.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- 172.21.15.61:0/1169348231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0af0d0 msgr2=0x7f31dc0af560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- 172.21.15.61:0/1169348231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0af0d0 0x7f31dc0af560 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f31d40032f0 tx=0x7f31d4009290).stop 2022-07-12T08:48:42.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- 172.21.15.61:0/1169348231 shutdown_connections 2022-07-12T08:48:42.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- 172.21.15.61:0/1169348231 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0b1650 0x7f31dc0b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- 172.21.15.61:0/1169348231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0af0d0 0x7f31dc0af560 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 --2- 172.21.15.61:0/1169348231 >> v2:172.21.15.111:3300/0 conn(0x7f31dc061840 0x7f31dc142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- 172.21.15.61:0/1169348231 >> 172.21.15.61:0/1169348231 conn(0x7f31dc0b57c0 msgr2=0x7f31dc0b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- 172.21.15.61:0/1169348231 shutdown_connections 2022-07-12T08:48:42.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.342+0000 7f31dbfff700 1 -- 172.21.15.61:0/1169348231 wait complete. 2022-07-12T08:48:42.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 Processor -- start 2022-07-12T08:48:42.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 -- start start 2022-07-12T08:48:42.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31dc061840 0x7f31dc0ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0af0d0 0x7f31dc13e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 0x7f31dc152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc13f210 con 0x7f31dc0b1650 2022-07-12T08:48:42.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc13edc0 con 0x7f31dc061840 2022-07-12T08:48:42.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31dbfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc13eb80 con 0x7f31dc0af0d0 2022-07-12T08:48:42.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31db7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 0x7f31dc152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31db7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 0x7f31dc152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58902/0 (socket says 172.21.15.61:58902) 2022-07-12T08:48:42.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31db7fe700 1 -- 172.21.15.61:0/712398704 learned_addr learned my addr 172.21.15.61:0/712398704 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31d3fff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0af0d0 0x7f31dc13e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.346+0000 7f31daffd700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31dc061840 0x7f31dc0ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31db7fe700 1 -- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0af0d0 msgr2=0x7f31dc13e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31db7fe700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0af0d0 0x7f31dc13e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31db7fe700 1 -- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31dc061840 msgr2=0x7f31dc0ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31db7fe700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31dc061840 0x7f31dc0ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31db7fe700 1 -- 172.21.15.61:0/712398704 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4009530 con 0x7f31dc0b1650 2022-07-12T08:48:42.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31daffd700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31dc061840 0x7f31dc0ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:42.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31db7fe700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 0x7f31dc152df0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f31d40032f0 tx=0x7f31d4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31d400b870 con 0x7f31dc0b1650 2022-07-12T08:48:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d4021670 con 0x7f31dc0b1650 2022-07-12T08:48:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc13eff0 con 0x7f31dc0b1650 2022-07-12T08:48:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31d40219d0 con 0x7f31dc0b1650 2022-07-12T08:48:42.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc13f440 con 0x7f31dc0b1650 2022-07-12T08:48:42.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f31d4028070 con 0x7f31dc0b1650 2022-07-12T08:48:42.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc057460 con 0x7f31dc0b1650 2022-07-12T08:48:42.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31d8ff9700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f31c00539b0 0x7f31c0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f31d406cf50 con 0x7f31dc0b1650 2022-07-12T08:48:42.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31daffd700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f31c00539b0 0x7f31c0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.350+0000 7f31daffd700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f31c00539b0 0x7f31c0055e70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f31cc00c000 tx=0x7f31cc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.358+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f31d4043f60 con 0x7f31dc0b1650 2022-07-12T08:48:42.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.486+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f31dc13f1f0 con 0x7f31dc0b1650 2022-07-12T08:48:42.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.486+0000 7f31d8ff9700 1 -- 172.21.15.61:0/712398704 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+10 (secure 0 0 0) 0x7f31d4015070 con 0x7f31dc0b1650 2022-07-12T08:48:42.493 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:42.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f31c00539b0 msgr2=0x7f31c0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f31c00539b0 0x7f31c0055e70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7f31cc00c000 tx=0x7f31cc008040).stop 2022-07-12T08:48:42.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 msgr2=0x7f31dc152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 0x7f31dc152df0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f31d40032f0 tx=0x7f31d4016000).stop 2022-07-12T08:48:42.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 shutdown_connections 2022-07-12T08:48:42.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f31dc0b1650 0x7f31dc152df0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f31dc0af0d0 0x7f31dc13e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f31c00539b0 0x7f31c0055e70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 --2- 172.21.15.61:0/712398704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f31dc061840 0x7f31dc0ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 >> 172.21.15.61:0/712398704 conn(0x7f31dc0b57c0 msgr2=0x7f31dc0b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 shutdown_connections 2022-07-12T08:48:42.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.490+0000 7f31dbfff700 1 -- 172.21.15.61:0/712398704 wait complete. 2022-07-12T08:48:42.506 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:48:42.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.610+0000 7fd097268700 1 Processor -- start 2022-07-12T08:48:42.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.610+0000 7fd097268700 1 -- start start 2022-07-12T08:48:42.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd097268700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd0900b5920 0x7fd0900aa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd097268700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0900abfa0 con 0x7fd0900b0240 2022-07-12T08:48:42.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd097268700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09005cd00 con 0x7fd0900b06b0 2022-07-12T08:48:42.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd097268700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09005c870 con 0x7fd0900b5920 2022-07-12T08:48:42.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd0900b0240 0x7fd09005faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47124/0 (socket says 172.21.15.61:47124) 2022-07-12T08:48:42.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 -- 172.21.15.61:0/2033478277 learned_addr learned my addr 172.21.15.61:0/2033478277 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:42.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd096266700 1 --2- 172.21.15.61:0/2033478277 >> v2:172.21.15.61:3300/0 conn(0x7fd0900b5920 0x7fd0900aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547361107 0 0) 0x7fd0900abfa0 con 0x7fd0900b0240 2022-07-12T08:48:42.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd078003260 con 0x7fd0900b0240 2022-07-12T08:48:42.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061311755 0 0) 0x7fd078003260 con 0x7fd0900b0240 2022-07-12T08:48:42.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 >> v2:172.21.15.61:3300/0 conn(0x7fd0900b5920 msgr2=0x7fd0900aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 --2- 172.21.15.61:0/2033478277 >> v2:172.21.15.61:3300/0 conn(0x7fd0900b5920 0x7fd0900aa500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 >> v1:172.21.15.111:6789/0 conn(0x7fd0900b06b0 legacy=0x7fd090060200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09005cad0 con 0x7fd0900b0240 2022-07-12T08:48:42.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fd08c001660 con 0x7fd0900b0240 2022-07-12T08:48:42.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 >> v1:172.21.15.61:6789/0 conn(0x7fd0900b0240 legacy=0x7fd09005faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078003750 0x7fd078003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd078004500 0x7fd078004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd078004e70 0x7fd07800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09005c870 con 0x7fd078004e70 2022-07-12T08:48:42.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07800bd20 con 0x7fd078004500 2022-07-12T08:48:42.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07800bee0 con 0x7fd078003750 2022-07-12T08:48:42.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd096266700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078003750 0x7fd078003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd078004500 0x7fd078004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd096a67700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd078004e70 0x7fd07800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 -- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078003750 msgr2=0x7fd078003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078003750 0x7fd078003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 -- 172.21.15.61:0/2033478277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd078004e70 msgr2=0x7fd07800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd078004e70 0x7fd07800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 -- 172.21.15.61:0/2033478277 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078003260 con 0x7fd078004500 2022-07-12T08:48:42.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd096266700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078003750 0x7fd078003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:42.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd096a67700 1 -- 172.21.15.61:0/2033478277 reap_dead start 2022-07-12T08:48:42.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095a65700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd078004500 0x7fd078004980 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd08c001370 tx=0x7fd08c001e10).ready entity=mon.1 client_cookie=ba7dbfa7c16f2b0c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd08c0071a0 con 0x7fd078004500 2022-07-12T08:48:42.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c007300 con 0x7fd078004500 2022-07-12T08:48:42.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd095264700 1 -- 172.21.15.61:0/2033478277 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd08c00c8f0 con 0x7fd078004500 2022-07-12T08:48:42.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd097268700 1 -- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd078004500 msgr2=0x7fd078004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.614+0000 7fd097268700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd078004500 0x7fd078004980 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fd08c001370 tx=0x7fd08c001e10).stop 2022-07-12T08:48:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- 172.21.15.61:0/2033478277 shutdown_connections 2022-07-12T08:48:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 --2- 172.21.15.61:0/2033478277 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd078004500 0x7fd078004980 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- 172.21.15.61:0/2033478277 >> 172.21.15.61:0/2033478277 conn(0x7fd090063340 msgr2=0x7fd0900b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- 172.21.15.61:0/2033478277 shutdown_connections 2022-07-12T08:48:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- 172.21.15.61:0/2033478277 wait complete. 2022-07-12T08:48:42.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 Processor -- start 2022-07-12T08:48:42.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- start start 2022-07-12T08:48:42.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078004500 0x7fd0900992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 0x7fd09013fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd090151d30 0x7fd0901521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0900ad850 con 0x7fd09013fa70 2022-07-12T08:48:42.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0900ae820 con 0x7fd090151d30 2022-07-12T08:48:42.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd097268700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0900aeab0 con 0x7fd078004500 2022-07-12T08:48:42.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 0x7fd09013fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 0x7fd09013fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58920/0 (socket says 172.21.15.61:58920) 2022-07-12T08:48:42.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 -- 172.21.15.61:0/527372902 learned_addr learned my addr 172.21.15.61:0/527372902 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd096a67700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd090151d30 0x7fd0901521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd096266700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078004500 0x7fd0900992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 -- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078004500 msgr2=0x7fd0900992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078004500 0x7fd0900992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 -- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd090151d30 msgr2=0x7fd0901521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd090151d30 0x7fd0901521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 -- 172.21.15.61:0/527372902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900af870 con 0x7fd09013fa70 2022-07-12T08:48:42.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd095a65700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 0x7fd09013fef0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7fd08c001370 tx=0x7fd08c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.618+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd08c00ce70 con 0x7fd09013fa70 2022-07-12T08:48:42.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c004520 con 0x7fd09013fa70 2022-07-12T08:48:42.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd08c004880 con 0x7fd09013fa70 2022-07-12T08:48:42.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd090152ea0 con 0x7fd09013fa70 2022-07-12T08:48:42.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0900979d0 con 0x7fd09013fa70 2022-07-12T08:48:42.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd090057460 con 0x7fd09013fa70 2022-07-12T08:48:42.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd08c010070 con 0x7fd09013fa70 2022-07-12T08:48:42.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd07f7fe700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd070053af0 0x7fd070055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd08c06cb10 con 0x7fd09013fa70 2022-07-12T08:48:42.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.622+0000 7fd096266700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd070053af0 0x7fd070055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.626+0000 7fd096266700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd070053af0 0x7fd070055fb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fd08000b000 tx=0x7fd080006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.630+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd08c043a70 con 0x7fd09013fa70 2022-07-12T08:48:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0900aef00 con 0x7fd09013fa70 2022-07-12T08:48:42.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd07f7fe700 1 -- 172.21.15.61:0/527372902 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd08c025070 con 0x7fd09013fa70 2022-07-12T08:48:42.764 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:42.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd070053af0 msgr2=0x7fd070055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd070053af0 0x7fd070055fb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7fd08000b000 tx=0x7fd080006df0).stop 2022-07-12T08:48:42.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 msgr2=0x7fd09013fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 0x7fd09013fef0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7fd08c001370 tx=0x7fd08c015000).stop 2022-07-12T08:48:42.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 shutdown_connections 2022-07-12T08:48:42.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd070053af0 0x7fd070055fb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd090151d30 0x7fd0901521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd09013fa70 0x7fd09013fef0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 --2- 172.21.15.61:0/527372902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd078004500 0x7fd0900992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 >> 172.21.15.61:0/527372902 conn(0x7fd090063340 msgr2=0x7fd0900b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 shutdown_connections 2022-07-12T08:48:42.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.758+0000 7fd097268700 1 -- 172.21.15.61:0/527372902 wait complete. 2022-07-12T08:48:42.785 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:48:42.786 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:42.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 Processor -- start 2022-07-12T08:48:42.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 -- start start 2022-07-12T08:48:42.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9e240ab290 0x7f9e240ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e2405f140 0x7f9e2405ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e240abc70 con 0x7f9e240afa60 2022-07-12T08:48:42.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2405f580 con 0x7f9e240ab290 2022-07-12T08:48:42.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e291ae700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2405c720 con 0x7f9e2405f140 2022-07-12T08:48:42.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.890+0000 7f9e23fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9e240ab290 0x7f9e240ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9e240ab290 0x7f9e240ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58926/0 (socket says 172.21.15.61:58926) 2022-07-12T08:48:42.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 -- 172.21.15.61:0/3576745139 learned_addr learned my addr 172.21.15.61:0/3576745139 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 -- 172.21.15.61:0/3576745139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e2405f140 msgr2=0x7f9e2405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e237fe700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e2405f140 0x7f9e2405ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e2405f140 0x7f9e2405ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 -- 172.21.15.61:0/3576745139 >> v1:172.21.15.111:6789/0 conn(0x7f9e240afa60 legacy=0x7f9e2414a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:42.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 -- 172.21.15.61:0/3576745139 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2405c9c0 con 0x7f9e240ab290 2022-07-12T08:48:42.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e237fe700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e2405f140 0x7f9e2405ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:42.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e23fff700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e240ab290 0x7f9e240ab670 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f9e2405cd40 tx=0x7f9e1c009290).ready entity=mon.0 client_cookie=c3c6ad3742169d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e22ffd700 1 -- 172.21.15.61:0/3576745139 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1c00b860 con 0x7f9e240ab290 2022-07-12T08:48:42.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e22ffd700 1 -- 172.21.15.61:0/3576745139 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c00b9c0 con 0x7f9e240ab290 2022-07-12T08:48:42.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e22ffd700 1 -- 172.21.15.61:0/3576745139 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1c00bb70 con 0x7f9e240ab290 2022-07-12T08:48:42.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 -- 172.21.15.61:0/3576745139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e240ab290 msgr2=0x7f9e240ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e240ab290 0x7f9e240ab670 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7f9e2405cd40 tx=0x7f9e1c009290).stop 2022-07-12T08:48:42.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 -- 172.21.15.61:0/3576745139 shutdown_connections 2022-07-12T08:48:42.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e2405f140 0x7f9e2405ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 --2- 172.21.15.61:0/3576745139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e240ab290 0x7f9e240ab670 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 -- 172.21.15.61:0/3576745139 >> 172.21.15.61:0/3576745139 conn(0x7f9e240b1950 msgr2=0x7f9e240b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:42.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 -- 172.21.15.61:0/3576745139 shutdown_connections 2022-07-12T08:48:42.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 -- 172.21.15.61:0/3576745139 wait complete. 2022-07-12T08:48:42.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 Processor -- start 2022-07-12T08:48:42.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.894+0000 7f9e291ae700 1 -- start start 2022-07-12T08:48:42.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 0x7f9e24098ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e240ab290 0x7f9e24156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e240afa60 0x7f9e24156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24149790 con 0x7f9e2405f140 2022-07-12T08:48:42.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24149a30 con 0x7f9e240ab290 2022-07-12T08:48:42.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e24065280 con 0x7f9e240afa60 2022-07-12T08:48:42.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 0x7f9e24098ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 0x7f9e24098ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58932/0 (socket says 172.21.15.61:58932) 2022-07-12T08:48:42.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 -- 172.21.15.61:0/232270549 learned_addr learned my addr 172.21.15.61:0/232270549 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:42.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e237fe700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e240ab290 0x7f9e24156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e289ad700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e240afa60 0x7f9e24156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 -- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e240afa60 msgr2=0x7f9e24156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e240afa60 0x7f9e24156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 -- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e240ab290 msgr2=0x7f9e24156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:42.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e240ab290 0x7f9e24156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:42.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 -- 172.21.15.61:0/232270549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c009530 con 0x7f9e2405f140 2022-07-12T08:48:42.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e23fff700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 0x7f9e24098ab0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f9e2405cd40 tx=0x7f9e1c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1c021630 con 0x7f9e2405f140 2022-07-12T08:48:42.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1c021790 con 0x7f9e2405f140 2022-07-12T08:48:42.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e24155e20 con 0x7f9e2405f140 2022-07-12T08:48:42.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1c021af0 con 0x7f9e2405f140 2022-07-12T08:48:42.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.898+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e24097830 con 0x7f9e2405f140 2022-07-12T08:48:42.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.902+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e24057460 con 0x7f9e2405f140 2022-07-12T08:48:42.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.902+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9e1c027020 con 0x7f9e2405f140 2022-07-12T08:48:42.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.902+0000 7f9e217fa700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e040539b0 0x7f9e04055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:42.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.902+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e1c06d7b0 con 0x7f9e2405f140 2022-07-12T08:48:42.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.902+0000 7f9e237fe700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e040539b0 0x7f9e04055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:42.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.902+0000 7f9e237fe700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e040539b0 0x7f9e04055e70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f9e1800d000 tx=0x7f9e18008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:42.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:42.906+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e1c0446b0 con 0x7f9e2405f140 2022-07-12T08:48:43.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.070+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e24068350 con 0x7f9e2405f140 2022-07-12T08:48:43.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.070+0000 7f9e217fa700 1 -- 172.21.15.61:0/232270549 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f9e1c015070 con 0x7f9e2405f140 2022-07-12T08:48:43.077 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:43.078 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":50,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:35.492783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6901":{"gid":6901,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/167461784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":167461784},{"type":"v1","addr":"172.21.15.111:6835","nonce":167461784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:43.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e040539b0 msgr2=0x7f9e04055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:43.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e040539b0 0x7f9e04055e70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f9e1800d000 tx=0x7f9e18008040).stop 2022-07-12T08:48:43.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 msgr2=0x7f9e24098ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:43.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 0x7f9e24098ab0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 rx=0x7f9e2405cd40 tx=0x7f9e1c016000).stop 2022-07-12T08:48:43.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 shutdown_connections 2022-07-12T08:48:43.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9e240afa60 0x7f9e24156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:43.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e040539b0 0x7f9e04055e70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:43.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9e240ab290 0x7f9e24156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:43.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 --2- 172.21.15.61:0/232270549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e2405f140 0x7f9e24098ab0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:43.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 >> 172.21.15.61:0/232270549 conn(0x7f9e240b1950 msgr2=0x7f9e24145940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:43.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 shutdown_connections 2022-07-12T08:48:43.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:43.074+0000 7f9e291ae700 1 -- 172.21.15.61:0/232270549 wait complete. 2022-07-12T08:48:43.086 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 50 2022-07-12T08:48:43.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:34.489352+0000', 'modified': '2022-07-12T08:48:35.492783+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6901}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6901': {'gid': 6901, 'name': 'b', 'rank': 0, 'incarnation': 49, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.111:6835/167461784', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6834', 'nonce': 167461784}, {'type': 'v1', 'addr': '172.21.15.111:6835', 'nonce': 167461784}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:48:43.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:48:43.297 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-12T08:48:43.413 INFO:teuthology.orchestra.run.smithi111.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":6901,"want_state":"up:active","state":"up:active","rank_uptime":8.909889647,"mdsmap_epoch":50,"osdmap_epoch":90,"osdmap_epoch_barrier":89,"uptime":13.297118161} 2022-07-12T08:48:43.414 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 6901, "mdsmap_epoch": 50, "osdmap_epoch": 90, "osdmap_epoch_barrier": 89, "rank_uptime": 8.909889647, "state": "up:active", "uptime": 13.297118161, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:48:43.415 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:48:43.415 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:48:43.416 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:48:43.417 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.418 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:43.428 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:48:43.496 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:43.497 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:43.516 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:43.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.517 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:48:43.526 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:48:43.527 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:43.577 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:48:43.596 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:48:43.597 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:48:43.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:48:43.642+0000 7f8ca228b080 -1 init, newargv = 0x55f691f5eb30 newargc=9 2022-07-12T08:48:43.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21316]: starting ceph client 2022-07-12T08:48:43.653 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:48:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:48:43.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.658 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:48:43.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21316]: starting fuse 2022-07-12T08:48:43.667 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:48:43.669 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:48:43.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.672 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:48:43.673 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:48:43.673 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:48:43.675 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:48:43.678 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:48:43.678 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:43.679 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:48:43.679 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:48:43.680 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:48:43.680 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:43.680 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:48:43.681 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:48:43.681 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:48:43.682 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:48:43.682 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:43.683 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:48:43.683 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:48:43.683 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:48:43.684 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:48:43.684 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:48:43.685 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:48:43.686 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:48:43.687 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:48:43.734 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21316.asok 2022-07-12T08:48:43.739 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21316.asok 2022-07-12T08:48:43.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.742 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21316.asok status 2022-07-12T08:48:43.878 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:48:43.879 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:48:43.880 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:48:43.880 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:48:43.881 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:48:43.881 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:48:43.882 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:48:43.882 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21316", 2022-07-12T08:48:43.883 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:48:43.883 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:43.884 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:48:43.884 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:48:43.885 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 7212, 2022-07-12T08:48:43.885 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:48:43.886 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:48:43.887 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:48:43.887 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 7212 2022-07-12T08:48:43.888 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:43.889 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:48:43.889 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:48:43.889 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:48:43.889 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2960097034 2022-07-12T08:48:43.890 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:48:43.890 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:43.890 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:48:43.890 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:48:43.891 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:48:43.892 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2960097034 2022-07-12T08:48:43.892 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:48:43.893 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.7212 172.21.15.130:0/2960097034", 2022-07-12T08:48:43.893 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/2960097034", 2022-07-12T08:48:43.894 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:48:43.894 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 50, 2022-07-12T08:48:43.895 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 90, 2022-07-12T08:48:43.895 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:48:43.896 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:48:43.896 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:48:43.897 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:48:43.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.900 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:43.901 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:43.914 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:48:43.915 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:43.915 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:43.916 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:43.985 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-12T08:48:43.986 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p foo 2022-07-12T08:48:44.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:48:44.015 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:48:44.015 DEBUG:teuthology.orchestra.run.smithi121:> import stat 2022-07-12T08:48:44.016 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:48:44.016 DEBUG:teuthology.orchestra.run.smithi121:> print(os.stat("/home/ubuntu/cephtest/mnt.0/foo").st_ino) 2022-07-12T08:48:44.016 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:48:44.135 INFO:teuthology.orchestra.run.smithi121.stdout:1099511627776 2022-07-12T08:48:44.329 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:44.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 Processor -- start 2022-07-12T08:48:44.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- start start 2022-07-12T08:48:44.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8c005c050 0x7fd8c00ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:44.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8c014a8f0 0x7fd8c01443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00ad2f0 con 0x7fd8c00b01f0 2022-07-12T08:48:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c00b0a40 con 0x7fd8c005c050 2022-07-12T08:48:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c005c430 con 0x7fd8c014a8f0 2022-07-12T08:48:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8c005c050 0x7fd8c00ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8c005c050 0x7fd8c00ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58940/0 (socket says 172.21.15.61:58940) 2022-07-12T08:48:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 -- 172.21.15.61:0/1456709353 learned_addr learned my addr 172.21.15.61:0/1456709353 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6855700 1 --2- 172.21.15.61:0/1456709353 >> v2:172.21.15.111:3300/0 conn(0x7fd8c014a8f0 0x7fd8c01443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 -- 172.21.15.61:0/1456709353 >> v2:172.21.15.111:3300/0 conn(0x7fd8c014a8f0 msgr2=0x7fd8c01443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 --2- 172.21.15.61:0/1456709353 >> v2:172.21.15.111:3300/0 conn(0x7fd8c014a8f0 0x7fd8c01443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 -- 172.21.15.61:0/1456709353 >> v1:172.21.15.111:6789/0 conn(0x7fd8c00b01f0 legacy=0x7fd8c005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 -- 172.21.15.61:0/1456709353 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c00b2420 con 0x7fd8c005c050 2022-07-12T08:48:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6855700 1 --2- 172.21.15.61:0/1456709353 >> v2:172.21.15.111:3300/0 conn(0x7fd8c014a8f0 0x7fd8c01443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c6054700 1 --2- 172.21.15.61:0/1456709353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c005c050 0x7fd8c00ac710 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7fd8b0002f70 tx=0x7fd8b0006eb0).ready entity=mon.0 client_cookie=f925f47371139596 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c5052700 1 -- 172.21.15.61:0/1456709353 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8b000d610 con 0x7fd8c005c050 2022-07-12T08:48:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c5052700 1 -- 172.21.15.61:0/1456709353 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b0000e90 con 0x7fd8c005c050 2022-07-12T08:48:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c5052700 1 -- 172.21.15.61:0/1456709353 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8b000d9a0 con 0x7fd8c005c050 2022-07-12T08:48:44.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1456709353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c005c050 msgr2=0x7fd8c00ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:44.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1456709353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c005c050 0x7fd8c00ac710 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7fd8b0002f70 tx=0x7fd8b0006eb0).stop 2022-07-12T08:48:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1456709353 shutdown_connections 2022-07-12T08:48:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1456709353 >> v2:172.21.15.111:3300/0 conn(0x7fd8c014a8f0 0x7fd8c01443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1456709353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c005c050 0x7fd8c00ac710 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1456709353 >> 172.21.15.61:0/1456709353 conn(0x7fd8c0062c00 msgr2=0x7fd8c0062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1456709353 shutdown_connections 2022-07-12T08:48:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.438+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1456709353 wait complete. 2022-07-12T08:48:44.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 Processor -- start 2022-07-12T08:48:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 -- start start 2022-07-12T08:48:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8c005c050 0x7fd8c0149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c00b01f0 0x7fd8c01565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 0x7fd8c015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:44.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c009f670 con 0x7fd8c014a8f0 2022-07-12T08:48:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c0063ff0 con 0x7fd8c00b01f0 2022-07-12T08:48:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c009f3d0 con 0x7fd8c005c050 2022-07-12T08:48:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 0x7fd8c015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:44.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 0x7fd8c015a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58946/0 (socket says 172.21.15.61:58946) 2022-07-12T08:48:44.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 -- 172.21.15.61:0/1040351601 learned_addr learned my addr 172.21.15.61:0/1040351601 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:44.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6054700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8c005c050 0x7fd8c0149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:44.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c5853700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c00b01f0 0x7fd8c01565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:44.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 -- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8c005c050 msgr2=0x7fd8c0149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:44.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8c005c050 0x7fd8c0149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 -- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c00b01f0 msgr2=0x7fd8c01565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:44.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c00b01f0 0x7fd8c01565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 -- 172.21.15.61:0/1040351601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b000b1e0 con 0x7fd8c014a8f0 2022-07-12T08:48:44.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c6855700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 0x7fd8c015a120 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fd8bc006940 tx=0x7fd8bc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:44.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8bc011780 con 0x7fd8c014a8f0 2022-07-12T08:48:44.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8bc0118e0 con 0x7fd8c014a8f0 2022-07-12T08:48:44.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c0159c80 con 0x7fd8c014a8f0 2022-07-12T08:48:44.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.442+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8bc011c40 con 0x7fd8c014a8f0 2022-07-12T08:48:44.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c009f0c0 con 0x7fd8c014a8f0 2022-07-12T08:48:44.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c0159e10 con 0x7fd8c014a8f0 2022-07-12T08:48:44.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd8bc01c460 con 0x7fd8c014a8f0 2022-07-12T08:48:44.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8b77fe700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8a0053a60 0x7fd8a0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:44.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd8bc06c360 con 0x7fd8c014a8f0 2022-07-12T08:48:44.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8c6054700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8a0053a60 0x7fd8a0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:44.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.446+0000 7fd8c6054700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8a0053a60 0x7fd8a0055f20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd8b001a000 tx=0x7fd8b0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:44.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.454+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8bc0431c0 con 0x7fd8c014a8f0 2022-07-12T08:48:44.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.618+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8c0057460 con 0x7fd8c014a8f0 2022-07-12T08:48:44.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.618+0000 7fd8b77fe700 1 -- 172.21.15.61:0/1040351601 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fd8bc0467e0 con 0x7fd8c014a8f0 2022-07-12T08:48:44.631 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:44.631 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":50,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:35.492783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6901":{"gid":6901,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/167461784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":167461784},{"type":"v1","addr":"172.21.15.111:6835","nonce":167461784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:44.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8a0053a60 msgr2=0x7fd8a0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:44.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8a0053a60 0x7fd8a0055f20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fd8b001a000 tx=0x7fd8b0015000).stop 2022-07-12T08:48:44.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 msgr2=0x7fd8c015a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:44.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 0x7fd8c015a120 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 rx=0x7fd8bc006940 tx=0x7fd8bc008040).stop 2022-07-12T08:48:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 shutdown_connections 2022-07-12T08:48:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8c014a8f0 0x7fd8c015a120 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8a0053a60 0x7fd8a0055f20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8c00b01f0 0x7fd8c01565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 --2- 172.21.15.61:0/1040351601 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8c005c050 0x7fd8c0149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 >> 172.21.15.61:0/1040351601 conn(0x7fd8c0062c00 msgr2=0x7fd8c00b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 shutdown_connections 2022-07-12T08:48:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:44.622+0000 7fd8c82b8700 1 -- 172.21.15.61:0/1040351601 wait complete. 2022-07-12T08:48:44.637 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 50 2022-07-12T08:48:44.638 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dump inode 0x10000000000 2022-07-12T08:48:44.755 INFO:teuthology.orchestra.run.smithi111.stdout:{"path":"/foo","ino":1099511627776,"rdev":0,"ctime":"2022-07-12T08:48:44.005121+0000","btime":"2022-07-12T08:48:44.005121+0000","mode":16877,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-12T08:48:44.005121+0000","atime":"2022-07-12T08:48:44.005121+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":7170,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":7170,"want_loner":7170,"mds_caps_wanted":[]} 2022-07-12T08:48:44.757 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2022-07-12T08:48:44.005121+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "backtrace_version": 2, "btime": "2022-07-12T08:48:44.005121+0000", "change_attr": 0, "client_caps": [ { "client_id": 7170, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-12T08:48:44.005121+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "flocklock": {}, "gid": 0, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 7170, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2022-07-12T08:48:44.005121+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/foo", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": 7170, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "xattrs": [] } 2022-07-12T08:48:44.758 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:48:44.758 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:44.781 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:44.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:44.823 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:44.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:44.867 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:48:44.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:44.868 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:44.869 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:44.877 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:48:44.878 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:44.879 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:48:44.879 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:48:44.880 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:44.937 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21847]: fuse finished with error 0 and tester_r 0 2022-07-12T08:48:44.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:44.940 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:44.941 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:44.946 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:48:44.948 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:48:45.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:48:45.067 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:45.117 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:45.119 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:45.119 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:45.127 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:48:45.128 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:45.129 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:48:45.130 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:48:45.130 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:45.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21316]: fuse finished with error 0 and tester_r 0 2022-07-12T08:48:45.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:45.152 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:45.152 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:45.201 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:48:45.203 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:48:45.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:48:45.247 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:45.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:45.256 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:45.257 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:45.270 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:45.272 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:48:45.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:48:45.273 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:45.280 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:45.282 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:48:45.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.390+0000 7f02be2fa700 1 Processor -- start 2022-07-12T08:48:45.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.390+0000 7f02be2fa700 1 -- start start 2022-07-12T08:48:45.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02be2fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02b80b5aa0 0x7f02b813fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:45.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02be2fa700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f02b805c240 0x7f02b80af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:45.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02be2fa700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80abf00 con 0x7f02b8067970 2022-07-12T08:48:45.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02be2fa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b80b5e80 con 0x7f02b80b5aa0 2022-07-12T08:48:45.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02be2fa700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b805d170 con 0x7f02b805c240 2022-07-12T08:48:45.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02b80b5aa0 0x7f02b813fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:45.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02b80b5aa0 0x7f02b813fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58954/0 (socket says 172.21.15.61:58954) 2022-07-12T08:48:45.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 -- 172.21.15.61:0/1841939169 learned_addr learned my addr 172.21.15.61:0/1841939169 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:45.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02bc897700 1 --2- 172.21.15.61:0/1841939169 >> v2:172.21.15.111:3300/0 conn(0x7f02b805c240 0x7f02b80af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:45.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 -- 172.21.15.61:0/1841939169 >> v2:172.21.15.111:3300/0 conn(0x7f02b805c240 msgr2=0x7f02b80af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:45.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 --2- 172.21.15.61:0/1841939169 >> v2:172.21.15.111:3300/0 conn(0x7f02b805c240 0x7f02b80af9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 -- 172.21.15.61:0/1841939169 >> v1:172.21.15.111:6789/0 conn(0x7f02b8067970 legacy=0x7f02b81403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:45.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 -- 172.21.15.61:0/1841939169 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b8144df0 con 0x7f02b80b5aa0 2022-07-12T08:48:45.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b7fff700 1 --2- 172.21.15.61:0/1841939169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b813fed0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f02b00032f0 tx=0x7f02b0009290).ready entity=mon.0 client_cookie=ec7bd7723b0f61a9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:45.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b6ffd700 1 -- 172.21.15.61:0/1841939169 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02b000b870 con 0x7f02b80b5aa0 2022-07-12T08:48:45.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b6ffd700 1 -- 172.21.15.61:0/1841939169 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b000b9d0 con 0x7f02b80b5aa0 2022-07-12T08:48:45.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.394+0000 7f02b6ffd700 1 -- 172.21.15.61:0/1841939169 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02b000bb40 con 0x7f02b80b5aa0 2022-07-12T08:48:45.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 -- 172.21.15.61:0/1841939169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 msgr2=0x7f02b813fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:45.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1841939169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b813fed0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f02b00032f0 tx=0x7f02b0009290).stop 2022-07-12T08:48:45.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 -- 172.21.15.61:0/1841939169 shutdown_connections 2022-07-12T08:48:45.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1841939169 >> v2:172.21.15.111:3300/0 conn(0x7f02b805c240 0x7f02b80af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1841939169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b813fed0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 -- 172.21.15.61:0/1841939169 >> 172.21.15.61:0/1841939169 conn(0x7f02b8063520 msgr2=0x7f02b80b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 -- 172.21.15.61:0/1841939169 shutdown_connections 2022-07-12T08:48:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 -- 172.21.15.61:0/1841939169 wait complete. 2022-07-12T08:48:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 Processor -- start 2022-07-12T08:48:45.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.398+0000 7f02be2fa700 1 -- start start 2022-07-12T08:48:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b805c240 0x7f02b8065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b8067970 0x7f02b80aac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b80ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80ab610 con 0x7f02b80b5aa0 2022-07-12T08:48:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80a0400 con 0x7f02b8067970 2022-07-12T08:48:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80a06a0 con 0x7f02b805c240 2022-07-12T08:48:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b80ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02b77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b8067970 0x7f02b80aac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b80ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58964/0 (socket says 172.21.15.61:58964) 2022-07-12T08:48:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 -- 172.21.15.61:0/1502902556 learned_addr learned my addr 172.21.15.61:0/1502902556 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:45.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02b7fff700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b805c240 0x7f02b8065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:45.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 -- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b805c240 msgr2=0x7f02b8065bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:45.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b805c240 0x7f02b8065bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 -- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b8067970 msgr2=0x7f02b80aac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:45.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b8067970 0x7f02b80aac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 -- 172.21.15.61:0/1502902556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0009530 con 0x7f02b80b5aa0 2022-07-12T08:48:45.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02bc897700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b80ab120 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f02a800c000 tx=0x7f02a8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:45.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02a8011750 con 0x7f02b80b5aa0 2022-07-12T08:48:45.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80118b0 con 0x7f02b80b5aa0 2022-07-12T08:48:45.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b8065030 con 0x7f02b80b5aa0 2022-07-12T08:48:45.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02a8011c10 con 0x7f02b80b5aa0 2022-07-12T08:48:45.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.402+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b8064d20 con 0x7f02b80b5aa0 2022-07-12T08:48:45.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.406+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b8066810 con 0x7f02b80b5aa0 2022-07-12T08:48:45.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.406+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f02a8011d70 con 0x7f02b80b5aa0 2022-07-12T08:48:45.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.406+0000 7f02b57fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0298053af0 0x7f0298055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:45.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.406+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f02a806c270 con 0x7f02b80b5aa0 2022-07-12T08:48:45.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.406+0000 7f02b7fff700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0298053af0 0x7f0298055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:45.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.406+0000 7f02b7fff700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0298053af0 0x7f0298055fb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f02b001b000 tx=0x7f02b0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:45.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.410+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02a80430c0 con 0x7f02b80b5aa0 2022-07-12T08:48:45.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.542+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f02b8057460 con 0x7f02b80b5aa0 2022-07-12T08:48:45.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.542+0000 7f02b57fa700 1 -- 172.21.15.61:0/1502902556 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v90) v1 ==== 81+0+33696 (secure 0 0 0) 0x7f02a80466e0 con 0x7f02b80b5aa0 2022-07-12T08:48:45.550 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:45.551 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:45.551 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 90, 2022-07-12T08:48:45.552 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:48:45.552 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:48:45.553 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:48:35.492446+0000", 2022-07-12T08:48:45.553 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:45.554 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:45.554 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:48:45.554 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:48:45.555 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:48:45.555 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:48:45.555 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:48:45.556 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:48:45.556 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:48:45.557 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:45.557 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:48:45.558 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:48:45.558 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:48:45.559 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:48:45.559 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:48:45.559 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 18, 2022-07-12T08:48:45.560 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:48:45.560 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:48:45.561 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:48:45.561 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:48:45.562 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:48:45.562 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.563 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:48:45.563 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:48:45.564 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:48:45.564 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:45.565 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:45.565 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:45.565 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:45.566 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:45.567 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:45.567 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:45.568 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:45.568 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:48:45.568 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:48:45.569 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:48:45.569 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:48:45.570 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:48:45.570 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:45.571 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:45.571 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:45.572 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:45.572 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:45.572 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:45.573 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:45.573 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.574 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:48:45.574 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:45.575 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:45.575 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:45.575 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:45.576 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:45.576 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:45.577 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:45.577 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:45.578 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:45.578 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:45.579 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:45.579 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:45.580 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:45.580 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:45.581 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:45.581 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:45.582 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:45.583 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:45.584 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:45.585 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:45.585 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:45.586 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:45.587 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:45.588 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:45.588 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:45.589 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:45.589 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.589 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:45.590 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:45.590 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:45.591 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:45.591 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:45.592 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:45.592 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:45.593 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:45.593 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:45.593 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:45.594 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:45.594 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:45.594 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:48:45.595 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.595 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:45.595 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:48:45.596 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.596 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.597 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:48:45.597 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:48:45.597 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:48:45.597 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:45.598 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:45.598 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:45.599 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:45.599 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:45.599 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:45.603 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:45.603 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.605 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:48:45.605 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:45.606 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:45.606 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:45.607 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:45.607 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:45.608 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:45.608 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:45.609 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:45.609 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:45.610 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:45.610 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:45.611 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:45.611 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:45.612 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:45.612 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:45.612 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:45.613 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:45.613 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:45.614 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:45.614 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:45.615 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:45.615 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:45.616 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:45.616 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:45.617 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:45.617 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:45.618 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.618 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:45.619 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:45.619 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:45.619 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:45.620 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:45.620 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:45.621 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:45.621 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:45.622 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:45.622 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:45.623 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:45.623 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:45.624 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:45.624 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:48:45.625 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.625 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.626 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.626 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 17, 2022-07-12T08:48:45.627 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:48:45.627 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:48:32.975784+0000", 2022-07-12T08:48:45.628 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:45.628 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:45.628 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:45.629 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:45.629 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:45.630 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:45.630 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:45.631 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:45.631 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:45.632 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:45.633 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:45.633 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:45.634 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:45.634 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:45.635 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:45.636 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:45.636 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:45.636 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:45.637 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:45.637 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:45.638 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.638 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "89", 2022-07-12T08:48:45.639 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:45.639 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:45.640 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:45.640 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:45.641 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:45.641 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:45.642 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:45.643 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:45.643 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:45.644 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:45.644 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:45.645 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:45.645 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:45.645 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:45.646 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:45.647 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:45.647 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:45.647 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:45.648 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:45.649 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:45.649 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:45.649 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:45.650 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:45.650 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:45.651 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:45.652 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:45.652 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.653 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:45.653 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:45.654 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:45.654 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:45.654 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:45.655 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:45.655 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:45.656 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:45.656 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:45.656 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:45.657 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:45.657 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:45.658 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:48:45.658 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:48:45.659 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:48:45.659 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.660 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:45.660 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:48:45.661 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:48:45.661 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.662 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.662 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.663 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.663 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 18, 2022-07-12T08:48:45.663 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:48:45.664 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:48:33.715444+0000", 2022-07-12T08:48:45.664 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:45.665 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:45.665 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:45.665 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:45.666 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:45.666 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:45.667 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:45.667 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:45.667 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:45.668 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:45.668 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:45.668 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:45.669 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:45.669 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:45.670 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:45.671 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:45.671 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:45.672 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:45.672 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:45.672 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:45.672 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "90", 2022-07-12T08:48:45.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:45.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:45.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:45.674 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:45.674 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:45.675 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:45.675 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:45.676 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:45.676 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:45.677 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:45.677 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:45.678 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:45.678 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:45.679 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:45.680 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:45.680 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:45.680 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:45.681 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:45.681 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:45.681 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:45.682 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:45.682 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:45.683 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:45.683 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:45.683 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:45.684 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:45.684 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.685 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:45.686 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:45.686 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:45.687 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:45.687 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:45.688 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:45.688 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:45.689 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:45.689 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:45.690 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:45.690 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:45.691 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:45.691 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:45.692 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:48:45.692 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:48:45.693 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.693 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.694 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.694 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:45.695 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:48:45.695 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:48:45.696 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.697 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.697 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.698 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.698 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.699 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.699 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.700 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:45.700 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.701 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.701 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.702 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.702 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:48:45.703 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.704 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:48:45.705 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.706 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.706 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.706 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.707 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.707 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.708 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.709 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:48:45.709 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.710 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.710 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.711 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.711 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:48:45.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.712 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.713 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.713 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.713 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.714 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.714 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.715 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.715 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:48:45.715 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.715 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.716 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.716 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.717 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:48:45.717 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.720 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.720 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.721 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.721 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.721 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.722 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.722 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:48:45.722 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.723 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.723 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.723 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.723 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:48:45.723 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:45.724 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.724 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.724 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.725 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:48:45.725 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:48:45.725 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:48:45.725 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:48:45.725 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.726 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.726 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.726 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.726 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.727 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:45.728 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:48:45.729 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.729 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.730 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.731 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.731 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.731 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.732 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.732 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:45.733 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.733 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.733 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.734 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.734 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.735 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.735 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:48:45.736 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.736 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.737 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.738 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.738 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:48:45.738 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.739 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.739 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.740 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.740 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.741 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.741 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.741 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.741 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:48:45.742 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.742 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.742 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.742 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.743 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:48:45.743 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.743 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.743 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.744 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.744 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.744 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.745 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.745 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.745 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:48:45.746 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.746 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.746 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.746 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.747 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:48:45.747 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.747 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.747 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.748 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.748 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.748 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.748 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.749 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.749 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:48:45.749 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.749 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.750 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.750 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.750 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:48:45.751 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:45.751 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.752 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.753 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:48:45.753 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:48:45.754 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:48:45.754 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:48:45.755 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.755 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.755 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.756 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.756 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.756 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.757 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:45.757 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:48:45.758 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.758 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.759 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.760 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.760 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.761 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.762 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.762 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:45.763 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.763 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.764 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.764 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.764 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.765 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.765 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:48:45.766 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.766 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.767 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.767 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.767 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:48:45.768 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.768 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.769 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.769 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.770 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.770 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.771 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.771 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.771 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:48:45.772 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.773 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.773 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.774 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.774 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:48:45.774 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.775 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.775 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.775 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.776 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.776 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.776 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.776 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.777 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:48:45.777 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.778 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.778 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.778 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.779 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:48:45.779 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.780 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.780 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.781 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.781 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.782 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.783 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.784 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.784 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:48:45.784 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.785 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.785 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.786 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.786 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:48:45.787 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:45.787 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.788 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.788 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.789 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:48:45.789 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:48:45.790 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:48:45.790 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:48:45.790 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.791 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.791 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.792 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.792 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.793 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.794 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:45.794 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:48:45.795 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.795 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.796 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.796 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.797 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.797 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.797 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.798 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:45.798 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.798 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.799 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.799 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.800 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.800 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.800 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:48:45.801 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.801 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.802 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.802 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:48:45.802 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.803 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.803 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.803 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.805 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.805 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.805 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.806 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.806 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:48:45.807 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.807 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.808 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.808 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.809 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:48:45.816 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.817 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.817 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.818 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.818 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.818 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.819 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.819 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.819 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:48:45.820 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.820 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.820 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.821 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.821 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:48:45.821 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.822 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.822 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.822 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.822 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.823 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.823 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.823 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.823 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:48:45.823 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.824 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.824 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.824 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:48:45.825 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:45.826 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.826 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.826 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:48:45.827 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:48:45.827 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:48:45.827 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:48:45.828 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.828 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.829 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.829 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.829 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.830 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.830 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:45.830 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:48:45.831 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.831 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.831 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.832 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.832 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.832 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.833 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.833 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-12T08:48:45.833 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.834 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.834 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.834 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.835 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.835 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.836 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:48:45.836 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.837 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.837 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.837 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.838 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:48:45.838 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.839 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.839 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.839 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.839 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.840 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.840 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.840 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.840 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:48:45.840 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.841 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.841 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.841 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.842 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:48:45.842 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.842 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.842 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.843 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.843 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.843 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.843 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.844 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.844 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:48:45.844 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.844 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.845 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.845 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.845 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:48:45.845 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.846 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.846 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.847 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.847 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.847 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.847 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.848 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.848 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:48:45.848 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.849 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.849 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:48:45.850 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:45.850 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.850 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.851 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:48:45.851 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:48:45.852 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:48:45.852 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:48:45.852 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.852 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.853 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.853 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.854 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.854 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.854 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:45.855 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:48:45.855 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.855 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.855 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.855 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.856 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.856 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.856 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.857 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:45.857 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.857 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.858 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.858 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.858 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.858 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.859 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:48:45.859 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.859 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.860 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.860 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.860 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:48:45.861 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.861 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.861 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.862 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.862 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.862 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.862 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.863 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:48:45.863 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.863 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.863 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.864 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.864 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:48:45.864 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.864 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.865 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.865 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.865 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.865 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.866 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.866 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.866 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:48:45.866 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.867 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.867 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.867 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.868 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:48:45.868 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.868 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.869 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.869 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.869 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.869 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.870 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.870 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.870 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:48:45.870 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.871 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.871 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.871 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.871 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:48:45.871 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:45.872 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.872 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.872 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.872 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:48:45.872 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:48:45.873 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:48:45.873 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:48:45.873 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.873 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.874 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.874 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.874 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.874 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.875 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:45.875 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:48:45.875 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.875 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.875 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.876 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.876 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.876 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.876 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.877 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-12T08:48:45.877 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.877 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.877 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.877 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.878 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.878 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:48:45.878 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.879 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.879 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.879 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:48:45.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.880 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.880 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.880 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.880 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.881 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.881 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:48:45.881 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.882 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.882 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.882 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.882 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:48:45.883 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.883 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.883 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.883 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.883 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.884 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.884 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:48:45.885 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.885 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.885 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.885 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.885 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:48:45.886 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.886 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.886 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.886 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.887 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.887 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.887 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.887 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.888 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:48:45.888 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.888 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.888 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.889 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.889 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:48:45.889 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:45.889 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.889 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.890 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.890 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:48:45.890 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:48:45.890 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:48:45.891 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:48:45.891 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.891 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.891 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.891 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.892 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.892 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:45.892 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:48:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:45.894 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:45.894 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:45.894 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:45.894 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:45.895 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:45.895 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:45.895 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.895 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.895 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.896 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:48:45.896 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.897 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.897 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:48:45.897 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.897 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.897 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.898 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.898 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:45.898 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.898 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.899 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.899 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:48:45.899 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.899 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.899 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.900 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:48:45.900 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.900 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.901 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.901 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.901 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:45.901 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.901 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.902 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.902 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:48:45.902 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.903 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.903 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.903 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:48:45.903 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.903 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.904 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.904 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.904 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:45.904 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:45.905 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.905 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:45.905 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:48:45.905 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.905 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.906 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.906 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:45.906 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:48:45.906 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:45.907 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.907 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.907 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.907 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:48:45.908 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:48:45.908 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:48:45.908 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:48:45.908 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:45.908 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:45.909 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:45.909 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:45.909 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.909 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:45.910 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:48:45.910 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.910 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:45.910 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.911 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.911 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.911 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.911 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.912 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:48:45.912 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.912 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.912 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.912 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:45.913 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.913 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.913 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.913 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.914 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.914 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:48:45.914 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.914 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.914 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.915 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:45.915 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.915 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.915 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.916 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.916 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.916 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:48:45.916 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.916 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.917 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:45.917 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.917 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.918 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.918 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.918 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.918 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:48:45.918 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.919 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.919 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:45.919 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.919 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.920 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.920 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.920 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.920 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:48:45.921 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.921 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.921 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.921 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:45.921 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.922 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.922 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.922 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.922 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.923 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:48:45.923 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.923 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.923 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:45.924 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.924 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.924 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.925 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.925 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.925 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:48:45.925 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.925 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.926 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:45.926 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:45.926 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:45.926 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:45.927 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:45.927 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:45.927 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:45.927 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:48:45.927 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:45.928 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.928 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:45.928 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:48:45.928 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:48:45.929 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:48:45.929 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:48:45.929 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:48:45.929 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:48:45.929 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:48:45.930 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:48:45.930 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:48:45.930 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:48:45.930 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:48:45.930 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:48:45.931 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:48:45.931 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:45.931 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:45.931 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:48:45.932 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:48:45.932 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:48:45.932 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:48:45.932 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:48:45.932 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:45.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0298053af0 msgr2=0x7f0298055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:45.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0298053af0 0x7f0298055fb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f02b001b000 tx=0x7f02b0016000).stop 2022-07-12T08:48:45.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 msgr2=0x7f02b80ab120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:45.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b80ab120 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 rx=0x7f02a800c000 tx=0x7f02a8006df0).stop 2022-07-12T08:48:45.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 shutdown_connections 2022-07-12T08:48:45.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02b80b5aa0 0x7f02b80ab120 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0298053af0 0x7f0298055fb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02b8067970 0x7f02b80aac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 --2- 172.21.15.61:0/1502902556 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02b805c240 0x7f02b8065bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:45.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 >> 172.21.15.61:0/1502902556 conn(0x7f02b8063520 msgr2=0x7f02b80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:45.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 shutdown_connections 2022-07-12T08:48:45.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:45.546+0000 7f02be2fa700 1 -- 172.21.15.61:0/1502902556 wait complete. 2022-07-12T08:48:45.938 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:46.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 Processor -- start 2022-07-12T08:48:46.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 -- start start 2022-07-12T08:48:46.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a440b06b0 0x7f7a4405c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a44144320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7a44144810 0x7f7a44140150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a44141f90 con 0x7f7a440b06b0 2022-07-12T08:48:46.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a44064150 con 0x7f7a44144810 2022-07-12T08:48:46.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.042+0000 7f7a4ae97700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4405c580 con 0x7f7a44064850 2022-07-12T08:48:46.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a440b06b0 0x7f7a4405c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a49434700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7a44144810 0x7f7a44140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a440b06b0 0x7f7a4405c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58970/0 (socket says 172.21.15.61:58970) 2022-07-12T08:48:46.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a43fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a44144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 -- 172.21.15.61:0/1126116295 learned_addr learned my addr 172.21.15.61:0/1126116295 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:46.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 -- 172.21.15.61:0/1126116295 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 msgr2=0x7f7a44144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:46.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 --2- 172.21.15.61:0/1126116295 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a44144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 -- 172.21.15.61:0/1126116295 >> v2:172.21.15.111:3300/0 conn(0x7f7a44144810 msgr2=0x7f7a44140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:46.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 --2- 172.21.15.61:0/1126116295 >> v2:172.21.15.111:3300/0 conn(0x7f7a44144810 0x7f7a44140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 -- 172.21.15.61:0/1126116295 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a440643b0 con 0x7f7a440b06b0 2022-07-12T08:48:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a49434700 1 --2- 172.21.15.61:0/1126116295 >> v2:172.21.15.111:3300/0 conn(0x7f7a44144810 0x7f7a44140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a48c33700 1 --2- 172.21.15.61:0/1126116295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a4405c940 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f7a38002f70 tx=0x7f7a38006eb0).ready entity=mon.0 client_cookie=df17de8533c8186b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a437fe700 1 -- 172.21.15.61:0/1126116295 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a3800d570 con 0x7f7a440b06b0 2022-07-12T08:48:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a437fe700 1 -- 172.21.15.61:0/1126116295 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38000e90 con 0x7f7a440b06b0 2022-07-12T08:48:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a437fe700 1 -- 172.21.15.61:0/1126116295 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a3800d900 con 0x7f7a440b06b0 2022-07-12T08:48:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 -- 172.21.15.61:0/1126116295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 msgr2=0x7f7a4405c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/1126116295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a4405c940 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f7a38002f70 tx=0x7f7a38006eb0).stop 2022-07-12T08:48:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 -- 172.21.15.61:0/1126116295 shutdown_connections 2022-07-12T08:48:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/1126116295 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a44144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/1126116295 >> v2:172.21.15.111:3300/0 conn(0x7f7a44144810 0x7f7a44140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/1126116295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a4405c940 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 -- 172.21.15.61:0/1126116295 >> 172.21.15.61:0/1126116295 conn(0x7f7a44062c00 msgr2=0x7f7a44062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 -- 172.21.15.61:0/1126116295 shutdown_connections 2022-07-12T08:48:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.046+0000 7f7a4ae97700 1 -- 172.21.15.61:0/1126116295 wait complete. 2022-07-12T08:48:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 Processor -- start 2022-07-12T08:48:46.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 -- start start 2022-07-12T08:48:46.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a4414a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a44143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a44144810 0x7f7a44155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a441416f0 con 0x7f7a440b06b0 2022-07-12T08:48:46.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a441490b0 con 0x7f7a44144810 2022-07-12T08:48:46.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a4ae97700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a44148130 con 0x7f7a44064850 2022-07-12T08:48:46.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a44143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a44143f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58976/0 (socket says 172.21.15.61:58976) 2022-07-12T08:48:46.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 -- 172.21.15.61:0/2781794420 learned_addr learned my addr 172.21.15.61:0/2781794420 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:46.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a48c33700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a4414a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a49434700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a44144810 0x7f7a44155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 -- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 msgr2=0x7f7a4414a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a4414a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 -- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a44144810 msgr2=0x7f7a44155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a44144810 0x7f7a44155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 -- 172.21.15.61:0/2781794420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c009470 con 0x7f7a440b06b0 2022-07-12T08:48:46.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a43fff700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a44143f00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f7a3400c000 tx=0x7f7a34006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:46.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.050+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a34010780 con 0x7f7a440b06b0 2022-07-12T08:48:46.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a340108e0 con 0x7f7a440b06b0 2022-07-12T08:48:46.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3800b1e0 con 0x7f7a440b06b0 2022-07-12T08:48:46.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a34010c40 con 0x7f7a440b06b0 2022-07-12T08:48:46.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a441492e0 con 0x7f7a440b06b0 2022-07-12T08:48:46.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a44057460 con 0x7f7a440b06b0 2022-07-12T08:48:46.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7a3401c460 con 0x7f7a440b06b0 2022-07-12T08:48:46.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a41ffb700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7a24053920 0x7f7a24055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.054+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7a3406c340 con 0x7f7a440b06b0 2022-07-12T08:48:46.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.058+0000 7f7a48c33700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7a24053920 0x7f7a24055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.058+0000 7f7a48c33700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7a24053920 0x7f7a24055de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f7a3801c000 tx=0x7f7a38015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:46.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.062+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7a340431a0 con 0x7f7a440b06b0 2022-07-12T08:48:46.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.230+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a4405c2c0 con 0x7f7a440b06b0 2022-07-12T08:48:46.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.230+0000 7f7a41ffb700 1 -- 172.21.15.61:0/2781794420 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f7a340467c0 con 0x7f7a440b06b0 2022-07-12T08:48:46.238 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:46.239 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":50,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:35.492783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6901},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6901":{"gid":6901,"name":"b","rank":0,"incarnation":49,"state":"up:active","state_seq":3,"addr":"172.21.15.111:6835/167461784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":167461784},{"type":"v1","addr":"172.21.15.111:6835","nonce":167461784}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:46.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7a24053920 msgr2=0x7f7a24055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7a24053920 0x7f7a24055de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f7a3801c000 tx=0x7f7a38015000).stop 2022-07-12T08:48:46.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 msgr2=0x7f7a44143f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a44143f00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f7a3400c000 tx=0x7f7a34006df0).stop 2022-07-12T08:48:46.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 shutdown_connections 2022-07-12T08:48:46.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7a24053920 0x7f7a24055de0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a44144810 0x7f7a44155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a440b06b0 0x7f7a44143f00 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 --2- 172.21.15.61:0/2781794420 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7a44064850 0x7f7a4414a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 >> 172.21.15.61:0/2781794420 conn(0x7f7a44062c00 msgr2=0x7f7a440b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:46.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 shutdown_connections 2022-07-12T08:48:46.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.234+0000 7f7a4ae97700 1 -- 172.21.15.61:0/2781794420 wait complete. 2022-07-12T08:48:46.248 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 50 2022-07-12T08:48:46.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:48:46.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.350+0000 7f3c92fe1700 1 Processor -- start 2022-07-12T08:48:46.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.350+0000 7f3c92fe1700 1 -- start start 2022-07-12T08:48:46.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c8c060d20 0x7f3c8c13f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3c8c0b1460 0x7f3c8c142230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c060830 con 0x7f3c8c060450 2022-07-12T08:48:46.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c13ef00 con 0x7f3c8c060d20 2022-07-12T08:48:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c13f1a0 con 0x7f3c8c0b1460 2022-07-12T08:48:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c90d7d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3c8c060450 0x7f3c8c0b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47200/0 (socket says 172.21.15.61:47200) 2022-07-12T08:48:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c8c060d20 0x7f3c8c13f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c90d7d700 1 -- 172.21.15.61:0/4246741342 learned_addr learned my addr 172.21.15.61:0/4246741342 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c9157e700 1 --2- 172.21.15.61:0/4246741342 >> v2:172.21.15.111:3300/0 conn(0x7f3c8c0b1460 0x7f3c8c142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8b7fe700 1 -- 172.21.15.61:0/4246741342 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007451802 0 0) 0x7f3c8c060830 con 0x7f3c8c060450 2022-07-12T08:48:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8bfff700 1 -- 172.21.15.61:0/4246741342 >> v2:172.21.15.111:3300/0 conn(0x7f3c8c0b1460 msgr2=0x7f3c8c142230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8bfff700 1 --2- 172.21.15.61:0/4246741342 >> v2:172.21.15.111:3300/0 conn(0x7f3c8c0b1460 0x7f3c8c142230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8bfff700 1 -- 172.21.15.61:0/4246741342 >> v1:172.21.15.61:6789/0 conn(0x7f3c8c060450 legacy=0x7f3c8c0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8bfff700 1 -- 172.21.15.61:0/4246741342 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c13f440 con 0x7f3c8c060d20 2022-07-12T08:48:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8bfff700 1 --2- 172.21.15.61:0/4246741342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c13f8e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f3c78002f70 tx=0x7f3c78006df0).ready entity=mon.0 client_cookie=923277321598b44 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8b7fe700 1 -- 172.21.15.61:0/4246741342 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c7800d570 con 0x7f3c8c060d20 2022-07-12T08:48:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8b7fe700 1 -- 172.21.15.61:0/4246741342 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78000e90 con 0x7f3c8c060d20 2022-07-12T08:48:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c8b7fe700 1 -- 172.21.15.61:0/4246741342 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c7800d8c0 con 0x7f3c8c060d20 2022-07-12T08:48:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- 172.21.15.61:0/4246741342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 msgr2=0x7f3c8c13f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/4246741342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c13f8e0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 rx=0x7f3c78002f70 tx=0x7f3c78006df0).stop 2022-07-12T08:48:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- 172.21.15.61:0/4246741342 shutdown_connections 2022-07-12T08:48:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/4246741342 >> v2:172.21.15.111:3300/0 conn(0x7f3c8c0b1460 0x7f3c8c142230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/4246741342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c13f8e0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- 172.21.15.61:0/4246741342 >> 172.21.15.61:0/4246741342 conn(0x7f3c8c0b5500 msgr2=0x7f3c8c0b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- 172.21.15.61:0/4246741342 shutdown_connections 2022-07-12T08:48:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.354+0000 7f3c92fe1700 1 -- 172.21.15.61:0/4246741342 wait complete. 2022-07-12T08:48:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 Processor -- start 2022-07-12T08:48:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 -- start start 2022-07-12T08:48:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c8c060450 0x7f3c8c093de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c063ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c0b1460 0x7f3c8c064390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c065250 con 0x7f3c8c060d20 2022-07-12T08:48:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c065fb0 con 0x7f3c8c0b1460 2022-07-12T08:48:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c09b000 con 0x7f3c8c060450 2022-07-12T08:48:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c063ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c063ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:58990/0 (socket says 172.21.15.61:58990) 2022-07-12T08:48:46.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 -- 172.21.15.61:0/2101219344 learned_addr learned my addr 172.21.15.61:0/2101219344 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:46.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c90d7d700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c8c060450 0x7f3c8c093de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c9157e700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c0b1460 0x7f3c8c064390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 -- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c8c060450 msgr2=0x7f3c8c093de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c8c060450 0x7f3c8c093de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 -- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c0b1460 msgr2=0x7f3c8c064390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:46.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c0b1460 0x7f3c8c064390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:46.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 -- 172.21.15.61:0/2101219344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7800b180 con 0x7f3c8c060d20 2022-07-12T08:48:46.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c8bfff700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c063ea0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f3c78002f70 tx=0x7f3c78014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:46.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c7800dea0 con 0x7f3c8c060d20 2022-07-12T08:48:46.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c78013e90 con 0x7f3c8c060d20 2022-07-12T08:48:46.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c09b2a0 con 0x7f3c8c060d20 2022-07-12T08:48:46.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c7801e5f0 con 0x7f3c8c060d20 2022-07-12T08:48:46.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.358+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c099fe0 con 0x7f3c8c060d20 2022-07-12T08:48:46.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.362+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c057460 con 0x7f3c8c060d20 2022-07-12T08:48:46.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.362+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3c78025030 con 0x7f3c8c060d20 2022-07-12T08:48:46.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.362+0000 7f3c89ffb700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c6c053920 0x7f3c6c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:46.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.362+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3c7806cca0 con 0x7f3c8c060d20 2022-07-12T08:48:46.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.362+0000 7f3c90d7d700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c6c053920 0x7f3c6c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:46.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.362+0000 7f3c90d7d700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c6c053920 0x7f3c6c055de0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f3c8000d000 tx=0x7f3c80009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:46.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.366+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c78043af0 con 0x7f3c8c060d20 2022-07-12T08:48:46.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:46.530+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c8c09b460 con 0x7f3c8c060d20 2022-07-12T08:48:47.436 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:48:47.430+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:48:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.442+0000 7f3c89ffb700 1 -- 172.21.15.61:0/2101219344 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v51) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3c78023070 con 0x7f3c8c060d20 2022-07-12T08:48:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c6c053920 msgr2=0x7f3c6c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c6c053920 0x7f3c6c055de0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f3c8000d000 tx=0x7f3c80009040).stop 2022-07-12T08:48:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 msgr2=0x7f3c8c063ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c063ea0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f3c78002f70 tx=0x7f3c78014000).stop 2022-07-12T08:48:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 shutdown_connections 2022-07-12T08:48:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3c6c053920 0x7f3c6c055de0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3c8c0b1460 0x7f3c8c064390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c8c060d20 0x7f3c8c063ea0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 --2- 172.21.15.61:0/2101219344 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3c8c060450 0x7f3c8c093de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 >> 172.21.15.61:0/2101219344 conn(0x7f3c8c0b5500 msgr2=0x7f3c8c0abb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:47.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 shutdown_connections 2022-07-12T08:48:47.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.446+0000 7f3c92fe1700 1 -- 172.21.15.61:0/2101219344 wait complete. 2022-07-12T08:48:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:48:47.468 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:47.475 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:48:47.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.570+0000 7f204b120700 1 Processor -- start 2022-07-12T08:48:47.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.570+0000 7f204b120700 1 -- start start 2022-07-12T08:48:47.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204b120700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20440af6e0 0x7f2044143480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204b120700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440afac0 con 0x7f204413cb80 2022-07-12T08:48:47.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204b120700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440611b0 con 0x7f2044060c30 2022-07-12T08:48:47.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204b120700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2044143970 con 0x7f20440af6e0 2022-07-12T08:48:47.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a11e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20440af6e0 0x7f2044143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a91f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f204413cb80 0x7f20440b0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47218/0 (socket says 172.21.15.61:47218) 2022-07-12T08:48:47.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a91f700 1 -- 172.21.15.61:0/2269711900 learned_addr learned my addr 172.21.15.61:0/2269711900 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:47.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204911c700 1 -- 172.21.15.61:0/2269711900 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732491216 0 0) 0x7f20440afac0 con 0x7f204413cb80 2022-07-12T08:48:47.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204911c700 1 -- 172.21.15.61:0/2269711900 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202c003260 con 0x7f204413cb80 2022-07-12T08:48:47.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a11e700 1 -- 172.21.15.61:0/2269711900 >> v1:172.21.15.111:6789/0 conn(0x7f2044060c30 legacy=0x7f204413c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204911c700 1 -- 172.21.15.61:0/2269711900 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988767134 0 0) 0x7f20440611b0 con 0x7f2044060c30 2022-07-12T08:48:47.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a11e700 1 -- 172.21.15.61:0/2269711900 >> v1:172.21.15.61:6789/0 conn(0x7f204413cb80 legacy=0x7f20440b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a11e700 1 -- 172.21.15.61:0/2269711900 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044061450 con 0x7f20440af6e0 2022-07-12T08:48:47.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204a11e700 1 --2- 172.21.15.61:0/2269711900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f2044143480 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f203c001b80 tx=0x7f2040006df0).ready entity=mon.0 client_cookie=341636c4ec735cac server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:47.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204911c700 1 -- 172.21.15.61:0/2269711900 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f203c0019e0 con 0x7f20440af6e0 2022-07-12T08:48:47.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204911c700 1 -- 172.21.15.61:0/2269711900 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040000e90 con 0x7f20440af6e0 2022-07-12T08:48:47.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.574+0000 7f204911c700 1 -- 172.21.15.61:0/2269711900 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2040011400 con 0x7f20440af6e0 2022-07-12T08:48:47.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- 172.21.15.61:0/2269711900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 msgr2=0x7f2044143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 --2- 172.21.15.61:0/2269711900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f2044143480 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7f203c001b80 tx=0x7f2040006df0).stop 2022-07-12T08:48:47.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- 172.21.15.61:0/2269711900 shutdown_connections 2022-07-12T08:48:47.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 --2- 172.21.15.61:0/2269711900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f2044143480 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- 172.21.15.61:0/2269711900 >> 172.21.15.61:0/2269711900 conn(0x7f20440b5780 msgr2=0x7f20440b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:47.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- 172.21.15.61:0/2269711900 shutdown_connections 2022-07-12T08:48:47.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- 172.21.15.61:0/2269711900 wait complete. 2022-07-12T08:48:47.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 Processor -- start 2022-07-12T08:48:47.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- start start 2022-07-12T08:48:47.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2044060c30 0x7f2044140f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f204413b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f204413cb80 0x7f204413b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2044141cb0 con 0x7f20440af6e0 2022-07-12T08:48:47.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204413f8a0 con 0x7f204413cb80 2022-07-12T08:48:47.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204b120700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204413efe0 con 0x7f2044060c30 2022-07-12T08:48:47.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204991d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f204413b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204991d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f204413b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59002/0 (socket says 172.21.15.61:59002) 2022-07-12T08:48:47.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204991d700 1 -- 172.21.15.61:0/2577332101 learned_addr learned my addr 172.21.15.61:0/2577332101 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:47.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204a91f700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f204413cb80 0x7f204413b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.578+0000 7f204a11e700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2044060c30 0x7f2044140f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204991d700 1 -- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2044060c30 msgr2=0x7f2044140f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204991d700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2044060c30 0x7f2044140f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204991d700 1 -- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f204413cb80 msgr2=0x7f204413b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204991d700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f204413cb80 0x7f204413b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204991d700 1 -- 172.21.15.61:0/2577332101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204000b180 con 0x7f20440af6e0 2022-07-12T08:48:47.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204991d700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f204413b3d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f2038000fe0 tx=0x7f2038007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:47.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2038012400 con 0x7f20440af6e0 2022-07-12T08:48:47.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2038012560 con 0x7f20440af6e0 2022-07-12T08:48:47.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20380128c0 con 0x7f20440af6e0 2022-07-12T08:48:47.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204413f280 con 0x7f20440af6e0 2022-07-12T08:48:47.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204413bdb0 con 0x7f20440af6e0 2022-07-12T08:48:47.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2044057460 con 0x7f20440af6e0 2022-07-12T08:48:47.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2038019030 con 0x7f20440af6e0 2022-07-12T08:48:47.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f20377fe700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20240539d0 0x7f2024055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f203806b100 con 0x7f20440af6e0 2022-07-12T08:48:47.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.582+0000 7f204a11e700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20240539d0 0x7f2024055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.586+0000 7f204a11e700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20240539d0 0x7f2024055e90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f2040019000 tx=0x7f20440610a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:47.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.590+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20380420c0 con 0x7f20440af6e0 2022-07-12T08:48:47.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.754+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2044068400 con 0x7f20440af6e0 2022-07-12T08:48:47.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.754+0000 7f20377fe700 1 -- 172.21.15.61:0/2577332101 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+1776 (secure 0 0 0) 0x7f20380456e0 con 0x7f20440af6e0 2022-07-12T08:48:47.769 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:47.769 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48}],"filesystems":[{"mdsmap":{"epoch":51,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:34.489352+0000","modified":"2022-07-12T08:48:47.435522+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-12T08:48:47.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20240539d0 msgr2=0x7f2024055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20240539d0 0x7f2024055e90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f2040019000 tx=0x7f20440610a0).stop 2022-07-12T08:48:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 msgr2=0x7f204413b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f204413b3d0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 rx=0x7f2038000fe0 tx=0x7f2038007b50).stop 2022-07-12T08:48:47.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 shutdown_connections 2022-07-12T08:48:47.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20240539d0 0x7f2024055e90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f204413cb80 0x7f204413b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20440af6e0 0x7f204413b3d0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 --2- 172.21.15.61:0/2577332101 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2044060c30 0x7f2044140f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 >> 172.21.15.61:0/2577332101 conn(0x7f20440b5780 msgr2=0x7f20440b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:47.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 shutdown_connections 2022-07-12T08:48:47.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.758+0000 7f204b120700 1 -- 172.21.15.61:0/2577332101 wait complete. 2022-07-12T08:48:47.778 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-12T08:48:47.781 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:48:47.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 Processor -- start 2022-07-12T08:48:47.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 -- start start 2022-07-12T08:48:47.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44b80b24b0 0x7f44b81445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44b80aaa50 0x7f44b8140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b8141a40 con 0x7f44b8140d20 2022-07-12T08:48:47.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b80b3190 con 0x7f44b80aaa50 2022-07-12T08:48:47.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bdb4b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b8143ea0 con 0x7f44b80b24b0 2022-07-12T08:48:47.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44b80aaa50 0x7f44b8140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44b80aaa50 0x7f44b8140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59012/0 (socket says 172.21.15.61:59012) 2022-07-12T08:48:47.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 -- 172.21.15.61:0/3847222764 learned_addr learned my addr 172.21.15.61:0/3847222764 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:47.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44bcb49700 1 --2- 172.21.15.61:0/3847222764 >> v2:172.21.15.111:3300/0 conn(0x7f44b80b24b0 0x7f44b81445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 -- 172.21.15.61:0/3847222764 >> v2:172.21.15.111:3300/0 conn(0x7f44b80b24b0 msgr2=0x7f44b81445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 --2- 172.21.15.61:0/3847222764 >> v2:172.21.15.111:3300/0 conn(0x7f44b80b24b0 0x7f44b81445e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 -- 172.21.15.61:0/3847222764 >> v1:172.21.15.111:6789/0 conn(0x7f44b8140d20 legacy=0x7f44b813d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.890+0000 7f44b7fff700 1 -- 172.21.15.61:0/3847222764 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8144140 con 0x7f44b80aaa50 2022-07-12T08:48:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44b7fff700 1 --2- 172.21.15.61:0/3847222764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80aaa50 0x7f44b8140830 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f44b80b14e0 tx=0x7f44ac006eb0).ready entity=mon.0 client_cookie=b9c9af8fe92848a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44b77fe700 1 -- 172.21.15.61:0/3847222764 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44ac00d610 con 0x7f44b80aaa50 2022-07-12T08:48:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44b77fe700 1 -- 172.21.15.61:0/3847222764 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ac000e90 con 0x7f44b80aaa50 2022-07-12T08:48:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44b77fe700 1 -- 172.21.15.61:0/3847222764 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44ac00d9a0 con 0x7f44b80aaa50 2022-07-12T08:48:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3847222764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80aaa50 msgr2=0x7f44b8140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3847222764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80aaa50 0x7f44b8140830 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 rx=0x7f44b80b14e0 tx=0x7f44ac006eb0).stop 2022-07-12T08:48:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3847222764 shutdown_connections 2022-07-12T08:48:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3847222764 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80aaa50 0x7f44b8140830 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3847222764 >> v2:172.21.15.111:3300/0 conn(0x7f44b80b24b0 0x7f44b81445e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3847222764 >> 172.21.15.61:0/3847222764 conn(0x7f44b80b5860 msgr2=0x7f44b80b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:47.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3847222764 shutdown_connections 2022-07-12T08:48:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3847222764 wait complete. 2022-07-12T08:48:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 Processor -- start 2022-07-12T08:48:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- start start 2022-07-12T08:48:47.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 0x7f44b80afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44b8140d20 0x7f44b8091df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44b80ae190 0x7f44b808f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80ae8a0 con 0x7f44b80b24b0 2022-07-12T08:48:47.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b8091410 con 0x7f44b80ae190 2022-07-12T08:48:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bdb4b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b80916b0 con 0x7f44b8140d20 2022-07-12T08:48:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 0x7f44b80afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 0x7f44b80afa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59016/0 (socket says 172.21.15.61:59016) 2022-07-12T08:48:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 -- 172.21.15.61:0/3895978691 learned_addr learned my addr 172.21.15.61:0/3895978691 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:47.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44b7fff700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44b8140d20 0x7f44b8091df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bd34a700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44b80ae190 0x7f44b808f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 -- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44b8140d20 msgr2=0x7f44b8091df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44b8140d20 0x7f44b8091df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 -- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44b80ae190 msgr2=0x7f44b808f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44b80ae190 0x7f44b808f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bcb49700 1 -- 172.21.15.61:0/3895978691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac00b1e0 con 0x7f44b80b24b0 2022-07-12T08:48:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44b7fff700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44b8140d20 0x7f44b8091df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.894+0000 7f44bd34a700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44b80ae190 0x7f44b808f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44bcb49700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 0x7f44b80afa00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f44a800c000 tx=0x7f44a8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44a8010780 con 0x7f44b80b24b0 2022-07-12T08:48:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a80108e0 con 0x7f44b80b24b0 2022-07-12T08:48:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b8091950 con 0x7f44b80b24b0 2022-07-12T08:48:47.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44a8010c40 con 0x7f44b80b24b0 2022-07-12T08:48:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b81530e0 con 0x7f44b80b24b0 2022-07-12T08:48:47.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b8091ae0 con 0x7f44b80b24b0 2022-07-12T08:48:47.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f44a801c460 con 0x7f44b80b24b0 2022-07-12T08:48:47.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b5ffb700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4498053a60 0x7f4498055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f44a806c520 con 0x7f44b80b24b0 2022-07-12T08:48:47.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.898+0000 7f44b7fff700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4498053a60 0x7f4498055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.902+0000 7f44b7fff700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4498053a60 0x7f4498055f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f44ac01a000 tx=0x7f44ac015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:47.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:47.906+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44a80432c0 con 0x7f44b80b24b0 2022-07-12T08:48:48.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.066+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f44b8057460 con 0x7f44b80b24b0 2022-07-12T08:48:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.446+0000 7f44b5ffb700 1 -- 172.21.15.61:0/3895978691 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v52) v1 ==== 104+0+0 (secure 0 0 0) 0x7f44a80468e0 con 0x7f44b80b24b0 2022-07-12T08:48:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4498053a60 msgr2=0x7f4498055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4498053a60 0x7f4498055f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f44ac01a000 tx=0x7f44ac015000).stop 2022-07-12T08:48:48.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 msgr2=0x7f44b80afa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 0x7f44b80afa00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f44a800c000 tx=0x7f44a8006df0).stop 2022-07-12T08:48:48.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 shutdown_connections 2022-07-12T08:48:48.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4498053a60 0x7f4498055f20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44b80ae190 0x7f44b808f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44b8140d20 0x7f44b8091df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 --2- 172.21.15.61:0/3895978691 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44b80b24b0 0x7f44b80afa00 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 >> 172.21.15.61:0/3895978691 conn(0x7f44b80b5860 msgr2=0x7f44b813e180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:48.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 shutdown_connections 2022-07-12T08:48:48.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.450+0000 7f44bdb4b700 1 -- 172.21.15.61:0/3895978691 wait complete. 2022-07-12T08:48:48.480 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:48:48.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c335c700 1 Processor -- start 2022-07-12T08:48:48.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c335c700 1 -- start start 2022-07-12T08:48:48.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c335c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f93bc0ab290 0x7f93bc0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c335c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc0abc70 con 0x7f93bc0afa60 2022-07-12T08:48:48.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c335c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93bc05c720 con 0x7f93bc05f140 2022-07-12T08:48:48.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c335c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93bc05f580 con 0x7f93bc0ab290 2022-07-12T08:48:48.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c2b5b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f93bc0afa60 0x7f93bc147530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47246/0 (socket says 172.21.15.61:47246) 2022-07-12T08:48:48.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c2b5b700 1 -- 172.21.15.61:0/2076277725 learned_addr learned my addr 172.21.15.61:0/2076277725 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:48.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c235a700 1 --2- 172.21.15.61:0/2076277725 >> v2:172.21.15.111:3300/0 conn(0x7f93bc0ab290 0x7f93bc0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979883054 0 0) 0x7f93bc0abc70 con 0x7f93bc0afa60 2022-07-12T08:48:48.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a4003260 con 0x7f93bc0afa60 2022-07-12T08:48:48.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755578256 0 0) 0x7f93bc05c720 con 0x7f93bc05f140 2022-07-12T08:48:48.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93bc0abc70 con 0x7f93bc05f140 2022-07-12T08:48:48.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4210886080 0 0) 0x7f93a4003260 con 0x7f93bc0afa60 2022-07-12T08:48:48.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 >> v2:172.21.15.111:3300/0 conn(0x7f93bc0ab290 msgr2=0x7f93bc0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:48.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 --2- 172.21.15.61:0/2076277725 >> v2:172.21.15.111:3300/0 conn(0x7f93bc0ab290 0x7f93bc0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 >> v1:172.21.15.111:6789/0 conn(0x7f93bc05f140 legacy=0x7f93bc05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc05c9c0 con 0x7f93bc0afa60 2022-07-12T08:48:48.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f93b40015e0 con 0x7f93bc0afa60 2022-07-12T08:48:48.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 >> v1:172.21.15.61:6789/0 conn(0x7f93bc0afa60 legacy=0x7f93bc147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93a4003930 0x7f93a4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93a4004500 0x7f93a4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93a400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc05f580 con 0x7f93a4004e70 2022-07-12T08:48:48.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a400bd20 con 0x7f93a4003930 2022-07-12T08:48:48.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a400bee0 con 0x7f93a4004500 2022-07-12T08:48:48.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.606+0000 7f93c2b5b700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93a400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c235a700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93a4003930 0x7f93a4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c1b59700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93a4004500 0x7f93a4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 -- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93a4004500 msgr2=0x7f93a4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:48.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93a4004500 0x7f93a4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 -- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93a4003930 msgr2=0x7f93a4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:48.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93a4003930 0x7f93a4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 -- 172.21.15.61:0/2076277725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4003260 con 0x7f93a4004e70 2022-07-12T08:48:48.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c1b59700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93a4004500 0x7f93a4004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:48.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c235a700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93a4003930 0x7f93a4003d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:48.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 -- 172.21.15.61:0/2076277725 reap_dead start 2022-07-12T08:48:48.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c2b5b700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93a400b750 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f93b40013b0 tx=0x7f93bc0aadc0).ready entity=mon.0 client_cookie=8ce89a2bc223b28 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:48.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93b40070d0 con 0x7f93a4004e70 2022-07-12T08:48:48.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4007230 con 0x7f93a4004e70 2022-07-12T08:48:48.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c1358700 1 -- 172.21.15.61:0/2076277725 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93b400c740 con 0x7f93a4004e70 2022-07-12T08:48:48.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 -- 172.21.15.61:0/2076277725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 msgr2=0x7f93a400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93a400b750 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f93b40013b0 tx=0x7f93bc0aadc0).stop 2022-07-12T08:48:48.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 -- 172.21.15.61:0/2076277725 shutdown_connections 2022-07-12T08:48:48.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 --2- 172.21.15.61:0/2076277725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93a400b750 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 -- 172.21.15.61:0/2076277725 >> 172.21.15.61:0/2076277725 conn(0x7f93bc0b1950 msgr2=0x7f93bc0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:48.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 -- 172.21.15.61:0/2076277725 shutdown_connections 2022-07-12T08:48:48.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 -- 172.21.15.61:0/2076277725 wait complete. 2022-07-12T08:48:48.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.610+0000 7f93c335c700 1 Processor -- start 2022-07-12T08:48:48.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 -- start start 2022-07-12T08:48:48.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93bc0646c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc0656e0 0x7f93bc14caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93bc14cfe0 0x7f93bc159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc1498c0 con 0x7f93a4004e70 2022-07-12T08:48:48.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc148ee0 con 0x7f93bc0656e0 2022-07-12T08:48:48.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93bc149180 con 0x7f93bc14cfe0 2022-07-12T08:48:48.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93bc0646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93bc0646c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59036/0 (socket says 172.21.15.61:59036) 2022-07-12T08:48:48.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 -- 172.21.15.61:0/108172799 learned_addr learned my addr 172.21.15.61:0/108172799 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:48.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c1b59700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc0656e0 0x7f93bc14caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c2b5b700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93bc14cfe0 0x7f93bc159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 -- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93bc14cfe0 msgr2=0x7f93bc159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93bc14cfe0 0x7f93bc159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 -- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc0656e0 msgr2=0x7f93bc14caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:48.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc0656e0 0x7f93bc14caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:48.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 -- 172.21.15.61:0/108172799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93bc149420 con 0x7f93a4004e70 2022-07-12T08:48:48.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c1b59700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc0656e0 0x7f93bc14caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:48.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c2b5b700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93bc14cfe0 0x7f93bc159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:48.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c235a700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93bc0646c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f93ac00b000 tx=0x7f93ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:48.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93ac012610 con 0x7f93a4004e70 2022-07-12T08:48:48.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac012770 con 0x7f93a4004e70 2022-07-12T08:48:48.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93bc14d460 con 0x7f93a4004e70 2022-07-12T08:48:48.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93ac012ad0 con 0x7f93a4004e70 2022-07-12T08:48:48.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.614+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93bc066a00 con 0x7f93a4004e70 2022-07-12T08:48:48.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.618+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93bc1495b0 con 0x7f93a4004e70 2022-07-12T08:48:48.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.618+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f93ac012c30 con 0x7f93a4004e70 2022-07-12T08:48:48.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.618+0000 7f93ab7fe700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f939c0539d0 0x7f939c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:48.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.618+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f93ac06c8d0 con 0x7f93a4004e70 2022-07-12T08:48:48.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.618+0000 7f93c1b59700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f939c0539d0 0x7f939c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:48.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.618+0000 7f93c1b59700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f939c0539d0 0x7f939c055e90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f93b8009000 tx=0x7f93b8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:48.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.622+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f93ac0436e0 con 0x7f93a4004e70 2022-07-12T08:48:48.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:48.746+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f93bc057460 con 0x7f93a4004e70 2022-07-12T08:48:49.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.450+0000 7f93ab7fe700 1 -- 172.21.15.61:0/108172799 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) v1 ==== 185+0+0 (secure 0 0 0) 0x7f93ac046d00 con 0x7f93a4004e70 2022-07-12T08:48:49.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f939c0539d0 msgr2=0x7f939c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:49.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f939c0539d0 0x7f939c055e90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f93b8009000 tx=0x7f93b8009040).stop 2022-07-12T08:48:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 msgr2=0x7f93bc0646c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:49.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93bc0646c0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f93ac00b000 tx=0x7f93ac008040).stop 2022-07-12T08:48:49.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 shutdown_connections 2022-07-12T08:48:49.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f93bc14cfe0 0x7f93bc159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f939c0539d0 0x7f939c055e90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f93bc0656e0 0x7f93bc14caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 --2- 172.21.15.61:0/108172799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93a4004e70 0x7f93bc0646c0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 >> 172.21.15.61:0/108172799 conn(0x7f93bc0b1950 msgr2=0x7f93bc0afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 shutdown_connections 2022-07-12T08:48:49.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.454+0000 7f93c335c700 1 -- 172.21.15.61:0/108172799 wait complete. 2022-07-12T08:48:49.468 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:48:49.480 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:48:49.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 Processor -- start 2022-07-12T08:48:49.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 -- start start 2022-07-12T08:48:49.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 0x7f3bc8060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3bc809bce0 0x7f3bc80af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc80aecb0 con 0x7f3bc80b3f60 2022-07-12T08:48:49.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc8143b00 con 0x7f3bc809bce0 2022-07-12T08:48:49.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bcd383700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc81435c0 con 0x7f3bc80af280 2022-07-12T08:48:49.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bccb82700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3bc80b3f60 0x7f3bc80b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47262/0 (socket says 172.21.15.61:47262) 2022-07-12T08:48:49.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.586+0000 7f3bccb82700 1 -- 172.21.15.61:0/640608030 learned_addr learned my addr 172.21.15.61:0/640608030 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:49.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc77fe700 1 --2- 172.21.15.61:0/640608030 >> v2:172.21.15.111:3300/0 conn(0x7f3bc809bce0 0x7f3bc80af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225771578 0 0) 0x7f3bc80aecb0 con 0x7f3bc80b3f60 2022-07-12T08:48:49.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb4003260 con 0x7f3bc80b3f60 2022-07-12T08:48:49.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 0x7f3bc8060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322227586 0 0) 0x7f3bb4003260 con 0x7f3bc80b3f60 2022-07-12T08:48:49.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 msgr2=0x7f3bc8060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:49.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 0x7f3bc8060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 >> v2:172.21.15.111:3300/0 conn(0x7f3bc809bce0 msgr2=0x7f3bc80af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:49.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 --2- 172.21.15.61:0/640608030 >> v2:172.21.15.111:3300/0 conn(0x7f3bc809bce0 0x7f3bc80af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc8143860 con 0x7f3bc80b3f60 2022-07-12T08:48:49.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f3bc0001660 con 0x7f3bc80b3f60 2022-07-12T08:48:49.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 >> v1:172.21.15.61:6789/0 conn(0x7f3bc80b3f60 legacy=0x7f3bc80b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:49.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb4003750 0x7f3bb4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 0x7f3bb4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb40049f0 0x7f3bb4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc81435c0 con 0x7f3bb4003750 2022-07-12T08:48:49.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4004330 con 0x7f3bb40049f0 2022-07-12T08:48:49.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb400b7a0 con 0x7f3bc80af280 2022-07-12T08:48:49.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb4003750 0x7f3bb4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bccb82700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb40049f0 0x7f3bb4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc77fe700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 0x7f3bb4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 -- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 msgr2=0x7f3bb4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:49.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bc80af280 0x7f3bb4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 -- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb40049f0 msgr2=0x7f3bb4004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:49.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bb40049f0 0x7f3bb4004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 -- 172.21.15.61:0/640608030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4003260 con 0x7f3bb4003750 2022-07-12T08:48:49.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb4003750 0x7f3bb4003b70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f3bb4004470 tx=0x7f3bbc006d90).ready entity=mon.0 client_cookie=361cdbd768d78d99 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:49.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bbc013450 con 0x7f3bb4003750 2022-07-12T08:48:49.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc0135b0 con 0x7f3bb4003750 2022-07-12T08:48:49.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bc6ffd700 1 -- 172.21.15.61:0/640608030 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bbc013910 con 0x7f3bb4003750 2022-07-12T08:48:49.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 -- 172.21.15.61:0/640608030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb4003750 msgr2=0x7f3bb4003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:49.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 --2- 172.21.15.61:0/640608030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb4003750 0x7f3bb4003b70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f3bb4004470 tx=0x7f3bbc006d90).stop 2022-07-12T08:48:49.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bccb82700 1 -- 172.21.15.61:0/640608030 reap_dead start 2022-07-12T08:48:49.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 -- 172.21.15.61:0/640608030 shutdown_connections 2022-07-12T08:48:49.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 -- 172.21.15.61:0/640608030 >> 172.21.15.61:0/640608030 conn(0x7f3bc80b92c0 msgr2=0x7f3bc80b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:49.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 -- 172.21.15.61:0/640608030 shutdown_connections 2022-07-12T08:48:49.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 -- 172.21.15.61:0/640608030 wait complete. 2022-07-12T08:48:49.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 Processor -- start 2022-07-12T08:48:49.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.590+0000 7f3bcd383700 1 -- start start 2022-07-12T08:48:49.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bb4003750 0x7f3bc81449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 0x7f3bc809ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bc80b0700 0x7f3bc8151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8143fe0 con 0x7f3bc80b1f60 2022-07-12T08:48:49.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8144280 con 0x7f3bc80b0700 2022-07-12T08:48:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8144520 con 0x7f3bb4003750 2022-07-12T08:48:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 0x7f3bc809ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 0x7f3bc809ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59054/0 (socket says 172.21.15.61:59054) 2022-07-12T08:48:49.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 -- 172.21.15.61:0/2953313106 learned_addr learned my addr 172.21.15.61:0/2953313106 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bb4003750 0x7f3bc81449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bccb82700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bc80b0700 0x7f3bc8151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 -- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bb4003750 msgr2=0x7f3bc81449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:49.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bb4003750 0x7f3bc81449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 -- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bc80b0700 msgr2=0x7f3bc8151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:49.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bc80b0700 0x7f3bc8151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:49.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 -- 172.21.15.61:0/2953313106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc809dff0 con 0x7f3bc80b1f60 2022-07-12T08:48:49.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc77fe700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 0x7f3bc809ee10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f3bb800b000 tx=0x7f3bb80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:49.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bb800fa10 con 0x7f3bc80b1f60 2022-07-12T08:48:49.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb800fb70 con 0x7f3bc80b1f60 2022-07-12T08:48:49.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc8151fa0 con 0x7f3bc80b1f60 2022-07-12T08:48:49.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bb800fd20 con 0x7f3bc80b1f60 2022-07-12T08:48:49.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc81524f0 con 0x7f3bc80b1f60 2022-07-12T08:48:49.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc81446e0 con 0x7f3bc80b1f60 2022-07-12T08:48:49.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.594+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3bb8024070 con 0x7f3bc80b1f60 2022-07-12T08:48:49.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.598+0000 7f3bc57fa700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ba8053ad0 0x7f3ba8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:49.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.598+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f3bb806c980 con 0x7f3bc80b1f60 2022-07-12T08:48:49.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.598+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ba8053ad0 0x7f3ba8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:49.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.598+0000 7f3bc7fff700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ba8053ad0 0x7f3ba8055f90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f3bbc01c000 tx=0x7f3bbc017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:49.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.606+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3bb8043a40 con 0x7f3bc80b1f60 2022-07-12T08:48:49.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:49.730+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3bc80afeb0 con 0x7f3bc80b1f60 2022-07-12T08:48:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.454+0000 7f3bc57fa700 1 -- 172.21.15.61:0/2953313106 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3bb8047060 con 0x7f3bc80b1f60 2022-07-12T08:48:50.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ba8053ad0 msgr2=0x7f3ba8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ba8053ad0 0x7f3ba8055f90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f3bbc01c000 tx=0x7f3bbc017000).stop 2022-07-12T08:48:50.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 msgr2=0x7f3bc809ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 0x7f3bc809ee10 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f3bb800b000 tx=0x7f3bb80091d0).stop 2022-07-12T08:48:50.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 shutdown_connections 2022-07-12T08:48:50.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ba8053ad0 0x7f3ba8055f90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3bc80b0700 0x7f3bc8151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bc80b1f60 0x7f3bc809ee10 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 --2- 172.21.15.61:0/2953313106 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3bb4003750 0x7f3bc81449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 >> 172.21.15.61:0/2953313106 conn(0x7f3bc80b92c0 msgr2=0x7f3bc805c940 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:50.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 shutdown_connections 2022-07-12T08:48:50.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.462+0000 7f3bcd383700 1 -- 172.21.15.61:0/2953313106 wait complete. 2022-07-12T08:48:50.471 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:48:50.480 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:48:50.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 Processor -- start 2022-07-12T08:48:50.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 -- start start 2022-07-12T08:48:50.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0cd805d060 0x7f0cd8060eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0cd80aafa0 0x7f0cd8146b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd805d440 con 0x7f0cd805f860 2022-07-12T08:48:50.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd80aa230 con 0x7f0cd80aafa0 2022-07-12T08:48:50.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd759e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd8146ff0 con 0x7f0cd805d060 2022-07-12T08:48:50.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd6d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0cd805f860 0x7f0cd805fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47282/0 (socket says 172.21.15.61:47282) 2022-07-12T08:48:50.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd6d9d700 1 -- 172.21.15.61:0/3748023852 learned_addr learned my addr 172.21.15.61:0/3748023852 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:50.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0cd80aafa0 0x7f0cd8146b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd659c700 1 --2- 172.21.15.61:0/3748023852 >> v2:172.21.15.111:3300/0 conn(0x7f0cd805d060 0x7f0cd8060eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd5d9b700 1 -- 172.21.15.61:0/3748023852 >> v2:172.21.15.111:3300/0 conn(0x7f0cd805d060 msgr2=0x7f0cd8060eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:50.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd559a700 1 -- 172.21.15.61:0/3748023852 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055109982 0 0) 0x7f0cd805d440 con 0x7f0cd805f860 2022-07-12T08:48:50.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd5d9b700 1 --2- 172.21.15.61:0/3748023852 >> v2:172.21.15.111:3300/0 conn(0x7f0cd805d060 0x7f0cd8060eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd5d9b700 1 -- 172.21.15.61:0/3748023852 >> v1:172.21.15.61:6789/0 conn(0x7f0cd805f860 legacy=0x7f0cd805fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd5d9b700 1 -- 172.21.15.61:0/3748023852 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8060c60 con 0x7f0cd80aafa0 2022-07-12T08:48:50.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.586+0000 7f0cd5d9b700 1 --2- 172.21.15.61:0/3748023852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd80aafa0 0x7f0cd8146b00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f0cc80032f0 tx=0x7f0cc80091d0).ready entity=mon.0 client_cookie=cb55fc33c6eca272 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:50.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd559a700 1 -- 172.21.15.61:0/3748023852 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc800b680 con 0x7f0cd80aafa0 2022-07-12T08:48:50.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd559a700 1 -- 172.21.15.61:0/3748023852 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc800b7e0 con 0x7f0cd80aafa0 2022-07-12T08:48:50.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd559a700 1 -- 172.21.15.61:0/3748023852 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc800b950 con 0x7f0cd80aafa0 2022-07-12T08:48:50.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 -- 172.21.15.61:0/3748023852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd80aafa0 msgr2=0x7f0cd8146b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3748023852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd80aafa0 0x7f0cd8146b00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 rx=0x7f0cc80032f0 tx=0x7f0cc80091d0).stop 2022-07-12T08:48:50.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 -- 172.21.15.61:0/3748023852 shutdown_connections 2022-07-12T08:48:50.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3748023852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd80aafa0 0x7f0cd8146b00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3748023852 >> v2:172.21.15.111:3300/0 conn(0x7f0cd805d060 0x7f0cd8060eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 -- 172.21.15.61:0/3748023852 >> 172.21.15.61:0/3748023852 conn(0x7f0cd80b1a50 msgr2=0x7f0cd80b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:50.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 -- 172.21.15.61:0/3748023852 shutdown_connections 2022-07-12T08:48:50.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 -- 172.21.15.61:0/3748023852 wait complete. 2022-07-12T08:48:50.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 Processor -- start 2022-07-12T08:48:50.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.590+0000 7f0cd759e700 1 -- start start 2022-07-12T08:48:50.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 0x7f0cd8063f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd805f860 0x7f0cd8145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0cd8067690 0x7f0cd8140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80af9e0 con 0x7f0cd805d060 2022-07-12T08:48:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8063a70 con 0x7f0cd805f860 2022-07-12T08:48:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80add30 con 0x7f0cd8067690 2022-07-12T08:48:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 0x7f0cd8063f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 0x7f0cd8063f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59066/0 (socket says 172.21.15.61:59066) 2022-07-12T08:48:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 -- 172.21.15.61:0/3273876155 learned_addr learned my addr 172.21.15.61:0/3273876155 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd5d9b700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd805f860 0x7f0cd8145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd6d9d700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0cd8067690 0x7f0cd8140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 -- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0cd8067690 msgr2=0x7f0cd8140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0cd8067690 0x7f0cd8140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 -- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd805f860 msgr2=0x7f0cd8145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd805f860 0x7f0cd8145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 -- 172.21.15.61:0/3273876155 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8009470 con 0x7f0cd805d060 2022-07-12T08:48:50.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd5d9b700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd805f860 0x7f0cd8145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:50.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd6d9d700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0cd8067690 0x7f0cd8140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:50.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd659c700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 0x7f0cd8063f10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f0cc000c000 tx=0x7f0cc0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:50.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc0010780 con 0x7f0cd805d060 2022-07-12T08:48:50.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc00108e0 con 0x7f0cd805d060 2022-07-12T08:48:50.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd80af520 con 0x7f0cd805d060 2022-07-12T08:48:50.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cc0010c40 con 0x7f0cd805d060 2022-07-12T08:48:50.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd8140820 con 0x7f0cd805d060 2022-07-12T08:48:50.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.594+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd80638d0 con 0x7f0cd805d060 2022-07-12T08:48:50.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.598+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0cc001c460 con 0x7f0cd805d060 2022-07-12T08:48:50.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.598+0000 7f0ccf7fe700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0cbc053af0 0x7f0cbc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.598+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0cc006c0d0 con 0x7f0cd805d060 2022-07-12T08:48:50.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.598+0000 7f0cd5d9b700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0cbc053af0 0x7f0cbc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.598+0000 7f0cd5d9b700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0cbc053af0 0x7f0cbc055fb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f0cc8017000 tx=0x7f0cc8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:50.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.602+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0cd80638d0 con 0x7f0cd805d060 2022-07-12T08:48:50.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.734+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0cd80638d0 con 0x7f0cd805d060 2022-07-12T08:48:50.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0ccf7fe700 1 -- 172.21.15.61:0/3273876155 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0cd80638d0 con 0x7f0cd805d060 2022-07-12T08:48:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0cbc053af0 msgr2=0x7f0cbc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0cbc053af0 0x7f0cbc055fb0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f0cc8017000 tx=0x7f0cc8016000).stop 2022-07-12T08:48:50.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 msgr2=0x7f0cd8063f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 0x7f0cd8063f10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 rx=0x7f0cc000c000 tx=0x7f0cc0006df0).stop 2022-07-12T08:48:50.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 shutdown_connections 2022-07-12T08:48:50.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0cd8067690 0x7f0cd8140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0cbc053af0 0x7f0cbc055fb0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0cd805f860 0x7f0cd8145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 --2- 172.21.15.61:0/3273876155 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd805d060 0x7f0cd8063f10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 >> 172.21.15.61:0/3273876155 conn(0x7f0cd80b1a50 msgr2=0x7f0cd81430a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 shutdown_connections 2022-07-12T08:48:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.742+0000 7f0cd759e700 1 -- 172.21.15.61:0/3273876155 wait complete. 2022-07-12T08:48:50.761 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-12T08:48:50.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f6860ad6700 1 Processor -- start 2022-07-12T08:48:50.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f6860ad6700 1 -- start start 2022-07-12T08:48:50.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f6860ad6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f685c144ad0 0x7f685c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f6860ad6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685c0ad2f0 con 0x7f685c05c050 2022-07-12T08:48:50.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f6860ad6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f685c05c430 con 0x7f685c0b01f0 2022-07-12T08:48:50.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f6860ad6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f685c0b0a40 con 0x7f685c144ad0 2022-07-12T08:48:50.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f685c144ad0 0x7f685c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f685c144ad0 0x7f685c0ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59078/0 (socket says 172.21.15.61:59078) 2022-07-12T08:48:50.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 -- 172.21.15.61:0/1952770303 learned_addr learned my addr 172.21.15.61:0/1952770303 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:50.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f685c05c050 0x7f685c05cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47294/0 (socket says 172.21.15.61:47294) 2022-07-12T08:48:50.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 -- 172.21.15.61:0/1952770303 >> v1:172.21.15.111:6789/0 conn(0x7f685c0b01f0 legacy=0x7f685c1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 -- 172.21.15.61:0/1952770303 >> v1:172.21.15.61:6789/0 conn(0x7f685c05c050 legacy=0x7f685c05cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 -- 172.21.15.61:0/1952770303 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c0b2420 con 0x7f685c144ad0 2022-07-12T08:48:50.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685ad9d700 1 --2- 172.21.15.61:0/1952770303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c144ad0 0x7f685c0ac710 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f6854002f70 tx=0x7f6854006eb0).ready entity=mon.0 client_cookie=509610f8161128f2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:50.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685959a700 1 -- 172.21.15.61:0/1952770303 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f685400d660 con 0x7f685c144ad0 2022-07-12T08:48:50.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685959a700 1 -- 172.21.15.61:0/1952770303 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6854000e90 con 0x7f685c144ad0 2022-07-12T08:48:50.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.862+0000 7f685959a700 1 -- 172.21.15.61:0/1952770303 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f685400d9f0 con 0x7f685c144ad0 2022-07-12T08:48:50.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- 172.21.15.61:0/1952770303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c144ad0 msgr2=0x7f685c0ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 --2- 172.21.15.61:0/1952770303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c144ad0 0x7f685c0ac710 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f6854002f70 tx=0x7f6854006eb0).stop 2022-07-12T08:48:50.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- 172.21.15.61:0/1952770303 shutdown_connections 2022-07-12T08:48:50.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 --2- 172.21.15.61:0/1952770303 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c144ad0 0x7f685c0ac710 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- 172.21.15.61:0/1952770303 >> 172.21.15.61:0/1952770303 conn(0x7f685c062c00 msgr2=0x7f685c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:50.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- 172.21.15.61:0/1952770303 shutdown_connections 2022-07-12T08:48:50.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- 172.21.15.61:0/1952770303 wait complete. 2022-07-12T08:48:50.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 Processor -- start 2022-07-12T08:48:50.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- start start 2022-07-12T08:48:50.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 0x7f685c14e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f685c0b01f0 0x7f685c0ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f685c144ad0 0x7f685c14a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c149280 con 0x7f685c05c050 2022-07-12T08:48:50.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c0aec90 con 0x7f685c0b01f0 2022-07-12T08:48:50.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6860ad6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f685c14b1d0 con 0x7f685c144ad0 2022-07-12T08:48:50.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f685a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 0x7f685c14e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f685a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 0x7f685c14e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59082/0 (socket says 172.21.15.61:59082) 2022-07-12T08:48:50.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f685a59c700 1 -- 172.21.15.61:0/2990871669 learned_addr learned my addr 172.21.15.61:0/2990871669 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:50.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f6859d9b700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f685c0b01f0 0x7f685c0ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.866+0000 7f685ad9d700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f685c144ad0 0x7f685c14a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f685a59c700 1 -- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f685c144ad0 msgr2=0x7f685c14a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f685a59c700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f685c144ad0 0x7f685c14a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f685a59c700 1 -- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f685c0b01f0 msgr2=0x7f685c0ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:50.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f685a59c700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f685c0b01f0 0x7f685c0ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:50.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f685a59c700 1 -- 172.21.15.61:0/2990871669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685400b1e0 con 0x7f685c05c050 2022-07-12T08:48:50.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f685a59c700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 0x7f685c14e360 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f685c0aee20 tx=0x7f6850007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:50.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68500070d0 con 0x7f685c05c050 2022-07-12T08:48:50.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6850007230 con 0x7f685c05c050 2022-07-12T08:48:50.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c14b470 con 0x7f685c05c050 2022-07-12T08:48:50.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6850005550 con 0x7f685c05c050 2022-07-12T08:48:50.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c146480 con 0x7f685c05c050 2022-07-12T08:48:50.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c14b030 con 0x7f685c05c050 2022-07-12T08:48:50.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f685001a070 con 0x7f685c05c050 2022-07-12T08:48:50.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f684b7fe700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6838053ad0 0x7f6838055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:50.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f685006ad70 con 0x7f685c05c050 2022-07-12T08:48:50.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.870+0000 7f6859d9b700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6838053ad0 0x7f6838055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:50.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.874+0000 7f6859d9b700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6838053ad0 0x7f6838055f90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f684c00d000 tx=0x7f684c000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:50.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:50.878+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f685c14b030 con 0x7f685c05c050 2022-07-12T08:48:51.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.046+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7f685c14b030 con 0x7f685c05c050 2022-07-12T08:48:51.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.458+0000 7f684b7fe700 1 -- 172.21.15.61:0/2990871669 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v346) v1 ==== 138+0+0 (secure 0 0 0) 0x7f685c14b030 con 0x7f685c05c050 2022-07-12T08:48:51.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6838053ad0 msgr2=0x7f6838055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6838053ad0 0x7f6838055f90 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f684c00d000 tx=0x7f684c000f40).stop 2022-07-12T08:48:51.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 msgr2=0x7f685c14e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 0x7f685c14e360 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7f685c0aee20 tx=0x7f6850007c40).stop 2022-07-12T08:48:51.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 shutdown_connections 2022-07-12T08:48:51.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f685c144ad0 0x7f685c14a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6838053ad0 0x7f6838055f90 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f685c0b01f0 0x7f685c0ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 --2- 172.21.15.61:0/2990871669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f685c05c050 0x7f685c14e360 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 >> 172.21.15.61:0/2990871669 conn(0x7f685c062c00 msgr2=0x7f685c0b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:51.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 shutdown_connections 2022-07-12T08:48:51.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.462+0000 7f6860ad6700 1 -- 172.21.15.61:0/2990871669 wait complete. 2022-07-12T08:48:51.480 INFO:tasks.cephfs_test_runner:test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:48:51.481 INFO:tasks.cephfs_test_runner:Starting test: test_filtered_df (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:48:51.482 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-12T08:48:51.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.586+0000 7f8ec4987700 1 Processor -- start 2022-07-12T08:48:51.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.586+0000 7f8ec4987700 1 -- start start 2022-07-12T08:48:51.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ec00af890 0x7f8ec005c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:51.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec0143610 con 0x7f8ec0060b50 2022-07-12T08:48:51.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec0143840 con 0x7f8ec00af4b0 2022-07-12T08:48:51.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ec00afd00 con 0x7f8ec00af890 2022-07-12T08:48:51.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebf59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ec00af890 0x7f8ec005c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:51.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebe59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8ec0060b50 0x7f8ec005bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47308/0 (socket says 172.21.15.61:47308) 2022-07-12T08:48:51.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebe59c700 1 -- 172.21.15.61:0/2613256060 learned_addr learned my addr 172.21.15.61:0/2613256060 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:51.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebf59e700 1 -- 172.21.15.61:0/2613256060 >> v1:172.21.15.111:6789/0 conn(0x7f8ec00af4b0 legacy=0x7f8ec0142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebdd9b700 1 -- 172.21.15.61:0/2613256060 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878492452 0 0) 0x7f8ec0143840 con 0x7f8ec00af4b0 2022-07-12T08:48:51.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebf59e700 1 -- 172.21.15.61:0/2613256060 >> v1:172.21.15.61:6789/0 conn(0x7f8ec0060b50 legacy=0x7f8ec005bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebf59e700 1 -- 172.21.15.61:0/2613256060 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec00afef0 con 0x7f8ec00af890 2022-07-12T08:48:51.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebf59e700 1 --2- 172.21.15.61:0/2613256060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00af890 0x7f8ec005c500 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f8eb4001110 tx=0x7f8eb0006df0).ready entity=mon.0 client_cookie=a1a33c3d4104d06 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:51.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebdd9b700 1 -- 172.21.15.61:0/2613256060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb000d5c0 con 0x7f8ec00af890 2022-07-12T08:48:51.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebdd9b700 1 -- 172.21.15.61:0/2613256060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0000e90 con 0x7f8ec00af890 2022-07-12T08:48:51.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ebdd9b700 1 -- 172.21.15.61:0/2613256060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb000d910 con 0x7f8ec00af890 2022-07-12T08:48:51.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- 172.21.15.61:0/2613256060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00af890 msgr2=0x7f8ec005c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 --2- 172.21.15.61:0/2613256060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00af890 0x7f8ec005c500 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f8eb4001110 tx=0x7f8eb0006df0).stop 2022-07-12T08:48:51.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- 172.21.15.61:0/2613256060 shutdown_connections 2022-07-12T08:48:51.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 --2- 172.21.15.61:0/2613256060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec00af890 0x7f8ec005c500 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- 172.21.15.61:0/2613256060 >> 172.21.15.61:0/2613256060 conn(0x7f8ec00b92c0 msgr2=0x7f8ec00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:51.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- 172.21.15.61:0/2613256060 shutdown_connections 2022-07-12T08:48:51.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.590+0000 7f8ec4987700 1 -- 172.21.15.61:0/2613256060 wait complete. 2022-07-12T08:48:51.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 Processor -- start 2022-07-12T08:48:51.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 -- start start 2022-07-12T08:48:51.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 0x7f8ec005caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:51.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8ec00af4b0 0x7f8ec009b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:51.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ec00af890 0x7f8ec009b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec009bd10 con 0x7f8ec0060b50 2022-07-12T08:48:51.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec00b17a0 con 0x7f8ec00af890 2022-07-12T08:48:51.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec01560b0 con 0x7f8ec00af4b0 2022-07-12T08:48:51.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 0x7f8ec005caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:51.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 0x7f8ec005caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59094/0 (socket says 172.21.15.61:59094) 2022-07-12T08:48:51.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 -- 172.21.15.61:0/3614671198 learned_addr learned my addr 172.21.15.61:0/3614671198 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:51.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebf59e700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ec00af890 0x7f8ec009b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:51.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebe59c700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8ec00af4b0 0x7f8ec009b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:51.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 -- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8ec00af4b0 msgr2=0x7f8ec009b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8ec00af4b0 0x7f8ec009b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 -- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ec00af890 msgr2=0x7f8ec009b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:51.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ec00af890 0x7f8ec009b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:51.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 -- 172.21.15.61:0/3614671198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb000b180 con 0x7f8ec0060b50 2022-07-12T08:48:51.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ebed9d700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 0x7f8ec005caf0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f8ec005d060 tx=0x7f8eb8007ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:51.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb800a580 con 0x7f8ec0060b50 2022-07-12T08:48:51.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb800a6e0 con 0x7f8ec0060b50 2022-07-12T08:48:51.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.594+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec00b09a0 con 0x7f8ec0060b50 2022-07-12T08:48:51.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8eb800a850 con 0x7f8ec0060b50 2022-07-12T08:48:51.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec00b1500 con 0x7f8ec0060b50 2022-07-12T08:48:51.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec0057460 con 0x7f8ec0060b50 2022-07-12T08:48:51.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8eb8010070 con 0x7f8ec0060b50 2022-07-12T08:48:51.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8eaffff700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e9c053af0 0x7f8e9c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:51.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8eb806b980 con 0x7f8ec0060b50 2022-07-12T08:48:51.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.598+0000 7f8ebe59c700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e9c053af0 0x7f8e9c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:51.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.602+0000 7f8ebe59c700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e9c053af0 0x7f8e9c055fb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8eb400b000 tx=0x7f8eb4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:51.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.606+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8eb8042bc0 con 0x7f8ec0060b50 2022-07-12T08:48:51.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:51.778+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7f8ec009bf80 con 0x7f8ec0060b50 2022-07-12T08:48:52.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.462+0000 7f8eaffff700 1 -- 172.21.15.61:0/3614671198 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v347) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8eb80461e0 con 0x7f8ec0060b50 2022-07-12T08:48:52.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e9c053af0 msgr2=0x7f8e9c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e9c053af0 0x7f8e9c055fb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f8eb400b000 tx=0x7f8eb4009040).stop 2022-07-12T08:48:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 msgr2=0x7f8ec005caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 0x7f8ec005caf0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 rx=0x7f8ec005d060 tx=0x7f8eb8007ed0).stop 2022-07-12T08:48:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 shutdown_connections 2022-07-12T08:48:52.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e9c053af0 0x7f8e9c055fb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8ec00af890 0x7f8ec009b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8ec00af4b0 0x7f8ec009b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 --2- 172.21.15.61:0/3614671198 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ec0060b50 0x7f8ec005caf0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 >> 172.21.15.61:0/3614671198 conn(0x7f8ec00b92c0 msgr2=0x7f8ec0148170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:52.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 shutdown_connections 2022-07-12T08:48:52.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.466+0000 7f8ec4987700 1 -- 172.21.15.61:0/3614671198 wait complete. 2022-07-12T08:48:52.487 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:48:52.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfaca3e700 1 Processor -- start 2022-07-12T08:48:52.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfaca3e700 1 -- start start 2022-07-12T08:48:52.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfaca3e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbfa809a1f0 0x7fbfa809a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:52.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfaca3e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa8099010 con 0x7fbfa8060c10 2022-07-12T08:48:52.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfaca3e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa814b780 con 0x7fbfa80b1000 2022-07-12T08:48:52.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfaca3e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa8060ff0 con 0x7fbfa809a1f0 2022-07-12T08:48:52.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbfa8060c10 0x7fbfa814b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47320/0 (socket says 172.21.15.61:47320) 2022-07-12T08:48:52.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa77fe700 1 -- 172.21.15.61:0/2414983380 learned_addr learned my addr 172.21.15.61:0/2414983380 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa7fff700 1 --2- 172.21.15.61:0/2414983380 >> v2:172.21.15.61:3300/0 conn(0x7fbfa809a1f0 0x7fbfa809a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa67fc700 1 -- 172.21.15.61:0/2414983380 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341963979 0 0) 0x7fbfa8099010 con 0x7fbfa8060c10 2022-07-12T08:48:52.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa67fc700 1 -- 172.21.15.61:0/2414983380 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf94003260 con 0x7fbfa8060c10 2022-07-12T08:48:52.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa67fc700 1 -- 172.21.15.61:0/2414983380 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676758544 0 0) 0x7fbf98001110 con 0x7fbfa80b1000 2022-07-12T08:48:52.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa7fff700 1 -- 172.21.15.61:0/2414983380 >> v1:172.21.15.111:6789/0 conn(0x7fbfa80b1000 legacy=0x7fbfa80af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa7fff700 1 -- 172.21.15.61:0/2414983380 >> v1:172.21.15.61:6789/0 conn(0x7fbfa8060c10 legacy=0x7fbfa814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa7fff700 1 -- 172.21.15.61:0/2414983380 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa814a930 con 0x7fbfa809a1f0 2022-07-12T08:48:52.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa7fff700 1 --2- 172.21.15.61:0/2414983380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa809a1f0 0x7fbfa809a660 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7fbf9c001800 tx=0x7fbfa00091d0).ready entity=mon.0 client_cookie=5f24abac44744d8e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:52.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa67fc700 1 -- 172.21.15.61:0/2414983380 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c001660 con 0x7fbfa809a1f0 2022-07-12T08:48:52.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa67fc700 1 -- 172.21.15.61:0/2414983380 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfa000b580 con 0x7fbfa809a1f0 2022-07-12T08:48:52.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.598+0000 7fbfa67fc700 1 -- 172.21.15.61:0/2414983380 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfa000b6f0 con 0x7fbfa809a1f0 2022-07-12T08:48:52.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 -- 172.21.15.61:0/2414983380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa809a1f0 msgr2=0x7fbfa809a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/2414983380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa809a1f0 0x7fbfa809a660 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7fbf9c001800 tx=0x7fbfa00091d0).stop 2022-07-12T08:48:52.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 -- 172.21.15.61:0/2414983380 shutdown_connections 2022-07-12T08:48:52.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/2414983380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa809a1f0 0x7fbfa809a660 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 -- 172.21.15.61:0/2414983380 >> 172.21.15.61:0/2414983380 conn(0x7fbfa805bd10 msgr2=0x7fbfa805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 -- 172.21.15.61:0/2414983380 shutdown_connections 2022-07-12T08:48:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 -- 172.21.15.61:0/2414983380 wait complete. 2022-07-12T08:48:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 Processor -- start 2022-07-12T08:48:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.602+0000 7fbfaca3e700 1 -- start start 2022-07-12T08:48:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 0x7fbfa8067280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfa809a1f0 0x7fbfa80af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa80b1000 0x7fbfa80ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8156b90 con 0x7fbfa8060c10 2022-07-12T08:48:52.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8144830 con 0x7fbfa80b1000 2022-07-12T08:48:52.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8156dc0 con 0x7fbfa809a1f0 2022-07-12T08:48:52.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 0x7fbfa8067280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:52.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 0x7fbfa8067280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59108/0 (socket says 172.21.15.61:59108) 2022-07-12T08:48:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 -- 172.21.15.61:0/3498018557 learned_addr learned my addr 172.21.15.61:0/3498018557 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa6ffd700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfa809a1f0 0x7fbfa80af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa7fff700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa80b1000 0x7fbfa80ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 -- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfa809a1f0 msgr2=0x7fbfa80af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfa809a1f0 0x7fbfa80af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 -- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa80b1000 msgr2=0x7fbfa80ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa80b1000 0x7fbfa80ac320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 -- 172.21.15.61:0/3498018557 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0009470 con 0x7fbfa8060c10 2022-07-12T08:48:52.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa77fe700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 0x7fbfa8067280 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7fbf9c0011d0 tx=0x7fbf9c001ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:52.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c007170 con 0x7fbfa8060c10 2022-07-12T08:48:52.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0072d0 con 0x7fbfa8060c10 2022-07-12T08:48:52.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa8062b40 con 0x7fbfa8060c10 2022-07-12T08:48:52.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf9c004920 con 0x7fbfa8060c10 2022-07-12T08:48:52.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa81469a0 con 0x7fbfa8060c10 2022-07-12T08:48:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.606+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa8156f80 con 0x7fbfa8060c10 2022-07-12T08:48:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.610+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbf9c01b070 con 0x7fbfa8060c10 2022-07-12T08:48:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.610+0000 7fbfa4ff9700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbf88053940 0x7fbf88055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.610+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fbf9c06b440 con 0x7fbfa8060c10 2022-07-12T08:48:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.610+0000 7fbfa6ffd700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbf88053940 0x7fbf88055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.610+0000 7fbfa6ffd700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbf88053940 0x7fbf88055e00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fbf9800d000 tx=0x7fbf98007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:52.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.614+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbf9c042510 con 0x7fbfa8060c10 2022-07-12T08:48:52.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.742+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fbfa806b2b0 con 0x7fbfa8060c10 2022-07-12T08:48:52.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.750+0000 7fbfa4ff9700 1 -- 172.21.15.61:0/3498018557 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbf9c019020 con 0x7fbfa8060c10 2022-07-12T08:48:52.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbf88053940 msgr2=0x7fbf88055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbf88053940 0x7fbf88055e00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fbf9800d000 tx=0x7fbf98007b90).stop 2022-07-12T08:48:52.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 msgr2=0x7fbfa8067280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:52.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 0x7fbfa8067280 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7fbf9c0011d0 tx=0x7fbf9c001ae0).stop 2022-07-12T08:48:52.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 shutdown_connections 2022-07-12T08:48:52.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbf88053940 0x7fbf88055e00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbfa80b1000 0x7fbfa80ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbfa809a1f0 0x7fbfa80af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 --2- 172.21.15.61:0/3498018557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfa8060c10 0x7fbfa8067280 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:52.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 >> 172.21.15.61:0/3498018557 conn(0x7fbfa805bd10 msgr2=0x7fbfa805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:52.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 shutdown_connections 2022-07-12T08:48:52.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:52.754+0000 7fbfaca3e700 1 -- 172.21.15.61:0/3498018557 wait complete. 2022-07-12T08:48:52.775 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:48:52.776 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:52.793 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:52.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:52.840 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:52.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:48:52.884 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:48:52.885 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:52.886 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:52.886 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:48:52.894 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:48:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:52.896 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:48:52.896 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:52.897 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:52.897 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:48:52.913 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:48:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:52.918 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:48:52.919 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:48:52.920 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:48:52.920 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:48:52.935 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:48:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:48:52.937 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:48:52.938 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:48:53.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.042+0000 7f19a8b5c700 1 Processor -- start 2022-07-12T08:48:53.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.042+0000 7f19a8b5c700 1 -- start start 2022-07-12T08:48:53.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f19a4061e50 0x7f19a414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4062ae0 0x7f19a4147070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a4060e30 con 0x7f19a4147560 2022-07-12T08:48:53.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a414a8b0 con 0x7f19a4061e50 2022-07-12T08:48:53.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a414ab50 con 0x7f19a4062ae0 2022-07-12T08:48:53.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a2d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f19a4147560 0x7f19a405ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47336/0 (socket says 172.21.15.61:47336) 2022-07-12T08:48:53.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a2d9d700 1 -- 172.21.15.61:0/650018708 learned_addr learned my addr 172.21.15.61:0/650018708 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:53.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a259c700 1 --2- 172.21.15.61:0/650018708 >> v2:172.21.15.61:3300/0 conn(0x7f19a4061e50 0x7f19a414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a1d9b700 1 --2- 172.21.15.61:0/650018708 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4062ae0 0x7f19a4147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a259c700 1 -- 172.21.15.61:0/650018708 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4062ae0 msgr2=0x7f19a4147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:53.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a259c700 1 --2- 172.21.15.61:0/650018708 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4062ae0 0x7f19a4147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a259c700 1 -- 172.21.15.61:0/650018708 >> v1:172.21.15.61:6789/0 conn(0x7f19a4147560 legacy=0x7f19a405ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a259c700 1 -- 172.21.15.61:0/650018708 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4061070 con 0x7f19a4061e50 2022-07-12T08:48:53.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a259c700 1 --2- 172.21.15.61:0/650018708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4061e50 0x7f19a414aff0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f199c0032f0 tx=0x7f199c0091d0).ready entity=mon.0 client_cookie=7db3862e33905e20 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a159a700 1 -- 172.21.15.61:0/650018708 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f199c009ea0 con 0x7f19a4061e50 2022-07-12T08:48:53.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a159a700 1 -- 172.21.15.61:0/650018708 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c00b8a0 con 0x7f19a4061e50 2022-07-12T08:48:53.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a159a700 1 -- 172.21.15.61:0/650018708 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f199c00ba10 con 0x7f19a4061e50 2022-07-12T08:48:53.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- 172.21.15.61:0/650018708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4061e50 msgr2=0x7f19a414aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/650018708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4061e50 0x7f19a414aff0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7f199c0032f0 tx=0x7f199c0091d0).stop 2022-07-12T08:48:53.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- 172.21.15.61:0/650018708 shutdown_connections 2022-07-12T08:48:53.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/650018708 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4062ae0 0x7f19a4147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/650018708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4061e50 0x7f19a414aff0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- 172.21.15.61:0/650018708 >> 172.21.15.61:0/650018708 conn(0x7f19a40b7290 msgr2=0x7f19a40b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:53.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- 172.21.15.61:0/650018708 shutdown_connections 2022-07-12T08:48:53.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.046+0000 7f19a8b5c700 1 -- 172.21.15.61:0/650018708 wait complete. 2022-07-12T08:48:53.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 Processor -- start 2022-07-12T08:48:53.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 -- start start 2022-07-12T08:48:53.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4061e50 0x7f19a40b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 0x7f19a409c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a4147560 0x7f19a40acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4099f40 con 0x7f19a4062ae0 2022-07-12T08:48:53.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40adcb0 con 0x7f19a4147560 2022-07-12T08:48:53.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a8b5c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40adf50 con 0x7f19a4061e50 2022-07-12T08:48:53.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 0x7f19a409c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 0x7f19a409c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59124/0 (socket says 172.21.15.61:59124) 2022-07-12T08:48:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a1d9b700 1 -- 172.21.15.61:0/3822367521 learned_addr learned my addr 172.21.15.61:0/3822367521 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a259c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4061e50 0x7f19a40b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.050+0000 7f19a2d9d700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a4147560 0x7f19a40acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a1d9b700 1 -- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4061e50 msgr2=0x7f19a40b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a1d9b700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4061e50 0x7f19a40b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a1d9b700 1 -- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a4147560 msgr2=0x7f19a40acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a1d9b700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a4147560 0x7f19a40acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a1d9b700 1 -- 172.21.15.61:0/3822367521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c009470 con 0x7f19a4062ae0 2022-07-12T08:48:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a1d9b700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 0x7f19a409c0a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f198c0030b0 tx=0x7f198c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f198c011780 con 0x7f19a4062ae0 2022-07-12T08:48:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c0118e0 con 0x7f19a4062ae0 2022-07-12T08:48:53.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a4099aa0 con 0x7f19a4062ae0 2022-07-12T08:48:53.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f198c011c40 con 0x7f19a4062ae0 2022-07-12T08:48:53.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a40b16e0 con 0x7f19a4062ae0 2022-07-12T08:48:53.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a40ac570 con 0x7f19a4062ae0 2022-07-12T08:48:53.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f198c01c460 con 0x7f19a4062ae0 2022-07-12T08:48:53.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f199b7fe700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1988053a60 0x7f1988055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f198c06bcb0 con 0x7f19a4062ae0 2022-07-12T08:48:53.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a259c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1988053a60 0x7f1988055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.054+0000 7f19a259c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1988053a60 0x7f1988055f20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f199c01b000 tx=0x7f199c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.062+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19a40ac570 con 0x7f19a4062ae0 2022-07-12T08:48:53.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.186+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f19a4099c30 con 0x7f19a4062ae0 2022-07-12T08:48:53.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f199b7fe700 1 -- 172.21.15.61:0/3822367521 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v93) v1 ==== 81+0+28876 (secure 0 0 0) 0x7f198c042e90 con 0x7f19a4062ae0 2022-07-12T08:48:53.194 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:53.195 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:53.195 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 93, 2022-07-12T08:48:53.196 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:48:53.196 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:48:53.197 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:48:50.451132+0000", 2022-07-12T08:48:53.197 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:53.198 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:48:53.198 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:48:53.199 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:48:53.199 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:48:53.199 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:48:53.200 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:48:53.200 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:48:53.200 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:48:53.201 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:53.201 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:48:53.201 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:48:53.201 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:48:53.201 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:48:53.202 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:48:53.202 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 18, 2022-07-12T08:48:53.202 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:48:53.202 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:48:53.203 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:48:53.203 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:48:53.203 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:48:53.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.204 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:48:53.204 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:48:53.204 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:48:53.204 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:53.204 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:53.205 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:53.205 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:53.205 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:53.205 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:53.206 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:53.206 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:53.206 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:48:53.207 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:48:53.207 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:48:53.207 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:48:53.207 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:48:53.208 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:53.208 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:53.208 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:53.208 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:53.208 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:53.209 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:53.209 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:53.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.209 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:48:53.210 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:53.210 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:53.211 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:53.211 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:53.211 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:53.211 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:53.211 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:53.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:53.212 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:53.212 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:53.212 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:53.213 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:53.213 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:53.213 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:53.213 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:53.214 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:53.214 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:53.214 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:53.214 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:53.215 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:53.215 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:53.215 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:53.215 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:53.215 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:53.216 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:53.216 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:53.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.217 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:53.217 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:53.218 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:53.218 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:53.218 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:53.219 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:53.219 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:53.220 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:53.220 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:53.220 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:53.221 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:53.221 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:48:53.222 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:48:53.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.223 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:53.223 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:48:53.224 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.225 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.225 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:48:53.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:48:53.226 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:48:53.227 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:48:53.227 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:48:53.227 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:48:53.228 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:48:53.228 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:48:53.230 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:48:53.231 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:48:53.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:48:53.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:48:53.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:48:53.232 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:48:53.232 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:48:53.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:48:53.233 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:48:53.234 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:48:53.235 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:48:53.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:48:53.236 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:48:53.236 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:48:53.236 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:48:53.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:48:53.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:48:53.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:48:53.238 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:48:53.238 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:48:53.239 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:48:53.240 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:48:53.241 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:48:53.241 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:48:53.242 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:48:53.242 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:48:53.243 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:48:53.243 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:48:53.243 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:48:53.244 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:48:53.244 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:48:53.244 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:48:53.244 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:48:53.245 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:48:53.245 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:48:53.245 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:48:53.246 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:48:53.246 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:48:53.246 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:48:53.246 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:48:53.247 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:48:53.247 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:48:53.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.247 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:48:53.248 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:48:53.248 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:48:53.248 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:48:53.248 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:48:53.248 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:48:53.249 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:48:53.249 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:48:53.249 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:48:53.250 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:48:53.250 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:48:53.251 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:48:53.251 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:48:53.251 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:48:53.252 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.252 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.252 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:53.252 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:48:53.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.253 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:53.253 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:48:53.254 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.254 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.255 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.256 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.257 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.257 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.258 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.258 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:53.259 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.259 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.260 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.261 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:48:53.262 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.262 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:48:53.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.265 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.265 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.266 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.268 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:48:53.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.269 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.269 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.270 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:48:53.270 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.271 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.271 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.271 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.272 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.272 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.273 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.273 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:48:53.274 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.275 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.276 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:48:53.276 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.277 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.277 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.278 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.278 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.279 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:48:53.280 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:48:53.282 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:48:53.283 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.283 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.284 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:48:53.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:48:53.284 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:48:53.285 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:48:53.285 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.286 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.286 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.286 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.287 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.288 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:53.288 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:48:53.289 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.289 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.289 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.290 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.291 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.291 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:53.292 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.292 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.292 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.292 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.293 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.293 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.293 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:48:53.293 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.294 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.294 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.294 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:48:53.295 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.295 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.295 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.295 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.296 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.296 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.296 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:48:53.297 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.297 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.298 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:48:53.298 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.298 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.299 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.299 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.299 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.300 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:48:53.300 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.301 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.301 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.301 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.302 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:48:53.302 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.302 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.303 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.304 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.304 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:48:53.305 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.306 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:48:53.306 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:48:53.306 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.307 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.307 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:48:53.307 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:48:53.308 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:48:53.308 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:48:53.308 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.308 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.309 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.309 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.310 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:53.310 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:48:53.310 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.311 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.311 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.311 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.312 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.312 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.312 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.312 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:53.313 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.314 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.314 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.315 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.315 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:48:53.315 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.316 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.317 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:48:53.317 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.318 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.318 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.319 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.319 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.320 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.320 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:48:53.321 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.322 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.323 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.323 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:48:53.324 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.324 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.325 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.326 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.326 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.327 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.327 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:48:53.328 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.330 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.330 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:48:53.331 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.331 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.332 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.333 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.333 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.334 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:48:53.335 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.335 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.336 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.336 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:48:53.337 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:48:53.337 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.338 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.338 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.339 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:48:53.339 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:48:53.340 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:48:53.340 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:48:53.341 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.341 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.342 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.342 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.343 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:53.344 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:48:53.344 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.345 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.345 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.346 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.346 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.346 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.347 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.347 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:53.348 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.348 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.349 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.349 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.350 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.350 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.351 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:48:53.351 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.351 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.352 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.352 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.353 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:48:53.353 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.353 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.354 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.354 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.355 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.355 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.356 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:48:53.357 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:48:53.358 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.358 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.358 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.359 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.359 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.359 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.360 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.360 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:48:53.360 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.360 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.361 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:48:53.361 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.361 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.362 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.362 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.362 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.363 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.363 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.363 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:48:53.363 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.364 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.364 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.364 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:48:53.364 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:48:53.365 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.365 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.365 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:48:53.366 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:48:53.366 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:48:53.366 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:48:53.366 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.366 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.367 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.367 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.367 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.367 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.368 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:53.368 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:48:53.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.368 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.368 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.369 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.369 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.369 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.369 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.370 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-12T08:48:53.370 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.370 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.370 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.371 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.371 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:48:53.371 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.372 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:48:53.373 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.373 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.373 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.373 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.374 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:48:53.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.375 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.375 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.375 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.376 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:48:53.376 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.376 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.376 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.376 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.377 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.377 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.378 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:48:53.378 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.378 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.378 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.379 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.379 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:48:53.379 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.379 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.380 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.380 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.380 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.381 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.381 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:48:53.381 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.382 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.382 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.382 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:48:53.382 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:48:53.383 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.383 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.383 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.383 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:48:53.384 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:48:53.384 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:48:53.384 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:48:53.384 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.385 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.385 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.385 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.385 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.386 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.386 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:53.386 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:48:53.386 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.386 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.387 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.387 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.387 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.387 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.388 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.388 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-12T08:48:53.388 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.388 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.389 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.389 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.389 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.390 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:48:53.390 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.390 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.390 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.390 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:48:53.391 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.391 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.391 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.392 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.392 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.392 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.392 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:48:53.393 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.393 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.394 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.394 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:48:53.394 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.394 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.395 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.395 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.395 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.395 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.395 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.396 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.396 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:48:53.396 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.396 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.397 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:48:53.397 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.398 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.398 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.398 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.398 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:48:53.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:48:53.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:48:53.401 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.401 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:48:53.402 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:48:53.402 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:48:53.402 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:48:53.402 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.403 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.403 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.403 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.404 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.404 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.404 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:53.404 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:48:53.405 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.405 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.405 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.405 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.405 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.406 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.406 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.406 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-12T08:48:53.406 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.407 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.407 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.407 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.407 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.408 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.408 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:48:53.408 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.408 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.409 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.409 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:48:53.409 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.409 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.410 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.410 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.410 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.410 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.411 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.411 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:48:53.411 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.412 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.412 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:48:53.412 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.413 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.413 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.413 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.414 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:48:53.414 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.415 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.415 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.415 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:48:53.416 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.416 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.416 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.417 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:48:53.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.418 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.418 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:48:53.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:48:53.419 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.419 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.419 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.420 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:48:53.420 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:48:53.420 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:48:53.420 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:48:53.421 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.421 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.421 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.421 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.422 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.422 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:53.422 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:48:53.422 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:48:53.423 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:48:53.423 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:48:53.423 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:48:53.423 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:48:53.424 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:48:53.424 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:48:53.424 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-12T08:48:53.424 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:48:53.424 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:48:53.425 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:48:53.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.425 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.426 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:48:53.426 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.426 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.426 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:48:53.427 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.427 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.427 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.428 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:48:53.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.428 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.429 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.429 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:48:53.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.429 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.429 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.430 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:48:53.430 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.431 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.431 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.431 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.431 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:48:53.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.432 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:48:53.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.433 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.433 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:48:53.434 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.434 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.434 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.434 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.435 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:48:53.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:48:53.435 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:48:53.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:48:53.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.436 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.436 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:48:53.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:48:53.437 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:48:53.437 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.437 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.438 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.438 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:48:53.438 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:48:53.438 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:48:53.438 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:48:53.439 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:48:53.439 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:48:53.439 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:48:53.439 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:53.440 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.440 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:53.440 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:48:53.440 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.441 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:48:53.441 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.441 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.441 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.441 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.442 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.442 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:48:53.442 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.442 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.443 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.443 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:48:53.443 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.443 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.444 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.444 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.444 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.444 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:48:53.444 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.445 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.445 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.445 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:48:53.445 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.446 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.446 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.446 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.446 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.446 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:48:53.447 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.447 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.447 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:48:53.448 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.448 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.448 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.448 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.448 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.449 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:48:53.449 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.449 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.449 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.450 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:48:53.450 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.450 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.450 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.451 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.451 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.451 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:48:53.452 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.453 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:48:53.453 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.453 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.453 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.453 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.454 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.454 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:48:53.454 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.454 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.455 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:48:53.455 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.455 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.456 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.456 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.456 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.456 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:48:53.456 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.457 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.457 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:53.457 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:48:53.457 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:48:53.458 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:48:53.458 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:48:53.458 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:48:53.458 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:48:53.459 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:48:53.459 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:48:53.459 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.459 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:48:53.459 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:48:53.460 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:48:53.460 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:48:53.460 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:48:53.460 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:48:53.461 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.111:6835/167461784": "2022-07-13T08:48:46.533786+0000", 2022-07-12T08:48:53.461 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.111:6834/167461784": "2022-07-13T08:48:46.533786+0000" 2022-07-12T08:48:53.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.461 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:48:53.461 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:48:53.462 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:48:53.462 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:48:53.462 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:48:53.462 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:48:53.463 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:48:53.463 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:48:53.463 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:53.463 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:53.464 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:48:53.464 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:48:53.464 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:48:53.464 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:48:53.465 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:48:53.465 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:53.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1988053a60 msgr2=0x7f1988055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1988053a60 0x7f1988055f20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f199c01b000 tx=0x7f199c016000).stop 2022-07-12T08:48:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 msgr2=0x7f19a409c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 0x7f19a409c0a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f198c0030b0 tx=0x7f198c008040).stop 2022-07-12T08:48:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 shutdown_connections 2022-07-12T08:48:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1988053a60 0x7f1988055f20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f19a4147560 0x7f19a40acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19a4062ae0 0x7f19a409c0a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 --2- 172.21.15.61:0/3822367521 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f19a4061e50 0x7f19a40b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 >> 172.21.15.61:0/3822367521 conn(0x7f19a40b7290 msgr2=0x7f19a405ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 shutdown_connections 2022-07-12T08:48:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.190+0000 7f19a8b5c700 1 -- 172.21.15.61:0/3822367521 wait complete. 2022-07-12T08:48:53.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:53.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 Processor -- start 2022-07-12T08:48:53.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 -- start start 2022-07-12T08:48:53.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f09ec0af280 0x7f09ec060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0aecb0 con 0x7f09ec09bce0 2022-07-12T08:48:53.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0afd70 con 0x7f09ec0af280 2022-07-12T08:48:53.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1958700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec143b00 con 0x7f09ec0b3f60 2022-07-12T08:48:53.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f0956700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f09ec0af280 0x7f09ec060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f0956700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f09ec0af280 0x7f09ec060680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59132/0 (socket says 172.21.15.61:59132) 2022-07-12T08:48:53.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f0956700 1 -- 172.21.15.61:0/4261663060 learned_addr learned my addr 172.21.15.61:0/4261663060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:53.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.574+0000 7f09f1157700 1 --2- 172.21.15.61:0/4261663060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09eb7fe700 1 -- 172.21.15.61:0/4261663060 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222085372 0 0) 0x7f09ec0aecb0 con 0x7f09ec09bce0 2022-07-12T08:48:53.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f0956700 1 -- 172.21.15.61:0/4261663060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 msgr2=0x7f09ec0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:53.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f0956700 1 --2- 172.21.15.61:0/4261663060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec0b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f0956700 1 -- 172.21.15.61:0/4261663060 >> v1:172.21.15.111:6789/0 conn(0x7f09ec09bce0 legacy=0x7f09ec0af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f0956700 1 -- 172.21.15.61:0/4261663060 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec1435c0 con 0x7f09ec0af280 2022-07-12T08:48:53.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f0956700 1 --2- 172.21.15.61:0/4261663060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec060680 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f09e0002f70 tx=0x7f09e0006e50).ready entity=mon.0 client_cookie=ff8075be97669e6e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09eb7fe700 1 -- 172.21.15.61:0/4261663060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09e000d5c0 con 0x7f09ec0af280 2022-07-12T08:48:53.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09eb7fe700 1 -- 172.21.15.61:0/4261663060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e0000e90 con 0x7f09ec0af280 2022-07-12T08:48:53.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09eb7fe700 1 -- 172.21.15.61:0/4261663060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09e000d910 con 0x7f09ec0af280 2022-07-12T08:48:53.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 -- 172.21.15.61:0/4261663060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 msgr2=0x7f09ec060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 --2- 172.21.15.61:0/4261663060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec060680 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f09e0002f70 tx=0x7f09e0006e50).stop 2022-07-12T08:48:53.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 -- 172.21.15.61:0/4261663060 shutdown_connections 2022-07-12T08:48:53.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 --2- 172.21.15.61:0/4261663060 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec0b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 --2- 172.21.15.61:0/4261663060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec060680 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 -- 172.21.15.61:0/4261663060 >> 172.21.15.61:0/4261663060 conn(0x7f09ec0b92c0 msgr2=0x7f09ec0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:53.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 -- 172.21.15.61:0/4261663060 shutdown_connections 2022-07-12T08:48:53.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 -- 172.21.15.61:0/4261663060 wait complete. 2022-07-12T08:48:53.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 Processor -- start 2022-07-12T08:48:53.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.578+0000 7f09f1958700 1 -- start start 2022-07-12T08:48:53.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec09bce0 0x7f09ec09e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0b36b0 con 0x7f09ec0af280 2022-07-12T08:48:53.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec151e00 con 0x7f09ec09bce0 2022-07-12T08:48:53.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec146f70 con 0x7f09ec0b3f60 2022-07-12T08:48:53.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec09bce0 0x7f09ec09e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1157700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec146590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59140/0 (socket says 172.21.15.61:59140) 2022-07-12T08:48:53.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09ebfff700 1 -- 172.21.15.61:0/3158319998 learned_addr learned my addr 172.21.15.61:0/3158319998 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:53.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 -- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 msgr2=0x7f09ec146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 -- 172.21.15.61:0/3158319998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 msgr2=0x7f09ec146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 -- 172.21.15.61:0/3158319998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e000b1e0 con 0x7f09ec09bce0 2022-07-12T08:48:53.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1157700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:53.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09ebfff700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:53.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f0956700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec09bce0 0x7f09ec09e110 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f09e0002f70 tx=0x7f09e0014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09e0013da0 con 0x7f09ec09bce0 2022-07-12T08:48:53.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e001e400 con 0x7f09ec09bce0 2022-07-12T08:48:53.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ec1423d0 con 0x7f09ec09bce0 2022-07-12T08:48:53.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09e001e760 con 0x7f09ec09bce0 2022-07-12T08:48:53.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.582+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec142050 con 0x7f09ec09bce0 2022-07-12T08:48:53.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.586+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec142210 con 0x7f09ec09bce0 2022-07-12T08:48:53.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.586+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f09e0025030 con 0x7f09ec09bce0 2022-07-12T08:48:53.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.586+0000 7f09e9ffb700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f09cc0539b0 0x7f09cc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.586+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f09e006c680 con 0x7f09ec09bce0 2022-07-12T08:48:53.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.586+0000 7f09ebfff700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f09cc0539b0 0x7f09cc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.586+0000 7f09ebfff700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f09cc0539b0 0x7f09cc055e70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f09dc00d000 tx=0x7f09dc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.594+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f09e00438c0 con 0x7f09ec09bce0 2022-07-12T08:48:53.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09ec151f40 con 0x7f09ec09bce0 2022-07-12T08:48:53.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09e9ffb700 1 -- 172.21.15.61:0/3158319998 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f09e0023090 con 0x7f09ec09bce0 2022-07-12T08:48:53.767 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:53.767 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48},{"gid":7248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2537284472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2537284472},{"type":"v1","addr":"172.21.15.111:6835","nonce":2537284472}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[]} 2022-07-12T08:48:53.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f09cc0539b0 msgr2=0x7f09cc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f09cc0539b0 0x7f09cc055e70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f09dc00d000 tx=0x7f09dc009040).stop 2022-07-12T08:48:53.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec09bce0 msgr2=0x7f09ec09e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec09bce0 0x7f09ec09e110 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f09e0002f70 tx=0x7f09e0014000).stop 2022-07-12T08:48:53.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 shutdown_connections 2022-07-12T08:48:53.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09ec0b3f60 0x7f09ec146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09ec0af280 0x7f09ec146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f09cc0539b0 0x7f09cc055e70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 --2- 172.21.15.61:0/3158319998 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09ec09bce0 0x7f09ec09e110 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 >> 172.21.15.61:0/3158319998 conn(0x7f09ec0b92c0 msgr2=0x7f09ec13f600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:53.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 shutdown_connections 2022-07-12T08:48:53.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.762+0000 7f09f1958700 1 -- 172.21.15.61:0/3158319998 wait complete. 2022-07-12T08:48:53.773 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 52 2022-07-12T08:48:53.781 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:48:53.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4bec9a9700 1 Processor -- start 2022-07-12T08:48:53.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4bec9a9700 1 -- start start 2022-07-12T08:48:53.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4bec9a9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4be80ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4bec9a9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be80ad2f0 con 0x7f4be80b01f0 2022-07-12T08:48:53.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4bec9a9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be805c430 con 0x7f4be814a8f0 2022-07-12T08:48:53.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4bec9a9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80b0a40 con 0x7f4be805c050 2022-07-12T08:48:53.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4be80b01f0 0x7f4be805cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47362/0 (socket says 172.21.15.61:47362) 2022-07-12T08:48:53.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 -- 172.21.15.61:0/4257513498 learned_addr learned my addr 172.21.15.61:0/4257513498 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:53.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be659c700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4be80ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969225003 0 0) 0x7f4be80ad2f0 con 0x7f4be80b01f0 2022-07-12T08:48:53.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc8003260 con 0x7f4be80b01f0 2022-07-12T08:48:53.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182505879 0 0) 0x7f4be805c430 con 0x7f4be814a8f0 2022-07-12T08:48:53.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be80ad2f0 con 0x7f4be814a8f0 2022-07-12T08:48:53.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236428111 0 0) 0x7f4bc8003260 con 0x7f4be80b01f0 2022-07-12T08:48:53.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 msgr2=0x7f4be80ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:53.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4be80ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 >> v1:172.21.15.111:6789/0 conn(0x7f4be814a8f0 legacy=0x7f4be81443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be80b2420 con 0x7f4be80b01f0 2022-07-12T08:48:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be659c700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4be80ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f4be00019e0 con 0x7f4be80b01f0 2022-07-12T08:48:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 >> v1:172.21.15.61:6789/0 conn(0x7f4be80b01f0 legacy=0x7f4be805cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc8003910 0x7f4bc8003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4bc8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4bc8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4bc8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80b0a40 con 0x7f4bc8004500 2022-07-12T08:48:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc800b750 con 0x7f4bc8003910 2022-07-12T08:48:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc800b8c0 con 0x7f4be805c050 2022-07-12T08:48:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be659c700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc8003910 0x7f4bc8003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 -- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 msgr2=0x7f4bc8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be6d9d700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4bc8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4bc8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 -- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc8003910 msgr2=0x7f4bc8003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc8003910 0x7f4bc8003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 -- 172.21.15.61:0/4257513498 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd000b180 con 0x7f4bc8004500 2022-07-12T08:48:53.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be6d9d700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be805c050 0x7f4bc8004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:53.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be659c700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bc8003910 0x7f4bc8003cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4bc8004980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f4be00016f0 tx=0x7f4be0007ac0).ready entity=mon.0 client_cookie=4ce8bc05537e7f94 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.877+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4be000ce10 con 0x7f4bc8004500 2022-07-12T08:48:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be0005990 con 0x7f4bc8004500 2022-07-12T08:48:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4be559a700 1 -- 172.21.15.61:0/4257513498 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4be0005cf0 con 0x7f4bc8004500 2022-07-12T08:48:53.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 -- 172.21.15.61:0/4257513498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 msgr2=0x7f4bc8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/4257513498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4bc8004980 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7f4be00016f0 tx=0x7f4be0007ac0).stop 2022-07-12T08:48:53.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4be6d9d700 1 -- 172.21.15.61:0/4257513498 reap_dead start 2022-07-12T08:48:53.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 -- 172.21.15.61:0/4257513498 shutdown_connections 2022-07-12T08:48:53.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 -- 172.21.15.61:0/4257513498 >> 172.21.15.61:0/4257513498 conn(0x7f4be8062c00 msgr2=0x7f4be8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:53.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 -- 172.21.15.61:0/4257513498 shutdown_connections 2022-07-12T08:48:53.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 -- 172.21.15.61:0/4257513498 wait complete. 2022-07-12T08:48:53.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 Processor -- start 2022-07-12T08:48:53.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.881+0000 7f4bec9a9700 1 -- start start 2022-07-12T08:48:53.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4be8066700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be81456f0 0x7f4be8064ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be8144640 0x7f4be8144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8066080 con 0x7f4bc8004500 2022-07-12T08:48:53.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8066e00 con 0x7f4be81456f0 2022-07-12T08:48:53.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be809f160 con 0x7f4be8144640 2022-07-12T08:48:53.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4be8066700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4be8066700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59154/0 (socket says 172.21.15.61:59154) 2022-07-12T08:48:53.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 -- 172.21.15.61:0/298880314 learned_addr learned my addr 172.21.15.61:0/298880314 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be6d9d700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be8144640 0x7f4be8144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be81456f0 0x7f4be8064ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 -- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be8144640 msgr2=0x7f4be8144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be8144640 0x7f4be8144aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 -- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be81456f0 msgr2=0x7f4be8064ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be81456f0 0x7f4be8064ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 -- 172.21.15.61:0/298880314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be809f400 con 0x7f4bc8004500 2022-07-12T08:48:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be81456f0 0x7f4be8064ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4be659c700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4be8066700 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f4bd000b000 tx=0x7f4bd0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd0010e90 con 0x7f4bc8004500 2022-07-12T08:48:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000e8d0 con 0x7f4bc8004500 2022-07-12T08:48:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be809f630 con 0x7f4bc8004500 2022-07-12T08:48:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd000ec30 con 0x7f4bc8004500 2022-07-12T08:48:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be8065120 con 0x7f4bc8004500 2022-07-12T08:48:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.885+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be8057460 con 0x7f4bc8004500 2022-07-12T08:48:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.889+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4bd0024070 con 0x7f4bc8004500 2022-07-12T08:48:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.889+0000 7f4bdf7fe700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4bcc053a40 0x7f4bcc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.889+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4bd006c590 con 0x7f4bc8004500 2022-07-12T08:48:53.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.889+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4bcc053a40 0x7f4bcc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:53.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.889+0000 7f4be5d9b700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4bcc053a40 0x7f4bcc055f00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f4be0019000 tx=0x7f4be0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:53.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:53.893+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4bd0043730 con 0x7f4bc8004500 2022-07-12T08:48:54.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.061+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be80447a0 con 0x7f4bc8004500 2022-07-12T08:48:54.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bdf7fe700 1 -- 172.21.15.61:0/298880314 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f4bd0046d50 con 0x7f4bc8004500 2022-07-12T08:48:54.071 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:54.071 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":52,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6203,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/425282604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":425282604},{"type":"v1","addr":"172.21.15.61:6833","nonce":425282604}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":48},{"gid":7248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2537284472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2537284472},{"type":"v1","addr":"172.21.15.111:6835","nonce":2537284472}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":52}],"filesystems":[]} 2022-07-12T08:48:54.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4bcc053a40 msgr2=0x7f4bcc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4bcc053a40 0x7f4bcc055f00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7f4be0019000 tx=0x7f4be0015000).stop 2022-07-12T08:48:54.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 msgr2=0x7f4be8066700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4be8066700 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7f4bd000b000 tx=0x7f4bd0008040).stop 2022-07-12T08:48:54.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 shutdown_connections 2022-07-12T08:48:54.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4be8144640 0x7f4be8144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4bcc053a40 0x7f4bcc055f00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be81456f0 0x7f4be8064ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 --2- 172.21.15.61:0/298880314 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc8004500 0x7f4be8066700 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 >> 172.21.15.61:0/298880314 conn(0x7f4be8062c00 msgr2=0x7f4be8145ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:54.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 shutdown_connections 2022-07-12T08:48:54.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.065+0000 7f4bec9a9700 1 -- 172.21.15.61:0/298880314 wait complete. 2022-07-12T08:48:54.086 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 52 2022-07-12T08:48:54.088 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:48:54.089 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:48:54.089 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:48:54.090 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:48:54.091 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:48:54.091 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:48:54.092 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:48:54.092 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:48:54.102 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:48:54.103 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:48:54.106 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:48:54.107 DEBUG:teuthology.parallel:result is None 2022-07-12T08:48:54.109 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:48:54.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:48:54.113 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:48:54.114 DEBUG:teuthology.parallel:result is None 2022-07-12T08:48:54.116 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:48:54.178 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:48:54.188 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:48:54.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805e4de700 1 Processor -- start 2022-07-12T08:48:54.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805e4de700 1 -- start start 2022-07-12T08:48:54.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805e4de700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f805813ca80 0x7f80580b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:54.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805e4de700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80580afb60 con 0x7f80580613a0 2022-07-12T08:48:54.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805e4de700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80581435a0 con 0x7f80580af780 2022-07-12T08:48:54.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805e4de700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8058143840 con 0x7f805813ca80 2022-07-12T08:48:54.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805ccdb700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f80580613a0 0x7f80580b0350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47394/0 (socket says 172.21.15.61:47394) 2022-07-12T08:48:54.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805ccdb700 1 -- 172.21.15.61:0/2391674693 learned_addr learned my addr 172.21.15.61:0/2391674693 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:54.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805dcdd700 1 --2- 172.21.15.61:0/2391674693 >> v2:172.21.15.111:3300/0 conn(0x7f805813ca80 0x7f80580b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:54.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819788068 0 0) 0x7f80580afb60 con 0x7f80580613a0 2022-07-12T08:48:54.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8040003260 con 0x7f80580613a0 2022-07-12T08:48:54.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942548008 0 0) 0x7f80581435a0 con 0x7f80580af780 2022-07-12T08:48:54.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80580afb60 con 0x7f80580af780 2022-07-12T08:48:54.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014555262 0 0) 0x7f8040003260 con 0x7f80580613a0 2022-07-12T08:48:54.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805dcdd700 1 -- 172.21.15.61:0/2391674693 >> v1:172.21.15.111:6789/0 conn(0x7f80580af780 legacy=0x7f805813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805dcdd700 1 -- 172.21.15.61:0/2391674693 >> v1:172.21.15.61:6789/0 conn(0x7f80580613a0 legacy=0x7f80580b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805dcdd700 1 -- 172.21.15.61:0/2391674693 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8058067430 con 0x7f805813ca80 2022-07-12T08:48:54.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f805dcdd700 1 --2- 172.21.15.61:0/2391674693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f805813ca80 0x7f80580b0a60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8048001110 tx=0x7f80500091d0).ready entity=mon.1 client_cookie=803417e03748d5cc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:54.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8048001660 con 0x7f805813ca80 2022-07-12T08:48:54.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805000f510 con 0x7f805813ca80 2022-07-12T08:48:54.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.221+0000 7f8047fff700 1 -- 172.21.15.61:0/2391674693 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805000f680 con 0x7f805813ca80 2022-07-12T08:48:54.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 -- 172.21.15.61:0/2391674693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f805813ca80 msgr2=0x7f80580b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 --2- 172.21.15.61:0/2391674693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f805813ca80 0x7f80580b0a60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f8048001110 tx=0x7f80500091d0).stop 2022-07-12T08:48:54.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 -- 172.21.15.61:0/2391674693 shutdown_connections 2022-07-12T08:48:54.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 --2- 172.21.15.61:0/2391674693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f805813ca80 0x7f80580b0a60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 -- 172.21.15.61:0/2391674693 >> 172.21.15.61:0/2391674693 conn(0x7f80580b5780 msgr2=0x7f80580b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:54.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 -- 172.21.15.61:0/2391674693 shutdown_connections 2022-07-12T08:48:54.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 -- 172.21.15.61:0/2391674693 wait complete. 2022-07-12T08:48:54.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 Processor -- start 2022-07-12T08:48:54.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.225+0000 7f805e4de700 1 -- start start 2022-07-12T08:48:54.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 0x7f80580b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:54.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80580af780 0x7f805813d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:54.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f805813ca80 0x7f80581558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:54.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058066d30 con 0x7f80580613a0 2022-07-12T08:48:54.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058155430 con 0x7f80580af780 2022-07-12T08:48:54.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f805813d6f0 con 0x7f805813ca80 2022-07-12T08:48:54.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 0x7f80580b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:54.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 0x7f80580b3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59180/0 (socket says 172.21.15.61:59180) 2022-07-12T08:48:54.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 -- 172.21.15.61:0/3079317524 learned_addr learned my addr 172.21.15.61:0/3079317524 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:54.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805dcdd700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f805813ca80 0x7f80581558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:54.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805ccdb700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80580af780 0x7f805813d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:54.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 -- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f805813ca80 msgr2=0x7f80581558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f805813ca80 0x7f80581558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 -- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80580af780 msgr2=0x7f805813d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:54.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80580af780 0x7f805813d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:54.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 -- 172.21.15.61:0/3079317524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8050009470 con 0x7f80580613a0 2022-07-12T08:48:54.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805d4dc700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 0x7f80580b3c30 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f8054001110 tx=0x7f8054007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:54.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8054012400 con 0x7f80580613a0 2022-07-12T08:48:54.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054012560 con 0x7f80580613a0 2022-07-12T08:48:54.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80580669b0 con 0x7f80580613a0 2022-07-12T08:48:54.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80540128c0 con 0x7f80580613a0 2022-07-12T08:48:54.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8058140380 con 0x7f80580613a0 2022-07-12T08:48:54.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8058066590 con 0x7f80580613a0 2022-07-12T08:48:54.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.229+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8054019030 con 0x7f80580613a0 2022-07-12T08:48:54.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.233+0000 7f80467fc700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8038053a40 0x7f8038055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:54.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.233+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f805406b010 con 0x7f80580613a0 2022-07-12T08:48:54.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.233+0000 7f805ccdb700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8038053a40 0x7f8038055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:54.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.233+0000 7f805ccdb700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8038053a40 0x7f8038055f00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f804800d000 tx=0x7f8048000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:54.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.237+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8058066590 con 0x7f80580613a0 2022-07-12T08:48:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:54.377+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f8058066590 con 0x7f80580613a0 2022-07-12T08:48:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.209+0000 7f80467fc700 1 -- 172.21.15.61:0/3079317524 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v96) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8058066590 con 0x7f80580613a0 2022-07-12T08:48:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8038053a40 msgr2=0x7f8038055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8038053a40 0x7f8038055f00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f804800d000 tx=0x7f8048000bc0).stop 2022-07-12T08:48:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 msgr2=0x7f80580b3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 0x7f80580b3c30 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 rx=0x7f8054001110 tx=0x7f8054007b60).stop 2022-07-12T08:48:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 shutdown_connections 2022-07-12T08:48:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f805813ca80 0x7f80581558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8038053a40 0x7f8038055f00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80580af780 0x7f805813d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 --2- 172.21.15.61:0/3079317524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80580613a0 0x7f80580b3c30 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 >> 172.21.15.61:0/3079317524 conn(0x7f80580b5780 msgr2=0x7f80580b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 shutdown_connections 2022-07-12T08:48:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.213+0000 7f805e4de700 1 -- 172.21.15.61:0/3079317524 wait complete. 2022-07-12T08:48:55.224 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:48:55.237 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:48:55.238 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:48:55.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 Processor -- start 2022-07-12T08:48:55.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 -- start start 2022-07-12T08:48:55.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f00100b5aa0 0x7f00100b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 0x7f00101474b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001005d3d0 con 0x7f001005bfc0 2022-07-12T08:48:55.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001014b590 con 0x7f00100b5aa0 2022-07-12T08:48:55.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f00178c5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00100b5e80 con 0x7f0010147000 2022-07-12T08:48:55.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0014e60700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f001005bfc0 0x7f001014ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47408/0 (socket says 172.21.15.61:47408) 2022-07-12T08:48:55.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0014e60700 1 -- 172.21.15.61:0/1388495093 learned_addr learned my addr 172.21.15.61:0/1388495093 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:55.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 0x7f00101474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015661700 1 --2- 172.21.15.61:0/1388495093 >> v2:172.21.15.111:3300/0 conn(0x7f00100b5aa0 0x7f00100b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089351842 0 0) 0x7f001005d3d0 con 0x7f001005bfc0 2022-07-12T08:48:55.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff4003260 con 0x7f001005bfc0 2022-07-12T08:48:55.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236039105 0 0) 0x7efff4003260 con 0x7f001005bfc0 2022-07-12T08:48:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 msgr2=0x7f00101474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 0x7f00101474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 >> v2:172.21.15.111:3300/0 conn(0x7f00100b5aa0 msgr2=0x7f00100b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 --2- 172.21.15.61:0/1388495093 >> v2:172.21.15.111:3300/0 conn(0x7f00100b5aa0 0x7f00100b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001014a740 con 0x7f001005bfc0 2022-07-12T08:48:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f000c001660 con 0x7f001005bfc0 2022-07-12T08:48:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 >> v1:172.21.15.61:6789/0 conn(0x7f001005bfc0 legacy=0x7f001014ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 0x7efff4003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4003c80 0x7efff4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efff40049f0 0x7efff4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001014b590 con 0x7efff40049f0 2022-07-12T08:48:55.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00100b5e80 con 0x7efff4003c80 2022-07-12T08:48:55.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff400b750 con 0x7f0010147000 2022-07-12T08:48:55.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015661700 1 --2- 172.21.15.61:0/1388495093 >> v2:172.21.15.111:3300/0 conn(0x7f00100b5aa0 0x7f00100b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:48:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efff40049f0 0x7efff4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0014e60700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4003c80 0x7efff4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015661700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 0x7efff4003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 -- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 msgr2=0x7efff4003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0010147000 0x7efff4003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 -- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4003c80 msgr2=0x7efff4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efff4003c80 0x7efff4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 -- 172.21.15.61:0/1388495093 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff4003260 con 0x7efff40049f0 2022-07-12T08:48:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.337+0000 7f0015e62700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efff40049f0 0x7efff4004e70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7efff40054d0 tx=0x7f00080091d0).ready entity=mon.0 client_cookie=125fd50e59ba5631 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f000800e7e0 con 0x7efff40049f0 2022-07-12T08:48:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000800e940 con 0x7efff40049f0 2022-07-12T08:48:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0007fff700 1 -- 172.21.15.61:0/1388495093 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f000800eca0 con 0x7efff40049f0 2022-07-12T08:48:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- 172.21.15.61:0/1388495093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efff40049f0 msgr2=0x7efff4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 --2- 172.21.15.61:0/1388495093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efff40049f0 0x7efff4004e70 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 rx=0x7efff40054d0 tx=0x7f00080091d0).stop 2022-07-12T08:48:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0015e62700 1 -- 172.21.15.61:0/1388495093 reap_dead start 2022-07-12T08:48:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- 172.21.15.61:0/1388495093 shutdown_connections 2022-07-12T08:48:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- 172.21.15.61:0/1388495093 >> 172.21.15.61:0/1388495093 conn(0x7f0010063520 msgr2=0x7f00100b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- 172.21.15.61:0/1388495093 shutdown_connections 2022-07-12T08:48:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- 172.21.15.61:0/1388495093 wait complete. 2022-07-12T08:48:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 Processor -- start 2022-07-12T08:48:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- start start 2022-07-12T08:48:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0010149900 0x7f0010156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 0x7f001009f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f001009fac0 0x7f0010159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010156c70 con 0x7f00100a0750 2022-07-12T08:48:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0010149ce0 con 0x7f0010149900 2022-07-12T08:48:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f00178c5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001009ff40 con 0x7f001009fac0 2022-07-12T08:48:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 0x7f001009f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 0x7f001009f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59204/0 (socket says 172.21.15.61:59204) 2022-07-12T08:48:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 -- 172.21.15.61:0/766398923 learned_addr learned my addr 172.21.15.61:0/766398923 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0015661700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0010149900 0x7f0010156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0015e62700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f001009fac0 0x7f0010159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 -- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f001009fac0 msgr2=0x7f0010159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f001009fac0 0x7f0010159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 -- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0010149900 msgr2=0x7f0010156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0010149900 0x7f0010156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0014e60700 1 -- 172.21.15.61:0/766398923 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00100a0ff0 con 0x7f00100a0750 2022-07-12T08:48:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.341+0000 7f0015661700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0010149900 0x7f0010156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f0014e60700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 0x7f001009f5d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f000c00d000 tx=0x7f00100ac100).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f000c0116b0 con 0x7f00100a0750 2022-07-12T08:48:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000c011810 con 0x7f00100a0750 2022-07-12T08:48:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00100a1290 con 0x7f00100a0750 2022-07-12T08:48:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f000c011b70 con 0x7f00100a0750 2022-07-12T08:48:55.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0010064500 con 0x7f00100a0750 2022-07-12T08:48:55.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f000c01b030 con 0x7f00100a0750 2022-07-12T08:48:55.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00067fc700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efff8053a60 0x7efff8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0010057460 con 0x7f00100a0750 2022-07-12T08:48:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f000c06b6c0 con 0x7f00100a0750 2022-07-12T08:48:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.345+0000 7f0015661700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efff8053a60 0x7efff8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.349+0000 7f0015661700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efff8053a60 0x7efff8055f20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f000000b000 tx=0x7f0000008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.353+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f000c046380 con 0x7f00100a0750 2022-07-12T08:48:55.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0010062690 con 0x7f00100a0750 2022-07-12T08:48:55.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00067fc700 1 -- 172.21.15.61:0/766398923 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f000c019020 con 0x7f00100a0750 2022-07-12T08:48:55.528 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:55.529 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:48:55.529 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:48:55.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.530 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:48:55.531 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:48:55.531 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.531 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:55.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.533 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.533 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.534 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:48:55.535 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:48:55.535 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:55.536 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.536 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.537 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.537 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.537 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.538 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.538 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:48:55.539 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:48:55.539 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.540 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.540 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.540 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.541 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.541 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.542 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.543 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:48:55.543 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:48:55.543 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.543 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.544 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.544 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.544 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.544 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.545 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:48:55.545 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:48:55.545 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.545 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.546 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.546 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.546 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.546 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.547 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.547 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:48:55.547 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:48:55.547 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.548 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.548 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.549 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.549 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.550 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.550 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:48:55.551 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:48:55.551 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.553 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.554 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.554 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:48:55.555 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:48:55.555 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.556 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.557 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.557 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.558 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:48:55.559 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:48:55.559 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.561 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.561 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.562 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.562 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.562 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:48:55.563 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:48:55.563 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.564 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.564 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.565 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.565 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.565 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.566 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.566 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:48:55.567 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:48:55.567 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.567 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:55.567 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:55.567 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:55.568 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.568 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.569 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:48:55.569 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:48:55.570 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.570 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:55.571 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:55.571 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:55.571 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.572 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.572 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.573 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:48:55.573 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:48:55.573 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.574 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:48:55.574 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:48:55.574 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:48:55.575 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.575 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.575 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.576 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:48:55.576 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:48:55.576 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.577 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:55.577 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:48:55.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:48:55.578 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.578 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.579 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.579 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.580 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:48:55.580 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:48:55.580 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:55.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:48:55.581 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.582 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.582 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:55.583 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:55.583 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:48:55.583 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:48:55.584 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:48:55.584 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:48:55.584 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:48:55.585 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:48:55.585 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.585 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:55.586 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:48:55.586 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:48:55.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efff8053a60 msgr2=0x7efff8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efff8053a60 0x7efff8055f20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f000000b000 tx=0x7f0000008040).stop 2022-07-12T08:48:55.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 msgr2=0x7f001009f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 0x7f001009f5d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 rx=0x7f000c00d000 tx=0x7f00100ac100).stop 2022-07-12T08:48:55.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 shutdown_connections 2022-07-12T08:48:55.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f001009fac0 0x7f0010159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00100a0750 0x7f001009f5d0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efff8053a60 0x7efff8055f20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 --2- 172.21.15.61:0/766398923 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0010149900 0x7f0010156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 >> 172.21.15.61:0/766398923 conn(0x7f0010063520 msgr2=0x7f00100aad20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:55.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 shutdown_connections 2022-07-12T08:48:55.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.521+0000 7f00178c5700 1 -- 172.21.15.61:0/766398923 wait complete. 2022-07-12T08:48:55.593 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:55.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.693+0000 7f8c669dc700 1 Processor -- start 2022-07-12T08:48:55.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.693+0000 7f8c669dc700 1 -- start start 2022-07-12T08:48:55.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c669dc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c600ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c669dc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c600ad2f0 con 0x7f8c600b01f0 2022-07-12T08:48:55.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c669dc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6005c430 con 0x7f8c6014a8f0 2022-07-12T08:48:55.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c669dc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c600b0a40 con 0x7f8c6005c050 2022-07-12T08:48:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8c600b01f0 0x7f8c6005cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47428/0 (socket says 172.21.15.61:47428) 2022-07-12T08:48:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5f7fe700 1 -- 172.21.15.61:0/1336614873 learned_addr learned my addr 172.21.15.61:0/1336614873 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577277083 0 0) 0x7f8c600ad2f0 con 0x7f8c600b01f0 2022-07-12T08:48:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c4c003260 con 0x7f8c600b01f0 2022-07-12T08:48:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c600ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2546586381 0 0) 0x7f8c4c003260 con 0x7f8c600b01f0 2022-07-12T08:48:55.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 msgr2=0x7f8c600ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c600ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 >> v1:172.21.15.111:6789/0 conn(0x7f8c6014a8f0 legacy=0x7f8c601443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600b2420 con 0x7f8c600b01f0 2022-07-12T08:48:55.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f8c50001660 con 0x7f8c600b01f0 2022-07-12T08:48:55.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 >> v1:172.21.15.61:6789/0 conn(0x7f8c600b01f0 legacy=0x7f8c6005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c4c003780 0x7f8c4c003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c004500 0x7f8c4c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c4c004ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c600b0a40 con 0x7f8c4c003780 2022-07-12T08:48:55.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c00b8e0 con 0x7f8c4c004500 2022-07-12T08:48:55.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c00ba50 con 0x7f8c6005c050 2022-07-12T08:48:55.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c4c003780 0x7f8c4c003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5f7fe700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c004500 0x7f8c4c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c64f79700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c4c004ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 -- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 msgr2=0x7f8c4c004ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c4c004ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 -- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c004500 msgr2=0x7f8c4c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c004500 0x7f8c4c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 -- 172.21.15.61:0/1336614873 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c003260 con 0x7f8c4c003780 2022-07-12T08:48:55.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c64f79700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c6005c050 0x7f8c4c004ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:55.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c4c003780 0x7f8c4c003ba0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f8c5400a000 tx=0x7f8c54008040).ready entity=mon.0 client_cookie=a1e7f932019551b8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:55.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c54013670 con 0x7f8c4c003780 2022-07-12T08:48:55.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c540137d0 con 0x7f8c4c003780 2022-07-12T08:48:55.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c5effd700 1 -- 172.21.15.61:0/1336614873 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c54013b30 con 0x7f8c4c003780 2022-07-12T08:48:55.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c669dc700 1 -- 172.21.15.61:0/1336614873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c4c003780 msgr2=0x7f8c4c003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c669dc700 1 --2- 172.21.15.61:0/1336614873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c4c003780 0x7f8c4c003ba0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7f8c5400a000 tx=0x7f8c54008040).stop 2022-07-12T08:48:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.697+0000 7f8c64f79700 1 -- 172.21.15.61:0/1336614873 reap_dead start 2022-07-12T08:48:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- 172.21.15.61:0/1336614873 shutdown_connections 2022-07-12T08:48:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- 172.21.15.61:0/1336614873 >> 172.21.15.61:0/1336614873 conn(0x7f8c60062c00 msgr2=0x7f8c60062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- 172.21.15.61:0/1336614873 shutdown_connections 2022-07-12T08:48:55.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- 172.21.15.61:0/1336614873 wait complete. 2022-07-12T08:48:55.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 Processor -- start 2022-07-12T08:48:55.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- start start 2022-07-12T08:48:55.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c003780 0x7f8c60159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c600af230 0x7f8c60141390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 0x7f8c60141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c600ad780 con 0x7f8c60157360 2022-07-12T08:48:55.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60141d70 con 0x7f8c4c003780 2022-07-12T08:48:55.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c669dc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c60144240 con 0x7f8c600af230 2022-07-12T08:48:55.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 0x7f8c60141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 0x7f8c60141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59226/0 (socket says 172.21.15.61:59226) 2022-07-12T08:48:55.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 -- 172.21.15.61:0/2641848324 learned_addr learned my addr 172.21.15.61:0/2641848324 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:55.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c003780 0x7f8c60159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c5f7fe700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c600af230 0x7f8c60141390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 -- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c600af230 msgr2=0x7f8c60141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c600af230 0x7f8c60141390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 -- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c003780 msgr2=0x7f8c60159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c003780 0x7f8c60159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 -- 172.21.15.61:0/2641848324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c60142160 con 0x7f8c60157360 2022-07-12T08:48:55.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c003780 0x7f8c60159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:55.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c5f7fe700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c600af230 0x7f8c60141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:55.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.701+0000 7f8c64f79700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 0x7f8c60141880 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f8c58001210 tx=0x7f8c58009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:55.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c58002680 con 0x7f8c60157360 2022-07-12T08:48:55.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c580027e0 con 0x7f8c60157360 2022-07-12T08:48:55.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c601444a0 con 0x7f8c60157360 2022-07-12T08:48:55.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c58002b40 con 0x7f8c60157360 2022-07-12T08:48:55.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c60148d40 con 0x7f8c60157360 2022-07-12T08:48:55.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c601449a0 con 0x7f8c60157360 2022-07-12T08:48:55.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8c58002ca0 con 0x7f8c60157360 2022-07-12T08:48:55.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5d7fa700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c40053a40 0x7f8c40055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8c5806bc10 con 0x7f8c60157360 2022-07-12T08:48:55.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.705+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c40053a40 0x7f8c40055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.709+0000 7f8c5ffff700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c40053a40 0x7f8c40055f00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f8c54019000 tx=0x7f8c54008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:55.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.713+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8c601449a0 con 0x7f8c60157360 2022-07-12T08:48:55.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.841+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c601449a0 con 0x7f8c60157360 2022-07-12T08:48:55.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.841+0000 7f8c5d7fa700 1 -- 172.21.15.61:0/2641848324 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+13230 (secure 0 0 0) 0x7f8c580253c0 con 0x7f8c60157360 2022-07-12T08:48:55.851 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:55.852 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":96,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:55.208626+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:55.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c40053a40 msgr2=0x7f8c40055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c40053a40 0x7f8c40055f00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f8c54019000 tx=0x7f8c54008040).stop 2022-07-12T08:48:55.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 msgr2=0x7f8c60141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:55.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 0x7f8c60141880 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f8c58001210 tx=0x7f8c58009040).stop 2022-07-12T08:48:55.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 shutdown_connections 2022-07-12T08:48:55.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8c60157360 0x7f8c60141880 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8c600af230 0x7f8c60141390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8c40053a40 0x7f8c40055f00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 --2- 172.21.15.61:0/2641848324 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8c4c003780 0x7f8c60159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 >> 172.21.15.61:0/2641848324 conn(0x7f8c60062c00 msgr2=0x7f8c601460c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:55.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 shutdown_connections 2022-07-12T08:48:55.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.845+0000 7f8c669dc700 1 -- 172.21.15.61:0/2641848324 wait complete. 2022-07-12T08:48:55.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:48:55.864 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:48:55.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.973+0000 7fc4e95b4700 1 Processor -- start 2022-07-12T08:48:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.973+0000 7fc4e95b4700 1 -- start start 2022-07-12T08:48:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4e40b3ea0 0x7fc4e40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4e405c810 0x7fc4e40b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:55.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e40b3120 con 0x7fc4e40b9db0 2022-07-12T08:48:55.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e405bd10 con 0x7fc4e405c810 2022-07-12T08:48:55.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e40b33c0 con 0x7fc4e40b3ea0 2022-07-12T08:48:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4e405c810 0x7fc4e40b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4e405c810 0x7fc4e40b4c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59234/0 (socket says 172.21.15.61:59234) 2022-07-12T08:48:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 -- 172.21.15.61:0/400489226 learned_addr learned my addr 172.21.15.61:0/400489226 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:55.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e27fc700 1 --2- 172.21.15.61:0/400489226 >> v2:172.21.15.111:3300/0 conn(0x7fc4e40b3ea0 0x7fc4e40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:55.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 -- 172.21.15.61:0/400489226 >> v2:172.21.15.111:3300/0 conn(0x7fc4e40b3ea0 msgr2=0x7fc4e40b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 --2- 172.21.15.61:0/400489226 >> v2:172.21.15.111:3300/0 conn(0x7fc4e40b3ea0 0x7fc4e40b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:55.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 -- 172.21.15.61:0/400489226 >> v1:172.21.15.111:6789/0 conn(0x7fc4e40b9db0 legacy=0x7fc4e405c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:55.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 -- 172.21.15.61:0/400489226 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e405cc60 con 0x7fc4e405c810 2022-07-12T08:48:56.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e27fc700 1 --2- 172.21.15.61:0/400489226 >> v2:172.21.15.111:3300/0 conn(0x7fc4e40b3ea0 0x7fc4e40b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:56.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e37fe700 1 --2- 172.21.15.61:0/400489226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e405c810 0x7fc4e40b4c90 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fc4e40b1230 tx=0x7fc4dc009290).ready entity=mon.0 client_cookie=d06fc0dd70027ed2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e1ffb700 1 -- 172.21.15.61:0/400489226 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4dc00b860 con 0x7fc4e405c810 2022-07-12T08:48:56.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e1ffb700 1 -- 172.21.15.61:0/400489226 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc00b9c0 con 0x7fc4e405c810 2022-07-12T08:48:56.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e1ffb700 1 -- 172.21.15.61:0/400489226 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4dc00bb30 con 0x7fc4e405c810 2022-07-12T08:48:56.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- 172.21.15.61:0/400489226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e405c810 msgr2=0x7fc4e40b4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/400489226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e405c810 0x7fc4e40b4c90 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 rx=0x7fc4e40b1230 tx=0x7fc4dc009290).stop 2022-07-12T08:48:56.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- 172.21.15.61:0/400489226 shutdown_connections 2022-07-12T08:48:56.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/400489226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e405c810 0x7fc4e40b4c90 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/400489226 >> v2:172.21.15.111:3300/0 conn(0x7fc4e40b3ea0 0x7fc4e40b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- 172.21.15.61:0/400489226 >> 172.21.15.61:0/400489226 conn(0x7fc4e4067960 msgr2=0x7fc4e40b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- 172.21.15.61:0/400489226 shutdown_connections 2022-07-12T08:48:56.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- 172.21.15.61:0/400489226 wait complete. 2022-07-12T08:48:56.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 Processor -- start 2022-07-12T08:48:56.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- start start 2022-07-12T08:48:56.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e405c810 0x7fc4e4145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 0x7fc4e40ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4e40b9db0 0x7fc4e41488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e40adb10 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4146470 con 0x7fc4e405c810 2022-07-12T08:48:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.977+0000 7fc4e95b4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e40acce0 con 0x7fc4e40b9db0 2022-07-12T08:48:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 0x7fc4e40ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 0x7fc4e40ad180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59238/0 (socket says 172.21.15.61:59238) 2022-07-12T08:48:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 -- 172.21.15.61:0/292398871 learned_addr learned my addr 172.21.15.61:0/292398871 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e2ffd700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e405c810 0x7fc4e4145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e37fe700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4e40b9db0 0x7fc4e41488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 -- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4e40b9db0 msgr2=0x7fc4e41488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4e40b9db0 0x7fc4e41488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 -- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e405c810 msgr2=0x7fc4e4145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e405c810 0x7fc4e4145c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 -- 172.21.15.61:0/292398871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc009530 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e27fc700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 0x7fc4e40ad180 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fc4d4006940 tx=0x7fc4d4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4d4011780 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d40118e0 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e40afc00 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4d4011c40 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e41473f0 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e40acea0 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc4d4012420 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4d3fff700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4c0053a40 0x7fc4c0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.981+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc4d406b840 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.985+0000 7fc4e2ffd700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4c0053a40 0x7fc4c0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.985+0000 7fc4e2ffd700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4c0053a40 0x7fc4c0055f00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fc4d800d000 tx=0x7fc4d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:55.985+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4d4042a70 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.117+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc4e4057460 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.117+0000 7fc4d3fff700 1 -- 172.21.15.61:0/292398871 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v96) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc4d4046090 con 0x7fc4e40b3ea0 2022-07-12T08:48:56.125 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:48:56.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4c0053a40 msgr2=0x7fc4c0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4c0053a40 0x7fc4c0055f00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7fc4d800d000 tx=0x7fc4d8009040).stop 2022-07-12T08:48:56.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 msgr2=0x7fc4e40ad180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 0x7fc4e40ad180 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7fc4d4006940 tx=0x7fc4d4008040).stop 2022-07-12T08:48:56.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 shutdown_connections 2022-07-12T08:48:56.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4e40b9db0 0x7fc4e41488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4e40b3ea0 0x7fc4e40ad180 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc4c0053a40 0x7fc4c0055f00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 --2- 172.21.15.61:0/292398871 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4e405c810 0x7fc4e4145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 >> 172.21.15.61:0/292398871 conn(0x7fc4e4067960 msgr2=0x7fc4e4143050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 shutdown_connections 2022-07-12T08:48:56.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.121+0000 7fc4e95b4700 1 -- 172.21.15.61:0/292398871 wait complete. 2022-07-12T08:48:56.144 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:48:56.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 Processor -- start 2022-07-12T08:48:56.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- start start 2022-07-12T08:48:56.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05880b0280 0x7f058814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f05880aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058805c3e0 con 0x7f05880b5920 2022-07-12T08:48:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05880b5d00 con 0x7f05880b0280 2022-07-12T08:48:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058805c610 con 0x7f05880aa830 2022-07-12T08:48:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058dba7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05880b0280 0x7f058814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058e3a8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f05880b5920 0x7f058814a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47462/0 (socket says 172.21.15.61:47462) 2022-07-12T08:48:56.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058e3a8700 1 -- 172.21.15.61:0/1546077940 learned_addr learned my addr 172.21.15.61:0/1546077940 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:56.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058eba9700 1 --2- 172.21.15.61:0/1546077940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f05880aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058dba7700 1 -- 172.21.15.61:0/1546077940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 msgr2=0x7f05880aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:56.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058dba7700 1 --2- 172.21.15.61:0/1546077940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f05880aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058d3a6700 1 -- 172.21.15.61:0/1546077940 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427439315 0 0) 0x7f058805c3e0 con 0x7f05880b5920 2022-07-12T08:48:56.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058dba7700 1 -- 172.21.15.61:0/1546077940 >> v1:172.21.15.61:6789/0 conn(0x7f05880b5920 legacy=0x7f058814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058dba7700 1 -- 172.21.15.61:0/1546077940 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058805c8b0 con 0x7f05880b0280 2022-07-12T08:48:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058dba7700 1 --2- 172.21.15.61:0/1546077940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f058814b0f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f0578002f70 tx=0x7f0578006df0).ready entity=mon.0 client_cookie=1fc522ab0fbc379 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058d3a6700 1 -- 172.21.15.61:0/1546077940 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057800d570 con 0x7f05880b0280 2022-07-12T08:48:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058d3a6700 1 -- 172.21.15.61:0/1546077940 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578000e90 con 0x7f05880b0280 2022-07-12T08:48:56.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058d3a6700 1 -- 172.21.15.61:0/1546077940 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057800d8c0 con 0x7f05880b0280 2022-07-12T08:48:56.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- 172.21.15.61:0/1546077940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 msgr2=0x7f058814b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1546077940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f058814b0f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f0578002f70 tx=0x7f0578006df0).stop 2022-07-12T08:48:56.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- 172.21.15.61:0/1546077940 shutdown_connections 2022-07-12T08:48:56.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1546077940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f05880aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1546077940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f058814b0f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- 172.21.15.61:0/1546077940 >> 172.21.15.61:0/1546077940 conn(0x7f0588063340 msgr2=0x7f05880b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- 172.21.15.61:0/1546077940 shutdown_connections 2022-07-12T08:48:56.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.253+0000 7f058f3aa700 1 -- 172.21.15.61:0/1546077940 wait complete. 2022-07-12T08:48:56.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.257+0000 7f058f3aa700 1 Processor -- start 2022-07-12T08:48:56.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.257+0000 7f058f3aa700 1 -- start start 2022-07-12T08:48:56.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.257+0000 7f058f3aa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f0588159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.257+0000 7f058f3aa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f0588063bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.257+0000 7f058f3aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05880b5920 0x7f05880640b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058f3aa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880acce0 con 0x7f05880b0280 2022-07-12T08:48:56.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058f3aa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880ac430 con 0x7f05880b5920 2022-07-12T08:48:56.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058f3aa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058815a490 con 0x7f05880aa830 2022-07-12T08:48:56.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f0588063bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f0588063bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59252/0 (socket says 172.21.15.61:59252) 2022-07-12T08:48:56.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 -- 172.21.15.61:0/1509887166 learned_addr learned my addr 172.21.15.61:0/1509887166 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:56.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058e3a8700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f0588159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058eba9700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05880b5920 0x7f05880640b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 -- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 msgr2=0x7f0588159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f0588159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 -- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05880b5920 msgr2=0x7f05880640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05880b5920 0x7f05880640b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 -- 172.21.15.61:0/1509887166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057800b180 con 0x7f05880b0280 2022-07-12T08:48:56.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058dba7700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f0588063bc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f0578002f70 tx=0x7f0578014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057800dea0 con 0x7f05880b0280 2022-07-12T08:48:56.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0578013de0 con 0x7f05880b0280 2022-07-12T08:48:56.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05880ac900 con 0x7f05880b0280 2022-07-12T08:48:56.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057801e5f0 con 0x7f05880b0280 2022-07-12T08:48:56.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0588144910 con 0x7f05880b0280 2022-07-12T08:48:56.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0588057460 con 0x7f05880b0280 2022-07-12T08:48:56.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0578025030 con 0x7f05880b0280 2022-07-12T08:48:56.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.261+0000 7f05777fe700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0568053a60 0x7f0568055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.265+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f057806c780 con 0x7f05880b0280 2022-07-12T08:48:56.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.265+0000 7f058e3a8700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0568053a60 0x7f0568055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.265+0000 7f058e3a8700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0568053a60 0x7f0568055f20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f058400d000 tx=0x7f0584009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.269+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0578043a10 con 0x7f05880b0280 2022-07-12T08:48:56.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.393+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f05880b0060 con 0x7f05880b0280 2022-07-12T08:48:56.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f05777fe700 1 -- 172.21.15.61:0/1509887166 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v96) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0578023020 con 0x7f05880b0280 2022-07-12T08:48:56.403 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:48:56.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0568053a60 msgr2=0x7f0568055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0568053a60 0x7f0568055f20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f058400d000 tx=0x7f0584009040).stop 2022-07-12T08:48:56.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 msgr2=0x7f0588063bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f0588063bc0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f0578002f70 tx=0x7f0578014000).stop 2022-07-12T08:48:56.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 shutdown_connections 2022-07-12T08:48:56.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0568053a60 0x7f0568055f20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05880b5920 0x7f05880640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05880b0280 0x7f0588063bc0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 --2- 172.21.15.61:0/1509887166 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05880aa830 0x7f0588159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 >> 172.21.15.61:0/1509887166 conn(0x7f0588063340 msgr2=0x7f05880b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 shutdown_connections 2022-07-12T08:48:56.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.397+0000 7f058f3aa700 1 -- 172.21.15.61:0/1509887166 wait complete. 2022-07-12T08:48:56.415 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:48:56.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4977fff700 1 Processor -- start 2022-07-12T08:48:56.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4977fff700 1 -- start start 2022-07-12T08:48:56.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4977fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f49780b3f60 0x7f49780b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4977fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49780aecb0 con 0x7f49780af280 2022-07-12T08:48:56.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4977fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49781435c0 con 0x7f497809bce0 2022-07-12T08:48:56.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4977fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49780afd70 con 0x7f49780b3f60 2022-07-12T08:48:56.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4976ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f49780af280 0x7f4978060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47476/0 (socket says 172.21.15.61:47476) 2022-07-12T08:48:56.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4976ffd700 1 -- 172.21.15.61:0/3548990412 learned_addr learned my addr 172.21.15.61:0/3548990412 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:56.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f49777fe700 1 --2- 172.21.15.61:0/3548990412 >> v2:172.21.15.111:3300/0 conn(0x7f49780b3f60 0x7f49780b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4975ffb700 1 -- 172.21.15.61:0/3548990412 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461584617 0 0) 0x7f49780aecb0 con 0x7f49780af280 2022-07-12T08:48:56.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f4975ffb700 1 -- 172.21.15.61:0/3548990412 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f495c003260 con 0x7f49780af280 2022-07-12T08:48:56.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f49777fe700 1 -- 172.21.15.61:0/3548990412 >> v1:172.21.15.111:6789/0 conn(0x7f497809bce0 legacy=0x7f49780af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f49777fe700 1 -- 172.21.15.61:0/3548990412 >> v1:172.21.15.61:6789/0 conn(0x7f49780af280 legacy=0x7f4978060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.505+0000 7f49777fe700 1 -- 172.21.15.61:0/3548990412 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978143860 con 0x7f49780b3f60 2022-07-12T08:48:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f49777fe700 1 --2- 172.21.15.61:0/3548990412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b4810 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f49700032f0 tx=0x7f49700091d0).ready entity=mon.1 client_cookie=532a3bd9a6d6f546 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4975ffb700 1 -- 172.21.15.61:0/3548990412 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f497000b6c0 con 0x7f49780b3f60 2022-07-12T08:48:56.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4975ffb700 1 -- 172.21.15.61:0/3548990412 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497000b820 con 0x7f49780b3f60 2022-07-12T08:48:56.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4975ffb700 1 -- 172.21.15.61:0/3548990412 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f497000b990 con 0x7f49780b3f60 2022-07-12T08:48:56.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 -- 172.21.15.61:0/3548990412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 msgr2=0x7f49780b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 --2- 172.21.15.61:0/3548990412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b4810 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f49700032f0 tx=0x7f49700091d0).stop 2022-07-12T08:48:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 -- 172.21.15.61:0/3548990412 shutdown_connections 2022-07-12T08:48:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 --2- 172.21.15.61:0/3548990412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b4810 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 -- 172.21.15.61:0/3548990412 >> 172.21.15.61:0/3548990412 conn(0x7f49780b92c0 msgr2=0x7f49780b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 -- 172.21.15.61:0/3548990412 shutdown_connections 2022-07-12T08:48:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 -- 172.21.15.61:0/3548990412 wait complete. 2022-07-12T08:48:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 Processor -- start 2022-07-12T08:48:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.509+0000 7f4977fff700 1 -- start start 2022-07-12T08:48:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f497809bce0 0x7f497809ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 0x7f4978155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49780b14a0 con 0x7f49780af280 2022-07-12T08:48:56.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978148900 con 0x7f49780b3f60 2022-07-12T08:48:56.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49780b0880 con 0x7f497809bce0 2022-07-12T08:48:56.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 0x7f4978155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 0x7f4978155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59266/0 (socket says 172.21.15.61:59266) 2022-07-12T08:48:56.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 -- 172.21.15.61:0/2609478414 learned_addr learned my addr 172.21.15.61:0/2609478414 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:56.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4976ffd700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f497809bce0 0x7f497809ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49777fe700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 -- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f497809bce0 msgr2=0x7f497809ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f497809bce0 0x7f497809ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 -- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 msgr2=0x7f49780b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 -- 172.21.15.61:0/2609478414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970009470 con 0x7f49780af280 2022-07-12T08:48:56.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f49767fc700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 0x7f4978155df0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f49780663d0 tx=0x7f4968007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4968012400 con 0x7f49780af280 2022-07-12T08:48:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4968012560 con 0x7f49780af280 2022-07-12T08:48:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49780b0b20 con 0x7f49780af280 2022-07-12T08:48:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49680128c0 con 0x7f49780af280 2022-07-12T08:48:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49780b1a80 con 0x7f49780af280 2022-07-12T08:48:56.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.513+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4978156310 con 0x7f49780af280 2022-07-12T08:48:56.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.517+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4968019030 con 0x7f49780af280 2022-07-12T08:48:56.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.517+0000 7f4967fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4954053a40 0x7f4954055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.517+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f496806ae60 con 0x7f49780af280 2022-07-12T08:48:56.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.517+0000 7f4976ffd700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4954053a40 0x7f4954055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.517+0000 7f4976ffd700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4954053a40 0x7f4954055f00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f496c00d000 tx=0x7f496c007a40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.521+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f49680420f0 con 0x7f49780af280 2022-07-12T08:48:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.649+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f49780b0cb0 con 0x7f49780af280 2022-07-12T08:48:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.649+0000 7f4967fff700 1 -- 172.21.15.61:0/2609478414 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v96) v1 ==== 84+0+143 (secure 0 0 0) 0x7f4968045710 con 0x7f49780af280 2022-07-12T08:48:56.657 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:56.658 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:48:56.658 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:56.658 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:48:56.659 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:48:56.659 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:48:56.659 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:48:56.659 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:48:56.660 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:48:56.660 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:48:56.660 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:48:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4954053a40 msgr2=0x7f4954055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4954053a40 0x7f4954055f00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f496c00d000 tx=0x7f496c007a40).stop 2022-07-12T08:48:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 msgr2=0x7f4978155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 0x7f4978155df0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 rx=0x7f49780663d0 tx=0x7f4968007c20).stop 2022-07-12T08:48:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 shutdown_connections 2022-07-12T08:48:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4954053a40 0x7f4954055f00 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49780b3f60 0x7f49780b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49780af280 0x7f4978155df0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 --2- 172.21.15.61:0/2609478414 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f497809bce0 0x7f497809ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 >> 172.21.15.61:0/2609478414 conn(0x7f49780b92c0 msgr2=0x7f49780b80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 shutdown_connections 2022-07-12T08:48:56.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.653+0000 7f4977fff700 1 -- 172.21.15.61:0/2609478414 wait complete. 2022-07-12T08:48:56.670 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:48:56.671 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:48:56.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.773+0000 7f6aea0f8700 1 Processor -- start 2022-07-12T08:48:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.773+0000 7f6aea0f8700 1 -- start start 2022-07-12T08:48:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae40b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ae409b9d0 0x7f6ae413f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae413fe80 con 0x7f6ae40b0e10 2022-07-12T08:48:56.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae40ac490 con 0x7f6ae409b9d0 2022-07-12T08:48:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae413f250 con 0x7f6ae40b21c0 2022-07-12T08:48:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ae409b9d0 0x7f6ae413f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ae409b9d0 0x7f6ae413f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59274/0 (socket says 172.21.15.61:59274) 2022-07-12T08:48:56.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 -- 172.21.15.61:0/866071913 learned_addr learned my addr 172.21.15.61:0/866071913 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:56.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae90f6700 1 --2- 172.21.15.61:0/866071913 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae40b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 -- 172.21.15.61:0/866071913 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 msgr2=0x7f6ae40b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:56.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 --2- 172.21.15.61:0/866071913 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae40b25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 -- 172.21.15.61:0/866071913 >> v1:172.21.15.111:6789/0 conn(0x7f6ae40b0e10 legacy=0x7f6ae40603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 -- 172.21.15.61:0/866071913 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae413f4f0 con 0x7f6ae409b9d0 2022-07-12T08:48:56.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ae88f5700 1 --2- 172.21.15.61:0/866071913 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae413f990 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f6ad4002f70 tx=0x7f6ad4006f40).ready entity=mon.0 client_cookie=c1debc9604e69cb6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ad3fff700 1 -- 172.21.15.61:0/866071913 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ad400da80 con 0x7f6ae409b9d0 2022-07-12T08:48:56.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ad3fff700 1 -- 172.21.15.61:0/866071913 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad4000e90 con 0x7f6ae409b9d0 2022-07-12T08:48:56.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6ad3fff700 1 -- 172.21.15.61:0/866071913 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ad400de10 con 0x7f6ae409b9d0 2022-07-12T08:48:56.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- 172.21.15.61:0/866071913 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 msgr2=0x7f6ae413f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/866071913 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae413f990 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f6ad4002f70 tx=0x7f6ad4006f40).stop 2022-07-12T08:48:56.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- 172.21.15.61:0/866071913 shutdown_connections 2022-07-12T08:48:56.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/866071913 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae413f990 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/866071913 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae40b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- 172.21.15.61:0/866071913 >> 172.21.15.61:0/866071913 conn(0x7f6ae40b57c0 msgr2=0x7f6ae40b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:56.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- 172.21.15.61:0/866071913 shutdown_connections 2022-07-12T08:48:56.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.777+0000 7f6aea0f8700 1 -- 172.21.15.61:0/866071913 wait complete. 2022-07-12T08:48:56.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 Processor -- start 2022-07-12T08:48:56.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 -- start start 2022-07-12T08:48:56.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae4142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae40b0e10 0x7f6ae4140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae4150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae4064190 con 0x7f6ae409b9d0 2022-07-12T08:48:56.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae408f700 con 0x7f6ae40b0e10 2022-07-12T08:48:56.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae4064380 con 0x7f6ae40b21c0 2022-07-12T08:48:56.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae4142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae4142ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59278/0 (socket says 172.21.15.61:59278) 2022-07-12T08:48:56.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 -- 172.21.15.61:0/1287669472 learned_addr learned my addr 172.21.15.61:0/1287669472 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:56.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae98f7700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae4150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae88f5700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae40b0e10 0x7f6ae4140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 -- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 msgr2=0x7f6ae4150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae4150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 -- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae40b0e10 msgr2=0x7f6ae4140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:56.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae40b0e10 0x7f6ae4140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:56.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 -- 172.21.15.61:0/1287669472 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad400b460 con 0x7f6ae409b9d0 2022-07-12T08:48:56.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae88f5700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae40b0e10 0x7f6ae4140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:48:56.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae98f7700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae4150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:48:56.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ae90f6700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae4142ed0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7f6ae000c000 tx=0x7f6ae0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae0010780 con 0x7f6ae409b9d0 2022-07-12T08:48:56.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae00108e0 con 0x7f6ae409b9d0 2022-07-12T08:48:56.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae4064620 con 0x7f6ae409b9d0 2022-07-12T08:48:56.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae0010c40 con 0x7f6ae409b9d0 2022-07-12T08:48:56.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.781+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae4065150 con 0x7f6ae409b9d0 2022-07-12T08:48:56.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.785+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6ae001c460 con 0x7f6ae409b9d0 2022-07-12T08:48:56.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.785+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae4057460 con 0x7f6ae409b9d0 2022-07-12T08:48:56.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.785+0000 7f6ad27fc700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ac4053920 0x7f6ac4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:56.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.785+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f6ae006c0d0 con 0x7f6ae409b9d0 2022-07-12T08:48:56.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.785+0000 7f6ae88f5700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ac4053920 0x7f6ac4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:56.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.785+0000 7f6ae88f5700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ac4053920 0x7f6ac4055de0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f6ad4019000 tx=0x7f6ae40b6350).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:56.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.793+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ae0043260 con 0x7f6ae409b9d0 2022-07-12T08:48:56.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:56.917+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f6ae4068470 con 0x7f6ae409b9d0 2022-07-12T08:48:57.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.785+0000 7f6ad27fc700 1 -- 172.21.15.61:0/1287669472 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v97) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6ae0046880 con 0x7f6ae409b9d0 2022-07-12T08:48:57.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ac4053920 msgr2=0x7f6ac4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:57.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ac4053920 0x7f6ac4055de0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f6ad4019000 tx=0x7f6ae40b6350).stop 2022-07-12T08:48:57.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 msgr2=0x7f6ae4142ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:57.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae4142ed0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 rx=0x7f6ae000c000 tx=0x7f6ae0006df0).stop 2022-07-12T08:48:57.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 shutdown_connections 2022-07-12T08:48:57.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6ae40b21c0 0x7f6ae4150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ac4053920 0x7f6ac4055de0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6ae40b0e10 0x7f6ae4140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 --2- 172.21.15.61:0/1287669472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ae409b9d0 0x7f6ae4142ed0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 >> 172.21.15.61:0/1287669472 conn(0x7f6ae40b57c0 msgr2=0x7f6ae40b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 shutdown_connections 2022-07-12T08:48:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.789+0000 7f6aea0f8700 1 -- 172.21.15.61:0/1287669472 wait complete. 2022-07-12T08:48:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:48:57.808 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:48:57.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 Processor -- start 2022-07-12T08:48:57.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 -- start start 2022-07-12T08:48:57.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8380b21c0 0x7fc8380b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:57.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc83809b9d0 0x7fc83813f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:57.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83813fe80 con 0x7fc8380b0e10 2022-07-12T08:48:57.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8380ac490 con 0x7fc8380b21c0 2022-07-12T08:48:57.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83d48b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83805c420 con 0x7fc83809b9d0 2022-07-12T08:48:57.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8380b21c0 0x7fc8380b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:57.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83cc8a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc8380b0e10 0x7fc83805bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47508/0 (socket says 172.21.15.61:47508) 2022-07-12T08:48:57.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc83cc8a700 1 -- 172.21.15.61:0/2990381552 learned_addr learned my addr 172.21.15.61:0/2990381552 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:57.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8380b21c0 0x7fc8380b25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59286/0 (socket says 172.21.15.61:59286) 2022-07-12T08:48:57.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc82f7fe700 1 --2- 172.21.15.61:0/2990381552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc83809b9d0 0x7fc83813f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:57.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 -- 172.21.15.61:0/2990381552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc83809b9d0 msgr2=0x7fc83813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:57.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 --2- 172.21.15.61:0/2990381552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc83809b9d0 0x7fc83813f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc8377fe700 1 -- 172.21.15.61:0/2990381552 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786705283 0 0) 0x7fc83813fe80 con 0x7fc8380b0e10 2022-07-12T08:48:57.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 -- 172.21.15.61:0/2990381552 >> v1:172.21.15.61:6789/0 conn(0x7fc8380b0e10 legacy=0x7fc83805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:57.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 -- 172.21.15.61:0/2990381552 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83813f250 con 0x7fc8380b21c0 2022-07-12T08:48:57.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc837fff700 1 --2- 172.21.15.61:0/2990381552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b21c0 0x7fc8380b25a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7fc83805c560 tx=0x7fc828006df0).ready entity=mon.0 client_cookie=dd6a812500e2b74d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:57.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc8377fe700 1 -- 172.21.15.61:0/2990381552 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82800d570 con 0x7fc8380b21c0 2022-07-12T08:48:57.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc8377fe700 1 -- 172.21.15.61:0/2990381552 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc828000e90 con 0x7fc8380b21c0 2022-07-12T08:48:57.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.917+0000 7fc8377fe700 1 -- 172.21.15.61:0/2990381552 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82800d8c0 con 0x7fc8380b21c0 2022-07-12T08:48:57.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- 172.21.15.61:0/2990381552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b21c0 msgr2=0x7fc8380b25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:57.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2990381552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b21c0 0x7fc8380b25a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7fc83805c560 tx=0x7fc828006df0).stop 2022-07-12T08:48:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- 172.21.15.61:0/2990381552 shutdown_connections 2022-07-12T08:48:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2990381552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc83809b9d0 0x7fc83813f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2990381552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b21c0 0x7fc8380b25a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- 172.21.15.61:0/2990381552 >> 172.21.15.61:0/2990381552 conn(0x7fc8380b57c0 msgr2=0x7fc8380b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:57.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- 172.21.15.61:0/2990381552 shutdown_connections 2022-07-12T08:48:57.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- 172.21.15.61:0/2990381552 wait complete. 2022-07-12T08:48:57.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 Processor -- start 2022-07-12T08:48:57.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- start start 2022-07-12T08:48:57.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc83809b9d0 0x7fc8380aad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:57.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 0x7fc838152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:57.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8380b21c0 0x7fc8381532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:57.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838142c20 con 0x7fc8380b0e10 2022-07-12T08:48:57.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838142ec0 con 0x7fc83809b9d0 2022-07-12T08:48:57.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83d48b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc838140b30 con 0x7fc8380b21c0 2022-07-12T08:48:57.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc82f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 0x7fc838152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:57.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc82f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 0x7fc838152de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59294/0 (socket says 172.21.15.61:59294) 2022-07-12T08:48:57.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc82f7fe700 1 -- 172.21.15.61:0/2233814491 learned_addr learned my addr 172.21.15.61:0/2233814491 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:57.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.921+0000 7fc83cc8a700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8380b21c0 0x7fc8381532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:57.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc837fff700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc83809b9d0 0x7fc8380aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:57.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc82f7fe700 1 -- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8380b21c0 msgr2=0x7fc8381532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:57.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc82f7fe700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8380b21c0 0x7fc8381532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc82f7fe700 1 -- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc83809b9d0 msgr2=0x7fc8380aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:57.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc82f7fe700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc83809b9d0 0x7fc8380aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:57.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc82f7fe700 1 -- 172.21.15.61:0/2233814491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82800b180 con 0x7fc8380b0e10 2022-07-12T08:48:57.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc82f7fe700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 0x7fc838152de0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7fc82000c000 tx=0x7fc820006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:57.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc820010780 con 0x7fc8380b0e10 2022-07-12T08:48:57.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8200108e0 con 0x7fc8380b0e10 2022-07-12T08:48:57.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8381427f0 con 0x7fc8380b0e10 2022-07-12T08:48:57.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc820010c40 con 0x7fc8380b0e10 2022-07-12T08:48:57.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8381406e0 con 0x7fc8380b0e10 2022-07-12T08:48:57.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.925+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838143080 con 0x7fc8380b0e10 2022-07-12T08:48:57.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.929+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc82001c460 con 0x7fc8380b0e10 2022-07-12T08:48:57.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.929+0000 7fc835ffb700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc81c0539b0 0x7fc81c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:57.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.929+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fc82006c070 con 0x7fc8380b0e10 2022-07-12T08:48:57.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.929+0000 7fc837fff700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc81c0539b0 0x7fc81c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:57.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.929+0000 7fc837fff700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc81c0539b0 0x7fc81c055e70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fc82801a000 tx=0x7fc828014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:57.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:57.937+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc820043140 con 0x7fc8380b0e10 2022-07-12T08:48:58.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.069+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fc838057460 con 0x7fc8380b0e10 2022-07-12T08:48:58.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.789+0000 7fc835ffb700 1 -- 172.21.15.61:0/2233814491 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v98) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc820046760 con 0x7fc8380b0e10 2022-07-12T08:48:58.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc81c0539b0 msgr2=0x7fc81c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:58.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc81c0539b0 0x7fc81c055e70 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7fc82801a000 tx=0x7fc828014000).stop 2022-07-12T08:48:58.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 msgr2=0x7fc838152de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:58.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 0x7fc838152de0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7fc82000c000 tx=0x7fc820006df0).stop 2022-07-12T08:48:58.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 shutdown_connections 2022-07-12T08:48:58.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc8380b21c0 0x7fc8381532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380b0e10 0x7fc838152de0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc81c0539b0 0x7fc81c055e70 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 --2- 172.21.15.61:0/2233814491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc83809b9d0 0x7fc8380aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 >> 172.21.15.61:0/2233814491 conn(0x7fc8380b57c0 msgr2=0x7fc8380b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:58.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 shutdown_connections 2022-07-12T08:48:58.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.801+0000 7fc83d48b700 1 -- 172.21.15.61:0/2233814491 wait complete. 2022-07-12T08:48:58.809 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:48:58.819 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:48:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.921+0000 7f72abaf1700 1 Processor -- start 2022-07-12T08:48:58.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.921+0000 7f72abaf1700 1 -- start start 2022-07-12T08:48:58.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f72a4066fc0 0x7f72a40b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:58.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 0x7f72a405c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:58.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40b2f20 con 0x7f72a40b45d0 2022-07-12T08:48:58.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a40b3170 con 0x7f72a4066fc0 2022-07-12T08:48:58.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40b4170 con 0x7f72a40b2940 2022-07-12T08:48:58.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 0x7f72a405c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:58.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 0x7f72a405c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:41500/0 (socket says 172.21.15.61:41500) 2022-07-12T08:48:58.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 -- 172.21.15.61:0/54842750 learned_addr learned my addr 172.21.15.61:0/54842750 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:58.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72a988d700 1 --2- 172.21.15.61:0/54842750 >> v2:172.21.15.111:3300/0 conn(0x7f72a4066fc0 0x7f72a40b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:58.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 -- 172.21.15.61:0/54842750 >> v2:172.21.15.111:3300/0 conn(0x7f72a4066fc0 msgr2=0x7f72a40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:58.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 --2- 172.21.15.61:0/54842750 >> v2:172.21.15.111:3300/0 conn(0x7f72a4066fc0 0x7f72a40b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 -- 172.21.15.61:0/54842750 >> v1:172.21.15.111:6789/0 conn(0x7f72a40b45d0 legacy=0x7f72a40b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:58.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 -- 172.21.15.61:0/54842750 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a40673a0 con 0x7f72a40b2940 2022-07-12T08:48:58.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72aa08e700 1 --2- 172.21.15.61:0/54842750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 0x7f72a405c090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f729c0032f0 tx=0x7f729c009290).ready entity=mon.2 client_cookie=7656ff0c39aae0bf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:58.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72a888b700 1 -- 172.21.15.61:0/54842750 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f729c00b8a0 con 0x7f72a40b2940 2022-07-12T08:48:58.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72a888b700 1 -- 172.21.15.61:0/54842750 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c00ba00 con 0x7f72a40b2940 2022-07-12T08:48:58.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72a888b700 1 -- 172.21.15.61:0/54842750 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f729c00bb70 con 0x7f72a40b2940 2022-07-12T08:48:58.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- 172.21.15.61:0/54842750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 msgr2=0x7f72a405c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:58.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 --2- 172.21.15.61:0/54842750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 0x7f72a405c090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f729c0032f0 tx=0x7f729c009290).stop 2022-07-12T08:48:58.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- 172.21.15.61:0/54842750 shutdown_connections 2022-07-12T08:48:58.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 --2- 172.21.15.61:0/54842750 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b2940 0x7f72a405c090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 --2- 172.21.15.61:0/54842750 >> v2:172.21.15.111:3300/0 conn(0x7f72a4066fc0 0x7f72a40b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- 172.21.15.61:0/54842750 >> 172.21.15.61:0/54842750 conn(0x7f72a40b94d0 msgr2=0x7f72a40b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:58.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- 172.21.15.61:0/54842750 shutdown_connections 2022-07-12T08:48:58.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.925+0000 7f72abaf1700 1 -- 172.21.15.61:0/54842750 wait complete. 2022-07-12T08:48:58.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 Processor -- start 2022-07-12T08:48:58.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 -- start start 2022-07-12T08:48:58.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a4066fc0 0x7f72a4112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:58.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72a40b2940 0x7f72a4141bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:58.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 0x7f72a41420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:58.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40ae770 con 0x7f72a40b2940 2022-07-12T08:48:58.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a4115390 con 0x7f72a4066fc0 2022-07-12T08:48:58.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40af100 con 0x7f72a40b45d0 2022-07-12T08:48:58.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 0x7f72a41420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:58.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72a908c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72a40b2940 0x7f72a4141bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:58.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 0x7f72a41420a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:41506/0 (socket says 172.21.15.61:41506) 2022-07-12T08:48:58.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 -- 172.21.15.61:0/2063608250 learned_addr learned my addr 172.21.15.61:0/2063608250 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:58.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72a988d700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a4066fc0 0x7f72a4112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:58.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 -- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a4066fc0 msgr2=0x7f72a4112a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:58.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a4066fc0 0x7f72a4112a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 -- 172.21.15.61:0/2063608250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72a40b2940 msgr2=0x7f72a4141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:58.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72a40b2940 0x7f72a4141bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:58.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 -- 172.21.15.61:0/2063608250 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c009530 con 0x7f72a40b45d0 2022-07-12T08:48:58.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72aa08e700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 0x7f72a41420a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f729c0032f0 tx=0x7f729c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:58.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f729c00b8a0 con 0x7f72a40b45d0 2022-07-12T08:48:58.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c021480 con 0x7f72a40b45d0 2022-07-12T08:48:58.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a4113250 con 0x7f72a40b45d0 2022-07-12T08:48:58.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f729c0217e0 con 0x7f72a40b45d0 2022-07-12T08:48:58.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.929+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a4142590 con 0x7f72a40b45d0 2022-07-12T08:48:58.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.933+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a4057460 con 0x7f72a40b45d0 2022-07-12T08:48:58.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.933+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f729c00f070 con 0x7f72a40b45d0 2022-07-12T08:48:58.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.933+0000 7f7296ffd700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7284053af0 0x7f7284055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:58.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.933+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f729c06d320 con 0x7f72a40b45d0 2022-07-12T08:48:58.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.933+0000 7f72a988d700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7284053af0 0x7f7284055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:58.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.933+0000 7f72a988d700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7284053af0 0x7f7284055fb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f729800b000 tx=0x7f7298006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:58.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:58.941+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f729c044320 con 0x7f72a40b45d0 2022-07-12T08:48:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.105+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f72a40b7bc0 con 0x7f72a40b45d0 2022-07-12T08:48:59.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.117+0000 7f7296ffd700 1 -- 172.21.15.61:0/2063608250 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 19 and data pool 20 v54) v1 ==== 189+0+0 (secure 0 0 0) 0x7f729c015070 con 0x7f72a40b45d0 2022-07-12T08:48:59.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7284053af0 msgr2=0x7f7284055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7284053af0 0x7f7284055fb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f729800b000 tx=0x7f7298006df0).stop 2022-07-12T08:48:59.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 msgr2=0x7f72a41420a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 0x7f72a41420a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f729c0032f0 tx=0x7f729c016000).stop 2022-07-12T08:48:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 shutdown_connections 2022-07-12T08:48:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f72a40b45d0 0x7f72a41420a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72a40b2940 0x7f72a4141bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7284053af0 0x7f7284055fb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 --2- 172.21.15.61:0/2063608250 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f72a4066fc0 0x7f72a4112a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 >> 172.21.15.61:0/2063608250 conn(0x7f72a40b94d0 msgr2=0x7f72a40b8860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:59.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 shutdown_connections 2022-07-12T08:48:59.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.125+0000 7f72abaf1700 1 -- 172.21.15.61:0/2063608250 wait complete. 2022-07-12T08:48:59.135 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 19 and data pool 20 2022-07-12T08:48:59.143 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:59.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd44877700 1 Processor -- start 2022-07-12T08:48:59.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd44877700 1 -- start start 2022-07-12T08:48:59.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd44877700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 0x7fdd400b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd44877700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd40140010 con 0x7fdd400b25c0 2022-07-12T08:48:59.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd44877700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd400668f0 con 0x7fdd401436b0 2022-07-12T08:48:59.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd44877700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40142f70 con 0x7fdd400b6780 2022-07-12T08:48:59.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdd400b25c0 0x7fdd4013f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47546/0 (socket says 172.21.15.61:47546) 2022-07-12T08:48:59.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3effd700 1 -- 172.21.15.61:0/1834428713 learned_addr learned my addr 172.21.15.61:0/1834428713 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:48:59.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3f7fe700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 0x7fdd400b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688412434 0 0) 0x7fdd40140010 con 0x7fdd400b25c0 2022-07-12T08:48:59.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd2c003260 con 0x7fdd400b25c0 2022-07-12T08:48:59.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543758839 0 0) 0x7fdd2c003260 con 0x7fdd400b25c0 2022-07-12T08:48:59.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 msgr2=0x7fdd400b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:59.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 0x7fdd400b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 >> v1:172.21.15.111:6789/0 conn(0x7fdd401436b0 legacy=0x7fdd40065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40143210 con 0x7fdd400b25c0 2022-07-12T08:48:59.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fdd34001660 con 0x7fdd400b25c0 2022-07-12T08:48:59.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 >> v1:172.21.15.61:6789/0 conn(0x7fdd400b25c0 legacy=0x7fdd4013f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.233+0000 7fdd3e7fc700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 0x7fdd2c003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd2c003cb0 0x7fdd2c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0049f0 0x7fdd2c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd40142f70 con 0x7fdd2c003cb0 2022-07-12T08:48:59.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c00b800 con 0x7fdd2c0049f0 2022-07-12T08:48:59.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c00b9c0 con 0x7fdd400b6780 2022-07-12T08:48:59.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd2c003cb0 0x7fdd2c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3f7fe700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 0x7fdd2c003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3ffff700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0049f0 0x7fdd2c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 -- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 msgr2=0x7fdd2c003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:59.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd400b6780 0x7fdd2c003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 -- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0049f0 msgr2=0x7fdd2c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:59.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c0049f0 0x7fdd2c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 -- 172.21.15.61:0/1834428713 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c003260 con 0x7fdd2c003cb0 2022-07-12T08:48:59.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3effd700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd2c003cb0 0x7fdd2c004500 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fdd34001370 tx=0x7fdd40063680).ready entity=mon.0 client_cookie=95e167632b12328c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd340070d0 con 0x7fdd2c003cb0 2022-07-12T08:48:59.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34007230 con 0x7fdd2c003cb0 2022-07-12T08:48:59.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3e7fc700 1 -- 172.21.15.61:0/1834428713 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd3400c8f0 con 0x7fdd2c003cb0 2022-07-12T08:48:59.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 -- 172.21.15.61:0/1834428713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd2c003cb0 msgr2=0x7fdd2c004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 --2- 172.21.15.61:0/1834428713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd2c003cb0 0x7fdd2c004500 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fdd34001370 tx=0x7fdd40063680).stop 2022-07-12T08:48:59.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd3ffff700 1 -- 172.21.15.61:0/1834428713 reap_dead start 2022-07-12T08:48:59.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 -- 172.21.15.61:0/1834428713 shutdown_connections 2022-07-12T08:48:59.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 -- 172.21.15.61:0/1834428713 >> 172.21.15.61:0/1834428713 conn(0x7fdd40064570 msgr2=0x7fdd400b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:59.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 -- 172.21.15.61:0/1834428713 shutdown_connections 2022-07-12T08:48:59.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 -- 172.21.15.61:0/1834428713 wait complete. 2022-07-12T08:48:59.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 Processor -- start 2022-07-12T08:48:59.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.237+0000 7fdd44877700 1 -- start start 2022-07-12T08:48:59.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c003cb0 0x7fdd40144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 0x7fdd4012bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd401298e0 0x7fdd400af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4012b350 con 0x7fdd401397e0 2022-07-12T08:48:59.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4012b5b0 con 0x7fdd2c003cb0 2022-07-12T08:48:59.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4012b850 con 0x7fdd401298e0 2022-07-12T08:48:59.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 0x7fdd4012bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c003cb0 0x7fdd40144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 0x7fdd4012bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59340/0 (socket says 172.21.15.61:59340) 2022-07-12T08:48:59.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 -- 172.21.15.61:0/850629675 learned_addr learned my addr 172.21.15.61:0/850629675 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:59.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3ffff700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd401298e0 0x7fdd400af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 -- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd401298e0 msgr2=0x7fdd400af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd401298e0 0x7fdd400af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 -- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c003cb0 msgr2=0x7fdd40144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c003cb0 0x7fdd40144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 -- 172.21.15.61:0/850629675 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd40129d60 con 0x7fdd401397e0 2022-07-12T08:48:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3effd700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 0x7fdd4012bcf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7fdd34001370 tx=0x7fdd34015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd340070d0 con 0x7fdd401397e0 2022-07-12T08:48:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34004710 con 0x7fdd401397e0 2022-07-12T08:48:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd34004a70 con 0x7fdd401397e0 2022-07-12T08:48:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd400af9c0 con 0x7fdd401397e0 2022-07-12T08:48:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd400afec0 con 0x7fdd401397e0 2022-07-12T08:48:59.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.241+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4012ba10 con 0x7fdd401397e0 2022-07-12T08:48:59.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.245+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdd34010070 con 0x7fdd401397e0 2022-07-12T08:48:59.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.245+0000 7fdd3cff9700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdd20053940 0x7fdd20055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.245+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdd3406cc10 con 0x7fdd401397e0 2022-07-12T08:48:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.245+0000 7fdd3f7fe700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdd20053940 0x7fdd20055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.245+0000 7fdd3f7fe700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdd20053940 0x7fdd20055e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fdd3000b000 tx=0x7fdd30006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.249+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd34043a60 con 0x7fdd401397e0 2022-07-12T08:48:59.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.377+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd40057460 con 0x7fdd401397e0 2022-07-12T08:48:59.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.377+0000 7fdd3cff9700 1 -- 172.21.15.61:0/850629675 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16102 (secure 0 0 0) 0x7fdd34025020 con 0x7fdd401397e0 2022-07-12T08:48:59.384 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:59.384 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":99,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:59.110972+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:56.975401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:58.124077+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:59.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdd20053940 msgr2=0x7fdd20055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdd20053940 0x7fdd20055e00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fdd3000b000 tx=0x7fdd30006df0).stop 2022-07-12T08:48:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 msgr2=0x7fdd4012bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 0x7fdd4012bcf0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7fdd34001370 tx=0x7fdd34015000).stop 2022-07-12T08:48:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 shutdown_connections 2022-07-12T08:48:59.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd401298e0 0x7fdd400af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd401397e0 0x7fdd4012bcf0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdd20053940 0x7fdd20055e00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 --2- 172.21.15.61:0/850629675 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd2c003cb0 0x7fdd40144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 >> 172.21.15.61:0/850629675 conn(0x7fdd40064570 msgr2=0x7fdd400b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:59.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 shutdown_connections 2022-07-12T08:48:59.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.381+0000 7fdd44877700 1 -- 172.21.15.61:0/850629675 wait complete. 2022-07-12T08:48:59.399 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:48:59.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 Processor -- start 2022-07-12T08:48:59.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 -- start start 2022-07-12T08:48:59.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcbcc0b6ac0 0x7fcbcc146220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 0x7fcbcc060430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbcc0b6ea0 con 0x7fcbcc1420b0 2022-07-12T08:48:59.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbcc146710 con 0x7fcbcc0b6ac0 2022-07-12T08:48:59.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.497+0000 7fcbd0d4c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc060200 con 0x7fcbcc05f550 2022-07-12T08:48:59.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbcb7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcbcc0b6ac0 0x7fcbcc146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 0x7fcbcc060430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 0x7fcbcc060430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:41542/0 (socket says 172.21.15.61:41542) 2022-07-12T08:48:59.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbcbfff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7fcbcc1420b0 0x7fcbcc142540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:32936/0 (socket says 172.21.15.61:32936) 2022-07-12T08:48:59.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbcb7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcbcc0b6ac0 0x7fcbcc146220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:37922/0 (socket says 172.21.15.61:37922) 2022-07-12T08:48:59.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 -- 172.21.15.61:0/2531284768 learned_addr learned my addr 172.21.15.61:0/2531284768 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:59.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 -- 172.21.15.61:0/2531284768 >> v2:172.21.15.111:3300/0 conn(0x7fcbcc0b6ac0 msgr2=0x7fcbcc146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:59.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 --2- 172.21.15.61:0/2531284768 >> v2:172.21.15.111:3300/0 conn(0x7fcbcc0b6ac0 0x7fcbcc146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 -- 172.21.15.61:0/2531284768 >> v1:172.21.15.111:6789/0 conn(0x7fcbcc1420b0 legacy=0x7fcbcc142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 -- 172.21.15.61:0/2531284768 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc0b1360 con 0x7fcbcc05f550 2022-07-12T08:48:59.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbc3fff700 1 --2- 172.21.15.61:0/2531284768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 0x7fcbcc060430 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fcbb4002f70 tx=0x7fcbb4006eb0).ready entity=mon.2 client_cookie=a50c392df6cdc7c9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbcaffd700 1 -- 172.21.15.61:0/2531284768 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb400d570 con 0x7fcbcc05f550 2022-07-12T08:48:59.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbcaffd700 1 -- 172.21.15.61:0/2531284768 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4000e90 con 0x7fcbcc05f550 2022-07-12T08:48:59.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbcaffd700 1 -- 172.21.15.61:0/2531284768 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbb400d900 con 0x7fcbcc05f550 2022-07-12T08:48:59.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/2531284768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 msgr2=0x7fcbcc060430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/2531284768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 0x7fcbcc060430 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fcbb4002f70 tx=0x7fcbb4006eb0).stop 2022-07-12T08:48:59.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/2531284768 shutdown_connections 2022-07-12T08:48:59.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/2531284768 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc05f550 0x7fcbcc060430 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/2531284768 >> v2:172.21.15.111:3300/0 conn(0x7fcbcc0b6ac0 0x7fcbcc146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/2531284768 >> 172.21.15.61:0/2531284768 conn(0x7fcbcc064570 msgr2=0x7fcbcc0b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:59.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/2531284768 shutdown_connections 2022-07-12T08:48:59.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/2531284768 wait complete. 2022-07-12T08:48:59.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 Processor -- start 2022-07-12T08:48:59.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.501+0000 7fcbd0d4c700 1 -- start start 2022-07-12T08:48:59.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 0x7fcbcc0ad9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbcc0b6ac0 0x7fcbcc1453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc1420b0 0x7fcbcc0af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc13f2a0 con 0x7fcbcc05f550 2022-07-12T08:48:59.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc13f000 con 0x7fcbcc0b6ac0 2022-07-12T08:48:59.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbcc0ad530 con 0x7fcbcc1420b0 2022-07-12T08:48:59.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 0x7fcbcc0ad9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 0x7fcbcc0ad9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59352/0 (socket says 172.21.15.61:59352) 2022-07-12T08:48:59.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 -- 172.21.15.61:0/3767245239 learned_addr learned my addr 172.21.15.61:0/3767245239 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:59.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcbfff700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc1420b0 0x7fcbcc0af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbc3fff700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbcc0b6ac0 0x7fcbcc1453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 -- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc1420b0 msgr2=0x7fcbcc0af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc1420b0 0x7fcbcc0af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 -- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbcc0b6ac0 msgr2=0x7fcbcc1453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbcc0b6ac0 0x7fcbcc1453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 -- 172.21.15.61:0/3767245239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb400b1e0 con 0x7fcbcc05f550 2022-07-12T08:48:59.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbcb7fe700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 0x7fcbcc0ad9d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7fcbbc00c000 tx=0x7fcbbc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbbc010780 con 0x7fcbcc05f550 2022-07-12T08:48:59.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbbc0108e0 con 0x7fcbcc05f550 2022-07-12T08:48:59.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbcc13fd80 con 0x7fcbcc05f550 2022-07-12T08:48:59.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbbc010c40 con 0x7fcbcc05f550 2022-07-12T08:48:59.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbcc0b0170 con 0x7fcbcc05f550 2022-07-12T08:48:59.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.505+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbcc0ad6f0 con 0x7fcbcc05f550 2022-07-12T08:48:59.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.509+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcbbc01c460 con 0x7fcbcc05f550 2022-07-12T08:48:59.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.509+0000 7fcbc97fa700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcbb0053ad0 0x7fcbb0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.509+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcbbc06c280 con 0x7fcbcc05f550 2022-07-12T08:48:59.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.509+0000 7fcbc3fff700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcbb0053ad0 0x7fcbb0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.509+0000 7fcbc3fff700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcbb0053ad0 0x7fcbb0055f90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fcbb401c000 tx=0x7fcbb4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.517+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbbc043180 con 0x7fcbcc05f550 2022-07-12T08:48:59.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.641+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbcc057460 con 0x7fcbcc05f550 2022-07-12T08:48:59.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbc97fa700 1 -- 172.21.15.61:0/3767245239 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16102 (secure 0 0 0) 0x7fcbbc0467a0 con 0x7fcbcc05f550 2022-07-12T08:48:59.651 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:48:59.652 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":99,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:48:59.110972+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:56.975401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:58.124077+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:48:59.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcbb0053ad0 msgr2=0x7fcbb0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcbb0053ad0 0x7fcbb0055f90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fcbb401c000 tx=0x7fcbb4015000).stop 2022-07-12T08:48:59.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 msgr2=0x7fcbcc0ad9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 0x7fcbcc0ad9d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7fcbbc00c000 tx=0x7fcbbc006df0).stop 2022-07-12T08:48:59.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 shutdown_connections 2022-07-12T08:48:59.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcbcc1420b0 0x7fcbcc0af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcbb0053ad0 0x7fcbb0055f90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbcc0b6ac0 0x7fcbcc1453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 --2- 172.21.15.61:0/3767245239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbcc05f550 0x7fcbcc0ad9d0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.645+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 >> 172.21.15.61:0/3767245239 conn(0x7fcbcc064570 msgr2=0x7fcbcc0b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:59.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.649+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 shutdown_connections 2022-07-12T08:48:59.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.649+0000 7fcbd0d4c700 1 -- 172.21.15.61:0/3767245239 wait complete. 2022-07-12T08:48:59.665 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:48:59.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.765+0000 7f1eb675c700 1 Processor -- start 2022-07-12T08:48:59.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.765+0000 7f1eb675c700 1 -- start start 2022-07-12T08:48:59.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b6ae0 0x7f1eb01437d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1eb00b1e40 0x7f1eb00b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb00b6ec0 con 0x7f1eb005f3f0 2022-07-12T08:48:59.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb00b2990 con 0x7f1eb00b1e40 2022-07-12T08:48:59.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb00b0bb0 con 0x7f1eb00b6ae0 2022-07-12T08:48:59.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1eb00b1e40 0x7f1eb00b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1eb00b1e40 0x7f1eb00b12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59362/0 (socket says 172.21.15.61:59362) 2022-07-12T08:48:59.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 -- 172.21.15.61:0/2638159873 learned_addr learned my addr 172.21.15.61:0/2638159873 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:59.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb575a700 1 --2- 172.21.15.61:0/2638159873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b6ae0 0x7f1eb01437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 -- 172.21.15.61:0/2638159873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b6ae0 msgr2=0x7f1eb01437d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:48:59.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 --2- 172.21.15.61:0/2638159873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b6ae0 0x7f1eb01437d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 -- 172.21.15.61:0/2638159873 >> v1:172.21.15.111:6789/0 conn(0x7f1eb005f3f0 legacy=0x7f1eb005f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 -- 172.21.15.61:0/2638159873 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb00b0e50 con 0x7f1eb00b1e40 2022-07-12T08:48:59.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1ea7fff700 1 -- 172.21.15.61:0/2638159873 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224614688 0 0) 0x7f1eb00b6ec0 con 0x7f1eb005f3f0 2022-07-12T08:48:59.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb4f59700 1 --2- 172.21.15.61:0/2638159873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b1e40 0x7f1eb00b12f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f1ea0002f70 tx=0x7f1ea0006eb0).ready entity=mon.0 client_cookie=5a7b16d32444370c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1ea7fff700 1 -- 172.21.15.61:0/2638159873 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea000d570 con 0x7f1eb00b1e40 2022-07-12T08:48:59.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1ea7fff700 1 -- 172.21.15.61:0/2638159873 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea0000e90 con 0x7f1eb00b1e40 2022-07-12T08:48:59.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1ea7fff700 1 -- 172.21.15.61:0/2638159873 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ea000d8c0 con 0x7f1eb00b1e40 2022-07-12T08:48:59.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- 172.21.15.61:0/2638159873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b1e40 msgr2=0x7f1eb00b12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 --2- 172.21.15.61:0/2638159873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b1e40 0x7f1eb00b12f0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 rx=0x7f1ea0002f70 tx=0x7f1ea0006eb0).stop 2022-07-12T08:48:59.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- 172.21.15.61:0/2638159873 shutdown_connections 2022-07-12T08:48:59.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 --2- 172.21.15.61:0/2638159873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b1e40 0x7f1eb00b12f0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 --2- 172.21.15.61:0/2638159873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb00b6ae0 0x7f1eb01437d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- 172.21.15.61:0/2638159873 >> 172.21.15.61:0/2638159873 conn(0x7f1eb0064590 msgr2=0x7f1eb00b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:48:59.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- 172.21.15.61:0/2638159873 shutdown_connections 2022-07-12T08:48:59.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.769+0000 7f1eb675c700 1 -- 172.21.15.61:0/2638159873 wait complete. 2022-07-12T08:48:59.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 Processor -- start 2022-07-12T08:48:59.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 -- start start 2022-07-12T08:48:59.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb005f3f0 0x7f1eb00ad420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b6ae0 0x7f1eb005c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 0x7f1eb0150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0111a80 con 0x7f1eb00b6ae0 2022-07-12T08:48:59.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb01501e0 con 0x7f1eb005ca90 2022-07-12T08:48:59.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb675c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb0150480 con 0x7f1eb005f3f0 2022-07-12T08:48:59.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb4f59700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b6ae0 0x7f1eb005c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 0x7f1eb0150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb575a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb005f3f0 0x7f1eb00ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 0x7f1eb0150920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:37942/0 (socket says 172.21.15.61:37942) 2022-07-12T08:48:59.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 -- 172.21.15.61:0/1134976033 learned_addr learned my addr 172.21.15.61:0/1134976033 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:48:59.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb4f59700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b6ae0 0x7f1eb005c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59370/0 (socket says 172.21.15.61:59370) 2022-07-12T08:48:59.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 -- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb005f3f0 msgr2=0x7f1eb00ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb005f3f0 0x7f1eb00ad420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 -- 172.21.15.61:0/1134976033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b6ae0 msgr2=0x7f1eb005c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:48:59.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b6ae0 0x7f1eb005c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:48:59.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 -- 172.21.15.61:0/1134976033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea000b1e0 con 0x7f1eb005ca90 2022-07-12T08:48:59.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1eb5f5b700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 0x7f1eb0150920 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f1eac000fe0 tx=0x7f1eac007ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.773+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eac012400 con 0x7f1eb005ca90 2022-07-12T08:48:59.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac012560 con 0x7f1eb005ca90 2022-07-12T08:48:59.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb0111700 con 0x7f1eb005ca90 2022-07-12T08:48:59.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eac0128c0 con 0x7f1eb005ca90 2022-07-12T08:48:59.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb00ad910 con 0x7f1eb005ca90 2022-07-12T08:48:59.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb01118f0 con 0x7f1eb005ca90 2022-07-12T08:48:59.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1eac019030 con 0x7f1eb005ca90 2022-07-12T08:48:59.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1ea67fc700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e98053920 0x7f1e98055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:48:59.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1eac06b300 con 0x7f1eb005ca90 2022-07-12T08:48:59.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.777+0000 7f1eb575a700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e98053920 0x7f1e98055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:48:59.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.781+0000 7f1eb575a700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e98053920 0x7f1e98055de0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f1ea800b000 tx=0x7f1ea80091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:48:59.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.785+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1eac0420f0 con 0x7f1eb005ca90 2022-07-12T08:48:59.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:48:59.953+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1eb0150640 con 0x7f1eb005ca90 2022-07-12T08:49:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.813+0000 7f1ea67fc700 1 -- 172.21.15.61:0/1134976033 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1eac045710 con 0x7f1eb005ca90 2022-07-12T08:49:00.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e98053920 msgr2=0x7f1e98055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e98053920 0x7f1e98055de0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f1ea800b000 tx=0x7f1ea80091d0).stop 2022-07-12T08:49:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 msgr2=0x7f1eb0150920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 0x7f1eb0150920 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f1eac000fe0 tx=0x7f1eac007ad0).stop 2022-07-12T08:49:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 shutdown_connections 2022-07-12T08:49:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1e98053920 0x7f1e98055de0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1eb005ca90 0x7f1eb0150920 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1eb00b6ae0 0x7f1eb005c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 --2- 172.21.15.61:0/1134976033 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1eb005f3f0 0x7f1eb00ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 >> 172.21.15.61:0/1134976033 conn(0x7f1eb0064590 msgr2=0x7f1eb013c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 shutdown_connections 2022-07-12T08:49:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.817+0000 7f1eb675c700 1 -- 172.21.15.61:0/1134976033 wait complete. 2022-07-12T08:49:00.841 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:00.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 Processor -- start 2022-07-12T08:49:00.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- start start 2022-07-12T08:49:00.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d7c0afc70 0x7f1d7c1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:00.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c05d480 con 0x7f1d7c0adb00 2022-07-12T08:49:00.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c0b16d0 con 0x7f1d7c0b1f80 2022-07-12T08:49:00.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0b0fc0 con 0x7f1d7c0afc70 2022-07-12T08:49:00.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d7c0afc70 0x7f1d7c1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:00.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d7c0afc70 0x7f1d7c1433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59376/0 (socket says 172.21.15.61:59376) 2022-07-12T08:49:00.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2978695380 learned_addr learned my addr 172.21.15.61:0/2978695380 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:00.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2978695380 >> v1:172.21.15.111:6789/0 conn(0x7f1d7c0b1f80 legacy=0x7f1d7c0ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:00.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2978695380 >> v1:172.21.15.61:6789/0 conn(0x7f1d7c0adb00 legacy=0x7f1d7c1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:00.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2978695380 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0b2360 con 0x7f1d7c0afc70 2022-07-12T08:49:00.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d72ffd700 1 --2- 172.21.15.61:0/2978695380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c1433e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f1d7c0b30c0 tx=0x7f1d64006eb0).ready entity=mon.0 client_cookie=fc8a98eeec565744 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:00.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d7affd700 1 -- 172.21.15.61:0/2978695380 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6400d5c0 con 0x7f1d7c0afc70 2022-07-12T08:49:00.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d7affd700 1 -- 172.21.15.61:0/2978695380 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d64000e90 con 0x7f1d7c0afc70 2022-07-12T08:49:00.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d7affd700 1 -- 172.21.15.61:0/2978695380 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6400d950 con 0x7f1d7c0afc70 2022-07-12T08:49:00.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2978695380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 msgr2=0x7f1d7c1433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:00.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2978695380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c1433e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f1d7c0b30c0 tx=0x7f1d64006eb0).stop 2022-07-12T08:49:00.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2978695380 shutdown_connections 2022-07-12T08:49:00.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2978695380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c1433e0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2978695380 >> 172.21.15.61:0/2978695380 conn(0x7f1d7c062c00 msgr2=0x7f1d7c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:00.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2978695380 shutdown_connections 2022-07-12T08:49:00.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.949+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2978695380 wait complete. 2022-07-12T08:49:00.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 Processor -- start 2022-07-12T08:49:00.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 -- start start 2022-07-12T08:49:00.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7c0adb00 0x7f1d7c1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:00.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c065f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:00.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d7c0b1f80 0x7f1d7c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:00.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0ab440 con 0x7f1d7c0afc70 2022-07-12T08:49:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c155b80 con 0x7f1d7c0b1f80 2022-07-12T08:49:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c066480 con 0x7f1d7c0adb00 2022-07-12T08:49:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c065f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:00.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c065f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59382/0 (socket says 172.21.15.61:59382) 2022-07-12T08:49:00.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2073655208 learned_addr learned my addr 172.21.15.61:0/2073655208 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:00.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d7b7fe700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7c0adb00 0x7f1d7c1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:00.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d7bfff700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d7c0b1f80 0x7f1d7c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:00.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7c0adb00 msgr2=0x7f1d7c1478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:00.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7c0adb00 0x7f1d7c1478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d7c0b1f80 msgr2=0x7f1d7c156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:00.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d7c0b1f80 0x7f1d7c156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:00.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 -- 172.21.15.61:0/2073655208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6400b1e0 con 0x7f1d7c0afc70 2022-07-12T08:49:00.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d72ffd700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c065f90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f1d7c0b30c0 tx=0x7f1d64015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:00.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64021070 con 0x7f1d7c0afc70 2022-07-12T08:49:00.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d640117e0 con 0x7f1d7c0afc70 2022-07-12T08:49:00.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c148100 con 0x7f1d7c0afc70 2022-07-12T08:49:00.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d64011b40 con 0x7f1d7c0afc70 2022-07-12T08:49:00.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.953+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c065640 con 0x7f1d7c0afc70 2022-07-12T08:49:00.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.957+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c0ab5b0 con 0x7f1d7c0afc70 2022-07-12T08:49:00.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.957+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1d6400f070 con 0x7f1d7c0afc70 2022-07-12T08:49:00.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.957+0000 7f1d797fa700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d600539d0 0x7f1d60055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:00.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.957+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1d6406d2b0 con 0x7f1d7c0afc70 2022-07-12T08:49:00.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.957+0000 7f1d7b7fe700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d600539d0 0x7f1d60055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:00.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.961+0000 7f1d7b7fe700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d600539d0 0x7f1d60055e90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f1d6c00c000 tx=0x7f1d6c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:00.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:00.965+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d7c0ab5b0 con 0x7f1d7c0afc70 2022-07-12T08:49:01.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.129+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d7c0ab580 con 0x7f1d7c0afc70 2022-07-12T08:49:01.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d797fa700 1 -- 172.21.15.61:0/2073655208 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f1d64044100 con 0x7f1d7c0afc70 2022-07-12T08:49:01.139 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:01.139 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:59.111339+0000","modified":"2022-07-12T08:49:00.805218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2528523999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2528523999},{"type":"v1","addr":"172.21.15.61:6833","nonce":2528523999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-12T08:49:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d600539d0 msgr2=0x7f1d60055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d600539d0 0x7f1d60055e90 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7f1d6c00c000 tx=0x7f1d6c008040).stop 2022-07-12T08:49:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 msgr2=0x7f1d7c065f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c065f90 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f1d7c0b30c0 tx=0x7f1d64015000).stop 2022-07-12T08:49:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 shutdown_connections 2022-07-12T08:49:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d600539d0 0x7f1d60055e90 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d7c0b1f80 0x7f1d7c156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d7c0afc70 0x7f1d7c065f90 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 --2- 172.21.15.61:0/2073655208 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d7c0adb00 0x7f1d7c1478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 >> 172.21.15.61:0/2073655208 conn(0x7f1d7c062c00 msgr2=0x7f1d7c0b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 shutdown_connections 2022-07-12T08:49:01.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.133+0000 7f1d81d3a700 1 -- 172.21.15.61:0/2073655208 wait complete. 2022-07-12T08:49:01.147 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-12T08:49:01.158 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:01.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 Processor -- start 2022-07-12T08:49:01.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 -- start start 2022-07-12T08:49:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2250144480 0x7f225005bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225013ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22500b3010 con 0x7f22501440a0 2022-07-12T08:49:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22500624b0 con 0x7f2250144480 2022-07-12T08:49:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f22557ab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2250144da0 con 0x7f22500b3630 2022-07-12T08:49:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2250144480 0x7f225005bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2250144480 0x7f225005bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59388/0 (socket says 172.21.15.61:59388) 2022-07-12T08:49:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 -- 172.21.15.61:0/3558097435 learned_addr learned my addr 172.21.15.61:0/3558097435 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f2254faa700 1 --2- 172.21.15.61:0/3558097435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225013ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 -- 172.21.15.61:0/3558097435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 msgr2=0x7f225013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 --2- 172.21.15.61:0/3558097435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225013ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 -- 172.21.15.61:0/3558097435 >> v1:172.21.15.111:6789/0 conn(0x7f22501440a0 legacy=0x7f22501413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 -- 172.21.15.61:0/3558097435 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22500b2d70 con 0x7f2250144480 2022-07-12T08:49:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224f7fe700 1 --2- 172.21.15.61:0/3558097435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2250144480 0x7f225005bd10 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f2240002f70 tx=0x7f2240006eb0).ready entity=mon.0 client_cookie=a958d56568e35abd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224effd700 1 -- 172.21.15.61:0/3558097435 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f224000d5c0 con 0x7f2250144480 2022-07-12T08:49:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224effd700 1 -- 172.21.15.61:0/3558097435 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2240000e90 con 0x7f2250144480 2022-07-12T08:49:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.253+0000 7f224effd700 1 -- 172.21.15.61:0/3558097435 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f224000d950 con 0x7f2250144480 2022-07-12T08:49:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 -- 172.21.15.61:0/3558097435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2250144480 msgr2=0x7f225005bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 --2- 172.21.15.61:0/3558097435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2250144480 0x7f225005bd10 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 rx=0x7f2240002f70 tx=0x7f2240006eb0).stop 2022-07-12T08:49:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 -- 172.21.15.61:0/3558097435 shutdown_connections 2022-07-12T08:49:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 --2- 172.21.15.61:0/3558097435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225013ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 --2- 172.21.15.61:0/3558097435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2250144480 0x7f225005bd10 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 -- 172.21.15.61:0/3558097435 >> 172.21.15.61:0/3558097435 conn(0x7f22500b70b0 msgr2=0x7f22500b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 -- 172.21.15.61:0/3558097435 shutdown_connections 2022-07-12T08:49:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 -- 172.21.15.61:0/3558097435 wait complete. 2022-07-12T08:49:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 Processor -- start 2022-07-12T08:49:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.257+0000 7f22557ab700 1 -- start start 2022-07-12T08:49:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225005f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 0x7f225009b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2250144480 0x7f225009bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225009a5c0 con 0x7f22501440a0 2022-07-12T08:49:01.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225009a860 con 0x7f2250144480 2022-07-12T08:49:01.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225009ab00 con 0x7f22500b3630 2022-07-12T08:49:01.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 0x7f225009b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 0x7f225009b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59396/0 (socket says 172.21.15.61:59396) 2022-07-12T08:49:01.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 -- 172.21.15.61:0/1042085866 learned_addr learned my addr 172.21.15.61:0/1042085866 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:01.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224ffff700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225005f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f2254faa700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2250144480 0x7f225009bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 -- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 msgr2=0x7f225005f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225005f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 -- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2250144480 msgr2=0x7f225009bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2250144480 0x7f225009bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 -- 172.21.15.61:0/1042085866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224000b1e0 con 0x7f22501440a0 2022-07-12T08:49:01.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224f7fe700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 0x7f225009b5c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f2240002f70 tx=0x7f2240015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22400117e0 con 0x7f22501440a0 2022-07-12T08:49:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2240011940 con 0x7f22501440a0 2022-07-12T08:49:01.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225005f7d0 con 0x7f22501440a0 2022-07-12T08:49:01.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2240011ca0 con 0x7f22501440a0 2022-07-12T08:49:01.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.261+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2250146990 con 0x7f22501440a0 2022-07-12T08:49:01.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.265+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2250057460 con 0x7f22501440a0 2022-07-12T08:49:01.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.265+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2240025030 con 0x7f22501440a0 2022-07-12T08:49:01.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.265+0000 7f224d7fa700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f22340539b0 0x7f2234055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.265+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f224006d0a0 con 0x7f22501440a0 2022-07-12T08:49:01.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.265+0000 7f224ffff700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f22340539b0 0x7f2234055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.265+0000 7f224ffff700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f22340539b0 0x7f2234055e70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f223800d000 tx=0x7f2238007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.273+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2240043ef0 con 0x7f22501440a0 2022-07-12T08:49:01.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.401+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22500b1fb0 con 0x7f22501440a0 2022-07-12T08:49:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.401+0000 7f224d7fa700 1 -- 172.21.15.61:0/1042085866 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16091 (secure 0 0 0) 0x7f2240023020 con 0x7f22501440a0 2022-07-12T08:49:01.409 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:01.409 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":100,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:00.114186+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:56.975401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:58.124077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:01.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f22340539b0 msgr2=0x7f2234055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f22340539b0 0x7f2234055e70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f223800d000 tx=0x7f2238007b90).stop 2022-07-12T08:49:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 msgr2=0x7f225009b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 0x7f225009b5c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7f2240002f70 tx=0x7f2240015000).stop 2022-07-12T08:49:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 shutdown_connections 2022-07-12T08:49:01.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f22340539b0 0x7f2234055e70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2250144480 0x7f225009bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22501440a0 0x7f225009b5c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 --2- 172.21.15.61:0/1042085866 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f22500b3630 0x7f225005f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 >> 172.21.15.61:0/1042085866 conn(0x7f22500b70b0 msgr2=0x7f22500614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 shutdown_connections 2022-07-12T08:49:01.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.405+0000 7f22557ab700 1 -- 172.21.15.61:0/1042085866 wait complete. 2022-07-12T08:49:01.426 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:49:01.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 Processor -- start 2022-07-12T08:49:01.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 -- start start 2022-07-12T08:49:01.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5f2c141e50 0x7f5f2c1422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2c0b56c0 con 0x7f5f2c0b52e0 2022-07-12T08:49:01.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2c142f00 con 0x7f5f2c141e50 2022-07-12T08:49:01.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f33229700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c0b1320 con 0x7f5f2c0b4050 2022-07-12T08:49:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5f2c141e50 0x7f5f2c1422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5f2c141e50 0x7f5f2c1422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59406/0 (socket says 172.21.15.61:59406) 2022-07-12T08:49:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f30fc5700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5f2c0b52e0 0x7f5f2c0b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47620/0 (socket says 172.21.15.61:47620) 2022-07-12T08:49:01.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 -- 172.21.15.61:0/4045320378 learned_addr learned my addr 172.21.15.61:0/4045320378 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:01.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f2bfff700 1 --2- 172.21.15.61:0/4045320378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 -- 172.21.15.61:0/4045320378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 msgr2=0x7f5f2c0b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:01.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 --2- 172.21.15.61:0/4045320378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c0b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 -- 172.21.15.61:0/4045320378 >> v1:172.21.15.61:6789/0 conn(0x7f5f2c0b52e0 legacy=0x7f5f2c0b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/4045320378 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307575082 0 0) 0x7f5f2c0b56c0 con 0x7f5f2c0b52e0 2022-07-12T08:49:01.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 -- 172.21.15.61:0/4045320378 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c0b4bd0 con 0x7f5f2c141e50 2022-07-12T08:49:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f317c6700 1 --2- 172.21.15.61:0/4045320378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c1422c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f5f2c0b1460 tx=0x7f5f24009290).ready entity=mon.0 client_cookie=f27b2b861ee3f198 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/4045320378 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2400b860 con 0x7f5f2c141e50 2022-07-12T08:49:01.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/4045320378 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2400b9c0 con 0x7f5f2c141e50 2022-07-12T08:49:01.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.533+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/4045320378 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2400bb30 con 0x7f5f2c141e50 2022-07-12T08:49:01.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- 172.21.15.61:0/4045320378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 msgr2=0x7f5f2c1422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 --2- 172.21.15.61:0/4045320378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c1422c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f5f2c0b1460 tx=0x7f5f24009290).stop 2022-07-12T08:49:01.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- 172.21.15.61:0/4045320378 shutdown_connections 2022-07-12T08:49:01.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 --2- 172.21.15.61:0/4045320378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c1422c0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 --2- 172.21.15.61:0/4045320378 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c0b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- 172.21.15.61:0/4045320378 >> 172.21.15.61:0/4045320378 conn(0x7f5f2c067020 msgr2=0x7f5f2c067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:01.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- 172.21.15.61:0/4045320378 shutdown_connections 2022-07-12T08:49:01.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- 172.21.15.61:0/4045320378 wait complete. 2022-07-12T08:49:01.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 Processor -- start 2022-07-12T08:49:01.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- start start 2022-07-12T08:49:01.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f2c0b52e0 0x7f5f2c13e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c13e280 con 0x7f5f2c141e50 2022-07-12T08:49:01.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c145c40 con 0x7f5f2c0b52e0 2022-07-12T08:49:01.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f33229700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c0af720 con 0x7f5f2c0b4050 2022-07-12T08:49:01.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f317c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f317c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59412/0 (socket says 172.21.15.61:59412) 2022-07-12T08:49:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f317c6700 1 -- 172.21.15.61:0/2720311536 learned_addr learned my addr 172.21.15.61:0/2720311536 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f2bfff700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f2c0b52e0 0x7f5f2c13e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f30fc5700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.537+0000 7f5f317c6700 1 -- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 msgr2=0x7f5f2c147a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f317c6700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c147a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f317c6700 1 -- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f2c0b52e0 msgr2=0x7f5f2c13e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f317c6700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f2c0b52e0 0x7f5f2c13e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f317c6700 1 -- 172.21.15.61:0/2720311536 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f24009530 con 0x7f5f2c141e50 2022-07-12T08:49:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f317c6700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c155f00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7f5f2c0b1460 tx=0x7f5f24016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f24021630 con 0x7f5f2c141e50 2022-07-12T08:49:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f24021790 con 0x7f5f2c141e50 2022-07-12T08:49:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c0af9c0 con 0x7f5f2c141e50 2022-07-12T08:49:01.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f24021af0 con 0x7f5f2c141e50 2022-07-12T08:49:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c13eac0 con 0x7f5f2c141e50 2022-07-12T08:49:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c057460 con 0x7f5f2c141e50 2022-07-12T08:49:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5f2400f070 con 0x7f5f2c141e50 2022-07-12T08:49:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f29ffb700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f0c053a40 0x7f5f0c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f2406d490 con 0x7f5f2c141e50 2022-07-12T08:49:01.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.541+0000 7f5f30fc5700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f0c053a40 0x7f5f0c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.545+0000 7f5f30fc5700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f0c053a40 0x7f5f0c055f00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f5f1800d000 tx=0x7f5f18007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.549+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f240442e0 con 0x7f5f2c141e50 2022-07-12T08:49:01.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.713+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5f2c05cbf0 con 0x7f5f2c141e50 2022-07-12T08:49:01.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.721+0000 7f5f29ffb700 1 -- 172.21.15.61:0/2720311536 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5f24015070 con 0x7f5f2c141e50 2022-07-12T08:49:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f0c053a40 msgr2=0x7f5f0c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f0c053a40 0x7f5f0c055f00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f5f1800d000 tx=0x7f5f18007a00).stop 2022-07-12T08:49:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 msgr2=0x7f5f2c155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c155f00 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7f5f2c0b1460 tx=0x7f5f24016000).stop 2022-07-12T08:49:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 shutdown_connections 2022-07-12T08:49:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c141e50 0x7f5f2c155f00 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5f0c053a40 0x7f5f0c055f00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f2c0b52e0 0x7f5f2c13e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 --2- 172.21.15.61:0/2720311536 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5f2c0b4050 0x7f5f2c147a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 >> 172.21.15.61:0/2720311536 conn(0x7f5f2c067020 msgr2=0x7f5f2c067400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 shutdown_connections 2022-07-12T08:49:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.725+0000 7f5f33229700 1 -- 172.21.15.61:0/2720311536 wait complete. 2022-07-12T08:49:01.735 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:49:01.744 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:49:01.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.865+0000 7fdba0e84700 1 Processor -- start 2022-07-12T08:49:01.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.865+0000 7fdba0e84700 1 -- start start 2022-07-12T08:49:01.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdba0e84700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdb9c0b1a90 0x7fdb9c0aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdba0e84700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 0x7fdb9c0aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdba0e84700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9c143fb0 con 0x7fdb9c144bc0 2022-07-12T08:49:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdba0e84700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b1e70 con 0x7fdb9c0b1a90 2022-07-12T08:49:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdba0e84700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c1441e0 con 0x7fdb9c061780 2022-07-12T08:49:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 0x7fdb9c0aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb9a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdb9c0b1a90 0x7fdb9c0aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb9a59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdb9c0b1a90 0x7fdb9c0aaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:37990/0 (socket says 172.21.15.61:37990) 2022-07-12T08:49:01.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 0x7fdb9c0aaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:41614/0 (socket says 172.21.15.61:41614) 2022-07-12T08:49:01.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb9a59c700 1 -- 172.21.15.61:0/1864542841 learned_addr learned my addr 172.21.15.61:0/1864542841 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:01.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 -- 172.21.15.61:0/1864542841 >> v2:172.21.15.111:3300/0 conn(0x7fdb9c0b1a90 msgr2=0x7fdb9c0aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:01.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 --2- 172.21.15.61:0/1864542841 >> v2:172.21.15.111:3300/0 conn(0x7fdb9c0b1a90 0x7fdb9c0aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 -- 172.21.15.61:0/1864542841 >> v1:172.21.15.111:6789/0 conn(0x7fdb9c144bc0 legacy=0x7fdb9c060490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 -- 172.21.15.61:0/1864542841 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c144480 con 0x7fdb9c061780 2022-07-12T08:49:01.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb99d9b700 1 --2- 172.21.15.61:0/1864542841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 0x7fdb9c0aaf40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdb90002f70 tx=0x7fdb90006df0).ready entity=mon.2 client_cookie=9bd9a0ad6a2322d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb9959a700 1 -- 172.21.15.61:0/1864542841 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9000d570 con 0x7fdb9c061780 2022-07-12T08:49:01.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb9959a700 1 -- 172.21.15.61:0/1864542841 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb90000e90 con 0x7fdb9c061780 2022-07-12T08:49:01.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.869+0000 7fdb9959a700 1 -- 172.21.15.61:0/1864542841 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb9000d8c0 con 0x7fdb9c061780 2022-07-12T08:49:01.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- 172.21.15.61:0/1864542841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 msgr2=0x7fdb9c0aaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 --2- 172.21.15.61:0/1864542841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 0x7fdb9c0aaf40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fdb90002f70 tx=0x7fdb90006df0).stop 2022-07-12T08:49:01.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- 172.21.15.61:0/1864542841 shutdown_connections 2022-07-12T08:49:01.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 --2- 172.21.15.61:0/1864542841 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c061780 0x7fdb9c0aaf40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 --2- 172.21.15.61:0/1864542841 >> v2:172.21.15.111:3300/0 conn(0x7fdb9c0b1a90 0x7fdb9c0aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- 172.21.15.61:0/1864542841 >> 172.21.15.61:0/1864542841 conn(0x7fdb9c05f140 msgr2=0x7fdb9c05f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:01.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- 172.21.15.61:0/1864542841 shutdown_connections 2022-07-12T08:49:01.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- 172.21.15.61:0/1864542841 wait complete. 2022-07-12T08:49:01.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 Processor -- start 2022-07-12T08:49:01.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- start start 2022-07-12T08:49:01.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 0x7fdb9c0669a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb9c0b1a90 0x7fdb9c13cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c144bc0 0x7fdb9c13d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c09ba40 con 0x7fdb9c061780 2022-07-12T08:49:01.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c140900 con 0x7fdb9c0b1a90 2022-07-12T08:49:01.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdba0e84700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c140c20 con 0x7fdb9c144bc0 2022-07-12T08:49:01.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdb9a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 0x7fdb9c0669a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdb9a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 0x7fdb9c0669a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59422/0 (socket says 172.21.15.61:59422) 2022-07-12T08:49:01.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.873+0000 7fdb9a59c700 1 -- 172.21.15.61:0/954983680 learned_addr learned my addr 172.21.15.61:0/954983680 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:01.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb99d9b700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb9c0b1a90 0x7fdb9c13cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9ad9d700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c144bc0 0x7fdb9c13d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9a59c700 1 -- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c144bc0 msgr2=0x7fdb9c13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9a59c700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c144bc0 0x7fdb9c13d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9a59c700 1 -- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb9c0b1a90 msgr2=0x7fdb9c13cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:01.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9a59c700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb9c0b1a90 0x7fdb9c13cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:01.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9a59c700 1 -- 172.21.15.61:0/954983680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9000b180 con 0x7fdb9c061780 2022-07-12T08:49:01.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb9a59c700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 0x7fdb9c0669a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7fdb8c00c000 tx=0x7fdb8c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c00fe30 con 0x7fdb9c061780 2022-07-12T08:49:01.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9c13d5c0 con 0x7fdb9c061780 2022-07-12T08:49:01.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c013610 con 0x7fdb9c061780 2022-07-12T08:49:01.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb8c013970 con 0x7fdb9c061780 2022-07-12T08:49:01.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9c1418a0 con 0x7fdb9c061780 2022-07-12T08:49:01.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c140de0 con 0x7fdb9c061780 2022-07-12T08:49:01.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdb8c013ad0 con 0x7fdb9c061780 2022-07-12T08:49:01.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.877+0000 7fdb8b7fe700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb78053920 0x7fdb78055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:01.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.881+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb8c06c0c0 con 0x7fdb9c061780 2022-07-12T08:49:01.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.881+0000 7fdb99d9b700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb78053920 0x7fdb78055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:01.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.881+0000 7fdb99d9b700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb78053920 0x7fdb78055de0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fdb9001a000 tx=0x7fdb90014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:01.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:01.885+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb8c042f80 con 0x7fdb9c061780 2022-07-12T08:49:02.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.061+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fdb9c057460 con 0x7fdb9c061780 2022-07-12T08:49:02.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdb8b7fe700 1 -- 172.21.15.61:0/954983680 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7fdb8c0465a0 con 0x7fdb9c061780 2022-07-12T08:49:02.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb78053920 msgr2=0x7fdb78055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb78053920 0x7fdb78055de0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7fdb9001a000 tx=0x7fdb90014000).stop 2022-07-12T08:49:02.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 msgr2=0x7fdb9c0669a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 0x7fdb9c0669a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7fdb8c00c000 tx=0x7fdb8c008040).stop 2022-07-12T08:49:02.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 shutdown_connections 2022-07-12T08:49:02.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb9c144bc0 0x7fdb9c13d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb78053920 0x7fdb78055de0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb9c0b1a90 0x7fdb9c13cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 --2- 172.21.15.61:0/954983680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb9c061780 0x7fdb9c0669a0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 >> 172.21.15.61:0/954983680 conn(0x7fdb9c05f140 msgr2=0x7fdb9c0b23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:02.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 shutdown_connections 2022-07-12T08:49:02.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.073+0000 7fdba0e84700 1 -- 172.21.15.61:0/954983680 wait complete. 2022-07-12T08:49:02.090 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:49:02.093 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:49:02.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f6498700 1 Processor -- start 2022-07-12T08:49:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f6498700 1 -- start start 2022-07-12T08:49:02.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f6498700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0f00b02a0 0x7fb0f00b0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f6498700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f00acab0 con 0x7fb0f0146420 2022-07-12T08:49:02.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f6498700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f00b2170 con 0x7fb0f00614a0 2022-07-12T08:49:02.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f6498700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f005c150 con 0x7fb0f00b02a0 2022-07-12T08:49:02.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0effff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0f00b02a0 0x7fb0f00b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f4a35700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb0f0146420 0x7fb0f00af510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47652/0 (socket says 172.21.15.61:47652) 2022-07-12T08:49:02.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.201+0000 7fb0f4a35700 1 -- 172.21.15.61:0/2791335985 learned_addr learned my addr 172.21.15.61:0/2791335985 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:02.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0effff700 1 -- 172.21.15.61:0/2791335985 >> v1:172.21.15.111:6789/0 conn(0x7fb0f00614a0 legacy=0x7fb0f00b1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0effff700 1 -- 172.21.15.61:0/2791335985 >> v1:172.21.15.61:6789/0 conn(0x7fb0f0146420 legacy=0x7fb0f00af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0effff700 1 -- 172.21.15.61:0/2791335985 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f00abfb0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0eeffd700 1 -- 172.21.15.61:0/2791335985 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643212712 0 0) 0x7fb0f00acab0 con 0x7fb0f0146420 2022-07-12T08:49:02.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0effff700 1 --2- 172.21.15.61:0/2791335985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f00b0680 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fb0e0002f70 tx=0x7fb0e0006df0).ready entity=mon.0 client_cookie=f2008be7bbc3f36f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:02.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0eeffd700 1 -- 172.21.15.61:0/2791335985 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e000d610 con 0x7fb0f00b02a0 2022-07-12T08:49:02.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0eeffd700 1 -- 172.21.15.61:0/2791335985 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e0000e90 con 0x7fb0f00b02a0 2022-07-12T08:49:02.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0eeffd700 1 -- 172.21.15.61:0/2791335985 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e000d960 con 0x7fb0f00b02a0 2022-07-12T08:49:02.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 -- 172.21.15.61:0/2791335985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 msgr2=0x7fb0f00b0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2791335985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f00b0680 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fb0e0002f70 tx=0x7fb0e0006df0).stop 2022-07-12T08:49:02.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 -- 172.21.15.61:0/2791335985 shutdown_connections 2022-07-12T08:49:02.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2791335985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f00b0680 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 -- 172.21.15.61:0/2791335985 >> 172.21.15.61:0/2791335985 conn(0x7fb0f005f140 msgr2=0x7fb0f005f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:02.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 -- 172.21.15.61:0/2791335985 shutdown_connections 2022-07-12T08:49:02.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.205+0000 7fb0f6498700 1 -- 172.21.15.61:0/2791335985 wait complete. 2022-07-12T08:49:02.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 Processor -- start 2022-07-12T08:49:02.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 -- start start 2022-07-12T08:49:02.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f00614a0 0x7fb0f00928e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f0141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0146420 0x7fb0f0156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f0092dd0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f0149ec0 con 0x7fb0f00614a0 2022-07-12T08:49:02.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f6498700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f0141e20 con 0x7fb0f0146420 2022-07-12T08:49:02.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f0141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f0141930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59438/0 (socket says 172.21.15.61:59438) 2022-07-12T08:49:02.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 -- 172.21.15.61:0/2059189649 learned_addr learned my addr 172.21.15.61:0/2059189649 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:02.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0f4a35700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0146420 0x7fb0f0156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0effff700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f00614a0 0x7fb0f00928e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 -- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0146420 msgr2=0x7fb0f0156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0146420 0x7fb0f0156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 -- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f00614a0 msgr2=0x7fb0f00928e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f00614a0 0x7fb0f00928e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 -- 172.21.15.61:0/2059189649 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e000b180 con 0x7fb0f00b02a0 2022-07-12T08:49:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.209+0000 7fb0ef7fe700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f0141930 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fb0f0067fe0 tx=0x7fb0e4007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e4012400 con 0x7fb0f00b02a0 2022-07-12T08:49:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4012560 con 0x7fb0f00b02a0 2022-07-12T08:49:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f0155660 con 0x7fb0f00b02a0 2022-07-12T08:49:02.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e40128c0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f0156530 con 0x7fb0f00b02a0 2022-07-12T08:49:02.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb0e4019030 con 0x7fb0f00b02a0 2022-07-12T08:49:02.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f0149af0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0ed7fa700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0d00539b0 0x7fb0d0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb0e406b200 con 0x7fb0f00b02a0 2022-07-12T08:49:02.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0effff700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0d00539b0 0x7fb0d0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.213+0000 7fb0effff700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0d00539b0 0x7fb0d0055e70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fb0e001c000 tx=0x7fb0e0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:02.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.217+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0f0149af0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.385+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb0f0149af0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.393+0000 7fb0ed7fa700 1 -- 172.21.15.61:0/2059189649 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb0f0149af0 con 0x7fb0f00b02a0 2022-07-12T08:49:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0d00539b0 msgr2=0x7fb0d0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0d00539b0 0x7fb0d0055e70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fb0e001c000 tx=0x7fb0e0015000).stop 2022-07-12T08:49:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 msgr2=0x7fb0f0141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f0141930 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fb0f0067fe0 tx=0x7fb0e4007c50).stop 2022-07-12T08:49:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 shutdown_connections 2022-07-12T08:49:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb0f0146420 0x7fb0f0156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0f00b02a0 0x7fb0f0141930 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb0d00539b0 0x7fb0d0055e70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 --2- 172.21.15.61:0/2059189649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb0f00614a0 0x7fb0f00928e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 >> 172.21.15.61:0/2059189649 conn(0x7fb0f005f140 msgr2=0x7fb0f0060490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:02.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 shutdown_connections 2022-07-12T08:49:02.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.397+0000 7fb0f6498700 1 -- 172.21.15.61:0/2059189649 wait complete. 2022-07-12T08:49:02.408 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:49:02.421 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:02.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 Processor -- start 2022-07-12T08:49:02.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 -- start start 2022-07-12T08:49:02.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d280b7b00 0x7f1d280b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1d28142f40 0x7f1d28143380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d280b7ee0 con 0x7f1d28062820 2022-07-12T08:49:02.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d28060e80 con 0x7f1d280b7b00 2022-07-12T08:49:02.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d2ca3f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d28061120 con 0x7f1d28142f40 2022-07-12T08:49:02.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d1effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1d28062820 0x7f1d2805fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47662/0 (socket says 172.21.15.61:47662) 2022-07-12T08:49:02.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d1effd700 1 -- 172.21.15.61:0/1907453231 learned_addr learned my addr 172.21.15.61:0/1907453231 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:02.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d277fe700 1 --2- 172.21.15.61:0/1907453231 >> v2:172.21.15.61:3300/0 conn(0x7f1d280b7b00 0x7f1d280b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.521+0000 7f1d27fff700 1 --2- 172.21.15.61:0/1907453231 >> v2:172.21.15.111:3300/0 conn(0x7f1d28142f40 0x7f1d28143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d26ffd700 1 -- 172.21.15.61:0/1907453231 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734861520 0 0) 0x7f1d280b7ee0 con 0x7f1d28062820 2022-07-12T08:49:02.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d26ffd700 1 -- 172.21.15.61:0/1907453231 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d08003260 con 0x7f1d28062820 2022-07-12T08:49:02.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d277fe700 1 -- 172.21.15.61:0/1907453231 >> v2:172.21.15.111:3300/0 conn(0x7f1d28142f40 msgr2=0x7f1d28143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:02.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d277fe700 1 --2- 172.21.15.61:0/1907453231 >> v2:172.21.15.111:3300/0 conn(0x7f1d28142f40 0x7f1d28143380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d277fe700 1 -- 172.21.15.61:0/1907453231 >> v1:172.21.15.61:6789/0 conn(0x7f1d28062820 legacy=0x7f1d2805fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d277fe700 1 -- 172.21.15.61:0/1907453231 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2805f920 con 0x7f1d280b7b00 2022-07-12T08:49:02.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d277fe700 1 --2- 172.21.15.61:0/1907453231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d280b29d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f1d18002f70 tx=0x7f1d18006df0).ready entity=mon.0 client_cookie=775bc548767ce322 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:02.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d26ffd700 1 -- 172.21.15.61:0/1907453231 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d1800d4f0 con 0x7f1d280b7b00 2022-07-12T08:49:02.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d26ffd700 1 -- 172.21.15.61:0/1907453231 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d18000e90 con 0x7f1d280b7b00 2022-07-12T08:49:02.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1907453231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 msgr2=0x7f1d280b29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1907453231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d280b29d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f1d18002f70 tx=0x7f1d18006df0).stop 2022-07-12T08:49:02.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1907453231 shutdown_connections 2022-07-12T08:49:02.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1907453231 >> v2:172.21.15.111:3300/0 conn(0x7f1d28142f40 0x7f1d28143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1907453231 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d280b29d0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1907453231 >> 172.21.15.61:0/1907453231 conn(0x7f1d28065550 msgr2=0x7f1d280b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:02.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1907453231 shutdown_connections 2022-07-12T08:49:02.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1907453231 wait complete. 2022-07-12T08:49:02.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 Processor -- start 2022-07-12T08:49:02.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.525+0000 7f1d2ca3f700 1 -- start start 2022-07-12T08:49:02.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d28062820 0x7f1d2809b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d28155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d28142f40 0x7f1d28151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28154f30 con 0x7f1d280b7b00 2022-07-12T08:49:02.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d281551d0 con 0x7f1d28142f40 2022-07-12T08:49:02.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d28155470 con 0x7f1d28062820 2022-07-12T08:49:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d28155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d28155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59450/0 (socket says 172.21.15.61:59450) 2022-07-12T08:49:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 -- 172.21.15.61:0/1962896553 learned_addr learned my addr 172.21.15.61:0/1962896553 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:02.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d27fff700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d28142f40 0x7f1d28151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d277fe700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d28062820 0x7f1d2809b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 -- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d28062820 msgr2=0x7f1d2809b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d28062820 0x7f1d2809b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 -- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d28142f40 msgr2=0x7f1d28151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d28142f40 0x7f1d28151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 -- 172.21.15.61:0/1962896553 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1800b180 con 0x7f1d280b7b00 2022-07-12T08:49:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d1effd700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d28155910 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f1d10001110 tx=0x7f1d10007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d100070d0 con 0x7f1d280b7b00 2022-07-12T08:49:02.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d10007230 con 0x7f1d280b7b00 2022-07-12T08:49:02.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d10005610 con 0x7f1d280b7b00 2022-07-12T08:49:02.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d28066d80 con 0x7f1d280b7b00 2022-07-12T08:49:02.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d28096f80 con 0x7f1d280b7b00 2022-07-12T08:49:02.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.529+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d28155630 con 0x7f1d280b7b00 2022-07-12T08:49:02.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.533+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1d10010070 con 0x7f1d280b7b00 2022-07-12T08:49:02.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.533+0000 7f1d257fa700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d0c053a40 0x7f1d0c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:02.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.533+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1d1006b710 con 0x7f1d280b7b00 2022-07-12T08:49:02.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.533+0000 7f1d277fe700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d0c053a40 0x7f1d0c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:02.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.533+0000 7f1d277fe700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d0c053a40 0x7f1d0c055f00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f1d1801a000 tx=0x7f1d18014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:02.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.537+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d10042560 con 0x7f1d280b7b00 2022-07-12T08:49:02.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.701+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d28097290 con 0x7f1d280b7b00 2022-07-12T08:49:02.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.701+0000 7f1d257fa700 1 -- 172.21.15.61:0/1962896553 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f1d10045b80 con 0x7f1d280b7b00 2022-07-12T08:49:02.718 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:02.719 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:59.111339+0000","modified":"2022-07-12T08:49:00.805218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2528523999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2528523999},{"type":"v1","addr":"172.21.15.61:6833","nonce":2528523999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-12T08:49:02.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d0c053a40 msgr2=0x7f1d0c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d0c053a40 0x7f1d0c055f00 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7f1d1801a000 tx=0x7f1d18014000).stop 2022-07-12T08:49:02.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 msgr2=0x7f1d28155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:02.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d28155910 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f1d10001110 tx=0x7f1d10007b80).stop 2022-07-12T08:49:02.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 shutdown_connections 2022-07-12T08:49:02.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1d0c053a40 0x7f1d0c055f00 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1d28142f40 0x7f1d28151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d280b7b00 0x7f1d28155910 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 --2- 172.21.15.61:0/1962896553 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1d28062820 0x7f1d2809b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:02.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 >> 172.21.15.61:0/1962896553 conn(0x7f1d28065550 msgr2=0x7f1d28143380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:02.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 shutdown_connections 2022-07-12T08:49:02.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:02.705+0000 7f1d2ca3f700 1 -- 172.21.15.61:0/1962896553 wait complete. 2022-07-12T08:49:02.729 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-12T08:49:02.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:59.111339+0000', 'modified': '2022-07-12T08:49:00.805218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7338}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7338': {'gid': 7338, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2528523999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2528523999}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2528523999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:49:02.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:49:02.932 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:49:03.053 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":7338,"want_state":"up:active","state":"up:active","rank_uptime":3.927555157,"mdsmap_epoch":56,"osdmap_epoch":100,"osdmap_epoch_barrier":100,"uptime":8.8627810599999997} 2022-07-12T08:49:03.053 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 7338, "mdsmap_epoch": 56, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "rank_uptime": 3.927555157, "state": "up:active", "uptime": 8.86278106, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:49:03.054 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:03.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 Processor -- start 2022-07-12T08:49:03.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 -- start start 2022-07-12T08:49:03.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740b1e40 0x7f537405d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f53740ad590 0x7f53741401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537413fd50 con 0x7f53740ac710 2022-07-12T08:49:03.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5374067900 con 0x7f53740ad590 2022-07-12T08:49:03.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53797e5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5374067ba0 con 0x7f53740b1e40 2022-07-12T08:49:03.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f53740ad590 0x7f53741401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f53740ad590 0x7f53741401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59462/0 (socket says 172.21.15.61:59462) 2022-07-12T08:49:03.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 -- 172.21.15.61:0/1216557778 learned_addr learned my addr 172.21.15.61:0/1216557778 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:03.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f5372ffd700 1 --2- 172.21.15.61:0/1216557778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740b1e40 0x7f537405d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 -- 172.21.15.61:0/1216557778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740b1e40 msgr2=0x7f537405d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:03.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 --2- 172.21.15.61:0/1216557778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740b1e40 0x7f537405d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 -- 172.21.15.61:0/1216557778 >> v1:172.21.15.111:6789/0 conn(0x7f53740ac710 legacy=0x7f53740ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 -- 172.21.15.61:0/1216557778 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53740675f0 con 0x7f53740ad590 2022-07-12T08:49:03.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53737fe700 1 --2- 172.21.15.61:0/1216557778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f53741401b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f536c0032f0 tx=0x7f536c009290).ready entity=mon.0 client_cookie=766567b69d641ff1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53727fc700 1 -- 172.21.15.61:0/1216557778 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f536c00b8a0 con 0x7f53740ad590 2022-07-12T08:49:03.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53727fc700 1 -- 172.21.15.61:0/1216557778 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c00ba00 con 0x7f53740ad590 2022-07-12T08:49:03.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.157+0000 7f53727fc700 1 -- 172.21.15.61:0/1216557778 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f536c00bb70 con 0x7f53740ad590 2022-07-12T08:49:03.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 -- 172.21.15.61:0/1216557778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 msgr2=0x7f53741401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 --2- 172.21.15.61:0/1216557778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f53741401b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 rx=0x7f536c0032f0 tx=0x7f536c009290).stop 2022-07-12T08:49:03.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 -- 172.21.15.61:0/1216557778 shutdown_connections 2022-07-12T08:49:03.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 --2- 172.21.15.61:0/1216557778 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f53741401b0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 --2- 172.21.15.61:0/1216557778 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740b1e40 0x7f537405d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 -- 172.21.15.61:0/1216557778 >> 172.21.15.61:0/1216557778 conn(0x7f5374062c00 msgr2=0x7f5374062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:03.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 -- 172.21.15.61:0/1216557778 shutdown_connections 2022-07-12T08:49:03.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 -- 172.21.15.61:0/1216557778 wait complete. 2022-07-12T08:49:03.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 Processor -- start 2022-07-12T08:49:03.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.161+0000 7f53797e5700 1 -- start start 2022-07-12T08:49:03.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740ac710 0x7f53740b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f5374141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53740b1e40 0x7f53741419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53741425e0 con 0x7f53740ad590 2022-07-12T08:49:03.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537413b590 con 0x7f53740b1e40 2022-07-12T08:49:03.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53740b4250 con 0x7f53740ac710 2022-07-12T08:49:03.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f5374141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f5374141500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59466/0 (socket says 172.21.15.61:59466) 2022-07-12T08:49:03.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f5372ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740ac710 0x7f53740b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 -- 172.21.15.61:0/555147049 learned_addr learned my addr 172.21.15.61:0/555147049 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53737fe700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53740b1e40 0x7f53741419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 -- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740ac710 msgr2=0x7f53740b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740ac710 0x7f53740b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 -- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53740b1e40 msgr2=0x7f53741419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53740b1e40 0x7f53741419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 -- 172.21.15.61:0/555147049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c009530 con 0x7f53740ad590 2022-07-12T08:49:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f536a7fc700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f5374141500 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f537405ce90 tx=0x7f535c007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f535c007030 con 0x7f53740ad590 2022-07-12T08:49:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c007190 con 0x7f53740ad590 2022-07-12T08:49:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f537413b830 con 0x7f53740ad590 2022-07-12T08:49:03.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f535c006510 con 0x7f53740ad590 2022-07-12T08:49:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5374066ee0 con 0x7f53740ad590 2022-07-12T08:49:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.165+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53740b4410 con 0x7f53740ad590 2022-07-12T08:49:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.169+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f535c010030 con 0x7f53740ad590 2022-07-12T08:49:03.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.169+0000 7f5370ff9700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5358053940 0x7f5358055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.169+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f535c06b930 con 0x7f53740ad590 2022-07-12T08:49:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.169+0000 7f5372ffd700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5358053940 0x7f5358055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.169+0000 7f5372ffd700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5358053940 0x7f5358055e00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f536400b000 tx=0x7f5364006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.173+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f535c042890 con 0x7f53740ad590 2022-07-12T08:49:03.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.301+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5374057460 con 0x7f53740ad590 2022-07-12T08:49:03.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.305+0000 7f5370ff9700 1 -- 172.21.15.61:0/555147049 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16091 (secure 0 0 0) 0x7f535c045eb0 con 0x7f53740ad590 2022-07-12T08:49:03.311 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:03.312 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":100,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:00.114186+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:56.975401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:58.124077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:03.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5358053940 msgr2=0x7f5358055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5358053940 0x7f5358055e00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f536400b000 tx=0x7f5364006df0).stop 2022-07-12T08:49:03.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 msgr2=0x7f5374141500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f5374141500 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f537405ce90 tx=0x7f535c007c10).stop 2022-07-12T08:49:03.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 shutdown_connections 2022-07-12T08:49:03.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5358053940 0x7f5358055e00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53740b1e40 0x7f53741419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53740ad590 0x7f5374141500 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 --2- 172.21.15.61:0/555147049 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53740ac710 0x7f53740b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 >> 172.21.15.61:0/555147049 conn(0x7f5374062c00 msgr2=0x7f53740aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:03.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 shutdown_connections 2022-07-12T08:49:03.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.309+0000 7f53797e5700 1 -- 172.21.15.61:0/555147049 wait complete. 2022-07-12T08:49:03.332 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:48:56.975401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:48:58.124077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:49:03.334 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:49:03.405 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:03.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:03.414 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:03.417 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:03.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:03.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.445+0000 7fcefd381700 1 Processor -- start 2022-07-12T08:49:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.445+0000 7fcefd381700 1 -- start start 2022-07-12T08:49:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 0x7fcef80b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80b02a0 con 0x7fcef8097920 2022-07-12T08:49:03.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef80b0510 con 0x7fcef8139410 2022-07-12T08:49:03.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef809b280 con 0x7fcef805c9a0 2022-07-12T08:49:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcef8097920 0x7fcef8138d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47692/0 (socket says 172.21.15.61:47692) 2022-07-12T08:49:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef77fe700 1 -- 172.21.15.61:0/3027739154 learned_addr learned my addr 172.21.15.61:0/3027739154 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef7fff700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 0x7fcef80b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143256575 0 0) 0x7fcef80b02a0 con 0x7fcef8097920 2022-07-12T08:49:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced8003260 con 0x7fcef8097920 2022-07-12T08:49:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3396633350 0 0) 0x7fced8003260 con 0x7fcef8097920 2022-07-12T08:49:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 msgr2=0x7fcef80b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 0x7fcef80b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 >> v1:172.21.15.111:6789/0 conn(0x7fcef8139410 legacy=0x7fcef8142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef80ab650 con 0x7fcef8097920 2022-07-12T08:49:03.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fcee8001660 con 0x7fcef8097920 2022-07-12T08:49:03.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 >> v1:172.21.15.61:6789/0 conn(0x7fcef8097920 legacy=0x7fcef8138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 0x7fced8003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8003c70 0x7fced8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fced80049f0 0x7fced8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced800b750 con 0x7fced80049f0 2022-07-12T08:49:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced800b890 con 0x7fced8003c70 2022-07-12T08:49:03.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced800ba50 con 0x7fcef805c9a0 2022-07-12T08:49:03.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fced80049f0 0x7fced8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef77fe700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8003c70 0x7fced8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef7fff700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 0x7fced8003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 -- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 msgr2=0x7fced8003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:03.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef805c9a0 0x7fced8003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 -- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8003c70 msgr2=0x7fced8004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:03.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fced8003c70 0x7fced8004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 -- 172.21.15.61:0/3027739154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8003260 con 0x7fced80049f0 2022-07-12T08:49:03.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fced80049f0 0x7fced8004e70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fcef00013a0 tx=0x7fcef0007d60).ready entity=mon.0 client_cookie=6264f4aa1a636450 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef00016c0 con 0x7fced80049f0 2022-07-12T08:49:03.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0006890 con 0x7fced80049f0 2022-07-12T08:49:03.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcef6ffd700 1 -- 172.21.15.61:0/3027739154 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef0006bf0 con 0x7fced80049f0 2022-07-12T08:49:03.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- 172.21.15.61:0/3027739154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fced80049f0 msgr2=0x7fced8004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 --2- 172.21.15.61:0/3027739154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fced80049f0 0x7fced8004e70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 rx=0x7fcef00013a0 tx=0x7fcef0007d60).stop 2022-07-12T08:49:03.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefcb80700 1 -- 172.21.15.61:0/3027739154 reap_dead start 2022-07-12T08:49:03.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- 172.21.15.61:0/3027739154 shutdown_connections 2022-07-12T08:49:03.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- 172.21.15.61:0/3027739154 >> 172.21.15.61:0/3027739154 conn(0x7fcef805bd10 msgr2=0x7fcef805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- 172.21.15.61:0/3027739154 shutdown_connections 2022-07-12T08:49:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.449+0000 7fcefd381700 1 -- 172.21.15.61:0/3027739154 wait complete. 2022-07-12T08:49:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 Processor -- start 2022-07-12T08:49:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 -- start start 2022-07-12T08:49:03.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef80656e0 0x7fcef8063750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 0x7fcef8150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef8150a30 0x7fcef8066120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8065ac0 con 0x7fcef813e540 2022-07-12T08:49:03.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef8153f90 con 0x7fcef8150a30 2022-07-12T08:49:03.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef814fe00 con 0x7fcef80656e0 2022-07-12T08:49:03.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 0x7fcef8150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 0x7fcef8150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59486/0 (socket says 172.21.15.61:59486) 2022-07-12T08:49:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 -- 172.21.15.61:0/3155677151 learned_addr learned my addr 172.21.15.61:0/3155677151 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefcb80700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef8150a30 0x7fcef8066120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef7fff700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef80656e0 0x7fcef8063750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 -- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef80656e0 msgr2=0x7fcef8063750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef80656e0 0x7fcef8063750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 -- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef8150a30 msgr2=0x7fcef8066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef8150a30 0x7fcef8066120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 -- 172.21.15.61:0/3155677151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef81500a0 con 0x7fcef813e540 2022-07-12T08:49:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef7fff700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef80656e0 0x7fcef8063750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:03.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefcb80700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef8150a30 0x7fcef8066120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:03.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef77fe700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 0x7fcef8150540 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7fcee800b000 tx=0x7fcee8001e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcee8010680 con 0x7fcef813e540 2022-07-12T08:49:03.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee8010040 con 0x7fcef813e540 2022-07-12T08:49:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef8150e90 con 0x7fcef813e540 2022-07-12T08:49:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.453+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcee80109d0 con 0x7fcef813e540 2022-07-12T08:49:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef813a020 con 0x7fcef813e540 2022-07-12T08:49:03.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcee8010b30 con 0x7fcef813e540 2022-07-12T08:49:03.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef8153df0 con 0x7fcef813e540 2022-07-12T08:49:03.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcef57fa700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcedc053940 0x7fcedc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcee806bc80 con 0x7fcef813e540 2022-07-12T08:49:03.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcef7fff700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcedc053940 0x7fcedc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.457+0000 7fcef7fff700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcedc053940 0x7fcedc055e00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7fcee000b000 tx=0x7fcee0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.465+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcef8153df0 con 0x7fcef813e540 2022-07-12T08:49:03.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fcef8153df0 con 0x7fcef813e540 2022-07-12T08:49:03.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcef57fa700 1 -- 172.21.15.61:0/3155677151 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v100) v1 ==== 110+0+10 (secure 0 0 0) 0x7fcef8153df0 con 0x7fcef813e540 2022-07-12T08:49:03.600 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:49:03.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcedc053940 msgr2=0x7fcedc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcedc053940 0x7fcedc055e00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7fcee000b000 tx=0x7fcee0008040).stop 2022-07-12T08:49:03.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 msgr2=0x7fcef8150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 0x7fcef8150540 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7fcee800b000 tx=0x7fcee8001e50).stop 2022-07-12T08:49:03.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 shutdown_connections 2022-07-12T08:49:03.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcedc053940 0x7fcedc055e00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcef8150a30 0x7fcef8066120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcef813e540 0x7fcef8150540 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 --2- 172.21.15.61:0/3155677151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcef80656e0 0x7fcef8063750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 >> 172.21.15.61:0/3155677151 conn(0x7fcef805bd10 msgr2=0x7fcef80aa050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:03.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 shutdown_connections 2022-07-12T08:49:03.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.593+0000 7fcefd381700 1 -- 172.21.15.61:0/3155677151 wait complete. 2022-07-12T08:49:03.612 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:49:03.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 Processor -- start 2022-07-12T08:49:03.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 -- start start 2022-07-12T08:49:03.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf940af3d0 0x7fcf94144200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf9405f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940b2220 con 0x7fcf940afb90 2022-07-12T08:49:03.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf94140b90 con 0x7fcf940af3d0 2022-07-12T08:49:03.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf9aa5e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940af7b0 con 0x7fcf9405bd10 2022-07-12T08:49:03.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf93fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf940af3d0 0x7fcf94144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf93fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf940af3d0 0x7fcf94144200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:38066/0 (socket says 172.21.15.61:38066) 2022-07-12T08:49:03.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf93fff700 1 -- 172.21.15.61:0/3804164938 learned_addr learned my addr 172.21.15.61:0/3804164938 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:03.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf98ffb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf9405f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf92ffd700 1 -- 172.21.15.61:0/3804164938 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957087587 0 0) 0x7fcf940b2220 con 0x7fcf940afb90 2022-07-12T08:49:03.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf92ffd700 1 -- 172.21.15.61:0/3804164938 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80003260 con 0x7fcf940afb90 2022-07-12T08:49:03.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf98ffb700 1 -- 172.21.15.61:0/3804164938 >> v2:172.21.15.111:3300/0 conn(0x7fcf940af3d0 msgr2=0x7fcf94144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:03.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf98ffb700 1 --2- 172.21.15.61:0/3804164938 >> v2:172.21.15.111:3300/0 conn(0x7fcf940af3d0 0x7fcf94144200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf98ffb700 1 -- 172.21.15.61:0/3804164938 >> v1:172.21.15.111:6789/0 conn(0x7fcf940afb90 legacy=0x7fcf9405f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.721+0000 7fcf98ffb700 1 -- 172.21.15.61:0/3804164938 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf940ac7c0 con 0x7fcf9405bd10 2022-07-12T08:49:03.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf98ffb700 1 --2- 172.21.15.61:0/3804164938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf9405f930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fcf8c0032f0 tx=0x7fcf8c0091d0).ready entity=mon.2 client_cookie=7d8e222f0695b79e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf92ffd700 1 -- 172.21.15.61:0/3804164938 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf8c00b600 con 0x7fcf9405bd10 2022-07-12T08:49:03.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf92ffd700 1 -- 172.21.15.61:0/3804164938 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c00b760 con 0x7fcf9405bd10 2022-07-12T08:49:03.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf92ffd700 1 -- 172.21.15.61:0/3804164938 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf8c00b8d0 con 0x7fcf9405bd10 2022-07-12T08:49:03.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/3804164938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 msgr2=0x7fcf9405f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/3804164938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf9405f930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fcf8c0032f0 tx=0x7fcf8c0091d0).stop 2022-07-12T08:49:03.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/3804164938 shutdown_connections 2022-07-12T08:49:03.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/3804164938 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf9405f930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/3804164938 >> v2:172.21.15.111:3300/0 conn(0x7fcf940af3d0 0x7fcf94144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/3804164938 >> 172.21.15.61:0/3804164938 conn(0x7fcf940b5a40 msgr2=0x7fcf940b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:03.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/3804164938 shutdown_connections 2022-07-12T08:49:03.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.725+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/3804164938 wait complete. 2022-07-12T08:49:03.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 Processor -- start 2022-07-12T08:49:03.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 -- start start 2022-07-12T08:49:03.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf940b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 0x7fcf940aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf940afb90 0x7fcf940a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940a8fe0 con 0x7fcf940af3d0 2022-07-12T08:49:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940adbf0 con 0x7fcf940afb90 2022-07-12T08:49:03.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf9aa5e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf94151de0 con 0x7fcf9405bd10 2022-07-12T08:49:03.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 0x7fcf940aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf93fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf940b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 0x7fcf940aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59500/0 (socket says 172.21.15.61:59500) 2022-07-12T08:49:03.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 -- 172.21.15.61:0/1225898449 learned_addr learned my addr 172.21.15.61:0/1225898449 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:03.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf98ffb700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf940afb90 0x7fcf940a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 -- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 msgr2=0x7fcf940b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf940b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 -- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf940afb90 msgr2=0x7fcf940a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf940afb90 0x7fcf940a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 -- 172.21.15.61:0/1225898449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c009470 con 0x7fcf940af3d0 2022-07-12T08:49:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.729+0000 7fcf937fe700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 0x7fcf940aa6c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7fcf88001110 tx=0x7fcf88007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf88012400 con 0x7fcf940af3d0 2022-07-12T08:49:03.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88012560 con 0x7fcf940af3d0 2022-07-12T08:49:03.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf940aabb0 con 0x7fcf940af3d0 2022-07-12T08:49:03.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf880128c0 con 0x7fcf940af3d0 2022-07-12T08:49:03.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf940aa090 con 0x7fcf940af3d0 2022-07-12T08:49:03.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcf88019030 con 0x7fcf940af3d0 2022-07-12T08:49:03.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf941522c0 con 0x7fcf940af3d0 2022-07-12T08:49:03.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf917fa700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf74053ad0 0x7fcf74055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:03.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf8806b2f0 con 0x7fcf940af3d0 2022-07-12T08:49:03.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf93fff700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf74053ad0 0x7fcf74055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:03.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.733+0000 7fcf93fff700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf74053ad0 0x7fcf74055f90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fcf84008000 tx=0x7fcf84008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:03.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.741+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf88042140 con 0x7fcf940af3d0 2022-07-12T08:49:03.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.865+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fcf940b6580 con 0x7fcf940af3d0 2022-07-12T08:49:03.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.865+0000 7fcf917fa700 1 -- 172.21.15.61:0/1225898449 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7fcf88045760 con 0x7fcf940af3d0 2022-07-12T08:49:03.872 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:03.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf74053ad0 msgr2=0x7fcf74055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf74053ad0 0x7fcf74055f90 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fcf84008000 tx=0x7fcf84008040).stop 2022-07-12T08:49:03.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 msgr2=0x7fcf940aa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:03.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 0x7fcf940aa6c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7fcf88001110 tx=0x7fcf88007ae0).stop 2022-07-12T08:49:03.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 shutdown_connections 2022-07-12T08:49:03.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf74053ad0 0x7fcf74055f90 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf940afb90 0x7fcf940a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf940af3d0 0x7fcf940aa6c0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 --2- 172.21.15.61:0/1225898449 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf9405bd10 0x7fcf940b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:03.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 >> 172.21.15.61:0/1225898449 conn(0x7fcf940b5a40 msgr2=0x7fcf9405f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:03.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 shutdown_connections 2022-07-12T08:49:03.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.869+0000 7fcf9aa5e700 1 -- 172.21.15.61:0/1225898449 wait complete. 2022-07-12T08:49:03.886 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:49:04.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 Processor -- start 2022-07-12T08:49:04.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 -- start start 2022-07-12T08:49:04.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f040b5920 0x7f6f0414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b0280 0x7f6f0414aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0405c3e0 con 0x7f6f0405faf0 2022-07-12T08:49:04.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f040ab840 con 0x7f6f040b5920 2022-07-12T08:49:04.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040b5d00 con 0x7f6f040b0280 2022-07-12T08:49:04.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f040b5920 0x7f6f0414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f040b5920 0x7f6f0414a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59506/0 (socket says 172.21.15.61:59506) 2022-07-12T08:49:04.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 -- 172.21.15.61:0/3763078372 learned_addr learned my addr 172.21.15.61:0/3763078372 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:04.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f01d9b700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b0280 0x7f6f0414aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 -- 172.21.15.61:0/3763078372 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b0280 msgr2=0x7f6f0414aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:04.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b0280 0x7f6f0414aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 -- 172.21.15.61:0/3763078372 >> v1:172.21.15.111:6789/0 conn(0x7f6f0405faf0 legacy=0x7f6f0405ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 -- 172.21.15.61:0/3763078372 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0405c610 con 0x7f6f040b5920 2022-07-12T08:49:04.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f01d9b700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b0280 0x7f6f0414aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:04.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0259c700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b5920 0x7f6f0414a9e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f6efc002f70 tx=0x7f6efc006eb0).ready entity=mon.0 client_cookie=a162dbdabd170f64 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0159a700 1 -- 172.21.15.61:0/3763078372 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6efc00d5c0 con 0x7f6f040b5920 2022-07-12T08:49:04.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0159a700 1 -- 172.21.15.61:0/3763078372 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc000e90 con 0x7f6f040b5920 2022-07-12T08:49:04.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.993+0000 7f6f0159a700 1 -- 172.21.15.61:0/3763078372 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6efc00d950 con 0x7f6f040b5920 2022-07-12T08:49:04.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 -- 172.21.15.61:0/3763078372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b5920 msgr2=0x7f6f0414a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b5920 0x7f6f0414a9e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 rx=0x7f6efc002f70 tx=0x7f6efc006eb0).stop 2022-07-12T08:49:04.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 -- 172.21.15.61:0/3763078372 shutdown_connections 2022-07-12T08:49:04.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b0280 0x7f6f0414aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3763078372 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b5920 0x7f6f0414a9e0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 -- 172.21.15.61:0/3763078372 >> 172.21.15.61:0/3763078372 conn(0x7f6f04063340 msgr2=0x7f6f040b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:04.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 -- 172.21.15.61:0/3763078372 shutdown_connections 2022-07-12T08:49:04.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 -- 172.21.15.61:0/3763078372 wait complete. 2022-07-12T08:49:04.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 Processor -- start 2022-07-12T08:49:04.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:03.997+0000 7f6f0359e700 1 -- start start 2022-07-12T08:49:04.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0405faf0 0x7f6f04156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 0x7f6f040ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b5920 0x7f6f040adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040ad150 con 0x7f6f040b0280 2022-07-12T08:49:04.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040ac640 con 0x7f6f0405faf0 2022-07-12T08:49:04.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040ae440 con 0x7f6f040b5920 2022-07-12T08:49:04.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 0x7f6f040ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 0x7f6f040ad7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59514/0 (socket says 172.21.15.61:59514) 2022-07-12T08:49:04.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 -- 172.21.15.61:0/3549698643 learned_addr learned my addr 172.21.15.61:0/3549698643 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0259c700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0405faf0 0x7f6f04156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f02d9d700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b5920 0x7f6f040adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 -- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b5920 msgr2=0x7f6f040adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b5920 0x7f6f040adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 -- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0405faf0 msgr2=0x7f6f04156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0405faf0 0x7f6f04156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 -- 172.21.15.61:0/3549698643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc00b1e0 con 0x7f6f040b0280 2022-07-12T08:49:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f01d9b700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 0x7f6f040ad7f0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f6eec007000 tx=0x7f6eec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6eec010580 con 0x7f6f040b0280 2022-07-12T08:49:04.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec0106e0 con 0x7f6f040b0280 2022-07-12T08:49:04.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f040ae670 con 0x7f6f040b0280 2022-07-12T08:49:04.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6eec010a40 con 0x7f6f040b0280 2022-07-12T08:49:04.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f04067a20 con 0x7f6f040b0280 2022-07-12T08:49:04.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.001+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f04057460 con 0x7f6f040b0280 2022-07-12T08:49:04.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.005+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6eec010ba0 con 0x7f6f040b0280 2022-07-12T08:49:04.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.005+0000 7f6efb7fe700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ee8053a60 0x7f6ee8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.005+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6eec06bfe0 con 0x7f6f040b0280 2022-07-12T08:49:04.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.005+0000 7f6f0259c700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ee8053a60 0x7f6ee8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.005+0000 7f6f0259c700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ee8053a60 0x7f6ee8055f20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f6efc01c000 tx=0x7f6efc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.013+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6eec042ea0 con 0x7f6f040b0280 2022-07-12T08:49:04.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.133+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6f0406b2b0 con 0x7f6f040b0280 2022-07-12T08:49:04.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.133+0000 7f6efb7fe700 1 -- 172.21.15.61:0/3549698643 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7f6eec0464c0 con 0x7f6f040b0280 2022-07-12T08:49:04.141 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:04.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ee8053a60 msgr2=0x7f6ee8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ee8053a60 0x7f6ee8055f20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f6efc01c000 tx=0x7f6efc015000).stop 2022-07-12T08:49:04.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 msgr2=0x7f6f040ad7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 0x7f6f040ad7f0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7f6eec007000 tx=0x7f6eec008040).stop 2022-07-12T08:49:04.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 shutdown_connections 2022-07-12T08:49:04.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f6f040b5920 0x7f6f040adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f040b0280 0x7f6f040ad7f0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6ee8053a60 0x7f6ee8055f20 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 --2- 172.21.15.61:0/3549698643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6f0405faf0 0x7f6f04156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 >> 172.21.15.61:0/3549698643 conn(0x7f6f04063340 msgr2=0x7f6f040985d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:04.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 shutdown_connections 2022-07-12T08:49:04.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.137+0000 7f6f0359e700 1 -- 172.21.15.61:0/3549698643 wait complete. 2022-07-12T08:49:04.154 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:49:04.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba969700 1 Processor -- start 2022-07-12T08:49:04.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba969700 1 -- start start 2022-07-12T08:49:04.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba969700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 0x7fbab405c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba969700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab4143500 con 0x7fbab40b6130 2022-07-12T08:49:04.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba969700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab4143290 con 0x7fbab4142eb0 2022-07-12T08:49:04.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba969700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b65c0 con 0x7fbab405d130 2022-07-12T08:49:04.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba168700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbab40b6130 0x7fbab40b1ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47742/0 (socket says 172.21.15.61:47742) 2022-07-12T08:49:04.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbaba168700 1 -- 172.21.15.61:0/3056183668 learned_addr learned my addr 172.21.15.61:0/3056183668 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:04.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.257+0000 7fbab9166700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 0x7fbab405c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941050870 0 0) 0x7fbab4143500 con 0x7fbab40b6130 2022-07-12T08:49:04.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba9c003260 con 0x7fbab40b6130 2022-07-12T08:49:04.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4164705040 0 0) 0x7fba9c003260 con 0x7fbab40b6130 2022-07-12T08:49:04.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 msgr2=0x7fbab405c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:04.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 0x7fbab405c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 >> v1:172.21.15.111:6789/0 conn(0x7fbab4142eb0 legacy=0x7fbab40b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab4143760 con 0x7fbab40b6130 2022-07-12T08:49:04.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fbaac0019e0 con 0x7fbab40b6130 2022-07-12T08:49:04.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 >> v1:172.21.15.61:6789/0 conn(0x7fbab40b6130 legacy=0x7fbab40b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba9c003780 0x7fba9c003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba9c004500 0x7fba9c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 0x7fba9c004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40b65c0 con 0x7fba9c003780 2022-07-12T08:49:04.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c00b800 con 0x7fba9c004500 2022-07-12T08:49:04.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba9c00b990 con 0x7fbab405d130 2022-07-12T08:49:04.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba9c003780 0x7fba9c003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9166700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba9c004500 0x7fba9c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba168700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 0x7fba9c004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 -- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 msgr2=0x7fba9c004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:04.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab405d130 0x7fba9c004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 -- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba9c004500 msgr2=0x7fba9c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:04.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba9c004500 0x7fba9c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 -- 172.21.15.61:0/3056183668 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c003260 con 0x7fba9c003780 2022-07-12T08:49:04.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab9967700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba9c003780 0x7fba9c003ba0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fba9c0054d0 tx=0x7fbaa4007a00).ready entity=mon.0 client_cookie=3c249aa89db9f0d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaa4007d10 con 0x7fba9c003780 2022-07-12T08:49:04.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa4007070 con 0x7fba9c003780 2022-07-12T08:49:04.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbab8965700 1 -- 172.21.15.61:0/3056183668 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaa40073d0 con 0x7fba9c003780 2022-07-12T08:49:04.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 -- 172.21.15.61:0/3056183668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba9c003780 msgr2=0x7fba9c003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 --2- 172.21.15.61:0/3056183668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fba9c003780 0x7fba9c003ba0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7fba9c0054d0 tx=0x7fbaa4007a00).stop 2022-07-12T08:49:04.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba168700 1 -- 172.21.15.61:0/3056183668 reap_dead start 2022-07-12T08:49:04.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 -- 172.21.15.61:0/3056183668 shutdown_connections 2022-07-12T08:49:04.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 -- 172.21.15.61:0/3056183668 >> 172.21.15.61:0/3056183668 conn(0x7fbab4066d70 msgr2=0x7fbab4067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:04.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 -- 172.21.15.61:0/3056183668 shutdown_connections 2022-07-12T08:49:04.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 -- 172.21.15.61:0/3056183668 wait complete. 2022-07-12T08:49:04.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 Processor -- start 2022-07-12T08:49:04.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.261+0000 7fbaba969700 1 -- start start 2022-07-12T08:49:04.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 0x7fbab4146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab41483d0 0x7fbab4148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbab4152520 0x7fbab413bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4155ab0 con 0x7fbab4155380 2022-07-12T08:49:04.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4155d10 con 0x7fbab4152520 2022-07-12T08:49:04.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab4155760 con 0x7fbab41483d0 2022-07-12T08:49:04.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 0x7fbab4146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9166700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab41483d0 0x7fbab4148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 0x7fbab4146ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59536/0 (socket says 172.21.15.61:59536) 2022-07-12T08:49:04.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 -- 172.21.15.61:0/2351880065 learned_addr learned my addr 172.21.15.61:0/2351880065 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:04.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba168700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbab4152520 0x7fbab413bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 -- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab41483d0 msgr2=0x7fbab4148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab41483d0 0x7fbab4148d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 -- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbab4152520 msgr2=0x7fbab413bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbab4152520 0x7fbab413bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 -- 172.21.15.61:0/2351880065 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40b1a50 con 0x7fbab4155380 2022-07-12T08:49:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbab9967700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 0x7fbab4146ad0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7fba9c0054d0 tx=0x7fbaa4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaa4020600 con 0x7fbab4155380 2022-07-12T08:49:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa4020760 con 0x7fbab4155380 2022-07-12T08:49:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbaa4020ac0 con 0x7fbab4155380 2022-07-12T08:49:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab40b1c40 con 0x7fbab4155380 2022-07-12T08:49:04.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab4152bf0 con 0x7fbab4155380 2022-07-12T08:49:04.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.265+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab4155f10 con 0x7fbab4155380 2022-07-12T08:49:04.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.269+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbaa4020c20 con 0x7fbab4155380 2022-07-12T08:49:04.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.269+0000 7fbaa2ffd700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fba94053940 0x7fba94055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.269+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbaa401c040 con 0x7fbab4155380 2022-07-12T08:49:04.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.269+0000 7fbab9166700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fba94053940 0x7fba94055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.269+0000 7fbab9166700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fba94053940 0x7fba94055e00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fbab000b000 tx=0x7fbab0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.273+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbaa40436a0 con 0x7fbab4155380 2022-07-12T08:49:04.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.397+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbab4057460 con 0x7fbab4155380 2022-07-12T08:49:04.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaa2ffd700 1 -- 172.21.15.61:0/2351880065 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7fbaa4025030 con 0x7fbab4155380 2022-07-12T08:49:04.406 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fba94053940 msgr2=0x7fba94055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fba94053940 0x7fba94055e00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fbab000b000 tx=0x7fbab0006df0).stop 2022-07-12T08:49:04.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 msgr2=0x7fbab4146ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 0x7fbab4146ad0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7fba9c0054d0 tx=0x7fbaa4015000).stop 2022-07-12T08:49:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 shutdown_connections 2022-07-12T08:49:04.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fba94053940 0x7fba94055e00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbab4152520 0x7fbab413bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbab41483d0 0x7fbab4148d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 --2- 172.21.15.61:0/2351880065 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbab4155380 0x7fbab4146ad0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 >> 172.21.15.61:0/2351880065 conn(0x7fbab4066d70 msgr2=0x7fbab40b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:04.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 shutdown_connections 2022-07-12T08:49:04.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.401+0000 7fbaba969700 1 -- 172.21.15.61:0/2351880065 wait complete. 2022-07-12T08:49:04.421 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:49:04.422 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:04.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 Processor -- start 2022-07-12T08:49:04.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 -- start start 2022-07-12T08:49:04.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b1080 0x7fa5e80b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 0x7fa5e80afb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80b5740 con 0x7fa5e80b4090 2022-07-12T08:49:04.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e805cc00 con 0x7fa5e80b1080 2022-07-12T08:49:04.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ee787700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80b1460 con 0x7fa5e8142f30 2022-07-12T08:49:04.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5e7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b1080 0x7fa5e80b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ecd24700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 0x7fa5e80afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ecd24700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 0x7fa5e80afb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:41740/0 (socket says 172.21.15.61:41740) 2022-07-12T08:49:04.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5e7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b1080 0x7fa5e80b62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:38114/0 (socket says 172.21.15.61:38114) 2022-07-12T08:49:04.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.521+0000 7fa5ecd24700 1 -- 172.21.15.61:0/1560380644 learned_addr learned my addr 172.21.15.61:0/1560380644 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:04.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ecd24700 1 -- 172.21.15.61:0/1560380644 >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b1080 msgr2=0x7fa5e80b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:04.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ecd24700 1 --2- 172.21.15.61:0/1560380644 >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b1080 0x7fa5e80b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ecd24700 1 -- 172.21.15.61:0/1560380644 >> v1:172.21.15.111:6789/0 conn(0x7fa5e80b4090 legacy=0x7fa5e805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ecd24700 1 -- 172.21.15.61:0/1560380644 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80b6880 con 0x7fa5e8142f30 2022-07-12T08:49:04.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ecd24700 1 --2- 172.21.15.61:0/1560380644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 0x7fa5e80afb90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa5e00032f0 tx=0x7fa5e0009290).ready entity=mon.2 client_cookie=46dba86c62af427f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/1560380644 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5e000b8a0 con 0x7fa5e8142f30 2022-07-12T08:49:04.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/1560380644 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e000ba00 con 0x7fa5e8142f30 2022-07-12T08:49:04.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5e6ffd700 1 -- 172.21.15.61:0/1560380644 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5e000bb70 con 0x7fa5e8142f30 2022-07-12T08:49:04.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 -- 172.21.15.61:0/1560380644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 msgr2=0x7fa5e80afb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 --2- 172.21.15.61:0/1560380644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 0x7fa5e80afb90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa5e00032f0 tx=0x7fa5e0009290).stop 2022-07-12T08:49:04.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 -- 172.21.15.61:0/1560380644 shutdown_connections 2022-07-12T08:49:04.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 --2- 172.21.15.61:0/1560380644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e8142f30 0x7fa5e80afb90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 --2- 172.21.15.61:0/1560380644 >> v2:172.21.15.111:3300/0 conn(0x7fa5e80b1080 0x7fa5e80b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 -- 172.21.15.61:0/1560380644 >> 172.21.15.61:0/1560380644 conn(0x7fa5e8067020 msgr2=0x7fa5e8067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:04.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 -- 172.21.15.61:0/1560380644 shutdown_connections 2022-07-12T08:49:04.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 -- 172.21.15.61:0/1560380644 wait complete. 2022-07-12T08:49:04.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 Processor -- start 2022-07-12T08:49:04.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.525+0000 7fa5ee787700 1 -- start start 2022-07-12T08:49:04.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b1080 0x7fa5e8145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b4090 0x7fa5e8141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 0x7fa5e8142360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8156a50 con 0x7fa5e8142f30 2022-07-12T08:49:04.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8115770 con 0x7fa5e80b4090 2022-07-12T08:49:04.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8142850 con 0x7fa5e80b1080 2022-07-12T08:49:04.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 0x7fa5e8142360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 0x7fa5e8142360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59550/0 (socket says 172.21.15.61:59550) 2022-07-12T08:49:04.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 -- 172.21.15.61:0/2077209804 learned_addr learned my addr 172.21.15.61:0/2077209804 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:04.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e7fff700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b1080 0x7fa5e8145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e77fe700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b4090 0x7fa5e8141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 -- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b1080 msgr2=0x7fa5e8145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b1080 0x7fa5e8145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 -- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b4090 msgr2=0x7fa5e8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b4090 0x7fa5e8141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 -- 172.21.15.61:0/2077209804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e0009530 con 0x7fa5e8142f30 2022-07-12T08:49:04.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e7fff700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b1080 0x7fa5e8145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:04.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e77fe700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b4090 0x7fa5e8141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:04.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ecd24700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 0x7fa5e8142360 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fa5e00032f0 tx=0x7fa5e0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5e0021430 con 0x7fa5e8142f30 2022-07-12T08:49:04.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e0021590 con 0x7fa5e8142f30 2022-07-12T08:49:04.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e80ae970 con 0x7fa5e8142f30 2022-07-12T08:49:04.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5e00218f0 con 0x7fa5e8142f30 2022-07-12T08:49:04.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e80aeec0 con 0x7fa5e8142f30 2022-07-12T08:49:04.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.529+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e81155b0 con 0x7fa5e8142f30 2022-07-12T08:49:04.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.533+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa5e0028070 con 0x7fa5e8142f30 2022-07-12T08:49:04.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.533+0000 7fa5e57fa700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a40 0x7fa5cc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:04.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.533+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5e006d1e0 con 0x7fa5e8142f30 2022-07-12T08:49:04.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.533+0000 7fa5e7fff700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a40 0x7fa5cc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:04.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.533+0000 7fa5e7fff700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a40 0x7fa5cc055f00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fa5d800b000 tx=0x7fa5d8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:04.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.541+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5e0044140 con 0x7fa5e8142f30 2022-07-12T08:49:04.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.701+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5e8115930 con 0x7fa5e8142f30 2022-07-12T08:49:04.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5e57fa700 1 -- 172.21.15.61:0/2077209804 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fa5e0015020 con 0x7fa5e8142f30 2022-07-12T08:49:04.710 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:04.710 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:59.111339+0000","modified":"2022-07-12T08:49:00.805218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2528523999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2528523999},{"type":"v1","addr":"172.21.15.61:6833","nonce":2528523999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-12T08:49:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a40 msgr2=0x7fa5cc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a40 0x7fa5cc055f00 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7fa5d800b000 tx=0x7fa5d8008040).stop 2022-07-12T08:49:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 msgr2=0x7fa5e8142360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 0x7fa5e8142360 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 rx=0x7fa5e00032f0 tx=0x7fa5e0016000).stop 2022-07-12T08:49:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 shutdown_connections 2022-07-12T08:49:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8142f30 0x7fa5e8142360 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5cc053a40 0x7fa5cc055f00 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e80b4090 0x7fa5e8141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 --2- 172.21.15.61:0/2077209804 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e80b1080 0x7fa5e8145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 >> 172.21.15.61:0/2077209804 conn(0x7fa5e8067020 msgr2=0x7fa5e8149bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 shutdown_connections 2022-07-12T08:49:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:04.705+0000 7fa5ee787700 1 -- 172.21.15.61:0/2077209804 wait complete. 2022-07-12T08:49:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-12T08:49:04.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:59.111339+0000', 'modified': '2022-07-12T08:49:00.805218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7338}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7338': {'gid': 7338, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2528523999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2528523999}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2528523999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:49:04.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:49:04.925 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:49:05.041 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":7338,"want_state":"up:active","state":"up:active","rank_uptime":5.9156806179999997,"mdsmap_epoch":56,"osdmap_epoch":100,"osdmap_epoch_barrier":100,"uptime":10.850905885} 2022-07-12T08:49:05.043 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 7338, "mdsmap_epoch": 56, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "rank_uptime": 5.915680618, "state": "up:active", "uptime": 10.850905885, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:49:05.044 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:49:05.044 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:49:05.045 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:49:05.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:05.047 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:05.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:49:05.117 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:05.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:05.134 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:05.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:05.138 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:49:05.186 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:49:05.187 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:05.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:49:05.214 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:05.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:05.267 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:49:05.264+0000 7fc25a2e1080 -1 init, newargv = 0x5580affef770 newargc=9 2022-07-12T08:49:05.268 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21937]: starting ceph client 2022-07-12T08:49:05.268 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:05.270 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:05.271 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:49:05.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21937]: starting fuse 2022-07-12T08:49:06.379 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:49:06.399 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:06.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:06.472 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:06.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:06.474 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:49:06.485 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:49:06.485 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:49:06.486 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:06.488 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:49:06.489 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:49:06.489 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:49:06.490 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:49:06.491 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:49:06.491 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:06.492 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:49:06.493 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:49:06.493 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:49:06.494 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:06.494 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:49:06.495 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:49:06.495 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:49:06.496 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:49:06.497 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:06.497 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:49:06.498 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:49:06.499 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:49:06.499 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:49:06.499 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:49:06.500 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:06.501 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:49:06.502 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:49:06.573 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.21937.asok 2022-07-12T08:49:06.578 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.21937.asok 2022-07-12T08:49:06.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:06.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.21937.asok status 2022-07-12T08:49:06.669 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:49:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:49:06.670 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:49:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:49:06.671 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:49:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:49:06.672 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:49:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "21937", 2022-07-12T08:49:06.673 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:49:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:06.675 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:49:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:49:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 6554, 2022-07-12T08:49:06.676 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:49:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:49:06.677 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:49:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 6554 2022-07-12T08:49:06.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:49:06.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:49:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:49:06.680 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2759205964 2022-07-12T08:49:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:49:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:06.681 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:49:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:49:06.682 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:49:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2759205964 2022-07-12T08:49:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:06.683 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.6554 172.21.15.121:0/2759205964", 2022-07-12T08:49:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/2759205964", 2022-07-12T08:49:06.684 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:49:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 56, 2022-07-12T08:49:06.685 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 100, 2022-07-12T08:49:06.687 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:49:06.688 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:49:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:49:06.689 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:49:06.691 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:06.693 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:06.693 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:06.703 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:49:06.704 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:06.705 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:06.706 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:06.765 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:06.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 Processor -- start 2022-07-12T08:49:06.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 -- start start 2022-07-12T08:49:06.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea680ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:06.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea6805f140 0x7fea6805ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:06.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea680abc70 con 0x7fea680afa60 2022-07-12T08:49:06.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6805f580 con 0x7fea6805f140 2022-07-12T08:49:06.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.869+0000 7fea6d89c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6805c720 con 0x7fea680ab290 2022-07-12T08:49:06.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea6805f140 0x7fea6805ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:06.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea6805f140 0x7fea6805ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59554/0 (socket says 172.21.15.61:59554) 2022-07-12T08:49:06.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d09b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fea680afa60 0x7fea6814a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47774/0 (socket says 172.21.15.61:47774) 2022-07-12T08:49:06.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 -- 172.21.15.61:0/2480023190 learned_addr learned my addr 172.21.15.61:0/2480023190 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:06.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6c89a700 1 --2- 172.21.15.61:0/2480023190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea680ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:06.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 -- 172.21.15.61:0/2480023190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 msgr2=0x7fea680ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:06.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 --2- 172.21.15.61:0/2480023190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea680ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:06.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 -- 172.21.15.61:0/2480023190 >> v1:172.21.15.61:6789/0 conn(0x7fea680afa60 legacy=0x7fea6814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:06.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea677fe700 1 -- 172.21.15.61:0/2480023190 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772713279 0 0) 0x7fea680abc70 con 0x7fea680afa60 2022-07-12T08:49:06.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 -- 172.21.15.61:0/2480023190 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6805c9c0 con 0x7fea6805f140 2022-07-12T08:49:06.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea67fff700 1 --2- 172.21.15.61:0/2480023190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6805f140 0x7fea6805ce60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7fea58002f70 tx=0x7fea58006eb0).ready entity=mon.0 client_cookie=27f2bf5944168700 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:06.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea677fe700 1 -- 172.21.15.61:0/2480023190 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea5800d570 con 0x7fea6805f140 2022-07-12T08:49:06.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea677fe700 1 -- 172.21.15.61:0/2480023190 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea58000e90 con 0x7fea6805f140 2022-07-12T08:49:06.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea677fe700 1 -- 172.21.15.61:0/2480023190 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea5800d8c0 con 0x7fea6805f140 2022-07-12T08:49:06.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 -- 172.21.15.61:0/2480023190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6805f140 msgr2=0x7fea6805ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:06.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 --2- 172.21.15.61:0/2480023190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6805f140 0x7fea6805ce60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 rx=0x7fea58002f70 tx=0x7fea58006eb0).stop 2022-07-12T08:49:06.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 -- 172.21.15.61:0/2480023190 shutdown_connections 2022-07-12T08:49:06.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 --2- 172.21.15.61:0/2480023190 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea6805f140 0x7fea6805ce60 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:06.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 --2- 172.21.15.61:0/2480023190 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea680ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:06.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 -- 172.21.15.61:0/2480023190 >> 172.21.15.61:0/2480023190 conn(0x7fea680b1950 msgr2=0x7fea680b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:06.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 -- 172.21.15.61:0/2480023190 shutdown_connections 2022-07-12T08:49:06.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.873+0000 7fea6d89c700 1 -- 172.21.15.61:0/2480023190 wait complete. 2022-07-12T08:49:06.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 Processor -- start 2022-07-12T08:49:06.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 -- start start 2022-07-12T08:49:06.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6805f140 0x7fea68099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:06.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea68155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:06.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 0x7fea68156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:06.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea68065f20 con 0x7fea680afa60 2022-07-12T08:49:06.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea68149db0 con 0x7fea6805f140 2022-07-12T08:49:06.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6814a050 con 0x7fea680ab290 2022-07-12T08:49:06.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 0x7fea68156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:06.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 0x7fea68156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59564/0 (socket says 172.21.15.61:59564) 2022-07-12T08:49:06.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 -- 172.21.15.61:0/471255765 learned_addr learned my addr 172.21.15.61:0/471255765 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:06.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6c89a700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6805f140 0x7fea68099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:06.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea67fff700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea68155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 -- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 msgr2=0x7fea68155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea68155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:06.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 -- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6805f140 msgr2=0x7fea68099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:06.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6805f140 0x7fea68099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:06.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 -- 172.21.15.61:0/471255765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5800b1e0 con 0x7fea680afa60 2022-07-12T08:49:06.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d09b700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 0x7fea68156260 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fea60001360 tx=0x7fea60007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:06.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea60005ae0 con 0x7fea680afa60 2022-07-12T08:49:06.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea60005c40 con 0x7fea680afa60 2022-07-12T08:49:06.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6814a2f0 con 0x7fea680afa60 2022-07-12T08:49:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea60007110 con 0x7fea680afa60 2022-07-12T08:49:06.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.877+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea68156970 con 0x7fea680afa60 2022-07-12T08:49:06.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.881+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6814a480 con 0x7fea680afa60 2022-07-12T08:49:06.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.881+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fea60010030 con 0x7fea680afa60 2022-07-12T08:49:06.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.881+0000 7fea65ffb700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea480539b0 0x7fea48055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:06.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.881+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fea6006be70 con 0x7fea680afa60 2022-07-12T08:49:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.881+0000 7fea6c89a700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea480539b0 0x7fea48055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.881+0000 7fea6c89a700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea480539b0 0x7fea48055e70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7fea5c00b000 tx=0x7fea5c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:06.889+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea60042cc0 con 0x7fea680afa60 2022-07-12T08:49:07.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.013+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea68156970 con 0x7fea680afa60 2022-07-12T08:49:07.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea65ffb700 1 -- 172.21.15.61:0/471255765 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16091 (secure 0 0 0) 0x7fea68156970 con 0x7fea680afa60 2022-07-12T08:49:07.024 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:07.025 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":100,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:00.114186+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:48:56.975401+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-12T08:48:58.124077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:07.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea480539b0 msgr2=0x7fea48055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea6d89c700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea480539b0 0x7fea48055e70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7fea5c00b000 tx=0x7fea5c008040).stop 2022-07-12T08:49:07.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 msgr2=0x7fea68156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea6d89c700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 0x7fea68156260 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7fea60001360 tx=0x7fea60007f40).stop 2022-07-12T08:49:07.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 shutdown_connections 2022-07-12T08:49:07.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.017+0000 7fea6d89c700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea680afa60 0x7fea68156260 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.021+0000 7fea6d89c700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fea680ab290 0x7fea68155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.021+0000 7fea6d89c700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fea480539b0 0x7fea48055e70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.021+0000 7fea6d89c700 1 --2- 172.21.15.61:0/471255765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fea6805f140 0x7fea68099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.021+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 >> 172.21.15.61:0/471255765 conn(0x7fea680b1950 msgr2=0x7fea680b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.021+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 shutdown_connections 2022-07-12T08:49:07.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.021+0000 7fea6d89c700 1 -- 172.21.15.61:0/471255765 wait complete. 2022-07-12T08:49:07.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:48:56.975401+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:48:58.124077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:49:07.039 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:49:07.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.141+0000 7fa5eecc5700 1 Processor -- start 2022-07-12T08:49:07.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.141+0000 7fa5eecc5700 1 -- start start 2022-07-12T08:49:07.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5e8060a70 0x7fa5e809a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e805cf30 0x7fa5e809ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80b14e0 con 0x7fa5e81440b0 2022-07-12T08:49:07.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e80b1780 con 0x7fa5e8060a70 2022-07-12T08:49:07.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e809a380 con 0x7fa5e805cf30 2022-07-12T08:49:07.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5e8060a70 0x7fa5e809a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5e8060a70 0x7fa5e809a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59568/0 (socket says 172.21.15.61:59568) 2022-07-12T08:49:07.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 -- 172.21.15.61:0/1367438505 learned_addr learned my addr 172.21.15.61:0/1367438505 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:07.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5ed4c2700 1 --2- 172.21.15.61:0/1367438505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e805cf30 0x7fa5e809ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 -- 172.21.15.61:0/1367438505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e805cf30 msgr2=0x7fa5e809ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:07.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 --2- 172.21.15.61:0/1367438505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e805cf30 0x7fa5e809ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 -- 172.21.15.61:0/1367438505 >> v1:172.21.15.111:6789/0 conn(0x7fa5e81440b0 legacy=0x7fa5e8144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 -- 172.21.15.61:0/1367438505 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8140fb0 con 0x7fa5e8060a70 2022-07-12T08:49:07.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5edcc3700 1 --2- 172.21.15.61:0/1367438505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e809a820 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fa5d8002f70 tx=0x7fa5d8006eb0).ready entity=mon.0 client_cookie=8bef7bd47f99e402 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eccc1700 1 -- 172.21.15.61:0/1367438505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d800d570 con 0x7fa5e8060a70 2022-07-12T08:49:07.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eccc1700 1 -- 172.21.15.61:0/1367438505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d8000e90 con 0x7fa5e8060a70 2022-07-12T08:49:07.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eccc1700 1 -- 172.21.15.61:0/1367438505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5d800d900 con 0x7fa5e8060a70 2022-07-12T08:49:07.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1367438505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 msgr2=0x7fa5e809a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1367438505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e809a820 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7fa5d8002f70 tx=0x7fa5d8006eb0).stop 2022-07-12T08:49:07.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1367438505 shutdown_connections 2022-07-12T08:49:07.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1367438505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e805cf30 0x7fa5e809ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1367438505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e809a820 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1367438505 >> 172.21.15.61:0/1367438505 conn(0x7fa5e805bd10 msgr2=0x7fa5e805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1367438505 shutdown_connections 2022-07-12T08:49:07.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.145+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1367438505 wait complete. 2022-07-12T08:49:07.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 Processor -- start 2022-07-12T08:49:07.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 -- start start 2022-07-12T08:49:07.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e805cf30 0x7fa5e8143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e80ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e81440b0 0x7fa5e814fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80aeca0 con 0x7fa5e8060a70 2022-07-12T08:49:07.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e80aef40 con 0x7fa5e805cf30 2022-07-12T08:49:07.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e8143720 con 0x7fa5e81440b0 2022-07-12T08:49:07.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e80ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e80ac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59574/0 (socket says 172.21.15.61:59574) 2022-07-12T08:49:07.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 -- 172.21.15.61:0/1280686956 learned_addr learned my addr 172.21.15.61:0/1280686956 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:07.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ee4c4700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e81440b0 0x7fa5e814fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5edcc3700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e805cf30 0x7fa5e8143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 -- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e81440b0 msgr2=0x7fa5e814fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e81440b0 0x7fa5e814fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 -- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e805cf30 msgr2=0x7fa5e8143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e805cf30 0x7fa5e8143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 -- 172.21.15.61:0/1280686956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d800b1e0 con 0x7fa5e8060a70 2022-07-12T08:49:07.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5edcc3700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e805cf30 0x7fa5e8143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:07.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ee4c4700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e81440b0 0x7fa5e814fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:07.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5ed4c2700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e80ac690 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7fa5e400c000 tx=0x7fa5e4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5e400c830 con 0x7fa5e8060a70 2022-07-12T08:49:07.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e400c990 con 0x7fa5e8060a70 2022-07-12T08:49:07.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e8142720 con 0x7fa5e8060a70 2022-07-12T08:49:07.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.149+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e8142150 con 0x7fa5e8060a70 2022-07-12T08:49:07.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa5e400ccf0 con 0x7fa5e8060a70 2022-07-12T08:49:07.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e80aead0 con 0x7fa5e8060a70 2022-07-12T08:49:07.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa5e401c460 con 0x7fa5e8060a70 2022-07-12T08:49:07.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5d6ffd700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5c80539b0 0x7fa5c8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa5e406c5d0 con 0x7fa5e8060a70 2022-07-12T08:49:07.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5edcc3700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5c80539b0 0x7fa5c8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.153+0000 7fa5edcc3700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5c80539b0 0x7fa5c8055e70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7fa5d801a000 tx=0x7fa5d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.161+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5e4043340 con 0x7fa5e8060a70 2022-07-12T08:49:07.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa5e8057460 con 0x7fa5e8060a70 2022-07-12T08:49:07.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5d6ffd700 1 -- 172.21.15.61:0/1280686956 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v100) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa5e4046960 con 0x7fa5e8060a70 2022-07-12T08:49:07.297 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:49:07.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5c80539b0 msgr2=0x7fa5c8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5c80539b0 0x7fa5c8055e70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7fa5d801a000 tx=0x7fa5d8015000).stop 2022-07-12T08:49:07.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 msgr2=0x7fa5e80ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e80ac690 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 rx=0x7fa5e400c000 tx=0x7fa5e4006df0).stop 2022-07-12T08:49:07.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 shutdown_connections 2022-07-12T08:49:07.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5e81440b0 0x7fa5e814fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5e8060a70 0x7fa5e80ac690 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa5c80539b0 0x7fa5c8055e70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 --2- 172.21.15.61:0/1280686956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa5e805cf30 0x7fa5e8143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 >> 172.21.15.61:0/1280686956 conn(0x7fa5e805bd10 msgr2=0x7fa5e809a740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 shutdown_connections 2022-07-12T08:49:07.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.289+0000 7fa5eecc5700 1 -- 172.21.15.61:0/1280686956 wait complete. 2022-07-12T08:49:07.308 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:49:07.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 Processor -- start 2022-07-12T08:49:07.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 -- start start 2022-07-12T08:49:07.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ca00b5920 0x7f2ca014a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 0x7f2ca00aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca005c3e0 con 0x7f2ca00b0280 2022-07-12T08:49:07.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ca00b5d00 con 0x7f2ca00b5920 2022-07-12T08:49:07.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.417+0000 7f2c9f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca005c610 con 0x7f2ca00aa830 2022-07-12T08:49:07.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 0x7f2ca00aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9e59c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ca00b5920 0x7f2ca014a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 0x7f2ca00aac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:41782/0 (socket says 172.21.15.61:41782) 2022-07-12T08:49:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 -- 172.21.15.61:0/3610811806 learned_addr learned my addr 172.21.15.61:0/3610811806 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9d59a700 1 -- 172.21.15.61:0/3610811806 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838005285 0 0) 0x7f2ca005c3e0 con 0x7f2ca00b0280 2022-07-12T08:49:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9d59a700 1 -- 172.21.15.61:0/3610811806 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c80003260 con 0x7f2ca00b0280 2022-07-12T08:49:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 -- 172.21.15.61:0/3610811806 >> v2:172.21.15.111:3300/0 conn(0x7f2ca00b5920 msgr2=0x7f2ca014a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:07.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 --2- 172.21.15.61:0/3610811806 >> v2:172.21.15.111:3300/0 conn(0x7f2ca00b5920 0x7f2ca014a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 -- 172.21.15.61:0/3610811806 >> v1:172.21.15.111:6789/0 conn(0x7f2ca00b0280 legacy=0x7f2ca014aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 -- 172.21.15.61:0/3610811806 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca005c8b0 con 0x7f2ca00aa830 2022-07-12T08:49:07.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9ed9d700 1 --2- 172.21.15.61:0/3610811806 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 0x7f2ca00aac60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2c980032f0 tx=0x7f2c98009290).ready entity=mon.2 client_cookie=1e176ac3ae67892d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9d59a700 1 -- 172.21.15.61:0/3610811806 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c9800b6c0 con 0x7f2ca00aa830 2022-07-12T08:49:07.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9d59a700 1 -- 172.21.15.61:0/3610811806 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9800b820 con 0x7f2ca00aa830 2022-07-12T08:49:07.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9d59a700 1 -- 172.21.15.61:0/3610811806 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c9800b990 con 0x7f2ca00aa830 2022-07-12T08:49:07.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 -- 172.21.15.61:0/3610811806 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 msgr2=0x7f2ca00aac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/3610811806 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 0x7f2ca00aac60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2c980032f0 tx=0x7f2c98009290).stop 2022-07-12T08:49:07.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 -- 172.21.15.61:0/3610811806 shutdown_connections 2022-07-12T08:49:07.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/3610811806 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00aa830 0x7f2ca00aac60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/3610811806 >> v2:172.21.15.111:3300/0 conn(0x7f2ca00b5920 0x7f2ca014a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 -- 172.21.15.61:0/3610811806 >> 172.21.15.61:0/3610811806 conn(0x7f2ca0063340 msgr2=0x7f2ca00b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 -- 172.21.15.61:0/3610811806 shutdown_connections 2022-07-12T08:49:07.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 -- 172.21.15.61:0/3610811806 wait complete. 2022-07-12T08:49:07.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 Processor -- start 2022-07-12T08:49:07.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.421+0000 7f2c9f59e700 1 -- start start 2022-07-12T08:49:07.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 0x7f2ca00ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00b0280 0x7f2ca0147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ca00b5920 0x7f2ca0148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0064260 con 0x7f2ca00aa830 2022-07-12T08:49:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0067720 con 0x7f2ca00b5920 2022-07-12T08:49:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca0063880 con 0x7f2ca00b0280 2022-07-12T08:49:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 0x7f2ca00ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 0x7f2ca00ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59588/0 (socket says 172.21.15.61:59588) 2022-07-12T08:49:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 -- 172.21.15.61:0/1798358437 learned_addr learned my addr 172.21.15.61:0/1798358437 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:07.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9dd9b700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00b0280 0x7f2ca0147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9ed9d700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ca00b5920 0x7f2ca0148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 -- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00b0280 msgr2=0x7f2ca0147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00b0280 0x7f2ca0147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 -- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ca00b5920 msgr2=0x7f2ca0148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ca00b5920 0x7f2ca0148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 -- 172.21.15.61:0/1798358437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c98009530 con 0x7f2ca00aa830 2022-07-12T08:49:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9e59c700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 0x7f2ca00ae3d0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f2c8800c000 tx=0x7f2c88006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c88011750 con 0x7f2ca00aa830 2022-07-12T08:49:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c880118b0 con 0x7f2ca00aa830 2022-07-12T08:49:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca0063b20 con 0x7f2ca00aa830 2022-07-12T08:49:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c88011c10 con 0x7f2ca00aa830 2022-07-12T08:49:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.425+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca01485d0 con 0x7f2ca00aa830 2022-07-12T08:49:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.429+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca0057460 con 0x7f2ca00aa830 2022-07-12T08:49:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.429+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2c88011d70 con 0x7f2ca00aa830 2022-07-12T08:49:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.429+0000 7f2c977fe700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c840539b0 0x7f2c84055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.429+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2c8806c260 con 0x7f2ca00aa830 2022-07-12T08:49:07.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.429+0000 7f2c9dd9b700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c840539b0 0x7f2c84055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.429+0000 7f2c9dd9b700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c840539b0 0x7f2c84055e70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f2c9000d000 tx=0x7f2c90007ab0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.433+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c88043160 con 0x7f2ca00aa830 2022-07-12T08:49:07.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.561+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2ca006b2b0 con 0x7f2ca00aa830 2022-07-12T08:49:07.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.561+0000 7f2c977fe700 1 -- 172.21.15.61:0/1798358437 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v100) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2c88046780 con 0x7f2ca00aa830 2022-07-12T08:49:07.569 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:07.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c840539b0 msgr2=0x7f2c84055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c840539b0 0x7f2c84055e70 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f2c9000d000 tx=0x7f2c90007ab0).stop 2022-07-12T08:49:07.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 msgr2=0x7f2ca00ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 0x7f2ca00ae3d0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f2c8800c000 tx=0x7f2c88006df0).stop 2022-07-12T08:49:07.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 shutdown_connections 2022-07-12T08:49:07.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c840539b0 0x7f2c84055e70 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ca00b5920 0x7f2ca0148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ca00b0280 0x7f2ca0147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 --2- 172.21.15.61:0/1798358437 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ca00aa830 0x7f2ca00ae3d0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 >> 172.21.15.61:0/1798358437 conn(0x7f2ca0063340 msgr2=0x7f2ca00b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 shutdown_connections 2022-07-12T08:49:07.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.565+0000 7f2c9f59e700 1 -- 172.21.15.61:0/1798358437 wait complete. 2022-07-12T08:49:07.582 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:49:07.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2cb4c62700 1 Processor -- start 2022-07-12T08:49:07.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2cb4c62700 1 -- start start 2022-07-12T08:49:07.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2cb4c62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2cb00b0280 0x7f2cb014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2cb4c62700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb005c3e0 con 0x7f2cb00b5920 2022-07-12T08:49:07.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2cb4c62700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb00b5d00 con 0x7f2cb00aa830 2022-07-12T08:49:07.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2cb4c62700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb005c610 con 0x7f2cb00b0280 2022-07-12T08:49:07.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2caeffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2cb00b0280 0x7f2cb014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2caf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2cb00b5920 0x7f2cb014a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47814/0 (socket says 172.21.15.61:47814) 2022-07-12T08:49:07.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.681+0000 7f2caf7fe700 1 -- 172.21.15.61:0/3585730010 learned_addr learned my addr 172.21.15.61:0/3585730010 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:07.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2caeffd700 1 -- 172.21.15.61:0/3585730010 >> v1:172.21.15.111:6789/0 conn(0x7f2cb00aa830 legacy=0x7f2cb00aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2caeffd700 1 -- 172.21.15.61:0/3585730010 >> v1:172.21.15.61:6789/0 conn(0x7f2cb00b5920 legacy=0x7f2cb014a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2caeffd700 1 -- 172.21.15.61:0/3585730010 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb005c8b0 con 0x7f2cb00b0280 2022-07-12T08:49:07.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2caeffd700 1 --2- 172.21.15.61:0/3585730010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00b0280 0x7f2cb014b0f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f2ca80032f0 tx=0x7f2ca80091d0).ready entity=mon.0 client_cookie=10cee67abbbfa1cd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cae7fc700 1 -- 172.21.15.61:0/3585730010 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca8009ea0 con 0x7f2cb00b0280 2022-07-12T08:49:07.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cae7fc700 1 -- 172.21.15.61:0/3585730010 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca800b8a0 con 0x7f2cb00b0280 2022-07-12T08:49:07.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cae7fc700 1 -- 172.21.15.61:0/3585730010 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca800ba10 con 0x7f2cb00b0280 2022-07-12T08:49:07.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 -- 172.21.15.61:0/3585730010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00b0280 msgr2=0x7f2cb014b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/3585730010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00b0280 0x7f2cb014b0f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7f2ca80032f0 tx=0x7f2ca80091d0).stop 2022-07-12T08:49:07.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 -- 172.21.15.61:0/3585730010 shutdown_connections 2022-07-12T08:49:07.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/3585730010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00b0280 0x7f2cb014b0f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 -- 172.21.15.61:0/3585730010 >> 172.21.15.61:0/3585730010 conn(0x7f2cb0063340 msgr2=0x7f2cb00b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 -- 172.21.15.61:0/3585730010 shutdown_connections 2022-07-12T08:49:07.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.685+0000 7f2cb4c62700 1 -- 172.21.15.61:0/3585730010 wait complete. 2022-07-12T08:49:07.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 Processor -- start 2022-07-12T08:49:07.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 -- start start 2022-07-12T08:49:07.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 0x7f2cb014c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb00b0280 0x7f2cb0098910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b5920 0x7f2cb01560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb014c550 con 0x7f2cb00aa830 2022-07-12T08:49:07.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb014b680 con 0x7f2cb00b0280 2022-07-12T08:49:07.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb014b920 con 0x7f2cb00b5920 2022-07-12T08:49:07.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 0x7f2cb014c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 0x7f2cb014c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59600/0 (socket says 172.21.15.61:59600) 2022-07-12T08:49:07.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 -- 172.21.15.61:0/523199493 learned_addr learned my addr 172.21.15.61:0/523199493 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:07.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caffff700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b5920 0x7f2cb01560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caeffd700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb00b0280 0x7f2cb0098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 -- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b5920 msgr2=0x7f2cb01560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b5920 0x7f2cb01560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 -- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb00b0280 msgr2=0x7f2cb0098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb00b0280 0x7f2cb0098910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 -- 172.21.15.61:0/523199493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca8009470 con 0x7f2cb00aa830 2022-07-12T08:49:07.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2caf7fe700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 0x7f2cb014c060 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f2cb00b5f30 tx=0x7f2ca4007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca400ce30 con 0x7f2cb00aa830 2022-07-12T08:49:07.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca40072e0 con 0x7f2cb00aa830 2022-07-12T08:49:07.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ca4005800 con 0x7f2cb00aa830 2022-07-12T08:49:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb014bbc0 con 0x7f2cb00aa830 2022-07-12T08:49:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.689+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb0064220 con 0x7f2cb00aa830 2022-07-12T08:49:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.693+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb00af3b0 con 0x7f2cb00aa830 2022-07-12T08:49:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.693+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2ca4010070 con 0x7f2cb00aa830 2022-07-12T08:49:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.693+0000 7f2cacff9700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c900539d0 0x7f2c90055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.693+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2ca406b4c0 con 0x7f2cb00aa830 2022-07-12T08:49:07.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.693+0000 7f2caeffd700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c900539d0 0x7f2c90055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.693+0000 7f2caeffd700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c900539d0 0x7f2c90055e90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f2ca8019000 tx=0x7f2ca8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.697+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2ca4042420 con 0x7f2cb00aa830 2022-07-12T08:49:07.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.821+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2cb006b2b0 con 0x7f2cb00aa830 2022-07-12T08:49:07.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.821+0000 7f2cacff9700 1 -- 172.21.15.61:0/523199493 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2ca40180a0 con 0x7f2cb00aa830 2022-07-12T08:49:07.831 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:07.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c900539d0 msgr2=0x7f2c90055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c900539d0 0x7f2c90055e90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f2ca8019000 tx=0x7f2ca8016000).stop 2022-07-12T08:49:07.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 msgr2=0x7f2cb014c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 0x7f2cb014c060 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7f2cb00b5f30 tx=0x7f2ca4007ca0).stop 2022-07-12T08:49:07.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 shutdown_connections 2022-07-12T08:49:07.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2cb00b5920 0x7f2cb01560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2c900539d0 0x7f2c90055e90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2cb00b0280 0x7f2cb0098910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 --2- 172.21.15.61:0/523199493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2cb00aa830 0x7f2cb014c060 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 >> 172.21.15.61:0/523199493 conn(0x7f2cb0063340 msgr2=0x7f2cb0098020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 shutdown_connections 2022-07-12T08:49:07.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.825+0000 7f2cb4c62700 1 -- 172.21.15.61:0/523199493 wait complete. 2022-07-12T08:49:07.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:49:07.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.953+0000 7f3079556700 1 Processor -- start 2022-07-12T08:49:07.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.953+0000 7f3079556700 1 -- start start 2022-07-12T08:49:07.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3074067750 0x7f3074067b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f30740af620 0x7f3074141c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30740ae280 con 0x7f30740b00e0 2022-07-12T08:49:07.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3074142cd0 con 0x7f30740af620 2022-07-12T08:49:07.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740ae520 con 0x7f3074067750 2022-07-12T08:49:07.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f306b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f30740b00e0 0x7f30740aef10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47826/0 (socket says 172.21.15.61:47826) 2022-07-12T08:49:07.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f306b7fe700 1 -- 172.21.15.61:0/915434001 learned_addr learned my addr 172.21.15.61:0/915434001 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:07.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3078d55700 1 --2- 172.21.15.61:0/915434001 >> v2:172.21.15.61:3300/0 conn(0x7f30740af620 0x7f3074141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3073fff700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3074067750 0x7f3074067b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f30737fe700 1 -- 172.21.15.61:0/915434001 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276387836 0 0) 0x7f30740ae280 con 0x7f30740b00e0 2022-07-12T08:49:07.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f30737fe700 1 -- 172.21.15.61:0/915434001 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3054003260 con 0x7f30740b00e0 2022-07-12T08:49:07.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3078d55700 1 -- 172.21.15.61:0/915434001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3074067750 msgr2=0x7f3074067b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:07.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3078d55700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3074067750 0x7f3074067b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3078d55700 1 -- 172.21.15.61:0/915434001 >> v1:172.21.15.61:6789/0 conn(0x7f30740b00e0 legacy=0x7f30740aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3078d55700 1 -- 172.21.15.61:0/915434001 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30740ae7c0 con 0x7f30740af620 2022-07-12T08:49:07.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3073fff700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3074067750 0x7f3074067b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3078d55700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740af620 0x7f3074141c20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f306c0032f0 tx=0x7f306c0091d0).ready entity=mon.0 client_cookie=b71ad1a4a70ad1b5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f30737fe700 1 -- 172.21.15.61:0/915434001 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f306c00b600 con 0x7f30740af620 2022-07-12T08:49:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f30737fe700 1 -- 172.21.15.61:0/915434001 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c00b760 con 0x7f30740af620 2022-07-12T08:49:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f30737fe700 1 -- 172.21.15.61:0/915434001 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f306c00b8d0 con 0x7f30740af620 2022-07-12T08:49:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- 172.21.15.61:0/915434001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740af620 msgr2=0x7f3074141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740af620 0x7f3074141c20 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7f306c0032f0 tx=0x7f306c0091d0).stop 2022-07-12T08:49:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- 172.21.15.61:0/915434001 shutdown_connections 2022-07-12T08:49:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740af620 0x7f3074141c20 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 --2- 172.21.15.61:0/915434001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3074067750 0x7f3074067b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- 172.21.15.61:0/915434001 >> 172.21.15.61:0/915434001 conn(0x7f3074063c50 msgr2=0x7f307405f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:07.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- 172.21.15.61:0/915434001 shutdown_connections 2022-07-12T08:49:07.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.957+0000 7f3079556700 1 -- 172.21.15.61:0/915434001 wait complete. 2022-07-12T08:49:07.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 Processor -- start 2022-07-12T08:49:07.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 -- start start 2022-07-12T08:49:07.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3074067750 0x7f30740b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30740af620 0x7f3074151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 0x7f30741554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307413d980 con 0x7f30740b00e0 2022-07-12T08:49:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30740ac5f0 con 0x7f3074067750 2022-07-12T08:49:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3079556700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30740ac350 con 0x7f30740af620 2022-07-12T08:49:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3078d55700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 0x7f30741554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3078d55700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 0x7f30741554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59618/0 (socket says 172.21.15.61:59618) 2022-07-12T08:49:07.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.961+0000 7f3078d55700 1 -- 172.21.15.61:0/4262016554 learned_addr learned my addr 172.21.15.61:0/4262016554 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:07.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f306b7fe700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30740af620 0x7f3074151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3073fff700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3074067750 0x7f30740b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3078d55700 1 -- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30740af620 msgr2=0x7f3074151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3078d55700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30740af620 0x7f3074151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3078d55700 1 -- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3074067750 msgr2=0x7f30740b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:07.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3078d55700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3074067750 0x7f30740b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3078d55700 1 -- 172.21.15.61:0/4262016554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306400b180 con 0x7f30740b00e0 2022-07-12T08:49:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3078d55700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 0x7f30741554e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f306c0032f0 tx=0x7f306c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f306c00b600 con 0x7f30740b00e0 2022-07-12T08:49:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c011b60 con 0x7f30740b00e0 2022-07-12T08:49:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c009470 con 0x7f30740b00e0 2022-07-12T08:49:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f306c01f3f0 con 0x7f30740b00e0 2022-07-12T08:49:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3074151e70 con 0x7f30740b00e0 2022-07-12T08:49:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.965+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3074057460 con 0x7f30740b00e0 2022-07-12T08:49:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.969+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f306c026030 con 0x7f30740b00e0 2022-07-12T08:49:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.969+0000 7f3071ffb700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3058053a60 0x7f3058055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:07.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.969+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f306c06c9f0 con 0x7f30740b00e0 2022-07-12T08:49:07.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.969+0000 7f3073fff700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3058053a60 0x7f3058055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:07.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.969+0000 7f3073fff700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3058053a60 0x7f3058055f20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f306400b000 tx=0x7f3064006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:07.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:07.973+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f306c06ee60 con 0x7f30740b00e0 2022-07-12T08:49:08.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f30740ac120 con 0x7f30740b00e0 2022-07-12T08:49:08.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3071ffb700 1 -- 172.21.15.61:0/4262016554 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) v1 ==== 100+0+10 (secure 0 0 0) 0x7f306c0240a0 con 0x7f30740b00e0 2022-07-12T08:49:08.107 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:08.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3058053a60 msgr2=0x7f3058055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3058053a60 0x7f3058055f20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7f306400b000 tx=0x7f3064006df0).stop 2022-07-12T08:49:08.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 msgr2=0x7f30741554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 0x7f30741554e0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 rx=0x7f306c0032f0 tx=0x7f306c016000).stop 2022-07-12T08:49:08.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 shutdown_connections 2022-07-12T08:49:08.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30740b00e0 0x7f30741554e0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f30740af620 0x7f3074151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3058053a60 0x7f3058055f20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 --2- 172.21.15.61:0/4262016554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3074067750 0x7f30740b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.101+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 >> 172.21.15.61:0/4262016554 conn(0x7f3074063c50 msgr2=0x7f3074143860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:08.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.105+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 shutdown_connections 2022-07-12T08:49:08.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.105+0000 7f3079556700 1 -- 172.21.15.61:0/4262016554 wait complete. 2022-07-12T08:49:08.122 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:49:08.123 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:08.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 Processor -- start 2022-07-12T08:49:08.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- start start 2022-07-12T08:49:08.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a800b0720 0x7f4a80143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:08.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4a800b2340 0x7f4a800b27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:08.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8005ff70 con 0x7f4a8005fb90 2022-07-12T08:49:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a800afa40 con 0x7f4a800b2340 2022-07-12T08:49:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800aecd0 con 0x7f4a800b0720 2022-07-12T08:49:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4a800b2340 0x7f4a800b27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4a800b2340 0x7f4a800b27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59626/0 (socket says 172.21.15.61:59626) 2022-07-12T08:49:08.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 -- 172.21.15.61:0/1548122457 learned_addr learned my addr 172.21.15.61:0/1548122457 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:08.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a85437700 1 --2- 172.21.15.61:0/1548122457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a800b0720 0x7f4a80143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:08.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 -- 172.21.15.61:0/1548122457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a800b0720 msgr2=0x7f4a80143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:08.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a84c36700 1 -- 172.21.15.61:0/1548122457 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102353147 0 0) 0x7f4a8005ff70 con 0x7f4a8005fb90 2022-07-12T08:49:08.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 --2- 172.21.15.61:0/1548122457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a800b0720 0x7f4a80143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 -- 172.21.15.61:0/1548122457 >> v1:172.21.15.111:6789/0 conn(0x7f4a8005fb90 legacy=0x7f4a80142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 -- 172.21.15.61:0/1548122457 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a800aef70 con 0x7f4a800b2340 2022-07-12T08:49:08.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a86439700 1 --2- 172.21.15.61:0/1548122457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800b27d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f4a70002f70 tx=0x7f4a70006eb0).ready entity=mon.0 client_cookie=a588c1145457c2c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a84c36700 1 -- 172.21.15.61:0/1548122457 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7000d570 con 0x7f4a800b2340 2022-07-12T08:49:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a84c36700 1 -- 172.21.15.61:0/1548122457 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a70000e90 con 0x7f4a800b2340 2022-07-12T08:49:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a84c36700 1 -- 172.21.15.61:0/1548122457 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a7000d8c0 con 0x7f4a800b2340 2022-07-12T08:49:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1548122457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 msgr2=0x7f4a800b27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1548122457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800b27d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f4a70002f70 tx=0x7f4a70006eb0).stop 2022-07-12T08:49:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1548122457 shutdown_connections 2022-07-12T08:49:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1548122457 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800b27d0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1548122457 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a800b0720 0x7f4a80143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1548122457 >> 172.21.15.61:0/1548122457 conn(0x7f4a80063cb0 msgr2=0x7f4a8005f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1548122457 shutdown_connections 2022-07-12T08:49:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.237+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1548122457 wait complete. 2022-07-12T08:49:08.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 Processor -- start 2022-07-12T08:49:08.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 -- start start 2022-07-12T08:49:08.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a8005fb90 0x7f4a80151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:08.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a800b0720 0x7f4a800ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:08.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:08.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800641f0 con 0x7f4a800b2340 2022-07-12T08:49:08.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80155110 con 0x7f4a800b0720 2022-07-12T08:49:08.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a80064420 con 0x7f4a8005fb90 2022-07-12T08:49:08.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:08.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800a85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59632/0 (socket says 172.21.15.61:59632) 2022-07-12T08:49:08.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 -- 172.21.15.61:0/1710953048 learned_addr learned my addr 172.21.15.61:0/1710953048 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:08.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a85c38700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a8005fb90 0x7f4a80151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:08.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a85437700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a800b0720 0x7f4a800ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:08.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 -- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a8005fb90 msgr2=0x7f4a80151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a8005fb90 0x7f4a80151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 -- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a800b0720 msgr2=0x7f4a800ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a800b0720 0x7f4a800ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 -- 172.21.15.61:0/1710953048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7000b1e0 con 0x7f4a800b2340 2022-07-12T08:49:08.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a86439700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800a85d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f4a70002f70 tx=0x7f4a70015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:08.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a700116e0 con 0x7f4a800b2340 2022-07-12T08:49:08.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a70011840 con 0x7f4a800b2340 2022-07-12T08:49:08.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a800a2ee0 con 0x7f4a800b2340 2022-07-12T08:49:08.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.241+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a70011ba0 con 0x7f4a800b2340 2022-07-12T08:49:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a800aab50 con 0x7f4a800b2340 2022-07-12T08:49:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4a70025030 con 0x7f4a800b2340 2022-07-12T08:49:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a80057460 con 0x7f4a800b2340 2022-07-12T08:49:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a76ffd700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a600539d0 0x7f4a60055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4a7006d060 con 0x7f4a800b2340 2022-07-12T08:49:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a85c38700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a600539d0 0x7f4a60055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.245+0000 7f4a85c38700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a600539d0 0x7f4a60055e90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f4a7c00d000 tx=0x7f4a7c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.249+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a70043eb0 con 0x7f4a800b2340 2022-07-12T08:49:08.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.409+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a80068470 con 0x7f4a800b2340 2022-07-12T08:49:08.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a76ffd700 1 -- 172.21.15.61:0/1710953048 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f4a70023020 con 0x7f4a800b2340 2022-07-12T08:49:08.418 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:08.418 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:59.111339+0000","modified":"2022-07-12T08:49:00.805218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2528523999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2528523999},{"type":"v1","addr":"172.21.15.61:6833","nonce":2528523999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-12T08:49:08.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a600539d0 msgr2=0x7f4a60055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a600539d0 0x7f4a60055e90 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f4a7c00d000 tx=0x7f4a7c009040).stop 2022-07-12T08:49:08.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 msgr2=0x7f4a800a85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:08.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800a85d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 rx=0x7f4a70002f70 tx=0x7f4a70015000).stop 2022-07-12T08:49:08.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 shutdown_connections 2022-07-12T08:49:08.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a800b2340 0x7f4a800a85d0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a600539d0 0x7f4a60055e90 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a800b0720 0x7f4a800ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 --2- 172.21.15.61:0/1710953048 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a8005fb90 0x7f4a80151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:08.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 >> 172.21.15.61:0/1710953048 conn(0x7f4a80063cb0 msgr2=0x7f4a801465f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:08.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 shutdown_connections 2022-07-12T08:49:08.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:08.413+0000 7f4a87e9c700 1 -- 172.21.15.61:0/1710953048 wait complete. 2022-07-12T08:49:08.424 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-12T08:49:08.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:48:59.111339+0000', 'modified': '2022-07-12T08:49:00.805218+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7338}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7338': {'gid': 7338, 'name': 'a', 'rank': 0, 'incarnation': 54, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2528523999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2528523999}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2528523999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:49:08.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:49:08.634 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:49:08.752 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":7338,"want_state":"up:active","state":"up:active","rank_uptime":9.6270044610000003,"mdsmap_epoch":56,"osdmap_epoch":100,"osdmap_epoch_barrier":100,"uptime":14.562232850999999} 2022-07-12T08:49:08.753 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 7338, "mdsmap_epoch": 56, "osdmap_epoch": 100, "osdmap_epoch_barrier": 100, "rank_uptime": 9.627004461, "state": "up:active", "uptime": 14.562232851, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:49:08.754 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:49:08.754 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:49:08.755 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:49:08.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:08.756 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:08.765 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:49:08.824 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:08.825 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:08.838 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:08.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:08.843 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:49:08.849 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:49:08.849 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:08.894 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:49:08.914 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:08.915 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:08.927 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:08.929 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:08.929 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:49:08.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:49:08.966+0000 7f31c0dce080 -1 init, newargv = 0x55bfb9ff9660 newargc=9 2022-07-12T08:49:08.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21398]: starting ceph client 2022-07-12T08:49:08.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21398]: starting fuse 2022-07-12T08:49:10.076 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:49:10.095 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:10.097 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:10.154 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:10.157 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:10.157 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:49:10.165 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:49:10.165 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:49:10.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:10.166 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:49:10.167 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:49:10.167 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:49:10.168 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:49:10.168 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:49:10.169 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:10.169 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:49:10.169 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:49:10.170 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:49:10.170 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:10.171 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:49:10.171 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:49:10.172 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:49:10.172 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:49:10.173 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:10.173 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:49:10.174 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:49:10.174 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:49:10.175 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:49:10.175 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:49:10.175 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:10.176 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:49:10.176 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:49:10.222 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21398.asok 2022-07-12T08:49:10.228 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21398.asok 2022-07-12T08:49:10.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:10.230 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21398.asok status 2022-07-12T08:49:10.365 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:49:10.366 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:49:10.366 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:49:10.366 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:49:10.367 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:49:10.367 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:49:10.368 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:49:10.368 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21398", 2022-07-12T08:49:10.368 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:49:10.368 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:10.369 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:49:10.369 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:49:10.369 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 7393, 2022-07-12T08:49:10.369 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:49:10.370 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:49:10.370 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:49:10.370 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 7393 2022-07-12T08:49:10.370 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:10.371 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:49:10.371 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:49:10.371 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:49:10.371 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 437704047 2022-07-12T08:49:10.372 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:49:10.372 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:10.373 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:49:10.373 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:49:10.373 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:49:10.373 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 437704047 2022-07-12T08:49:10.374 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:10.374 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.7393 172.21.15.130:0/437704047", 2022-07-12T08:49:10.375 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/437704047", 2022-07-12T08:49:10.375 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:49:10.375 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 56, 2022-07-12T08:49:10.376 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 100, 2022-07-12T08:49:10.376 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:49:10.376 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:49:10.376 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:49:10.376 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:49:10.378 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:10.378 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:10.379 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:10.386 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:49:10.388 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:10.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:10.389 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:10.453 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-12T08:49:10.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467ba65700 1 Processor -- start 2022-07-12T08:49:10.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467ba65700 1 -- start start 2022-07-12T08:49:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467ba65700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4674140b00 0x7f46740b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467ba65700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b6fc0 con 0x7f46740b1680 2022-07-12T08:49:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467ba65700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b1b20 con 0x7f46740b6be0 2022-07-12T08:49:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467ba65700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46740b1dc0 con 0x7f4674140b00 2022-07-12T08:49:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467a002700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4674140b00 0x7f46740b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f4679000700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f46740b1680 0x7f46740663c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47856/0 (socket says 172.21.15.61:47856) 2022-07-12T08:49:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f4679000700 1 -- 172.21.15.61:0/1174154408 learned_addr learned my addr 172.21.15.61:0/1174154408 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467a002700 1 -- 172.21.15.61:0/1174154408 >> v1:172.21.15.111:6789/0 conn(0x7f46740b6be0 legacy=0x7f46741403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467a002700 1 -- 172.21.15.61:0/1174154408 >> v1:172.21.15.61:6789/0 conn(0x7f46740b1680 legacy=0x7f46740663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.557+0000 7f467a002700 1 -- 172.21.15.61:0/1174154408 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740b2060 con 0x7f4674140b00 2022-07-12T08:49:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 --2- 172.21.15.61:0/1174154408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740b0d40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f466c0032f0 tx=0x7f466c0091d0).ready entity=mon.0 client_cookie=406f8feeb6e7ef22 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:10.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f4663fff700 1 -- 172.21.15.61:0/1174154408 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f466c009ea0 con 0x7f4674140b00 2022-07-12T08:49:10.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f4663fff700 1 -- 172.21.15.61:0/1174154408 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c00ba90 con 0x7f4674140b00 2022-07-12T08:49:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f4663fff700 1 -- 172.21.15.61:0/1174154408 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f466c00bdf0 con 0x7f4674140b00 2022-07-12T08:49:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- 172.21.15.61:0/1174154408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 msgr2=0x7f46740b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 --2- 172.21.15.61:0/1174154408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740b0d40 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f466c0032f0 tx=0x7f466c0091d0).stop 2022-07-12T08:49:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- 172.21.15.61:0/1174154408 shutdown_connections 2022-07-12T08:49:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 --2- 172.21.15.61:0/1174154408 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740b0d40 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- 172.21.15.61:0/1174154408 >> 172.21.15.61:0/1174154408 conn(0x7f46740645c0 msgr2=0x7f46740b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- 172.21.15.61:0/1174154408 shutdown_connections 2022-07-12T08:49:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- 172.21.15.61:0/1174154408 wait complete. 2022-07-12T08:49:10.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 Processor -- start 2022-07-12T08:49:10.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- start start 2022-07-12T08:49:10.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46740b1680 0x7f46740a3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46740b6be0 0x7f46741529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740aa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46740ab010 con 0x7f4674140b00 2022-07-12T08:49:10.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4674151dd0 con 0x7f46740b6be0 2022-07-12T08:49:10.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467ba65700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4674152000 con 0x7f46740b1680 2022-07-12T08:49:10.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740aa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740aa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59646/0 (socket says 172.21.15.61:59646) 2022-07-12T08:49:10.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 -- 172.21.15.61:0/445614205 learned_addr learned my addr 172.21.15.61:0/445614205 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:10.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f4679000700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46740b6be0 0x7f46741529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f4679801700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46740b1680 0x7f46740a3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 -- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46740b1680 msgr2=0x7f46740a3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46740b1680 0x7f46740a3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 -- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46740b6be0 msgr2=0x7f46741529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.561+0000 7f467a002700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46740b6be0 0x7f46741529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f467a002700 1 -- 172.21.15.61:0/445614205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f466c009470 con 0x7f4674140b00 2022-07-12T08:49:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f4679000700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46740b6be0 0x7f46741529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f467a002700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740aa860 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f466c0032f0 tx=0x7f466c018000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f466c01d070 con 0x7f4674140b00 2022-07-12T08:49:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c00b8a0 con 0x7f4674140b00 2022-07-12T08:49:10.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46741522a0 con 0x7f4674140b00 2022-07-12T08:49:10.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f466c009ea0 con 0x7f4674140b00 2022-07-12T08:49:10.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46740a9720 con 0x7f4674140b00 2022-07-12T08:49:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4674057460 con 0x7f4674140b00 2022-07-12T08:49:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.565+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f466c018800 con 0x7f4674140b00 2022-07-12T08:49:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.569+0000 7f46627fc700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4654053a60 0x7f4654055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.569+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f466c06d900 con 0x7f4674140b00 2022-07-12T08:49:10.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.569+0000 7f4679801700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4654053a60 0x7f4654055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.569+0000 7f4679801700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4654053a60 0x7f4654055f20 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f467000b000 tx=0x7f4670009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:10.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.577+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f466c0447c0 con 0x7f4674140b00 2022-07-12T08:49:10.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.741+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f46740ab1a0 con 0x7f4674140b00 2022-07-12T08:49:10.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.741+0000 7f46627fc700 1 -- 172.21.15.61:0/445614205 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1940 (secure 0 0 0) 0x7f466c027020 con 0x7f4674140b00 2022-07-12T08:49:10.750 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:10.751 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:10.751 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:10.752 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-12T08:49:10.752 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761054081024, 2022-07-12T08:49:10.752 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6531186688, 2022-07-12T08:49:10.753 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6531239936, 2022-07-12T08:49:10.753 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085088135674595833, 2022-07-12T08:49:10.753 INFO:teuthology.orchestra.run.smithi061.stdout: "num_osds": 8, 2022-07-12T08:49:10.754 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_osds": 0, 2022-07-12T08:49:10.754 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_omap_osds": 0 2022-07-12T08:49:10.755 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:10.755 INFO:teuthology.orchestra.run.smithi061.stdout: "stats_by_class": { 2022-07-12T08:49:10.755 INFO:teuthology.orchestra.run.smithi061.stdout: "ssd": { 2022-07-12T08:49:10.756 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-12T08:49:10.756 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761054081024, 2022-07-12T08:49:10.757 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6531186688, 2022-07-12T08:49:10.757 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6531239936, 2022-07-12T08:49:10.757 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085088135674595833 2022-07-12T08:49:10.758 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:10.758 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:10.758 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:49:10.759 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:10.759 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "device_health_metrics", 2022-07-12T08:49:10.760 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 1, 2022-07-12T08:49:10.760 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:10.760 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:10.761 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:10.761 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:10.761 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:10.762 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:10.762 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337290752 2022-07-12T08:49:10.763 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:10.763 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:10.763 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:10.764 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "rbd", 2022-07-12T08:49:10.764 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 2, 2022-07-12T08:49:10.764 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:10.765 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:10.765 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:10.766 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:10.766 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:10.766 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:10.767 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337290752 2022-07-12T08:49:10.767 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:10.767 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:10.768 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:10.768 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_metadata", 2022-07-12T08:49:10.769 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 19, 2022-07-12T08:49:10.769 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:10.769 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 2304, 2022-07-12T08:49:10.770 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 22, 2022-07-12T08:49:10.770 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 3, 2022-07-12T08:49:10.770 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 2304, 2022-07-12T08:49:10.771 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 3.1881570716763008e-09, 2022-07-12T08:49:10.771 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337290752 2022-07-12T08:49:10.771 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:10.771 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:10.772 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:10.772 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_data", 2022-07-12T08:49:10.772 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 20, 2022-07-12T08:49:10.772 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:10.773 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:10.773 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:10.773 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:10.773 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:10.773 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:10.774 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337290752 2022-07-12T08:49:10.774 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:10.774 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:10.774 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:10.775 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:10.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4654053a60 msgr2=0x7f4654055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4654053a60 0x7f4654055f20 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f467000b000 tx=0x7f4670009040).stop 2022-07-12T08:49:10.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 msgr2=0x7f46740aa860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740aa860 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 rx=0x7f466c0032f0 tx=0x7f466c018000).stop 2022-07-12T08:49:10.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 shutdown_connections 2022-07-12T08:49:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4674140b00 0x7f46740aa860 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4654053a60 0x7f4654055f20 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46740b6be0 0x7f46741529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 --2- 172.21.15.61:0/445614205 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46740b1680 0x7f46740a3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 >> 172.21.15.61:0/445614205 conn(0x7f46740645c0 msgr2=0x7f4674141cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 shutdown_connections 2022-07-12T08:49:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.745+0000 7f467ba65700 1 -- 172.21.15.61:0/445614205 wait complete. 2022-07-12T08:49:10.780 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data size -f json-pretty 2022-07-12T08:49:10.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 Processor -- start 2022-07-12T08:49:10.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 -- start start 2022-07-12T08:49:10.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48a00b7810 0x7f48a00b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a0143d80 0x7f48a005ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a01441d0 con 0x7f48a00b2010 2022-07-12T08:49:10.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a005bd10 con 0x7f48a00b7810 2022-07-12T08:49:10.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a005bfb0 con 0x7f48a0143d80 2022-07-12T08:49:10.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489dd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f48a00b2010 0x7f48a013a590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47872/0 (socket says 172.21.15.61:47872) 2022-07-12T08:49:10.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489dd9b700 1 -- 172.21.15.61:0/264388093 learned_addr learned my addr 172.21.15.61:0/264388093 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489e59c700 1 --2- 172.21.15.61:0/264388093 >> v2:172.21.15.61:3300/0 conn(0x7f48a00b7810 0x7f48a00b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489ed9d700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a0143d80 0x7f48a005ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489d59a700 1 -- 172.21.15.61:0/264388093 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83046659 0 0) 0x7f48a01441d0 con 0x7f48a00b2010 2022-07-12T08:49:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489d59a700 1 -- 172.21.15.61:0/264388093 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4884003260 con 0x7f48a00b2010 2022-07-12T08:49:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489e59c700 1 -- 172.21.15.61:0/264388093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a0143d80 msgr2=0x7f48a005ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489e59c700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a0143d80 0x7f48a005ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489e59c700 1 -- 172.21.15.61:0/264388093 >> v1:172.21.15.61:6789/0 conn(0x7f48a00b2010 legacy=0x7f48a013a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489d59a700 1 -- 172.21.15.61:0/264388093 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575273089 0 0) 0x7f4884003260 con 0x7f48a00b2010 2022-07-12T08:49:10.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489e59c700 1 -- 172.21.15.61:0/264388093 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a005c250 con 0x7f48a00b7810 2022-07-12T08:49:10.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.889+0000 7f489ed9d700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a0143d80 0x7f48a005ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:10.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489e59c700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a00b7810 0x7f48a00b7bf0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f4890001820 tx=0x7f4894006df0).ready entity=mon.0 client_cookie=2bc6df7cebda4156 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:10.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489d59a700 1 -- 172.21.15.61:0/264388093 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890001680 con 0x7f48a00b7810 2022-07-12T08:49:10.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489d59a700 1 -- 172.21.15.61:0/264388093 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4894000e90 con 0x7f48a00b7810 2022-07-12T08:49:10.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489d59a700 1 -- 172.21.15.61:0/264388093 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4894012460 con 0x7f48a00b7810 2022-07-12T08:49:10.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 -- 172.21.15.61:0/264388093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a00b7810 msgr2=0x7f48a00b7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a00b7810 0x7f48a00b7bf0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7f4890001820 tx=0x7f4894006df0).stop 2022-07-12T08:49:10.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 -- 172.21.15.61:0/264388093 shutdown_connections 2022-07-12T08:49:10.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a0143d80 0x7f48a005ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 --2- 172.21.15.61:0/264388093 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a00b7810 0x7f48a00b7bf0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 -- 172.21.15.61:0/264388093 >> 172.21.15.61:0/264388093 conn(0x7f48a0065650 msgr2=0x7f48a00b7100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 -- 172.21.15.61:0/264388093 shutdown_connections 2022-07-12T08:49:10.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.893+0000 7f489f59e700 1 -- 172.21.15.61:0/264388093 wait complete. 2022-07-12T08:49:10.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 Processor -- start 2022-07-12T08:49:10.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 -- start start 2022-07-12T08:49:10.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a00b2010 0x7f48a00b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a00b7810 0x7f48a014f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a0143d80 0x7f48a013d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0150a80 con 0x7f48a0143d80 2022-07-12T08:49:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00b1120 con 0x7f48a00b7810 2022-07-12T08:49:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a00b0200 con 0x7f48a00b2010 2022-07-12T08:49:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a0143d80 0x7f48a013d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a00b7810 0x7f48a014f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a0143d80 0x7f48a013d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59660/0 (socket says 172.21.15.61:59660) 2022-07-12T08:49:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489ed9d700 1 -- 172.21.15.61:0/945301027 learned_addr learned my addr 172.21.15.61:0/945301027 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489e59c700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a00b2010 0x7f48a00b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 -- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a00b2010 msgr2=0x7f48a00b6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a00b2010 0x7f48a00b6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 -- 172.21.15.61:0/945301027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a0143d80 msgr2=0x7f48a013d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a0143d80 0x7f48a013d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 -- 172.21.15.61:0/945301027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4898009470 con 0x7f48a00b7810 2022-07-12T08:49:10.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.897+0000 7f489dd9b700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a00b7810 0x7f48a014f800 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f48900018e0 tx=0x7f489000c740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:10.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890007170 con 0x7f48a00b7810 2022-07-12T08:49:10.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48900072d0 con 0x7f48a00b7810 2022-07-12T08:49:10.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489400b180 con 0x7f48a00b7810 2022-07-12T08:49:10.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890004710 con 0x7f48a00b7810 2022-07-12T08:49:10.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a014fcf0 con 0x7f48a00b7810 2022-07-12T08:49:10.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a0057460 con 0x7f48a00b7810 2022-07-12T08:49:10.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4890004870 con 0x7f48a00b7810 2022-07-12T08:49:10.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f488f7fe700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f487c053a40 0x7f487c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:10.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f489006b850 con 0x7f48a00b7810 2022-07-12T08:49:10.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.901+0000 7f489e59c700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f487c053a40 0x7f487c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:10.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.905+0000 7f489e59c700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f487c053a40 0x7f487c055f00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f489401a000 tx=0x7f4894014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:10.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:10.913+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f48900426b0 con 0x7f48a00b7810 2022-07-12T08:49:11.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.037+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"} v 0) v1 -- 0x7f48a006b2b0 con 0x7f48a00b7810 2022-07-12T08:49:11.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.037+0000 7f488f7fe700 1 -- 172.21.15.61:0/945301027 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"}]=0 v100) v1 ==== 123+0+64 (secure 0 0 0) 0x7f4890045cd0 con 0x7f48a00b7810 2022-07-12T08:49:11.046 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:11.046 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:11.047 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": "cephfs_data", 2022-07-12T08:49:11.047 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_id": 20, 2022-07-12T08:49:11.047 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2 2022-07-12T08:49:11.047 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:11.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f487c053a40 msgr2=0x7f487c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:11.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f487c053a40 0x7f487c055f00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f489401a000 tx=0x7f4894014000).stop 2022-07-12T08:49:11.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a00b7810 msgr2=0x7f48a014f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:11.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a00b7810 0x7f48a014f800 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f48900018e0 tx=0x7f489000c740).stop 2022-07-12T08:49:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 shutdown_connections 2022-07-12T08:49:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48a0143d80 0x7f48a013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f487c053a40 0x7f487c055f00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48a00b7810 0x7f48a014f800 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 --2- 172.21.15.61:0/945301027 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48a00b2010 0x7f48a00b6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 >> 172.21.15.61:0/945301027 conn(0x7f48a0065650 msgr2=0x7f48a00ad150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:11.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 shutdown_connections 2022-07-12T08:49:11.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.041+0000 7f489f59e700 1 -- 172.21.15.61:0/945301027 wait complete. 2022-07-12T08:49:11.059 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-12T08:49:11.060 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo df . 2022-07-12T08:49:11.106 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 1K-blocks Used Available Use% Mounted on 2022-07-12T08:49:11.106 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-fuse 352866304 0 352866304 0% /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:11.108 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:49:11.108 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:11.128 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:11.150 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:11.173 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:11.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:11.217 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:49:11.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:11.218 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:11.218 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:11.227 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:49:11.228 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:11.229 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:49:11.230 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:49:11.230 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:11.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[21937]: fuse finished with error 0 and tester_r 0 2022-07-12T08:49:11.249 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:11.249 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:11.250 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:11.298 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:49:11.299 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:49:11.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:11.418 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:11.426 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:11.427 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:11.428 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:11.437 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:49:11.439 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:11.440 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:49:11.441 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:49:11.441 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:11.458 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21398]: fuse finished with error 0 and tester_r 0 2022-07-12T08:49:11.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:11.461 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:11.462 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:11.533 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:49:11.534 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:49:11.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:11.571 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:11.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:11.597 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:11.597 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:49:11.621 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:49:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:11.628 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:49:11.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:11.630 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:49:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:11.684 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:49:11.685 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:49:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 Processor -- start 2022-07-12T08:49:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 -- start start 2022-07-12T08:49:11.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ac805d060 0x7f7ac8063140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7ac8146300 0x7f7ac8146760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac814a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac8061b20 con 0x7f7ac805d060 2022-07-12T08:49:11.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac805d440 con 0x7f7ac8146300 2022-07-12T08:49:11.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7acf4e8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80b3010 con 0x7f7ac814a4c0 2022-07-12T08:49:11.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7ace4e6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ac805d060 0x7f7ac8063140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7ace4e6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ac805d060 0x7f7ac8063140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59664/0 (socket says 172.21.15.61:59664) 2022-07-12T08:49:11.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.797+0000 7f7ace4e6700 1 -- 172.21.15.61:0/2387620797 learned_addr learned my addr 172.21.15.61:0/2387620797 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:11.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7ace4e6700 1 -- 172.21.15.61:0/2387620797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 msgr2=0x7f7ac814a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:11.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acece7700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac814a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7ace4e6700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac814a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7ace4e6700 1 -- 172.21.15.61:0/2387620797 >> v2:172.21.15.111:3300/0 conn(0x7f7ac8146300 msgr2=0x7f7ac8146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acdce5700 1 --2- 172.21.15.61:0/2387620797 >> v2:172.21.15.111:3300/0 conn(0x7f7ac8146300 0x7f7ac8146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7ace4e6700 1 --2- 172.21.15.61:0/2387620797 >> v2:172.21.15.111:3300/0 conn(0x7f7ac8146300 0x7f7ac8146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7ace4e6700 1 -- 172.21.15.61:0/2387620797 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac80b32b0 con 0x7f7ac805d060 2022-07-12T08:49:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acdce5700 1 --2- 172.21.15.61:0/2387620797 >> v2:172.21.15.111:3300/0 conn(0x7f7ac8146300 0x7f7ac8146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acece7700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7ace4e6700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac805d060 0x7f7ac8063140 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f7ab80032f0 tx=0x7f7ab8009290).ready entity=mon.0 client_cookie=c898500f6d02713b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acd4e4700 1 -- 172.21.15.61:0/2387620797 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ab800b870 con 0x7f7ac805d060 2022-07-12T08:49:11.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acd4e4700 1 -- 172.21.15.61:0/2387620797 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab800b9d0 con 0x7f7ac805d060 2022-07-12T08:49:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acd4e4700 1 -- 172.21.15.61:0/2387620797 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ab800bb40 con 0x7f7ac805d060 2022-07-12T08:49:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 -- 172.21.15.61:0/2387620797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac805d060 msgr2=0x7f7ac8063140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac805d060 0x7f7ac8063140 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 rx=0x7f7ab80032f0 tx=0x7f7ab8009290).stop 2022-07-12T08:49:11.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 -- 172.21.15.61:0/2387620797 shutdown_connections 2022-07-12T08:49:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac814a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/2387620797 >> v2:172.21.15.111:3300/0 conn(0x7f7ac8146300 0x7f7ac8146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/2387620797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac805d060 0x7f7ac8063140 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 -- 172.21.15.61:0/2387620797 >> 172.21.15.61:0/2387620797 conn(0x7f7ac80b3b40 msgr2=0x7f7ac80b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:11.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 -- 172.21.15.61:0/2387620797 shutdown_connections 2022-07-12T08:49:11.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 -- 172.21.15.61:0/2387620797 wait complete. 2022-07-12T08:49:11.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 Processor -- start 2022-07-12T08:49:11.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.801+0000 7f7acf4e8700 1 -- start start 2022-07-12T08:49:11.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac805d060 0x7f7ac8060270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 0x7f7ac8159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac815a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac8096f70 con 0x7f7ac8146300 2022-07-12T08:49:11.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac8097210 con 0x7f7ac805d060 2022-07-12T08:49:11.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80974b0 con 0x7f7ac814a4c0 2022-07-12T08:49:11.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 0x7f7ac8159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 0x7f7ac8159b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59670/0 (socket says 172.21.15.61:59670) 2022-07-12T08:49:11.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 -- 172.21.15.61:0/564073511 learned_addr learned my addr 172.21.15.61:0/564073511 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:11.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7ace4e6700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac805d060 0x7f7ac8060270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acece7700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac815a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 -- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 msgr2=0x7f7ac815a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:11.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac815a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 -- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac805d060 msgr2=0x7f7ac8060270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:11.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac805d060 0x7f7ac8060270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:11.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 -- 172.21.15.61:0/564073511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8009530 con 0x7f7ac8146300 2022-07-12T08:49:11.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acdce5700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 0x7f7ac8159b70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f7abc007000 tx=0x7f7abc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:11.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7abc010580 con 0x7f7ac8146300 2022-07-12T08:49:11.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc0106e0 con 0x7f7ac8146300 2022-07-12T08:49:11.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac8060760 con 0x7f7ac8146300 2022-07-12T08:49:11.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7abc010a40 con 0x7f7ac8146300 2022-07-12T08:49:11.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac809ced0 con 0x7f7ac8146300 2022-07-12T08:49:11.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac80b3550 con 0x7f7ac8146300 2022-07-12T08:49:11.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.805+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7abc010ba0 con 0x7f7ac8146300 2022-07-12T08:49:11.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.809+0000 7f7ac77fe700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7aa8053920 0x7f7aa8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:11.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.809+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7abc06bf80 con 0x7f7ac8146300 2022-07-12T08:49:11.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.809+0000 7f7ace4e6700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7aa8053920 0x7f7aa8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:11.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.809+0000 7f7ace4e6700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7aa8053920 0x7f7aa8055de0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f7ab801b000 tx=0x7f7ab8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:11.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.813+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7abc042d70 con 0x7f7ac8146300 2022-07-12T08:49:11.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.941+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7ac8096d90 con 0x7f7ac8146300 2022-07-12T08:49:11.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7ac77fe700 1 -- 172.21.15.61:0/564073511 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v100) v1 ==== 81+0+33698 (secure 0 0 0) 0x7f7abc046390 con 0x7f7ac8146300 2022-07-12T08:49:11.951 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:11.951 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:11.952 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 100, 2022-07-12T08:49:11.952 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:49:11.952 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:49:11.952 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:49:00.114186+0000", 2022-07-12T08:49:11.953 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:49:11.953 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:49:11.953 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:49:11.953 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:49:11.954 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:49:11.954 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:49:11.954 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:49:11.954 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:49:11.954 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:49:11.955 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:11.955 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:49:11.955 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:49:11.955 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:49:11.956 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:49:11.956 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:49:11.956 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 20, 2022-07-12T08:49:11.956 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:49:11.957 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:49:11.957 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:49:11.957 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:49:11.957 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:49:11.957 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:11.958 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:49:11.958 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:49:11.958 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:49:11.958 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:11.959 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:11.959 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:11.959 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:11.959 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:11.960 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:11.960 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:11.960 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:11.960 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:49:11.961 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:49:11.961 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:49:11.961 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:49:11.961 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:49:11.962 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:11.962 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:11.962 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:11.962 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:11.963 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:11.963 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:11.963 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:11.963 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:11.964 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:49:11.964 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:11.964 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:11.965 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:11.965 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:11.965 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:11.966 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:11.966 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:11.966 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:11.967 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:11.967 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:11.967 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:11.968 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:11.968 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:11.968 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:11.969 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:11.969 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:11.969 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:11.969 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:11.970 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:11.970 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:11.970 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:11.971 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:11.971 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:11.972 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:11.972 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:11.973 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:11.974 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:11.975 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:11.976 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:11.977 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:11.978 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:11.979 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:11.980 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:11.980 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:11.981 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:11.981 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:11.982 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:11.982 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:11.982 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:49:11.982 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:49:11.983 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:11.983 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:11.983 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:49:11.983 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:11.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:11.984 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:11.984 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:49:11.985 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:49:11.985 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:49:11.985 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:11.986 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:11.986 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:11.987 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:11.987 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:11.987 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:11.988 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:11.988 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:11.988 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:11.989 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:11.989 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:11.989 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:11.990 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:11.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:11.991 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:11.991 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:11.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:11.992 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:11.992 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:11.992 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:11.993 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:11.993 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:49:11.993 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:11.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:11.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:11.994 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:11.995 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:11.995 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:11.995 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:11.996 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:11.996 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:11.996 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:11.997 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:11.997 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:11.997 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:11.998 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:11.998 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:11.998 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:11.998 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:11.999 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:11.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:11.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:12.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:12.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:12.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:12.000 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:12.001 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:12.001 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:12.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.001 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:12.002 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:12.002 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:12.002 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:12.002 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:12.003 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:12.003 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:12.003 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:12.003 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:12.004 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:12.004 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:12.004 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:49:12.004 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:12.005 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:49:12.005 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.005 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.005 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.006 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 19, 2022-07-12T08:49:12.006 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:49:12.006 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:48:56.975401+0000", 2022-07-12T08:49:12.006 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:12.007 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:12.007 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:12.007 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:12.008 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:12.008 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:12.008 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:12.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:12.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:12.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:12.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:12.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:12.009 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:12.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:12.010 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:12.011 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:12.011 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:12.011 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:12.012 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:12.012 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:12.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "99", 2022-07-12T08:49:12.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:12.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:12.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:12.014 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:12.014 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:12.014 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:12.014 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:12.015 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:12.015 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:12.015 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:12.015 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:12.016 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:12.016 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:12.017 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:12.017 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:12.017 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:12.018 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:12.018 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:12.019 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:12.019 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:12.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:12.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:12.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:12.021 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:12.022 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:12.022 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:12.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.023 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:12.023 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:12.024 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:12.024 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:12.025 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:12.025 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:12.026 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:12.026 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:12.026 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:12.027 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:12.027 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:12.028 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:49:12.028 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:49:12.029 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:49:12.029 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:49:12.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.030 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:12.030 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:49:12.030 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:49:12.031 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.031 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.032 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 20, 2022-07-12T08:49:12.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:49:12.033 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:48:58.124077+0000", 2022-07-12T08:49:12.034 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:12.034 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:12.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:12.035 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:12.035 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:12.035 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:12.035 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:12.036 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:12.036 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:12.036 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:12.036 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:12.037 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:12.037 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:12.038 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:12.038 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:12.038 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:12.039 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:12.039 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:12.039 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:12.040 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:12.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.041 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "100", 2022-07-12T08:49:12.041 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:12.041 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:12.042 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:12.042 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:12.043 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:12.043 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:12.043 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:12.044 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:12.044 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:12.045 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:12.045 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:12.046 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:12.047 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:12.047 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:12.047 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:12.048 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:12.049 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:12.049 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:12.049 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:12.050 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:12.050 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:12.051 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:12.051 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:12.051 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:12.052 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:12.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:12.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.053 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:12.054 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:12.054 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:12.054 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:12.055 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:12.055 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:12.056 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:12.056 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:12.057 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:12.057 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:12.057 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:12.057 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:49:12.058 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:12.058 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:49:12.058 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:49:12.058 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.059 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.059 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.060 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:12.060 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:49:12.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.060 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:49:12.061 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:49:12.061 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.061 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.061 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.062 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.062 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.063 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.063 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.064 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:12.064 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.064 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.065 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:49:12.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.068 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:49:12.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.070 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.070 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.071 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.071 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.073 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.073 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:49:12.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.075 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.075 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:49:12.076 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.076 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.077 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.078 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.078 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.079 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:49:12.080 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.080 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.081 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.081 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:49:12.082 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.082 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.082 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.083 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.083 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.084 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.084 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:49:12.085 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.085 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.086 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.086 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:49:12.087 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:12.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.088 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.089 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:49:12.089 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:49:12.090 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:49:12.090 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:49:12.091 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.091 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.092 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.092 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.093 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:49:12.094 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:49:12.095 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.095 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.096 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.096 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.096 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.097 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.098 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:12.098 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.099 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.099 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.101 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:49:12.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.102 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.102 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.103 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:49:12.104 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.104 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.105 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.105 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.107 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:49:12.108 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:49:12.109 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.112 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:49:12.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.113 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.113 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:49:12.114 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.114 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.114 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.115 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.115 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.115 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.116 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.116 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:49:12.117 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.117 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:49:12.118 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:12.118 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.119 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.119 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:49:12.120 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:49:12.120 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:49:12.120 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:49:12.120 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.120 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.121 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.121 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.122 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:49:12.122 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:49:12.122 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.123 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.123 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.123 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.123 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.123 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 97, 2022-07-12T08:49:12.124 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.124 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.125 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.125 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.125 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:49:12.126 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.126 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.127 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:49:12.127 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.128 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.128 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.129 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.130 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:49:12.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.131 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:49:12.132 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.132 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.132 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.133 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.133 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.133 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.134 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:49:12.134 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:49:12.135 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.135 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.136 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.137 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:49:12.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:49:12.141 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:12.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.141 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.142 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:49:12.142 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:49:12.143 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:49:12.143 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:49:12.144 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.144 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.145 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.145 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.146 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:49:12.147 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:49:12.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.148 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.148 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.149 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.150 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.150 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.151 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:12.151 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.151 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.152 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.153 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:49:12.154 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.156 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:49:12.157 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.158 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.159 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.160 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.161 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.162 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:49:12.163 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.164 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.164 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.164 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.165 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:49:12.165 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.166 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.166 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.167 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.167 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.168 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.168 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:49:12.168 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.169 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.170 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:49:12.171 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.171 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.172 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.173 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.173 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.174 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.174 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:49:12.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:49:12.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:12.179 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.180 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.181 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:49:12.181 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:49:12.182 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:49:12.182 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:49:12.182 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.183 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.183 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.184 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.184 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.184 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.184 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:49:12.185 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:49:12.185 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.185 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.185 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.186 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.186 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.186 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 97, 2022-07-12T08:49:12.187 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.187 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.187 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:49:12.190 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.190 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.191 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.191 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.191 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:49:12.192 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.192 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.193 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.193 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.194 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.194 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.195 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:49:12.196 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.198 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:49:12.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.199 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.200 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.200 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.201 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.201 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.202 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:49:12.202 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.203 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:49:12.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.205 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.206 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.206 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.207 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.208 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:49:12.209 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:49:12.211 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:12.211 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.212 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.213 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:49:12.213 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:49:12.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:49:12.214 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:49:12.214 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.215 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.215 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.215 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.216 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:49:12.217 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:49:12.217 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.217 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.217 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.218 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.218 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.218 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.218 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.219 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 97, 2022-07-12T08:49:12.219 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.219 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.219 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.220 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:49:12.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.221 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.221 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.222 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:49:12.222 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.223 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.223 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.223 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.223 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.224 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:49:12.224 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.225 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.225 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.225 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:49:12.226 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.226 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.226 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.226 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.226 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.227 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.227 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:49:12.228 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.228 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:49:12.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.229 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.229 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.230 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:49:12.231 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.231 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.231 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:49:12.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:12.232 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.232 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.233 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.233 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:49:12.233 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:49:12.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:49:12.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:49:12.234 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.234 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.234 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.235 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.235 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.235 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:49:12.236 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:49:12.236 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.236 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.236 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.236 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.237 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.237 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:12.238 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.238 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.238 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.238 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.239 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.239 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:49:12.239 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.240 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:49:12.240 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.240 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.241 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.241 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.242 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:49:12.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:49:12.244 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.244 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.244 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.245 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:49:12.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.248 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:49:12.249 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.250 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.250 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.251 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.251 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.252 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:49:12.253 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.254 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.255 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:49:12.255 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:12.255 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.256 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.257 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:49:12.257 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:49:12.257 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:49:12.258 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:49:12.258 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.259 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.259 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.259 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.260 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.260 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:49:12.261 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:49:12.261 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:12.262 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:12.262 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:12.262 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:12.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:12.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:12.263 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:12.264 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:12.264 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:12.265 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:12.265 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:12.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.266 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.267 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:49:12.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.268 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.269 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.269 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.269 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:49:12.271 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.271 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.271 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.272 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:12.272 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.273 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.274 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:49:12.274 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.275 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.275 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:49:12.276 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.276 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.276 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.277 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:12.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.278 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:49:12.278 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.279 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:49:12.280 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.281 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.281 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.282 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:12.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:12.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.283 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:12.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:49:12.284 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:12.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:49:12.285 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:12.286 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.286 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.287 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:49:12.287 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:49:12.287 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:49:12.287 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:49:12.288 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:12.288 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:12.288 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:12.288 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:12.289 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.289 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:12.289 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:49:12.289 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.290 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:49:12.290 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.290 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.291 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.291 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.291 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:49:12.292 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.292 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:49:12.292 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.293 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.293 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.293 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.293 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:49:12.294 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.295 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.295 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.296 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:49:12.296 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.296 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.297 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.297 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.298 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.298 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:49:12.299 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.300 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:49:12.301 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.301 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.302 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.302 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.303 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:49:12.304 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.306 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:49:12.306 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.307 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.307 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.308 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.308 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.308 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:49:12.309 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.310 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:49:12.310 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.310 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.311 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.311 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.311 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.312 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:49:12.312 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.313 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.313 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:49:12.314 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.314 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.314 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.314 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.315 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.315 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:49:12.315 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:12.316 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:49:12.316 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:12.316 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:12.317 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:12.317 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:12.318 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:12.318 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:49:12.319 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:12.319 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.320 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:12.321 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:49:12.321 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:49:12.322 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:49:12.322 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:49:12.322 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:49:12.323 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:49:12.323 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:49:12.324 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:49:12.324 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:49:12.325 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:49:12.325 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:49:12.325 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:49:12.326 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:49:12.326 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:12.327 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:12.327 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:49:12.327 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:49:12.328 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:49:12.328 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:49:12.328 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:49:12.329 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7aa8053920 msgr2=0x7f7aa8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7aa8053920 0x7f7aa8055de0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f7ab801b000 tx=0x7f7ab8016000).stop 2022-07-12T08:49:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 msgr2=0x7f7ac8159b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 0x7f7ac8159b70 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f7abc007000 tx=0x7f7abc008040).stop 2022-07-12T08:49:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 shutdown_connections 2022-07-12T08:49:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7ac814a4c0 0x7f7ac815a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ac8146300 0x7f7ac8159b70 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7aa8053920 0x7f7aa8055de0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 --2- 172.21.15.61:0/564073511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7ac805d060 0x7f7ac8060270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 >> 172.21.15.61:0/564073511 conn(0x7f7ac80b3b40 msgr2=0x7f7ac8062420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:12.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 shutdown_connections 2022-07-12T08:49:12.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:11.945+0000 7f7acf4e8700 1 -- 172.21.15.61:0/564073511 wait complete. 2022-07-12T08:49:12.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:12.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150b59e700 1 Processor -- start 2022-07-12T08:49:12.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150b59e700 1 -- start start 2022-07-12T08:49:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f150c0b6b60 0x7f150c065a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c140010 con 0x7f150c0b25c0 2022-07-12T08:49:12.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150c0668f0 con 0x7f150c0b6780 2022-07-12T08:49:12.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150c142f70 con 0x7f150c0b6b60 2022-07-12T08:49:12.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f1509d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f150c0b25c0 0x7f150c1436b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47896/0 (socket says 172.21.15.61:47896) 2022-07-12T08:49:12.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f1509d9b700 1 -- 172.21.15.61:0/221178051 learned_addr learned my addr 172.21.15.61:0/221178051 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:12.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.445+0000 7f150ad9d700 1 --2- 172.21.15.61:0/221178051 >> v2:172.21.15.111:3300/0 conn(0x7f150c0b6b60 0x7f150c065a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150959a700 1 -- 172.21.15.61:0/221178051 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249910724 0 0) 0x7f150c140010 con 0x7f150c0b25c0 2022-07-12T08:49:12.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150959a700 1 -- 172.21.15.61:0/221178051 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f0003260 con 0x7f150c0b25c0 2022-07-12T08:49:12.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150ad9d700 1 -- 172.21.15.61:0/221178051 >> v1:172.21.15.111:6789/0 conn(0x7f150c0b6780 legacy=0x7f150c13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150ad9d700 1 -- 172.21.15.61:0/221178051 >> v1:172.21.15.61:6789/0 conn(0x7f150c0b25c0 legacy=0x7f150c1436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150ad9d700 1 -- 172.21.15.61:0/221178051 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c143210 con 0x7f150c0b6b60 2022-07-12T08:49:12.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150ad9d700 1 --2- 172.21.15.61:0/221178051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b6b60 0x7f150c065a80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f150c143590 tx=0x7f1504006df0).ready entity=mon.1 client_cookie=e390edd3719b738d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:12.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150959a700 1 -- 172.21.15.61:0/221178051 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150400d610 con 0x7f150c0b6b60 2022-07-12T08:49:12.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150959a700 1 -- 172.21.15.61:0/221178051 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1504000e90 con 0x7f150c0b6b60 2022-07-12T08:49:12.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150959a700 1 -- 172.21.15.61:0/221178051 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150400d960 con 0x7f150c0b6b60 2022-07-12T08:49:12.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 -- 172.21.15.61:0/221178051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b6b60 msgr2=0x7f150c065a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 --2- 172.21.15.61:0/221178051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b6b60 0x7f150c065a80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7f150c143590 tx=0x7f1504006df0).stop 2022-07-12T08:49:12.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 -- 172.21.15.61:0/221178051 shutdown_connections 2022-07-12T08:49:12.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 --2- 172.21.15.61:0/221178051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b6b60 0x7f150c065a80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 -- 172.21.15.61:0/221178051 >> 172.21.15.61:0/221178051 conn(0x7f150c064570 msgr2=0x7f150c0b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:12.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 -- 172.21.15.61:0/221178051 shutdown_connections 2022-07-12T08:49:12.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 -- 172.21.15.61:0/221178051 wait complete. 2022-07-12T08:49:12.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 Processor -- start 2022-07-12T08:49:12.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.449+0000 7f150b59e700 1 -- start start 2022-07-12T08:49:12.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b25c0 0x7f150c1525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 0x7f150c0ad690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f150c0b6b60 0x7f150c155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c152ac0 con 0x7f150c0b6780 2022-07-12T08:49:12.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c151e90 con 0x7f150c0b25c0 2022-07-12T08:49:12.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c152130 con 0x7f150c0b6b60 2022-07-12T08:49:12.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 0x7f150c0ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 0x7f150c0ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59686/0 (socket says 172.21.15.61:59686) 2022-07-12T08:49:12.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 -- 172.21.15.61:0/4197670315 learned_addr learned my addr 172.21.15.61:0/4197670315 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:12.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150a59c700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b25c0 0x7f150c1525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150ad9d700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f150c0b6b60 0x7f150c155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 -- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f150c0b6b60 msgr2=0x7f150c155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f150c0b6b60 0x7f150c155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 -- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b25c0 msgr2=0x7f150c1525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b25c0 0x7f150c1525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 -- 172.21.15.61:0/4197670315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150400b180 con 0x7f150c0b6780 2022-07-12T08:49:12.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f1509d9b700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 0x7f150c0ad690 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f14f4001490 tx=0x7f14f4007f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:12.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14f4005ba0 con 0x7f150c0b6780 2022-07-12T08:49:12.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4005d00 con 0x7f150c0b6780 2022-07-12T08:49:12.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150c0b5a80 con 0x7f150c0b6780 2022-07-12T08:49:12.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14f40072c0 con 0x7f150c0b6780 2022-07-12T08:49:12.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150c0adb80 con 0x7f150c0b6780 2022-07-12T08:49:12.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.453+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f150c144970 con 0x7f150c0b6780 2022-07-12T08:49:12.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.457+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f14f401a070 con 0x7f150c0b6780 2022-07-12T08:49:12.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.457+0000 7f15037fe700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14e4053920 0x7f14e4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.457+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f14f406b260 con 0x7f150c0b6780 2022-07-12T08:49:12.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.457+0000 7f150a59c700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14e4053920 0x7f14e4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.457+0000 7f150a59c700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14e4053920 0x7f14e4055de0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f14fc00d000 tx=0x7f14fc000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:12.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.465+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14f40421c0 con 0x7f150c0b6780 2022-07-12T08:49:12.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f150c057460 con 0x7f150c0b6780 2022-07-12T08:49:12.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f15037fe700 1 -- 172.21.15.61:0/4197670315 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f14f40457e0 con 0x7f150c0b6780 2022-07-12T08:49:12.631 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:12.631 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:59.111339+0000","modified":"2022-07-12T08:49:00.805218+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7338},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7338":{"gid":7338,"name":"a","rank":0,"incarnation":54,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2528523999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2528523999},{"type":"v1","addr":"172.21.15.61:6833","nonce":2528523999}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-12T08:49:12.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14e4053920 msgr2=0x7f14e4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14e4053920 0x7f14e4055de0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f14fc00d000 tx=0x7f14fc000bc0).stop 2022-07-12T08:49:12.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 msgr2=0x7f150c0ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 0x7f150c0ad690 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f14f4001490 tx=0x7f14f4007f00).stop 2022-07-12T08:49:12.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 shutdown_connections 2022-07-12T08:49:12.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f150c0b6b60 0x7f150c155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f150c0b6780 0x7f150c0ad690 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14e4053920 0x7f14e4055de0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 --2- 172.21.15.61:0/4197670315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f150c0b25c0 0x7f150c1525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 >> 172.21.15.61:0/4197670315 conn(0x7f150c064570 msgr2=0x7f150c0b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:12.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 shutdown_connections 2022-07-12T08:49:12.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.625+0000 7f150b59e700 1 -- 172.21.15.61:0/4197670315 wait complete. 2022-07-12T08:49:12.636 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-12T08:49:12.644 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:49:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 Processor -- start 2022-07-12T08:49:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 -- start start 2022-07-12T08:49:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4100b01f0 0x7ff4100ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff41014a8f0 0x7ff4101443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100ad2f0 con 0x7ff41005c050 2022-07-12T08:49:12.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4100b0a40 con 0x7ff4100b01f0 2022-07-12T08:49:12.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff418021700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41005c430 con 0x7ff41014a8f0 2022-07-12T08:49:12.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff4155bc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4100b01f0 0x7ff4100ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff4155bc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4100b01f0 0x7ff4100ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59696/0 (socket says 172.21.15.61:59696) 2022-07-12T08:49:12.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff4155bc700 1 -- 172.21.15.61:0/2094429576 learned_addr learned my addr 172.21.15.61:0/2094429576 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:12.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.741+0000 7ff4165be700 1 --2- 172.21.15.61:0/2094429576 >> v2:172.21.15.111:3300/0 conn(0x7ff41014a8f0 0x7ff4101443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff4155bc700 1 -- 172.21.15.61:0/2094429576 >> v2:172.21.15.111:3300/0 conn(0x7ff41014a8f0 msgr2=0x7ff4101443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:12.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff4155bc700 1 --2- 172.21.15.61:0/2094429576 >> v2:172.21.15.111:3300/0 conn(0x7ff41014a8f0 0x7ff4101443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff4155bc700 1 -- 172.21.15.61:0/2094429576 >> v1:172.21.15.111:6789/0 conn(0x7ff41005c050 legacy=0x7ff41005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff4155bc700 1 -- 172.21.15.61:0/2094429576 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4100b2420 con 0x7ff4100b01f0 2022-07-12T08:49:12.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff4155bc700 1 --2- 172.21.15.61:0/2094429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4100b01f0 0x7ff4100ac710 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7ff400002f70 tx=0x7ff400006eb0).ready entity=mon.0 client_cookie=1f56c7bdf74c58b8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:12.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff414dbb700 1 -- 172.21.15.61:0/2094429576 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff40000d5c0 con 0x7ff4100b01f0 2022-07-12T08:49:12.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff414dbb700 1 -- 172.21.15.61:0/2094429576 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff400000e90 con 0x7ff4100b01f0 2022-07-12T08:49:12.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff414dbb700 1 -- 172.21.15.61:0/2094429576 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff40000d950 con 0x7ff4100b01f0 2022-07-12T08:49:12.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 -- 172.21.15.61:0/2094429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4100b01f0 msgr2=0x7ff4100ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 --2- 172.21.15.61:0/2094429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4100b01f0 0x7ff4100ac710 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7ff400002f70 tx=0x7ff400006eb0).stop 2022-07-12T08:49:12.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 -- 172.21.15.61:0/2094429576 shutdown_connections 2022-07-12T08:49:12.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 --2- 172.21.15.61:0/2094429576 >> v2:172.21.15.111:3300/0 conn(0x7ff41014a8f0 0x7ff4101443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 --2- 172.21.15.61:0/2094429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4100b01f0 0x7ff4100ac710 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 -- 172.21.15.61:0/2094429576 >> 172.21.15.61:0/2094429576 conn(0x7ff410062c00 msgr2=0x7ff410062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:12.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 -- 172.21.15.61:0/2094429576 shutdown_connections 2022-07-12T08:49:12.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 -- 172.21.15.61:0/2094429576 wait complete. 2022-07-12T08:49:12.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 Processor -- start 2022-07-12T08:49:12.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.745+0000 7ff418021700 1 -- start start 2022-07-12T08:49:12.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 0x7ff4101565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4100b01f0 0x7ff410141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff41014a8f0 0x7ff41015a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff410142220 con 0x7ff41005c050 2022-07-12T08:49:12.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100aed40 con 0x7ff4100b01f0 2022-07-12T08:49:12.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff410159d60 con 0x7ff41014a8f0 2022-07-12T08:49:12.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 0x7ff4101565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 0x7ff4101565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59698/0 (socket says 172.21.15.61:59698) 2022-07-12T08:49:12.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 -- 172.21.15.61:0/2168555447 learned_addr learned my addr 172.21.15.61:0/2168555447 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:12.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff4155bc700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4100b01f0 0x7ff410141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff4165be700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff41014a8f0 0x7ff41015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 -- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff41014a8f0 msgr2=0x7ff41015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff41014a8f0 0x7ff41015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 -- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4100b01f0 msgr2=0x7ff410141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:12.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4100b01f0 0x7ff410141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 -- 172.21.15.61:0/2168555447 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40000b1e0 con 0x7ff41005c050 2022-07-12T08:49:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff415dbd700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 0x7ff4101565f0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7ff4100b62e0 tx=0x7ff4080014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff40800e4a0 con 0x7ff41005c050 2022-07-12T08:49:12.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40800e600 con 0x7ff41005c050 2022-07-12T08:49:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff40800e960 con 0x7ff41005c050 2022-07-12T08:49:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff410156ae0 con 0x7ff41005c050 2022-07-12T08:49:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff410141a20 con 0x7ff41005c050 2022-07-12T08:49:12.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.749+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4100aef10 con 0x7ff41005c050 2022-07-12T08:49:12.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.753+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff408019030 con 0x7ff41005c050 2022-07-12T08:49:12.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.753+0000 7ff406ffd700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff3f8053ad0 0x7ff3f8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:12.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.753+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff40806bab0 con 0x7ff41005c050 2022-07-12T08:49:12.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.753+0000 7ff4155bc700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff3f8053ad0 0x7ff3f8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:12.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.753+0000 7ff4155bc700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff3f8053ad0 0x7ff3f8055f90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7ff40001c000 tx=0x7ff400015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:12.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.757+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff408042a10 con 0x7ff41005c050 2022-07-12T08:49:12.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:12.925+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff410057460 con 0x7ff41005c050 2022-07-12T08:49:13.228 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:49:13.225+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:49:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.233+0000 7ff406ffd700 1 -- 172.21.15.61:0/2168555447 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff408017070 con 0x7ff41005c050 2022-07-12T08:49:13.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff3f8053ad0 msgr2=0x7ff3f8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff3f8053ad0 0x7ff3f8055f90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7ff40001c000 tx=0x7ff400015000).stop 2022-07-12T08:49:13.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 msgr2=0x7ff4101565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 0x7ff4101565f0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7ff4100b62e0 tx=0x7ff4080014a0).stop 2022-07-12T08:49:13.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 shutdown_connections 2022-07-12T08:49:13.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff41014a8f0 0x7ff41015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff3f8053ad0 0x7ff3f8055f90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4100b01f0 0x7ff410141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 --2- 172.21.15.61:0/2168555447 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff41005c050 0x7ff4101565f0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 >> 172.21.15.61:0/2168555447 conn(0x7ff410062c00 msgr2=0x7ff4100b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:13.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 shutdown_connections 2022-07-12T08:49:13.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.237+0000 7ff418021700 1 -- 172.21.15.61:0/2168555447 wait complete. 2022-07-12T08:49:13.248 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:49:13.260 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:13.268 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:49:13.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 Processor -- start 2022-07-12T08:49:13.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- start start 2022-07-12T08:49:13.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f233c14a520 0x7f233c14a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f233c0b0f60 0x7f233c1407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233c0b0b30 con 0x7f233c060ca0 2022-07-12T08:49:13.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233c0b1660 con 0x7f233c14a520 2022-07-12T08:49:13.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233c061e00 con 0x7f233c0b0f60 2022-07-12T08:49:13.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f233c14a520 0x7f233c14a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f233c14a520 0x7f233c14a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59718/0 (socket says 172.21.15.61:59718) 2022-07-12T08:49:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 -- 172.21.15.61:0/2036087653 learned_addr learned my addr 172.21.15.61:0/2036087653 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23420b5700 1 --1- 172.21.15.61:0/2036087653 >> v1:172.21.15.61:6789/0 conn(0x7f233c060ca0 0x7f233c140cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47940/0 (socket says 172.21.15.61:47940) 2022-07-12T08:49:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23410b3700 1 --2- 172.21.15.61:0/2036087653 >> v2:172.21.15.111:3300/0 conn(0x7f233c0b0f60 0x7f233c1407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 -- 172.21.15.61:0/2036087653 >> v2:172.21.15.111:3300/0 conn(0x7f233c0b0f60 msgr2=0x7f233c1407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 --2- 172.21.15.61:0/2036087653 >> v2:172.21.15.111:3300/0 conn(0x7f233c0b0f60 0x7f233c1407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 -- 172.21.15.61:0/2036087653 >> v1:172.21.15.61:6789/0 conn(0x7f233c060ca0 legacy=0x7f233c140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 -- 172.21.15.61:0/2036087653 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233c14af10 con 0x7f233c14a520 2022-07-12T08:49:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23418b4700 1 --2- 172.21.15.61:0/2036087653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c14a520 0x7f233c14a900 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f233c061f40 tx=0x7f2338006eb0).ready entity=mon.0 client_cookie=168e2752c3372fc8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23408b2700 1 -- 172.21.15.61:0/2036087653 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f233800d610 con 0x7f233c14a520 2022-07-12T08:49:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23408b2700 1 -- 172.21.15.61:0/2036087653 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2338000e90 con 0x7f233c14a520 2022-07-12T08:49:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23408b2700 1 -- 172.21.15.61:0/2036087653 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f233800d960 con 0x7f233c14a520 2022-07-12T08:49:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- 172.21.15.61:0/2036087653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c14a520 msgr2=0x7f233c14a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 --2- 172.21.15.61:0/2036087653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c14a520 0x7f233c14a900 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 rx=0x7f233c061f40 tx=0x7f2338006eb0).stop 2022-07-12T08:49:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- 172.21.15.61:0/2036087653 shutdown_connections 2022-07-12T08:49:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 --2- 172.21.15.61:0/2036087653 >> v2:172.21.15.111:3300/0 conn(0x7f233c0b0f60 0x7f233c1407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 --2- 172.21.15.61:0/2036087653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c14a520 0x7f233c14a900 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- 172.21.15.61:0/2036087653 >> 172.21.15.61:0/2036087653 conn(0x7f233c0b5780 msgr2=0x7f233c0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.361+0000 7f23428b6700 1 -- 172.21.15.61:0/2036087653 shutdown_connections 2022-07-12T08:49:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 -- 172.21.15.61:0/2036087653 wait complete. 2022-07-12T08:49:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 Processor -- start 2022-07-12T08:49:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 -- start start 2022-07-12T08:49:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 0x7f233c143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f233c0b0f60 0x7f233c13f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f233c14a520 0x7f233c13fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c156ce0 con 0x7f233c060ca0 2022-07-12T08:49:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233c141cf0 con 0x7f233c14a520 2022-07-12T08:49:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23428b6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233c141f90 con 0x7f233c0b0f60 2022-07-12T08:49:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 0x7f233c143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 0x7f233c143580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59726/0 (socket says 172.21.15.61:59726) 2022-07-12T08:49:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23410b3700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f233c0b0f60 0x7f233c13f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 -- 172.21.15.61:0/1612192151 learned_addr learned my addr 172.21.15.61:0/1612192151 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23420b5700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f233c14a520 0x7f233c13fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 -- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f233c0b0f60 msgr2=0x7f233c13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f233c0b0f60 0x7f233c13f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 -- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f233c14a520 msgr2=0x7f233c13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f233c14a520 0x7f233c13fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 -- 172.21.15.61:0/1612192151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233800b1e0 con 0x7f233c060ca0 2022-07-12T08:49:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.365+0000 7f23418b4700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 0x7f233c143580 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f233c061f40 tx=0x7f2338015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2338021070 con 0x7f233c060ca0 2022-07-12T08:49:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2338006b40 con 0x7f233c060ca0 2022-07-12T08:49:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2338002f50 con 0x7f233c060ca0 2022-07-12T08:49:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233c14d1b0 con 0x7f233c060ca0 2022-07-12T08:49:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233c157110 con 0x7f233c060ca0 2022-07-12T08:49:13.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233c142dd0 con 0x7f233c060ca0 2022-07-12T08:49:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f233800f070 con 0x7f233c060ca0 2022-07-12T08:49:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f232effd700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f231c053ad0 0x7f231c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.369+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f233806d020 con 0x7f233c060ca0 2022-07-12T08:49:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.373+0000 7f23410b3700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f231c053ad0 0x7f231c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.373+0000 7f23410b3700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f231c053ad0 0x7f231c055f90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f233400d000 tx=0x7f2334008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:13.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.377+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f233c142dd0 con 0x7f233c060ca0 2022-07-12T08:49:13.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.537+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233c156e70 con 0x7f233c060ca0 2022-07-12T08:49:13.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f232effd700 1 -- 172.21.15.61:0/1612192151 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+1783 (secure 0 0 0) 0x7f233c156e70 con 0x7f233c060ca0 2022-07-12T08:49:13.546 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:13.546 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":57,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[{"mdsmap":{"epoch":57,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:48:59.111339+0000","modified":"2022-07-12T08:49:13.227185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":101,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-12T08:49:13.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f231c053ad0 msgr2=0x7f231c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f231c053ad0 0x7f231c055f90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f233400d000 tx=0x7f2334008040).stop 2022-07-12T08:49:13.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 msgr2=0x7f233c143580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 0x7f233c143580 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f233c061f40 tx=0x7f2338015000).stop 2022-07-12T08:49:13.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 shutdown_connections 2022-07-12T08:49:13.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f231c053ad0 0x7f231c055f90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f233c14a520 0x7f233c13fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f233c0b0f60 0x7f233c13f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 --2- 172.21.15.61:0/1612192151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f233c060ca0 0x7f233c143580 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 >> 172.21.15.61:0/1612192151 conn(0x7f233c0b5780 msgr2=0x7f233c148280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:13.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 shutdown_connections 2022-07-12T08:49:13.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.541+0000 7f23428b6700 1 -- 172.21.15.61:0/1612192151 wait complete. 2022-07-12T08:49:13.554 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 57 2022-07-12T08:49:13.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:49:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc088c03700 1 Processor -- start 2022-07-12T08:49:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc088c03700 1 -- start start 2022-07-12T08:49:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc088c03700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc0840ab440 0x7fc084151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc088c03700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840ac440 con 0x7fc084147d50 2022-07-12T08:49:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc088c03700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840afe40 con 0x7fc0840ab060 2022-07-12T08:49:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc088c03700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0840b09b0 con 0x7fc0840ab440 2022-07-12T08:49:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc082d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc0840ab440 0x7fc084151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc082d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc0840ab440 0x7fc084151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59734/0 (socket says 172.21.15.61:59734) 2022-07-12T08:49:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc081d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc084147d50 0x7fc08405bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47952/0 (socket says 172.21.15.61:47952) 2022-07-12T08:49:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc082d9d700 1 -- 172.21.15.61:0/2917663136 learned_addr learned my addr 172.21.15.61:0/2917663136 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc082d9d700 1 -- 172.21.15.61:0/2917663136 >> v1:172.21.15.111:6789/0 conn(0x7fc0840ab060 legacy=0x7fc0840b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc082d9d700 1 -- 172.21.15.61:0/2917663136 >> v1:172.21.15.61:6789/0 conn(0x7fc084147d50 legacy=0x7fc08405bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.669+0000 7fc082d9d700 1 -- 172.21.15.61:0/2917663136 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0840ab8b0 con 0x7fc0840ab440 2022-07-12T08:49:13.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc082d9d700 1 --2- 172.21.15.61:0/2917663136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab440 0x7fc084151ea0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fc07c0032f0 tx=0x7fc07c009290).ready entity=mon.0 client_cookie=160356fc927ab2b8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:13.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08159a700 1 -- 172.21.15.61:0/2917663136 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc07c00b8a0 con 0x7fc0840ab440 2022-07-12T08:49:13.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08159a700 1 -- 172.21.15.61:0/2917663136 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c00ba00 con 0x7fc0840ab440 2022-07-12T08:49:13.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08159a700 1 -- 172.21.15.61:0/2917663136 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc07c00bb70 con 0x7fc0840ab440 2022-07-12T08:49:13.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- 172.21.15.61:0/2917663136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab440 msgr2=0x7fc084151ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 --2- 172.21.15.61:0/2917663136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab440 0x7fc084151ea0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 rx=0x7fc07c0032f0 tx=0x7fc07c009290).stop 2022-07-12T08:49:13.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- 172.21.15.61:0/2917663136 shutdown_connections 2022-07-12T08:49:13.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 --2- 172.21.15.61:0/2917663136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab440 0x7fc084151ea0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- 172.21.15.61:0/2917663136 >> 172.21.15.61:0/2917663136 conn(0x7fc0840b5290 msgr2=0x7fc0840b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:13.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- 172.21.15.61:0/2917663136 shutdown_connections 2022-07-12T08:49:13.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- 172.21.15.61:0/2917663136 wait complete. 2022-07-12T08:49:13.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 Processor -- start 2022-07-12T08:49:13.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- start start 2022-07-12T08:49:13.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 0x7fc084149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0840ab440 0x7fc084156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc084147d50 0x7fc0840ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0841564c0 con 0x7fc0840ab060 2022-07-12T08:49:13.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc084156760 con 0x7fc084147d50 2022-07-12T08:49:13.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc084156a00 con 0x7fc0840ab440 2022-07-12T08:49:13.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 0x7fc084149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 0x7fc084149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59738/0 (socket says 172.21.15.61:59738) 2022-07-12T08:49:13.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 -- 172.21.15.61:0/3435482537 learned_addr learned my addr 172.21.15.61:0/3435482537 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:13.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc082d9d700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc084147d50 0x7fc0840ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc081d9b700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0840ab440 0x7fc084156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 -- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0840ab440 msgr2=0x7fc084156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0840ab440 0x7fc084156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 -- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc084147d50 msgr2=0x7fc0840ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:13.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc084147d50 0x7fc0840ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:13.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 -- 172.21.15.61:0/3435482537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c009530 con 0x7fc0840ab060 2022-07-12T08:49:13.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc082d9d700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc084147d50 0x7fc0840ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:13.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc081d9b700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0840ab440 0x7fc084156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:13.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc08259c700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 0x7fc084149a10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fc0840b51e0 tx=0x7fc074007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:13.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc07400ce30 con 0x7fc0840ab060 2022-07-12T08:49:13.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07400f470 con 0x7fc0840ab060 2022-07-12T08:49:13.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc07400f7d0 con 0x7fc0840ab060 2022-07-12T08:49:13.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.673+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0841489f0 con 0x7fc0840ab060 2022-07-12T08:49:13.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0840acdc0 con 0x7fc0840ab060 2022-07-12T08:49:13.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0840ad070 con 0x7fc0840ab060 2022-07-12T08:49:13.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc074019030 con 0x7fc0840ab060 2022-07-12T08:49:13.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc0737fe700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:13.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc07406b330 con 0x7fc0840ab060 2022-07-12T08:49:13.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc081d9b700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:13.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.677+0000 7fc081d9b700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fc078008000 tx=0x7fc078009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:13.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.685+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0740421e0 con 0x7fc0840ab060 2022-07-12T08:49:13.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:13.845+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc084156bc0 con 0x7fc0840ab060 2022-07-12T08:49:14.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.245+0000 7fc0737fe700 1 -- 172.21.15.61:0/3435482537 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc074045800 con 0x7fc0840ab060 2022-07-12T08:49:14.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 msgr2=0x7fc060055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:14.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fc078008000 tx=0x7fc078009040).stop 2022-07-12T08:49:14.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 msgr2=0x7fc084149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:14.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 0x7fc084149a10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7fc0840b51e0 tx=0x7fc074007c50).stop 2022-07-12T08:49:14.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 shutdown_connections 2022-07-12T08:49:14.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc060053a40 0x7fc060055f00 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc084147d50 0x7fc0840ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc0840ab440 0x7fc084156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 --2- 172.21.15.61:0/3435482537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0840ab060 0x7fc084149a10 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 >> 172.21.15.61:0/3435482537 conn(0x7fc0840b5290 msgr2=0x7fc0840b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:14.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 shutdown_connections 2022-07-12T08:49:14.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.249+0000 7fc088c03700 1 -- 172.21.15.61:0/3435482537 wait complete. 2022-07-12T08:49:14.271 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:49:14.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.385+0000 7fc388b92700 1 Processor -- start 2022-07-12T08:49:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.385+0000 7fc388b92700 1 -- start start 2022-07-12T08:49:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc38405d060 0x7fc384145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc384067860 0x7fc384146760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3840afca0 con 0x7fc3840b3c60 2022-07-12T08:49:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38405d440 con 0x7fc38405d060 2022-07-12T08:49:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3840aff00 con 0x7fc384067860 2022-07-12T08:49:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc38405d060 0x7fc384145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc38405d060 0x7fc384145680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59746/0 (socket says 172.21.15.61:59746) 2022-07-12T08:49:14.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 -- 172.21.15.61:0/4013902080 learned_addr learned my addr 172.21.15.61:0/4013902080 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:14.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc383fff700 1 --2- 172.21.15.61:0/4013902080 >> v2:172.21.15.111:3300/0 conn(0x7fc384067860 0x7fc384146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:14.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 -- 172.21.15.61:0/4013902080 >> v2:172.21.15.111:3300/0 conn(0x7fc384067860 msgr2=0x7fc384146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:14.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 --2- 172.21.15.61:0/4013902080 >> v2:172.21.15.111:3300/0 conn(0x7fc384067860 0x7fc384146760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 -- 172.21.15.61:0/4013902080 >> v1:172.21.15.111:6789/0 conn(0x7fc3840b3c60 legacy=0x7fc384145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 -- 172.21.15.61:0/4013902080 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3840b4200 con 0x7fc38405d060 2022-07-12T08:49:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3837fe700 1 --2- 172.21.15.61:0/4013902080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc38405d060 0x7fc384145680 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fc374002f70 tx=0x7fc374006eb0).ready entity=mon.0 client_cookie=88bee55136891439 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:14.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3827fc700 1 -- 172.21.15.61:0/4013902080 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37400d5c0 con 0x7fc38405d060 2022-07-12T08:49:14.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3827fc700 1 -- 172.21.15.61:0/4013902080 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374000e90 con 0x7fc38405d060 2022-07-12T08:49:14.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc3827fc700 1 -- 172.21.15.61:0/4013902080 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37400d950 con 0x7fc38405d060 2022-07-12T08:49:14.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- 172.21.15.61:0/4013902080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc38405d060 msgr2=0x7fc384145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:14.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 --2- 172.21.15.61:0/4013902080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc38405d060 0x7fc384145680 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fc374002f70 tx=0x7fc374006eb0).stop 2022-07-12T08:49:14.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- 172.21.15.61:0/4013902080 shutdown_connections 2022-07-12T08:49:14.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 --2- 172.21.15.61:0/4013902080 >> v2:172.21.15.111:3300/0 conn(0x7fc384067860 0x7fc384146760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 --2- 172.21.15.61:0/4013902080 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc38405d060 0x7fc384145680 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- 172.21.15.61:0/4013902080 >> 172.21.15.61:0/4013902080 conn(0x7fc3840b67c0 msgr2=0x7fc384066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- 172.21.15.61:0/4013902080 shutdown_connections 2022-07-12T08:49:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.389+0000 7fc388b92700 1 -- 172.21.15.61:0/4013902080 wait complete. 2022-07-12T08:49:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 Processor -- start 2022-07-12T08:49:14.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 -- start start 2022-07-12T08:49:14.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38405d060 0x7fc384144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:14.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc384067860 0x7fc3840922d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:14.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 0x7fc3840927c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38409db60 con 0x7fc3840b3c60 2022-07-12T08:49:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3841526a0 con 0x7fc38405d060 2022-07-12T08:49:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc388b92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc384092d70 con 0x7fc384067860 2022-07-12T08:49:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 0x7fc3840927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:14.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 0x7fc3840927c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59754/0 (socket says 172.21.15.61:59754) 2022-07-12T08:49:14.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 -- 172.21.15.61:0/752586022 learned_addr learned my addr 172.21.15.61:0/752586022 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:14.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc382ffd700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc384067860 0x7fc3840922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc3837fe700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38405d060 0x7fc384144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 -- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc384067860 msgr2=0x7fc3840922d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc384067860 0x7fc3840922d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 -- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38405d060 msgr2=0x7fc384144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38405d060 0x7fc384144140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:14.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 -- 172.21.15.61:0/752586022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37400b1e0 con 0x7fc3840b3c60 2022-07-12T08:49:14.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc3837fe700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38405d060 0x7fc384144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:14.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.393+0000 7fc383fff700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 0x7fc3840927c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7fc37c006e00 tx=0x7fc37c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:14.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37c00eeb0 con 0x7fc3840b3c60 2022-07-12T08:49:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c00c850 con 0x7fc3840b3c60 2022-07-12T08:49:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc384152890 con 0x7fc3840b3c60 2022-07-12T08:49:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc37c00cbb0 con 0x7fc3840b3c60 2022-07-12T08:49:14.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc384152d30 con 0x7fc3840b3c60 2022-07-12T08:49:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384152a20 con 0x7fc3840b3c60 2022-07-12T08:49:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc37c022070 con 0x7fc3840b3c60 2022-07-12T08:49:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc380ff9700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc368053a40 0x7fc368055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:14.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc37c06c620 con 0x7fc3840b3c60 2022-07-12T08:49:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc3837fe700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc368053a40 0x7fc368055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.397+0000 7fc3837fe700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc368053a40 0x7fc368055f00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fc37401a000 tx=0x7fc374015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.405+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc37c043490 con 0x7fc3840b3c60 2022-07-12T08:49:14.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:14.533+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc384057460 con 0x7fc3840b3c60 2022-07-12T08:49:15.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc380ff9700 1 -- 172.21.15.61:0/752586022 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v102) v1 ==== 185+0+0 (secure 0 0 0) 0x7fc37c046ab0 con 0x7fc3840b3c60 2022-07-12T08:49:15.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc368053a40 msgr2=0x7fc368055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:15.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc388b92700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc368053a40 0x7fc368055f00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7fc37401a000 tx=0x7fc374015000).stop 2022-07-12T08:49:15.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 msgr2=0x7fc3840927c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:15.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc388b92700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 0x7fc3840927c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7fc37c006e00 tx=0x7fc37c0091d0).stop 2022-07-12T08:49:15.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 shutdown_connections 2022-07-12T08:49:15.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.257+0000 7fc388b92700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3840b3c60 0x7fc3840927c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.261+0000 7fc388b92700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc384067860 0x7fc3840922d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.261+0000 7fc388b92700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc368053a40 0x7fc368055f00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.261+0000 7fc388b92700 1 --2- 172.21.15.61:0/752586022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc38405d060 0x7fc384144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.261+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 >> 172.21.15.61:0/752586022 conn(0x7fc3840b67c0 msgr2=0x7fc38405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:15.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.261+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 shutdown_connections 2022-07-12T08:49:15.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.261+0000 7fc388b92700 1 -- 172.21.15.61:0/752586022 wait complete. 2022-07-12T08:49:15.276 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:49:15.283 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:49:15.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 Processor -- start 2022-07-12T08:49:15.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 -- start start 2022-07-12T08:49:15.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f29940b4840 0x7f29941420b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29940ba290 con 0x7f29940b3180 2022-07-12T08:49:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29940b1410 con 0x7f29940b4840 2022-07-12T08:49:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2998ae2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29940b4010 con 0x7f29940b9eb0 2022-07-12T08:49:15.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f299259c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:15.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2992d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f29940b4840 0x7f29941420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:15.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2991d9b700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f29940b3180 0x7f29940b2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:33346/0 (socket says 172.21.15.61:33346) 2022-07-12T08:49:15.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.397+0000 7f2991d9b700 1 -- 172.21.15.61:0/3479787330 learned_addr learned my addr 172.21.15.61:0/3479787330 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:15.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2992d9d700 1 -- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 msgr2=0x7f29940b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2992d9d700 1 --2- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2992d9d700 1 -- 172.21.15.61:0/3479787330 >> v1:172.21.15.111:6789/0 conn(0x7f29940b3180 legacy=0x7f29940b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f299159a700 1 -- 172.21.15.61:0/3479787330 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611578289 0 0) 0x7f29940ba290 con 0x7f29940b3180 2022-07-12T08:49:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2992d9d700 1 -- 172.21.15.61:0/3479787330 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29940b42b0 con 0x7f29940b4840 2022-07-12T08:49:15.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2992d9d700 1 --2- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b4840 0x7f29941420b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f298c0032f0 tx=0x7f298c0091d0).ready entity=mon.1 client_cookie=d52979ad57445fcb server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:15.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f299159a700 1 -- 172.21.15.61:0/3479787330 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f298c00b6c0 con 0x7f29940b4840 2022-07-12T08:49:15.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f299159a700 1 -- 172.21.15.61:0/3479787330 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c00b820 con 0x7f29940b4840 2022-07-12T08:49:15.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f299159a700 1 -- 172.21.15.61:0/3479787330 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f298c00b990 con 0x7f29940b4840 2022-07-12T08:49:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 -- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b4840 msgr2=0x7f29941420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 --2- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b4840 0x7f29941420b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f298c0032f0 tx=0x7f298c0091d0).stop 2022-07-12T08:49:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 -- 172.21.15.61:0/3479787330 shutdown_connections 2022-07-12T08:49:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 --2- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b4840 0x7f29941420b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 --2- 172.21.15.61:0/3479787330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 -- 172.21.15.61:0/3479787330 >> 172.21.15.61:0/3479787330 conn(0x7f29940678c0 msgr2=0x7f29940b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 -- 172.21.15.61:0/3479787330 shutdown_connections 2022-07-12T08:49:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.401+0000 7f2998ae2700 1 -- 172.21.15.61:0/3479787330 wait complete. 2022-07-12T08:49:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 Processor -- start 2022-07-12T08:49:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 -- start start 2022-07-12T08:49:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b3180 0x7f29940acc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 0x7f29940ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2994146be0 con 0x7f29940b4840 2022-07-12T08:49:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29940afa50 con 0x7f29940b3180 2022-07-12T08:49:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2994146940 con 0x7f29940b9eb0 2022-07-12T08:49:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 0x7f29940ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 0x7f29940ad830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59766/0 (socket says 172.21.15.61:59766) 2022-07-12T08:49:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 -- 172.21.15.61:0/2836320343 learned_addr learned my addr 172.21.15.61:0/2836320343 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2992d9d700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f299259c700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b3180 0x7f29940acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 -- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 msgr2=0x7f29940add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940add20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 -- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b3180 msgr2=0x7f29940acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b3180 0x7f29940acc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 -- 172.21.15.61:0/2836320343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c009470 con 0x7f29940b4840 2022-07-12T08:49:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2991d9b700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 0x7f29940ad830 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f297c001000 tx=0x7f297c007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f297c0070d0 con 0x7f29940b4840 2022-07-12T08:49:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c007230 con 0x7f29940b4840 2022-07-12T08:49:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29940af5e0 con 0x7f29940b4840 2022-07-12T08:49:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.405+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f297c0055d0 con 0x7f29940b4840 2022-07-12T08:49:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2994146510 con 0x7f29940b4840 2022-07-12T08:49:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29940af770 con 0x7f29940b4840 2022-07-12T08:49:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f297c010070 con 0x7f29940b4840 2022-07-12T08:49:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f298b7fe700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2978053a40 0x7f2978055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f297c06b280 con 0x7f29940b4840 2022-07-12T08:49:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f299259c700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2978053a40 0x7f2978055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:15.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.409+0000 7f299259c700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2978053a40 0x7f2978055f00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f2984006000 tx=0x7f2984006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:15.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.413+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f297c042340 con 0x7f29940b4840 2022-07-12T08:49:15.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:15.541+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2994057460 con 0x7f29940b4840 2022-07-12T08:49:16.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f298b7fe700 1 -- 172.21.15.61:0/2836320343 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v103) v1 ==== 173+0+0 (secure 0 0 0) 0x7f297c045960 con 0x7f29940b4840 2022-07-12T08:49:16.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2978053a40 msgr2=0x7f2978055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2978053a40 0x7f2978055f00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f2984006000 tx=0x7f2984006d90).stop 2022-07-12T08:49:16.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 msgr2=0x7f29940ad830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 0x7f29940ad830 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7f297c001000 tx=0x7f297c007b40).stop 2022-07-12T08:49:16.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 shutdown_connections 2022-07-12T08:49:16.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29940b9eb0 0x7f29940add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29940b4840 0x7f29940ad830 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2978053a40 0x7f2978055f00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 --2- 172.21.15.61:0/2836320343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29940b3180 0x7f29940acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.269+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 >> 172.21.15.61:0/2836320343 conn(0x7f29940678c0 msgr2=0x7f29940b8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:16.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.273+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 shutdown_connections 2022-07-12T08:49:16.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.273+0000 7f2998ae2700 1 -- 172.21.15.61:0/2836320343 wait complete. 2022-07-12T08:49:16.285 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:49:16.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:49:16.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.401+0000 7f7f14d4c700 1 Processor -- start 2022-07-12T08:49:16.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.401+0000 7f7f14d4c700 1 -- start start 2022-07-12T08:49:16.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10060eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f10146b00 0x7f7f10142de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1005d440 con 0x7f7f100aafa0 2022-07-12T08:49:16.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f100aa230 con 0x7f7f10146b00 2022-07-12T08:49:16.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f10060c60 con 0x7f7f1005d060 2022-07-12T08:49:16.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f10146b00 0x7f7f10142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f10146b00 0x7f7f10142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59772/0 (socket says 172.21.15.61:59772) 2022-07-12T08:49:16.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 -- 172.21.15.61:0/2439584239 learned_addr learned my addr 172.21.15.61:0/2439584239 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0f7fe700 1 --2- 172.21.15.61:0/2439584239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10060eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 -- 172.21.15.61:0/2439584239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 msgr2=0x7f7f10060eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 --2- 172.21.15.61:0/2439584239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10060eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 -- 172.21.15.61:0/2439584239 >> v1:172.21.15.111:6789/0 conn(0x7f7f100aafa0 legacy=0x7f7f1005f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 -- 172.21.15.61:0/2439584239 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f10146120 con 0x7f7f10146b00 2022-07-12T08:49:16.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0ffff700 1 --2- 172.21.15.61:0/2439584239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f10146b00 0x7f7f10142de0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f7f100aae40 tx=0x7f7f08009290).ready entity=mon.0 client_cookie=b3c94d72fad501d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:16.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0e7fc700 1 -- 172.21.15.61:0/2439584239 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f0800b860 con 0x7f7f10146b00 2022-07-12T08:49:16.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0e7fc700 1 -- 172.21.15.61:0/2439584239 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0800b9c0 con 0x7f7f10146b00 2022-07-12T08:49:16.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f0e7fc700 1 -- 172.21.15.61:0/2439584239 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f0800bb30 con 0x7f7f10146b00 2022-07-12T08:49:16.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2439584239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f10146b00 msgr2=0x7f7f10142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2439584239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f10146b00 0x7f7f10142de0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7f7f100aae40 tx=0x7f7f08009290).stop 2022-07-12T08:49:16.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2439584239 shutdown_connections 2022-07-12T08:49:16.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2439584239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f10146b00 0x7f7f10142de0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2439584239 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10060eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2439584239 >> 172.21.15.61:0/2439584239 conn(0x7f7f100b1a50 msgr2=0x7f7f100b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2439584239 shutdown_connections 2022-07-12T08:49:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.405+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2439584239 wait complete. 2022-07-12T08:49:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 Processor -- start 2022-07-12T08:49:16.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 -- start start 2022-07-12T08:49:16.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10064da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 0x7f7f100aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f10146b00 0x7f7f100b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100657c0 con 0x7f7f100aafa0 2022-07-12T08:49:16.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f100643c0 con 0x7f7f10146b00 2022-07-12T08:49:16.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f10064660 con 0x7f7f1005d060 2022-07-12T08:49:16.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 0x7f7f100aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 0x7f7f100aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59782/0 (socket says 172.21.15.61:59782) 2022-07-12T08:49:16.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 -- 172.21.15.61:0/2905642956 learned_addr learned my addr 172.21.15.61:0/2905642956 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:16.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0f7fe700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10064da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0ffff700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f10146b00 0x7f7f100b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 -- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 msgr2=0x7f7f10064da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10064da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 -- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f10146b00 msgr2=0x7f7f100b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f10146b00 0x7f7f100b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 -- 172.21.15.61:0/2905642956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08009530 con 0x7f7f100aafa0 2022-07-12T08:49:16.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0f7fe700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10064da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:16.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0ffff700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f10146b00 0x7f7f100b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:16.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0effd700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 0x7f7f100aeaf0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f7f100b2b40 tx=0x7f7f00007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:16.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f0000ce30 con 0x7f7f100aafa0 2022-07-12T08:49:16.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0000f470 con 0x7f7f100aafa0 2022-07-12T08:49:16.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f10064900 con 0x7f7f100aafa0 2022-07-12T08:49:16.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f0000f7d0 con 0x7f7f100aafa0 2022-07-12T08:49:16.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.409+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f100676d0 con 0x7f7f100aafa0 2022-07-12T08:49:16.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.413+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f10057460 con 0x7f7f100aafa0 2022-07-12T08:49:16.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.413+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7f00019030 con 0x7f7f100aafa0 2022-07-12T08:49:16.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.413+0000 7f7f0cff9700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ef0053a60 0x7f7ef0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.413+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7f0006afc0 con 0x7f7f100aafa0 2022-07-12T08:49:16.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.413+0000 7f7f0f7fe700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ef0053a60 0x7f7ef0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.413+0000 7f7f0f7fe700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ef0053a60 0x7f7ef0055f20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f7f0400b000 tx=0x7f7f04008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:16.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.417+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7f000421a0 con 0x7f7f100aafa0 2022-07-12T08:49:16.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.557+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f7f10068350 con 0x7f7f100aafa0 2022-07-12T08:49:16.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.565+0000 7f7f0cff9700 1 -- 172.21.15.61:0/2905642956 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7f000457c0 con 0x7f7f100aafa0 2022-07-12T08:49:16.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ef0053a60 msgr2=0x7f7ef0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ef0053a60 0x7f7ef0055f20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f7f0400b000 tx=0x7f7f04008040).stop 2022-07-12T08:49:16.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 msgr2=0x7f7f100aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 0x7f7f100aeaf0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f7f100b2b40 tx=0x7f7f00007c50).stop 2022-07-12T08:49:16.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 shutdown_connections 2022-07-12T08:49:16.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ef0053a60 0x7f7ef0055f20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f10146b00 0x7f7f100b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f100aafa0 0x7f7f100aeaf0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 --2- 172.21.15.61:0/2905642956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7f1005d060 0x7f7f10064da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 >> 172.21.15.61:0/2905642956 conn(0x7f7f100b1a50 msgr2=0x7f7f100b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:16.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 shutdown_connections 2022-07-12T08:49:16.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.569+0000 7f7f14d4c700 1 -- 172.21.15.61:0/2905642956 wait complete. 2022-07-12T08:49:16.589 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-12T08:49:16.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 Processor -- start 2022-07-12T08:49:16.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- start start 2022-07-12T08:49:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f400ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8f400b01f0 0x7f8f4014a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f4005cb00 0x7f8f4005cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400b0a40 con 0x7f8f4005cb00 2022-07-12T08:49:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f4005d430 con 0x7f8f400b01f0 2022-07-12T08:49:16.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400ad2f0 con 0x7f8f4005c050 2022-07-12T08:49:16.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f4005cb00 0x7f8f4005cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f4005cb00 0x7f8f4005cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59790/0 (socket says 172.21.15.61:59790) 2022-07-12T08:49:16.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 -- 172.21.15.61:0/3932296023 learned_addr learned my addr 172.21.15.61:0/3932296023 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:16.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f3ffff700 1 --2- 172.21.15.61:0/3932296023 >> v2:172.21.15.111:3300/0 conn(0x7f8f400b01f0 0x7f8f4014a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f44f41700 1 --2- 172.21.15.61:0/3932296023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f400ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 -- 172.21.15.61:0/3932296023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 msgr2=0x7f8f400ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:16.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 --2- 172.21.15.61:0/3932296023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f400ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 -- 172.21.15.61:0/3932296023 >> v2:172.21.15.111:3300/0 conn(0x7f8f400b01f0 msgr2=0x7f8f4014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 --2- 172.21.15.61:0/3932296023 >> v2:172.21.15.111:3300/0 conn(0x7f8f400b01f0 0x7f8f4014a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 -- 172.21.15.61:0/3932296023 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4005c430 con 0x7f8f4005cb00 2022-07-12T08:49:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f3ffff700 1 --2- 172.21.15.61:0/3932296023 >> v2:172.21.15.111:3300/0 conn(0x7f8f400b01f0 0x7f8f4014a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f45742700 1 --2- 172.21.15.61:0/3932296023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f4005cb00 0x7f8f4005cf40 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7f8f4005d570 tx=0x7f8f38006eb0).ready entity=mon.0 client_cookie=4445321b24a5996e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:16.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f3f7fe700 1 -- 172.21.15.61:0/3932296023 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f3800d570 con 0x7f8f4005cb00 2022-07-12T08:49:16.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f3f7fe700 1 -- 172.21.15.61:0/3932296023 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38000e90 con 0x7f8f4005cb00 2022-07-12T08:49:16.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f3f7fe700 1 -- 172.21.15.61:0/3932296023 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f3800d8c0 con 0x7f8f4005cb00 2022-07-12T08:49:16.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- 172.21.15.61:0/3932296023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f4005cb00 msgr2=0x7f8f4005cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- 172.21.15.61:0/3932296023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f4005cb00 0x7f8f4005cf40 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 rx=0x7f8f4005d570 tx=0x7f8f38006eb0).stop 2022-07-12T08:49:16.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- 172.21.15.61:0/3932296023 shutdown_connections 2022-07-12T08:49:16.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- 172.21.15.61:0/3932296023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f4005cb00 0x7f8f4005cf40 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- 172.21.15.61:0/3932296023 >> v2:172.21.15.111:3300/0 conn(0x7f8f400b01f0 0x7f8f4014a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 --2- 172.21.15.61:0/3932296023 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f400ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.689+0000 7f8f471a5700 1 -- 172.21.15.61:0/3932296023 >> 172.21.15.61:0/3932296023 conn(0x7f8f40062c00 msgr2=0x7f8f40062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:16.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 -- 172.21.15.61:0/3932296023 shutdown_connections 2022-07-12T08:49:16.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 -- 172.21.15.61:0/3932296023 wait complete. 2022-07-12T08:49:16.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 Processor -- start 2022-07-12T08:49:16.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 -- start start 2022-07-12T08:49:16.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f40148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f4005cb00 0x7f8f40159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 0x7f8f40159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400652b0 con 0x7f8f400b01f0 2022-07-12T08:49:16.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f40065550 con 0x7f8f4005cb00 2022-07-12T08:49:16.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f471a5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f40064fe0 con 0x7f8f4005c050 2022-07-12T08:49:16.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 0x7f8f40159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 0x7f8f40159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59796/0 (socket says 172.21.15.61:59796) 2022-07-12T08:49:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 -- 172.21.15.61:0/4013778531 learned_addr learned my addr 172.21.15.61:0/4013778531 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f44f41700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f40148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f3ffff700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f4005cb00 0x7f8f40159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 -- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 msgr2=0x7f8f40148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f40148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 -- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f4005cb00 msgr2=0x7f8f40159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:16.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f4005cb00 0x7f8f40159900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:16.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 -- 172.21.15.61:0/4013778531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3800b1e0 con 0x7f8f400b01f0 2022-07-12T08:49:16.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f3ffff700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f4005cb00 0x7f8f40159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:16.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f44f41700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f40148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:16.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f45742700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 0x7f8f40159df0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f8f4005d570 tx=0x7f8f38015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:16.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f38021070 con 0x7f8f400b01f0 2022-07-12T08:49:16.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.693+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38011760 con 0x7f8f400b01f0 2022-07-12T08:49:16.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f38011ac0 con 0x7f8f400b01f0 2022-07-12T08:49:16.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f40064930 con 0x7f8f400b01f0 2022-07-12T08:49:16.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4015a2e0 con 0x7f8f400b01f0 2022-07-12T08:49:16.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f40065710 con 0x7f8f400b01f0 2022-07-12T08:49:16.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f3800f070 con 0x7f8f400b01f0 2022-07-12T08:49:16.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f3dffb700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f240539b0 0x7f8f24055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:16.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8f3806cf50 con 0x7f8f400b01f0 2022-07-12T08:49:16.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f44f41700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f240539b0 0x7f8f24055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:16.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.697+0000 7f8f44f41700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f240539b0 0x7f8f24055e70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f8f2800c000 tx=0x7f8f28008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:16.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.705+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f38044190 con 0x7f8f400b01f0 2022-07-12T08:49:16.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:16.869+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7f8f40057460 con 0x7f8f400b01f0 2022-07-12T08:49:17.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.269+0000 7f8f3dffb700 1 -- 172.21.15.61:0/4013778531 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v371) v1 ==== 127+0+0 (secure 0 0 0) 0x7f8f380477b0 con 0x7f8f400b01f0 2022-07-12T08:49:17.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f240539b0 msgr2=0x7f8f24055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:17.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f240539b0 0x7f8f24055e70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f8f2800c000 tx=0x7f8f28008040).stop 2022-07-12T08:49:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 msgr2=0x7f8f40159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 0x7f8f40159df0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f8f4005d570 tx=0x7f8f38015000).stop 2022-07-12T08:49:17.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 shutdown_connections 2022-07-12T08:49:17.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f400b01f0 0x7f8f40159df0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f240539b0 0x7f8f24055e70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f4005cb00 0x7f8f40159900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 --2- 172.21.15.61:0/4013778531 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f4005c050 0x7f8f40148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 >> 172.21.15.61:0/4013778531 conn(0x7f8f40062c00 msgr2=0x7f8f40147b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:17.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 shutdown_connections 2022-07-12T08:49:17.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.273+0000 7f8f471a5700 1 -- 172.21.15.61:0/4013778531 wait complete. 2022-07-12T08:49:17.300 INFO:tasks.cephfs_test_runner:test_filtered_df (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:49:17.301 INFO:tasks.cephfs_test_runner:Starting test: test_fs_new (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:49:17.302 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-12T08:49:17.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 Processor -- start 2022-07-12T08:49:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- start start 2022-07-12T08:49:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf24060b00 0x7faf24143b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7faf241442b0 0x7faf240981e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf24061030 con 0x7faf24143720 2022-07-12T08:49:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf240b1d20 con 0x7faf24060b00 2022-07-12T08:49:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf240612d0 con 0x7faf241442b0 2022-07-12T08:49:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf24060b00 0x7faf24143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf24060b00 0x7faf24143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59800/0 (socket says 172.21.15.61:59800) 2022-07-12T08:49:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 -- 172.21.15.61:0/179806431 learned_addr learned my addr 172.21.15.61:0/179806431 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2ac2e700 1 --2- 172.21.15.61:0/179806431 >> v2:172.21.15.111:3300/0 conn(0x7faf241442b0 0x7faf240981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 -- 172.21.15.61:0/179806431 >> v2:172.21.15.111:3300/0 conn(0x7faf241442b0 msgr2=0x7faf240981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 --2- 172.21.15.61:0/179806431 >> v2:172.21.15.111:3300/0 conn(0x7faf241442b0 0x7faf240981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 -- 172.21.15.61:0/179806431 >> v1:172.21.15.111:6789/0 conn(0x7faf24143720 legacy=0x7faf240603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 -- 172.21.15.61:0/179806431 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf24140f40 con 0x7faf24060b00 2022-07-12T08:49:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf29c2c700 1 --2- 172.21.15.61:0/179806431 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24060b00 0x7faf24143b00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7faf1c0032f0 tx=0x7faf1c009290).ready entity=mon.0 client_cookie=3afd12610d8fb2d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2942b700 1 -- 172.21.15.61:0/179806431 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf1c00b870 con 0x7faf24060b00 2022-07-12T08:49:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2942b700 1 -- 172.21.15.61:0/179806431 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1c00b9d0 con 0x7faf24060b00 2022-07-12T08:49:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2942b700 1 -- 172.21.15.61:0/179806431 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf1c00bb40 con 0x7faf24060b00 2022-07-12T08:49:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- 172.21.15.61:0/179806431 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24060b00 msgr2=0x7faf24143b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 --2- 172.21.15.61:0/179806431 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24060b00 0x7faf24143b00 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 rx=0x7faf1c0032f0 tx=0x7faf1c009290).stop 2022-07-12T08:49:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- 172.21.15.61:0/179806431 shutdown_connections 2022-07-12T08:49:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 --2- 172.21.15.61:0/179806431 >> v2:172.21.15.111:3300/0 conn(0x7faf241442b0 0x7faf240981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 --2- 172.21.15.61:0/179806431 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24060b00 0x7faf24143b00 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- 172.21.15.61:0/179806431 >> 172.21.15.61:0/179806431 conn(0x7faf24064720 msgr2=0x7faf240b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:17.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.413+0000 7faf2c691700 1 -- 172.21.15.61:0/179806431 shutdown_connections 2022-07-12T08:49:17.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 -- 172.21.15.61:0/179806431 wait complete. 2022-07-12T08:49:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 Processor -- start 2022-07-12T08:49:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 -- start start 2022-07-12T08:49:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf24060b00 0x7faf240667a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 0x7faf240aa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf241442b0 0x7faf240aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf240afdc0 con 0x7faf24143720 2022-07-12T08:49:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf240acbf0 con 0x7faf241442b0 2022-07-12T08:49:17.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2c691700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf240ac950 con 0x7faf24060b00 2022-07-12T08:49:17.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 0x7faf240aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:17.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 0x7faf240aa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59808/0 (socket says 172.21.15.61:59808) 2022-07-12T08:49:17.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 -- 172.21.15.61:0/3018556330 learned_addr learned my addr 172.21.15.61:0/3018556330 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:17.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2ac2e700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf241442b0 0x7faf240aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:17.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf2a42d700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf24060b00 0x7faf240667a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:17.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 -- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf24060b00 msgr2=0x7faf240667a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:17.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf24060b00 0x7faf240667a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 -- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf241442b0 msgr2=0x7faf240aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:17.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf241442b0 0x7faf240aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.417+0000 7faf29c2c700 1 -- 172.21.15.61:0/3018556330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1c009530 con 0x7faf24143720 2022-07-12T08:49:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf29c2c700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 0x7faf240aa250 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7faf1c0032f0 tx=0x7faf1c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf1c021440 con 0x7faf24143720 2022-07-12T08:49:17.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1c0215a0 con 0x7faf24143720 2022-07-12T08:49:17.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf1c021900 con 0x7faf24143720 2022-07-12T08:49:17.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf240671d0 con 0x7faf24143720 2022-07-12T08:49:17.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf240aadd0 con 0x7faf24143720 2022-07-12T08:49:17.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faf1c029070 con 0x7faf24143720 2022-07-12T08:49:17.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf240ac7b0 con 0x7faf24143720 2022-07-12T08:49:17.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf237fe700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf04053920 0x7faf04055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:17.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.421+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7faf1c06d120 con 0x7faf24143720 2022-07-12T08:49:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.425+0000 7faf2a42d700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf04053920 0x7faf04055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.425+0000 7faf2a42d700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf04053920 0x7faf04055de0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7faf1800d000 tx=0x7faf18000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:17.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.425+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf240ac7b0 con 0x7faf24143720 2022-07-12T08:49:17.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:17.593+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7faf240ac7b0 con 0x7faf24143720 2022-07-12T08:49:18.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.273+0000 7faf237fe700 1 -- 172.21.15.61:0/3018556330 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v372) v1 ==== 125+0+0 (secure 0 0 0) 0x7faf240ac7b0 con 0x7faf24143720 2022-07-12T08:49:18.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf04053920 msgr2=0x7faf04055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf04053920 0x7faf04055de0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7faf1800d000 tx=0x7faf18000bc0).stop 2022-07-12T08:49:18.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 msgr2=0x7faf240aa250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 0x7faf240aa250 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7faf1c0032f0 tx=0x7faf1c016000).stop 2022-07-12T08:49:18.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 shutdown_connections 2022-07-12T08:49:18.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf04053920 0x7faf04055de0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf241442b0 0x7faf240aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf24143720 0x7faf240aa250 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 --2- 172.21.15.61:0/3018556330 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf24060b00 0x7faf240667a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 >> 172.21.15.61:0/3018556330 conn(0x7faf24064720 msgr2=0x7faf240982a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:18.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 shutdown_connections 2022-07-12T08:49:18.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.277+0000 7faf2c691700 1 -- 172.21.15.61:0/3018556330 wait complete. 2022-07-12T08:49:18.299 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:49:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.401+0000 7f7cc8638700 1 Processor -- start 2022-07-12T08:49:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.401+0000 7f7cc8638700 1 -- start start 2022-07-12T08:49:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7cc00b2f60 0x7f7cc013fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7cc0060360 0x7f7cc013c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc0143090 con 0x7f7cc00b2250 2022-07-12T08:49:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc00b3340 con 0x7f7cc00b2f60 2022-07-12T08:49:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc00664d0 con 0x7f7cc0060360 2022-07-12T08:49:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7cc00b2f60 0x7f7cc013fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7cc00b2f60 0x7f7cc013fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59816/0 (socket says 172.21.15.61:59816) 2022-07-12T08:49:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 -- 172.21.15.61:0/1240288206 learned_addr learned my addr 172.21.15.61:0/1240288206 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc6bd5700 1 --2- 172.21.15.61:0/1240288206 >> v2:172.21.15.111:3300/0 conn(0x7f7cc0060360 0x7f7cc013c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 -- 172.21.15.61:0/1240288206 >> v2:172.21.15.111:3300/0 conn(0x7f7cc0060360 msgr2=0x7f7cc013c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 --2- 172.21.15.61:0/1240288206 >> v2:172.21.15.111:3300/0 conn(0x7f7cc0060360 0x7f7cc013c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 -- 172.21.15.61:0/1240288206 >> v1:172.21.15.111:6789/0 conn(0x7f7cc00b2250 legacy=0x7f7cc0143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 -- 172.21.15.61:0/1240288206 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc01432c0 con 0x7f7cc00b2f60 2022-07-12T08:49:18.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc63d4700 1 --2- 172.21.15.61:0/1240288206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013fac0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f7cb4002f70 tx=0x7f7cb4006eb0).ready entity=mon.0 client_cookie=fc1f056cc0c22d42 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:18.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc53d2700 1 -- 172.21.15.61:0/1240288206 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cb400d5c0 con 0x7f7cc00b2f60 2022-07-12T08:49:18.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc53d2700 1 -- 172.21.15.61:0/1240288206 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4000e90 con 0x7f7cc00b2f60 2022-07-12T08:49:18.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc53d2700 1 -- 172.21.15.61:0/1240288206 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cb400d950 con 0x7f7cc00b2f60 2022-07-12T08:49:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- 172.21.15.61:0/1240288206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc00b2f60 msgr2=0x7f7cc013fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 --2- 172.21.15.61:0/1240288206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013fac0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f7cb4002f70 tx=0x7f7cb4006eb0).stop 2022-07-12T08:49:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- 172.21.15.61:0/1240288206 shutdown_connections 2022-07-12T08:49:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 --2- 172.21.15.61:0/1240288206 >> v2:172.21.15.111:3300/0 conn(0x7f7cc0060360 0x7f7cc013c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 --2- 172.21.15.61:0/1240288206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013fac0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- 172.21.15.61:0/1240288206 >> 172.21.15.61:0/1240288206 conn(0x7f7cc0063af0 msgr2=0x7f7cc0063f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- 172.21.15.61:0/1240288206 shutdown_connections 2022-07-12T08:49:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.405+0000 7f7cc8638700 1 -- 172.21.15.61:0/1240288206 wait complete. 2022-07-12T08:49:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 Processor -- start 2022-07-12T08:49:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 -- start start 2022-07-12T08:49:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 0x7f7cc013c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7cc00b2250 0x7f7cc013eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc00b0250 con 0x7f7cc0060360 2022-07-12T08:49:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc013baf0 con 0x7f7cc00b2f60 2022-07-12T08:49:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc8638700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc013b850 con 0x7f7cc00b2250 2022-07-12T08:49:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc63d4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 0x7f7cc013c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc63d4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 0x7f7cc013c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59822/0 (socket says 172.21.15.61:59822) 2022-07-12T08:49:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc63d4700 1 -- 172.21.15.61:0/2620128673 learned_addr learned my addr 172.21.15.61:0/2620128673 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.409+0000 7f7cc5bd3700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7cc00b2250 0x7f7cc013eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc6bd5700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc63d4700 1 -- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7cc00b2250 msgr2=0x7f7cc013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc63d4700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7cc00b2250 0x7f7cc013eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc63d4700 1 -- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc00b2f60 msgr2=0x7f7cc013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc63d4700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc63d4700 1 -- 172.21.15.61:0/2620128673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb400b1e0 con 0x7f7cc0060360 2022-07-12T08:49:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc63d4700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 0x7f7cc013c5f0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7f7cb4002f70 tx=0x7f7cb4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cb4011990 con 0x7f7cc0060360 2022-07-12T08:49:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4011af0 con 0x7f7cc0060360 2022-07-12T08:49:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc00af6e0 con 0x7f7cc0060360 2022-07-12T08:49:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cb4011e50 con 0x7f7cc0060360 2022-07-12T08:49:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc00afb80 con 0x7f7cc0060360 2022-07-12T08:49:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7cb4025030 con 0x7f7cc0060360 2022-07-12T08:49:18.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc0057460 con 0x7f7cc0060360 2022-07-12T08:49:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cb37fe700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ca0053a40 0x7f7ca0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7cb406cdf0 con 0x7f7cc0060360 2022-07-12T08:49:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.413+0000 7f7cc5bd3700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ca0053a40 0x7f7ca0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.417+0000 7f7cc5bd3700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ca0053a40 0x7f7ca0055f00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f7cbc00b000 tx=0x7f7cbc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:18.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.421+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7cb4044030 con 0x7f7cc0060360 2022-07-12T08:49:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.545+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7cc0140a50 con 0x7f7cc0060360 2022-07-12T08:49:18.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.553+0000 7f7cb37fe700 1 -- 172.21.15.61:0/2620128673 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7cb4023020 con 0x7f7cc0060360 2022-07-12T08:49:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ca0053a40 msgr2=0x7f7ca0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ca0053a40 0x7f7ca0055f00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f7cbc00b000 tx=0x7f7cbc009040).stop 2022-07-12T08:49:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 msgr2=0x7f7cc013c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 0x7f7cc013c5f0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 rx=0x7f7cb4002f70 tx=0x7f7cb4015000).stop 2022-07-12T08:49:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 shutdown_connections 2022-07-12T08:49:18.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7ca0053a40 0x7f7ca0055f00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7cc00b2f60 0x7f7cc013f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7cc00b2250 0x7f7cc013eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 --2- 172.21.15.61:0/2620128673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cc0060360 0x7f7cc013c5f0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 >> 172.21.15.61:0/2620128673 conn(0x7f7cc0063af0 msgr2=0x7f7cc005f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:18.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 shutdown_connections 2022-07-12T08:49:18.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.557+0000 7f7cc8638700 1 -- 172.21.15.61:0/2620128673 wait complete. 2022-07-12T08:49:18.576 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:49:18.577 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:18.594 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:18.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:18.637 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:18.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:18.681 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:49:18.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:18.682 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:18.683 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:18.692 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:49:18.693 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:18.694 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:18.694 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:18.695 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:18.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:18.705 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:49:18.705 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:18.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:18.706 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:18.707 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:49:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:18.717 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:49:18.717 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:49:18.718 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:49:18.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.821+0000 7f66e8d92700 1 Processor -- start 2022-07-12T08:49:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.821+0000 7f66e8d92700 1 -- start start 2022-07-12T08:49:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f66e40b4390 0x7f66e4140570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f66e41431e0 0x7f66e4143600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e40b3e80 con 0x7f66e40b6250 2022-07-12T08:49:18.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e4147390 con 0x7f66e41431e0 2022-07-12T08:49:18.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e40b6630 con 0x7f66e40b4390 2022-07-12T08:49:18.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f66e41431e0 0x7f66e4143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f66e41431e0 0x7f66e4143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59834/0 (socket says 172.21.15.61:59834) 2022-07-12T08:49:18.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 -- 172.21.15.61:0/380418138 learned_addr learned my addr 172.21.15.61:0/380418138 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:18.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e1d9b700 1 --2- 172.21.15.61:0/380418138 >> v2:172.21.15.111:3300/0 conn(0x7f66e40b4390 0x7f66e4140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 -- 172.21.15.61:0/380418138 >> v2:172.21.15.111:3300/0 conn(0x7f66e40b4390 msgr2=0x7f66e4140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:18.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 --2- 172.21.15.61:0/380418138 >> v2:172.21.15.111:3300/0 conn(0x7f66e40b4390 0x7f66e4140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 -- 172.21.15.61:0/380418138 >> v1:172.21.15.111:6789/0 conn(0x7f66e40b6250 legacy=0x7f66e413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 -- 172.21.15.61:0/380418138 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e41467b0 con 0x7f66e41431e0 2022-07-12T08:49:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e1d9b700 1 --2- 172.21.15.61:0/380418138 >> v2:172.21.15.111:3300/0 conn(0x7f66e40b4390 0x7f66e4140570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e2d9d700 1 --2- 172.21.15.61:0/380418138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e41431e0 0x7f66e4143600 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7f66dc003340 tx=0x7f66dc009290).ready entity=mon.0 client_cookie=c29dfdb61a82d0ba server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e159a700 1 -- 172.21.15.61:0/380418138 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66dc00b870 con 0x7f66e41431e0 2022-07-12T08:49:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e159a700 1 -- 172.21.15.61:0/380418138 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66dc00b9d0 con 0x7f66e41431e0 2022-07-12T08:49:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e159a700 1 -- 172.21.15.61:0/380418138 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66dc00bb40 con 0x7f66e41431e0 2022-07-12T08:49:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- 172.21.15.61:0/380418138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e41431e0 msgr2=0x7f66e4143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 --2- 172.21.15.61:0/380418138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e41431e0 0x7f66e4143600 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7f66dc003340 tx=0x7f66dc009290).stop 2022-07-12T08:49:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- 172.21.15.61:0/380418138 shutdown_connections 2022-07-12T08:49:18.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 --2- 172.21.15.61:0/380418138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e41431e0 0x7f66e4143600 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 --2- 172.21.15.61:0/380418138 >> v2:172.21.15.111:3300/0 conn(0x7f66e40b4390 0x7f66e4140570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- 172.21.15.61:0/380418138 >> 172.21.15.61:0/380418138 conn(0x7f66e4066f00 msgr2=0x7f66e40672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:18.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- 172.21.15.61:0/380418138 shutdown_connections 2022-07-12T08:49:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.825+0000 7f66e8d92700 1 -- 172.21.15.61:0/380418138 wait complete. 2022-07-12T08:49:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 Processor -- start 2022-07-12T08:49:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 -- start start 2022-07-12T08:49:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e40b4390 0x7f66e40b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 0x7f66e4140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66e41431e0 0x7f66e4155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4140fe0 con 0x7f66e40b6250 2022-07-12T08:49:18.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40b7e00 con 0x7f66e40b4390 2022-07-12T08:49:18.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e8d92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e40b8070 con 0x7f66e41431e0 2022-07-12T08:49:18.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 0x7f66e4140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 0x7f66e4140af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59838/0 (socket says 172.21.15.61:59838) 2022-07-12T08:49:18.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e1d9b700 1 -- 172.21.15.61:0/2596630259 learned_addr learned my addr 172.21.15.61:0/2596630259 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:18.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e259c700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e40b4390 0x7f66e40b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.829+0000 7f66e2d9d700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66e41431e0 0x7f66e4155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e1d9b700 1 -- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66e41431e0 msgr2=0x7f66e4155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e1d9b700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66e41431e0 0x7f66e4155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e1d9b700 1 -- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e40b4390 msgr2=0x7f66e40b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e1d9b700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e40b4390 0x7f66e40b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e1d9b700 1 -- 172.21.15.61:0/2596630259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc009530 con 0x7f66e40b6250 2022-07-12T08:49:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e1d9b700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 0x7f66e4140af0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f66e40b7f90 tx=0x7f66d4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66d4011780 con 0x7f66e40b6250 2022-07-12T08:49:18.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d40118e0 con 0x7f66e40b6250 2022-07-12T08:49:18.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e41556d0 con 0x7f66e40b6250 2022-07-12T08:49:18.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66d4011c40 con 0x7f66e40b6250 2022-07-12T08:49:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e40acd80 con 0x7f66e40b6250 2022-07-12T08:49:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e4142bb0 con 0x7f66e40b6250 2022-07-12T08:49:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f66d401c460 con 0x7f66e40b6250 2022-07-12T08:49:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66d37fe700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66c00539b0 0x7f66c0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:18.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f66d406bcd0 con 0x7f66e40b6250 2022-07-12T08:49:18.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.833+0000 7f66e259c700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66c00539b0 0x7f66c0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:18.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.837+0000 7f66e259c700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66c00539b0 0x7f66c0055e70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f66d800d000 tx=0x7f66d8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:18.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.841+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66e4142bb0 con 0x7f66e40b6250 2022-07-12T08:49:18.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.973+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f66e4142bb0 con 0x7f66e40b6250 2022-07-12T08:49:18.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.973+0000 7f66d37fe700 1 -- 172.21.15.61:0/2596630259 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v103) v1 ==== 81+0+28877 (secure 0 0 0) 0x7f66e4142bb0 con 0x7f66e40b6250 2022-07-12T08:49:18.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:18.981 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:18.982 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 103, 2022-07-12T08:49:18.982 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:49:18.982 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:49:18.982 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:49:16.257213+0000", 2022-07-12T08:49:18.983 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:49:18.983 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:49:18.983 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:49:18.984 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:49:18.984 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:49:18.985 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:49:18.985 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:49:18.985 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:49:18.985 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:49:18.986 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:18.986 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:49:18.986 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:49:18.986 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:49:18.987 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:49:18.987 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:49:18.987 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 20, 2022-07-12T08:49:18.987 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:49:18.988 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:49:18.988 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:49:18.988 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:49:18.988 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:49:18.988 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:18.989 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:49:18.989 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:49:18.989 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:49:18.989 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:18.990 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:18.990 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:18.991 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:18.991 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:18.991 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:18.991 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:18.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:18.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:49:18.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:49:18.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:49:18.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:49:18.993 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:49:18.993 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:18.993 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:18.993 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:18.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:18.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:18.994 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:18.994 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:18.995 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:18.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:49:18.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:18.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:18.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:18.996 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:18.996 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:18.996 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:18.996 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:18.996 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:18.997 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:18.997 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:18.997 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:18.997 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:18.998 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:18.998 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:18.999 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:18.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:18.999 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:19.000 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:19.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:19.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:19.001 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:19.001 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:19.002 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:19.002 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:19.003 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:19.003 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:19.003 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.004 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:19.004 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:19.004 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:19.004 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:19.005 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:19.005 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:19.005 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:19.005 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:19.006 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:19.006 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:19.006 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:19.006 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:49:19.007 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:49:19.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.007 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:19.007 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:49:19.007 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.008 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.008 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:49:19.008 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:49:19.009 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:49:19.009 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:19.009 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:19.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:19.010 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:19.010 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:19.010 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:19.010 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:19.011 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:19.011 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:19.012 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:19.012 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:19.012 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:19.012 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:19.013 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:19.013 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:19.014 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:19.014 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:19.014 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:19.015 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:19.015 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:19.015 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:49:19.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:19.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:19.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:19.017 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:19.017 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:19.017 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:19.018 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:19.018 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:19.019 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:19.019 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:19.020 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:19.020 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:19.020 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:19.020 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:19.021 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:19.021 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:19.021 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:19.021 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:19.022 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:19.022 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:19.022 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:19.023 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:19.023 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:19.023 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:19.023 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:19.023 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:19.024 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.024 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:19.024 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:19.025 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:19.025 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:19.025 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:19.026 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:19.026 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:19.027 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:19.027 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:19.028 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:19.028 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:19.029 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:49:19.029 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:19.029 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:49:19.030 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.030 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.031 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:19.031 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:49:19.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.032 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:49:19.032 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:49:19.033 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.033 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.034 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.034 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.035 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.035 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.036 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:19.036 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.037 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.037 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.037 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.038 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.038 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:49:19.039 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.039 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.041 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:49:19.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.042 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.042 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.042 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.043 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.044 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:49:19.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:49:19.048 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.048 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.048 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.049 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.049 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.051 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.051 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:49:19.051 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.052 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.052 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:49:19.053 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.053 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.053 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.054 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.054 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.054 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.055 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.055 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.055 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:49:19.055 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.056 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.056 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:49:19.057 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:19.057 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.058 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.059 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:49:19.059 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:49:19.060 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:49:19.060 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:49:19.060 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.060 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.061 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.061 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.062 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:49:19.062 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:49:19.062 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.063 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.063 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.063 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.063 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.063 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.064 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.064 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:19.064 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.064 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.065 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.065 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:49:19.066 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.066 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.067 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:49:19.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.067 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.067 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.068 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.068 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.068 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:49:19.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.069 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.070 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.070 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:49:19.070 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.070 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.070 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.071 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.071 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.072 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:49:19.072 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.073 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.073 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:49:19.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.074 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.074 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.075 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.075 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.075 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.075 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:49:19.076 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.076 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.077 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:49:19.077 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:19.078 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.078 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.079 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:49:19.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:49:19.080 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:49:19.081 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:49:19.081 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.082 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.082 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.082 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.084 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:49:19.084 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:49:19.085 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.085 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.085 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.086 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.086 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.087 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.087 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.087 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 97, 2022-07-12T08:49:19.088 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.088 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.089 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.090 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.090 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.090 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:49:19.091 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.092 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.092 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:49:19.093 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.093 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.093 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.094 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.094 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.094 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.095 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.095 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.095 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:49:19.096 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:49:19.098 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.098 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.099 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.100 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.101 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.101 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:49:19.102 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.102 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.103 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.103 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:49:19.104 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.104 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.105 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.106 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.107 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:49:19.108 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.109 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.110 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:49:19.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:19.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.111 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:49:19.111 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:49:19.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:49:19.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:49:19.112 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.112 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.112 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.112 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.113 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.113 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:49:19.113 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:49:19.114 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.114 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.114 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.115 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.116 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.116 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:19.117 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.117 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.118 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.118 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.119 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:49:19.120 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.120 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.120 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.121 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.121 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:49:19.121 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.122 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.123 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.123 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:49:19.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.126 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:49:19.127 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.128 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.128 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.129 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.130 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:49:19.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.132 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:49:19.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.133 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.133 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.135 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:49:19.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:49:19.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:19.139 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.140 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:49:19.141 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:49:19.141 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:49:19.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:49:19.142 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.142 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.143 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.145 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:49:19.145 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:49:19.145 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.146 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.146 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.147 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.148 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.148 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.149 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 97, 2022-07-12T08:49:19.149 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.149 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.150 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.150 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.151 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:49:19.152 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.152 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.153 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:49:19.154 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.154 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.155 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.155 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.156 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.156 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:49:19.157 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.157 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.157 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:49:19.158 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.158 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.158 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.158 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.159 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.159 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.159 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.159 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.160 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:49:19.160 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.161 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.161 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:49:19.162 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.162 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.163 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.163 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.163 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.164 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.164 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.164 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:49:19.164 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.164 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.165 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.165 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.165 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:49:19.165 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:19.166 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.166 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.166 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.166 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:49:19.167 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:49:19.167 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:49:19.167 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:49:19.167 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.168 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.168 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.168 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.169 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.170 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:49:19.170 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:49:19.171 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.171 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.172 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.172 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.173 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.173 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.174 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.174 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 97, 2022-07-12T08:49:19.174 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.174 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.174 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.175 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:49:19.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.177 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.178 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:49:19.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.179 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.179 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.180 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.181 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.182 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:49:19.183 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.184 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.184 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.185 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:49:19.185 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.185 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.186 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.186 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.186 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.187 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.187 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.187 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:49:19.188 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.189 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.190 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:49:19.190 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.190 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.191 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.191 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.192 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.192 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.193 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.193 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:49:19.194 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.194 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.195 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.195 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:49:19.196 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:19.196 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.197 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.197 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:49:19.198 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:49:19.198 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:49:19.198 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:49:19.199 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.199 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.200 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.200 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.201 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.201 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:49:19.202 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:49:19.202 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.203 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.203 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.204 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.204 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.204 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.204 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.205 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:19.205 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.205 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.205 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.206 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.206 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:49:19.206 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.207 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.207 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:49:19.208 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.208 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.208 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.208 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.209 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.209 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:49:19.210 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:49:19.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.212 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.212 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.214 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:49:19.216 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:49:19.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.218 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.218 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:49:19.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:49:19.221 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:19.222 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.222 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.222 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:49:19.223 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:49:19.223 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:49:19.224 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:49:19.224 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.224 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.224 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.225 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.225 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.225 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:49:19.226 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:49:19.226 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:19.226 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:19.226 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:19.226 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:19.227 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:19.227 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:19.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:19.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-12T08:49:19.228 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:19.228 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:19.228 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:19.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:49:19.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:49:19.230 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.231 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.231 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.231 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.231 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:19.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.232 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:49:19.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.233 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.233 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.233 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:49:19.233 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.234 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.234 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:19.234 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:49:19.235 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.236 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:49:19.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.237 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.237 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.237 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:19.238 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:19.238 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.238 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:19.238 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:49:19.238 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.239 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.239 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:19.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:49:19.239 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:19.240 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.240 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.240 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:49:19.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:49:19.241 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:49:19.241 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:49:19.241 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:19.242 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:19.242 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:19.242 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:19.242 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.242 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:19.243 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:49:19.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.243 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:49:19.243 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.244 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.244 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.244 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.244 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.245 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:49:19.245 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.245 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:49:19.246 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.246 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.246 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.246 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.247 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.247 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:49:19.247 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.248 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:49:19.248 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.248 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.248 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.248 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.249 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.249 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:49:19.249 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.250 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.250 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:49:19.250 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.250 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.251 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.251 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.251 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.251 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:49:19.251 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.252 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:49:19.252 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.252 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.253 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.253 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.253 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.253 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:49:19.254 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.254 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.255 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:49:19.255 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.255 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.255 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.255 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.256 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.256 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:49:19.256 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.257 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.257 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:49:19.257 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.257 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.257 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.258 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.258 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.258 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:49:19.259 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:19.259 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:49:19.260 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:19.260 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:19.260 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:19.260 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:19.261 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:19.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:49:19.261 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:19.261 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.262 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:19.262 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:49:19.262 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:49:19.262 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:49:19.263 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:49:19.263 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:49:19.264 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2528523999": "2022-07-13T08:49:12.930514+0000", 2022-07-12T08:49:19.264 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2528523999": "2022-07-13T08:49:12.930514+0000" 2022-07-12T08:49:19.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.264 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:49:19.264 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:49:19.265 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:49:19.265 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:49:19.265 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:49:19.265 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:49:19.265 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:49:19.266 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:49:19.266 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:19.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:19.266 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:49:19.267 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:49:19.267 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:49:19.267 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:49:19.267 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:49:19.268 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:19.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66c00539b0 msgr2=0x7f66c0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66c00539b0 0x7f66c0055e70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f66d800d000 tx=0x7f66d8000bc0).stop 2022-07-12T08:49:19.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 msgr2=0x7f66e4140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 0x7f66e4140af0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7f66e40b7f90 tx=0x7f66d4008040).stop 2022-07-12T08:49:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 shutdown_connections 2022-07-12T08:49:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f66e41431e0 0x7f66e4155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66e40b6250 0x7f66e4140af0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f66c00539b0 0x7f66c0055e70 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 --2- 172.21.15.61:0/2596630259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66e40b4390 0x7f66e40b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 >> 172.21.15.61:0/2596630259 conn(0x7f66e4066f00 msgr2=0x7f66e4146980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:19.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 shutdown_connections 2022-07-12T08:49:19.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:18.977+0000 7f66e8d92700 1 -- 172.21.15.61:0/2596630259 wait complete. 2022-07-12T08:49:19.276 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:19.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.381+0000 7f36624ad700 1 Processor -- start 2022-07-12T08:49:19.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.381+0000 7f36624ad700 1 -- start start 2022-07-12T08:49:19.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f365c0af280 0x7f365c060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c09bce0 0x7f365c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365c0aecb0 con 0x7f365c0b3f60 2022-07-12T08:49:19.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365c143b00 con 0x7f365c0af280 2022-07-12T08:49:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c1435c0 con 0x7f365c09bce0 2022-07-12T08:49:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3660caa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c09bce0 0x7f365c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3661cac700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f365c0b3f60 0x7f365c0b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:33436/0 (socket says 172.21.15.61:33436) 2022-07-12T08:49:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3661cac700 1 -- 172.21.15.61:0/3372627808 learned_addr learned my addr 172.21.15.61:0/3372627808 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36614ab700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f365c0af280 0x7f365c060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f364bfff700 1 -- 172.21.15.61:0/3372627808 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621938358 0 0) 0x7f365c0aecb0 con 0x7f365c0b3f60 2022-07-12T08:49:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f364bfff700 1 -- 172.21.15.61:0/3372627808 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3644003260 con 0x7f365c0b3f60 2022-07-12T08:49:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3660caa700 1 -- 172.21.15.61:0/3372627808 >> v2:172.21.15.111:3300/0 conn(0x7f365c0af280 msgr2=0x7f365c060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3660caa700 1 --2- 172.21.15.61:0/3372627808 >> v2:172.21.15.111:3300/0 conn(0x7f365c0af280 0x7f365c060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3660caa700 1 -- 172.21.15.61:0/3372627808 >> v1:172.21.15.111:6789/0 conn(0x7f365c0b3f60 legacy=0x7f365c0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3660caa700 1 -- 172.21.15.61:0/3372627808 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365c143860 con 0x7f365c09bce0 2022-07-12T08:49:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f3660caa700 1 --2- 172.21.15.61:0/3372627808 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c09bce0 0x7f365c0af660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3658002f70 tx=0x7f3658006df0).ready entity=mon.2 client_cookie=ad64f8454a3dd1fe server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:19.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f364bfff700 1 -- 172.21.15.61:0/3372627808 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365800d4a0 con 0x7f365c09bce0 2022-07-12T08:49:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f364bfff700 1 -- 172.21.15.61:0/3372627808 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658000e90 con 0x7f365c09bce0 2022-07-12T08:49:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f364bfff700 1 -- 172.21.15.61:0/3372627808 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365800d7f0 con 0x7f365c09bce0 2022-07-12T08:49:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- 172.21.15.61:0/3372627808 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c09bce0 msgr2=0x7f365c0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 --2- 172.21.15.61:0/3372627808 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c09bce0 0x7f365c0af660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f3658002f70 tx=0x7f3658006df0).stop 2022-07-12T08:49:19.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- 172.21.15.61:0/3372627808 shutdown_connections 2022-07-12T08:49:19.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 --2- 172.21.15.61:0/3372627808 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c09bce0 0x7f365c0af660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 --2- 172.21.15.61:0/3372627808 >> v2:172.21.15.111:3300/0 conn(0x7f365c0af280 0x7f365c060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- 172.21.15.61:0/3372627808 >> 172.21.15.61:0/3372627808 conn(0x7f365c0b92c0 msgr2=0x7f365c0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- 172.21.15.61:0/3372627808 shutdown_connections 2022-07-12T08:49:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.385+0000 7f36624ad700 1 -- 172.21.15.61:0/3372627808 wait complete. 2022-07-12T08:49:19.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 Processor -- start 2022-07-12T08:49:19.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 -- start start 2022-07-12T08:49:19.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c09bce0 0x7f365c155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 0x7f365c09c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c0b3f60 0x7f365c09e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c09ce40 con 0x7f365c0af280 2022-07-12T08:49:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c13d660 con 0x7f365c09bce0 2022-07-12T08:49:19.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365c09da90 con 0x7f365c0b3f60 2022-07-12T08:49:19.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3660caa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 0x7f365c09c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3660caa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 0x7f365c09c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59852/0 (socket says 172.21.15.61:59852) 2022-07-12T08:49:19.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3660caa700 1 -- 172.21.15.61:0/945976849 learned_addr learned my addr 172.21.15.61:0/945976849 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:19.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c0b3f60 0x7f365c09e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36614ab700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c09bce0 0x7f365c155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 -- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c09bce0 msgr2=0x7f365c155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c09bce0 0x7f365c155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 -- 172.21.15.61:0/945976849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 msgr2=0x7f365c09c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 0x7f365c09c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 -- 172.21.15.61:0/945976849 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365800b180 con 0x7f365c0b3f60 2022-07-12T08:49:19.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3660caa700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 0x7f365c09c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:19.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f3661cac700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c0b3f60 0x7f365c09e380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3654001490 tx=0x7f3654007dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:19.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3654005ba0 con 0x7f365c0b3f60 2022-07-12T08:49:19.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3654005d00 con 0x7f365c0b3f60 2022-07-12T08:49:19.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36540072c0 con 0x7f365c0b3f60 2022-07-12T08:49:19.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c09dd30 con 0x7f365c0b3f60 2022-07-12T08:49:19.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.389+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c09ea90 con 0x7f365c0b3f60 2022-07-12T08:49:19.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.393+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c057460 con 0x7f365c0b3f60 2022-07-12T08:49:19.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.393+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3654010070 con 0x7f365c0b3f60 2022-07-12T08:49:19.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.393+0000 7f364a7fc700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f363c053ad0 0x7f363c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.393+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f365406b470 con 0x7f365c0b3f60 2022-07-12T08:49:19.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.393+0000 7f36614ab700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f363c053ad0 0x7f363c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.393+0000 7f36614ab700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f363c053ad0 0x7f363c055f90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f364c006000 tx=0x7f364c006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:19.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.401+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3654042610 con 0x7f365c0b3f60 2022-07-12T08:49:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.565+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f365c1561a0 con 0x7f365c0b3f60 2022-07-12T08:49:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.565+0000 7f364a7fc700 1 -- 172.21.15.61:0/945976849 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f3654018020 con 0x7f365c0b3f60 2022-07-12T08:49:19.573 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:19.573 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7596,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3971554960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3971554960},{"type":"v1","addr":"172.21.15.61:6833","nonce":3971554960}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[]} 2022-07-12T08:49:19.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f363c053ad0 msgr2=0x7f363c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f363c053ad0 0x7f363c055f90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f364c006000 tx=0x7f364c006d90).stop 2022-07-12T08:49:19.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c0b3f60 msgr2=0x7f365c09e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c0b3f60 0x7f365c09e380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f3654001490 tx=0x7f3654007dc0).stop 2022-07-12T08:49:19.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 shutdown_connections 2022-07-12T08:49:19.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f365c0b3f60 0x7f365c09e380 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f365c0af280 0x7f365c09c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f363c053ad0 0x7f363c055f90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 --2- 172.21.15.61:0/945976849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f365c09bce0 0x7f365c155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 >> 172.21.15.61:0/945976849 conn(0x7f365c0b92c0 msgr2=0x7f365c0b7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:19.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 shutdown_connections 2022-07-12T08:49:19.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.569+0000 7f36624ad700 1 -- 172.21.15.61:0/945976849 wait complete. 2022-07-12T08:49:19.580 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 59 2022-07-12T08:49:19.593 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:19.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc8de4700 1 Processor -- start 2022-07-12T08:49:19.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc8de4700 1 -- start start 2022-07-12T08:49:19.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc8de4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 0x7f4cc40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc8de4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc40ad2f0 con 0x7f4cc405c050 2022-07-12T08:49:19.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc8de4700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc405c430 con 0x7f4cc40b01f0 2022-07-12T08:49:19.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc8de4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40b0a40 con 0x7f4cc4144ad0 2022-07-12T08:49:19.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4cc405c050 0x7f4cc405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48078/0 (socket says 172.21.15.61:48078) 2022-07-12T08:49:19.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc259c700 1 -- 172.21.15.61:0/1644460657 learned_addr learned my addr 172.21.15.61:0/1644460657 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:19.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 0x7f4cc40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398726527 0 0) 0x7f4cc40ad2f0 con 0x7f4cc405c050 2022-07-12T08:49:19.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca4003260 con 0x7f4cc405c050 2022-07-12T08:49:19.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397996644 0 0) 0x7f4cc405c430 con 0x7f4cc40b01f0 2022-07-12T08:49:19.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc40ad2f0 con 0x7f4cc40b01f0 2022-07-12T08:49:19.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016382032 0 0) 0x7f4ca4003260 con 0x7f4cc405c050 2022-07-12T08:49:19.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 msgr2=0x7f4cc40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:19.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 0x7f4cc40ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 >> v1:172.21.15.111:6789/0 conn(0x7f4cc40b01f0 legacy=0x7f4cc41443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc40b2420 con 0x7f4cc405c050 2022-07-12T08:49:19.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f4cb4001660 con 0x7f4cc405c050 2022-07-12T08:49:19.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 >> v1:172.21.15.61:6789/0 conn(0x7f4cc405c050 legacy=0x7f4cc405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 0x7f4ca4003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca4003e70 0x7f4ca4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40049f0 0x7f4ca4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40b0a40 con 0x7f4ca40049f0 2022-07-12T08:49:19.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca400b830 con 0x7f4ca4003e70 2022-07-12T08:49:19.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca400b9f0 con 0x7f4cc4144ad0 2022-07-12T08:49:19.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc259c700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 0x7f4ca4003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40049f0 0x7f4ca4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cb9d9b700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca4003e70 0x7f4ca4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 msgr2=0x7f4ca4003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:19.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4144ad0 0x7f4ca4003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca4003e70 msgr2=0x7f4ca4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:19.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca4003e70 0x7f4ca4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/1644460657 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4003260 con 0x7f4ca40049f0 2022-07-12T08:49:19.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cb9d9b700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ca4003e70 0x7f4ca4004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:19.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40049f0 0x7f4ca4004e70 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f4cbc00c000 tx=0x7f4cbc0091d0).ready entity=mon.0 client_cookie=5243811652932efc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:19.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cbc00e6e0 con 0x7f4ca40049f0 2022-07-12T08:49:19.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc00e840 con 0x7f4ca40049f0 2022-07-12T08:49:19.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.689+0000 7f4cc1d9b700 1 -- 172.21.15.61:0/1644460657 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cbc00eba0 con 0x7f4ca40049f0 2022-07-12T08:49:19.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- 172.21.15.61:0/1644460657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40049f0 msgr2=0x7f4ca4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/1644460657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ca40049f0 0x7f4ca4004e70 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 rx=0x7f4cbc00c000 tx=0x7f4cbc0091d0).stop 2022-07-12T08:49:19.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/1644460657 reap_dead start 2022-07-12T08:49:19.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- 172.21.15.61:0/1644460657 shutdown_connections 2022-07-12T08:49:19.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- 172.21.15.61:0/1644460657 >> 172.21.15.61:0/1644460657 conn(0x7f4cc4062c00 msgr2=0x7f4cc4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:19.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- 172.21.15.61:0/1644460657 shutdown_connections 2022-07-12T08:49:19.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- 172.21.15.61:0/1644460657 wait complete. 2022-07-12T08:49:19.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 Processor -- start 2022-07-12T08:49:19.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- start start 2022-07-12T08:49:19.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4143b70 0x7f4cc415a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc40a1cb0 0x7f4cc4149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 0x7f4cc414a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc414a800 con 0x7f4cc4149e60 2022-07-12T08:49:19.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc415ac30 con 0x7f4cc40a1cb0 2022-07-12T08:49:19.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc8de4700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4143f50 con 0x7f4cc4143b70 2022-07-12T08:49:19.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 0x7f4cc414a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 0x7f4cc414a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59872/0 (socket says 172.21.15.61:59872) 2022-07-12T08:49:19.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/2176448486 learned_addr learned my addr 172.21.15.61:0/2176448486 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:19.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cc259c700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4143b70 0x7f4cc415a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.693+0000 7f4cb9d9b700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc40a1cb0 0x7f4cc4149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4143b70 msgr2=0x7f4cc415a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4143b70 0x7f4cc415a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc40a1cb0 msgr2=0x7f4cc4149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc40a1cb0 0x7f4cc4149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc2d9d700 1 -- 172.21.15.61:0/2176448486 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc4145080 con 0x7f4cc4149e60 2022-07-12T08:49:19.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc2d9d700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 0x7f4cc414a310 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f4cc4144fa0 tx=0x7f4cbc01a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:19.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cbc00e6e0 con 0x7f4cc4149e60 2022-07-12T08:49:19.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc01db00 con 0x7f4cc4149e60 2022-07-12T08:49:19.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc4145320 con 0x7f4cc4149e60 2022-07-12T08:49:19.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cbc01de60 con 0x7f4cc4149e60 2022-07-12T08:49:19.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc4142170 con 0x7f4cc4149e60 2022-07-12T08:49:19.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc4057460 con 0x7f4cc4149e60 2022-07-12T08:49:19.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4cbc02d070 con 0x7f4cc4149e60 2022-07-12T08:49:19.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cbbfff700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ca8053a60 0x7f4ca8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:19.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4cbc06d0a0 con 0x7f4cc4149e60 2022-07-12T08:49:19.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.697+0000 7f4cc259c700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ca8053a60 0x7f4ca8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:19.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.701+0000 7f4cc259c700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ca8053a60 0x7f4ca8055f20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f4cb4005000 tx=0x7f4cc405c7f0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:19.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.705+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4cbc0442e0 con 0x7f4cc4149e60 2022-07-12T08:49:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.865+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cc40447a0 con 0x7f4cc4149e60 2022-07-12T08:49:19.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cbbfff700 1 -- 172.21.15.61:0/2176448486 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f4cbc016030 con 0x7f4cc4149e60 2022-07-12T08:49:19.874 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:19.875 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6431,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1002916949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1002916949},{"type":"v1","addr":"172.21.15.111:6835","nonce":1002916949}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53},{"gid":7596,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3971554960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3971554960},{"type":"v1","addr":"172.21.15.61:6833","nonce":3971554960}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[]} 2022-07-12T08:49:19.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ca8053a60 msgr2=0x7f4ca8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ca8053a60 0x7f4ca8055f20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f4cb4005000 tx=0x7f4cc405c7f0).stop 2022-07-12T08:49:19.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 msgr2=0x7f4cc414a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:19.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 0x7f4cc414a310 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7f4cc4144fa0 tx=0x7f4cbc01a000).stop 2022-07-12T08:49:19.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 shutdown_connections 2022-07-12T08:49:19.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cc4149e60 0x7f4cc414a310 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4ca8053a60 0x7f4ca8055f20 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4cc40a1cb0 0x7f4cc4149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 --2- 172.21.15.61:0/2176448486 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4cc4143b70 0x7f4cc415a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:19.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.869+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 >> 172.21.15.61:0/2176448486 conn(0x7f4cc4062c00 msgr2=0x7f4cc40ad6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:19.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.873+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 shutdown_connections 2022-07-12T08:49:19.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:19.873+0000 7f4cc8de4700 1 -- 172.21.15.61:0/2176448486 wait complete. 2022-07-12T08:49:19.881 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 59 2022-07-12T08:49:19.892 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:49:19.892 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:49:19.893 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:49:19.893 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:49:19.893 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:49:19.894 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:49:19.895 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:49:19.895 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:49:19.904 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:49:19.905 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:49:19.910 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:49:19.910 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:49:19.950 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:49:19.951 DEBUG:teuthology.parallel:result is None 2022-07-12T08:49:19.953 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:49:19.955 DEBUG:teuthology.parallel:result is None 2022-07-12T08:49:19.956 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:49:20.027 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:49:20.044 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:49:20.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.061+0000 7f4404587700 1 Processor -- start 2022-07-12T08:49:20.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.061+0000 7f4404587700 1 -- start start 2022-07-12T08:49:20.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.061+0000 7f4404587700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f43fc0ad3b0 0x7f43fc142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:20.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.061+0000 7f4404587700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43fc0ad790 con 0x7f43fc060630 2022-07-12T08:49:20.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.061+0000 7f4404587700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43fc0b03b0 con 0x7f43fc061460 2022-07-12T08:49:20.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.061+0000 7f4404587700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43fc061300 con 0x7f43fc0ad3b0 2022-07-12T08:49:20.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402323700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f43fc060630 0x7f43fc1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48108/0 (socket says 172.21.15.61:48108) 2022-07-12T08:49:20.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402323700 1 -- 172.21.15.61:0/842599713 learned_addr learned my addr 172.21.15.61:0/842599713 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:20.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402b24700 1 --2- 172.21.15.61:0/842599713 >> v2:172.21.15.111:3300/0 conn(0x7f43fc0ad3b0 0x7f43fc142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:20.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4401321700 1 -- 172.21.15.61:0/842599713 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541274212 0 0) 0x7f43fc0ad790 con 0x7f43fc060630 2022-07-12T08:49:20.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4401321700 1 -- 172.21.15.61:0/842599713 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e4003260 con 0x7f43fc060630 2022-07-12T08:49:20.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402b24700 1 -- 172.21.15.61:0/842599713 >> v1:172.21.15.111:6789/0 conn(0x7f43fc061460 legacy=0x7f43fc13f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:20.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402b24700 1 -- 172.21.15.61:0/842599713 >> v1:172.21.15.61:6789/0 conn(0x7f43fc060630 legacy=0x7f43fc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:20.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402b24700 1 -- 172.21.15.61:0/842599713 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0b0650 con 0x7f43fc0ad3b0 2022-07-12T08:49:20.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4402b24700 1 --2- 172.21.15.61:0/842599713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc0ad3b0 0x7f43fc142cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f43f4002f70 tx=0x7f43f4006df0).ready entity=mon.1 client_cookie=e663759d7df3567c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:20.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4401321700 1 -- 172.21.15.61:0/842599713 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f400d610 con 0x7f43fc0ad3b0 2022-07-12T08:49:20.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4401321700 1 -- 172.21.15.61:0/842599713 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f4000e90 con 0x7f43fc0ad3b0 2022-07-12T08:49:20.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4401321700 1 -- 172.21.15.61:0/842599713 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f400d960 con 0x7f43fc0ad3b0 2022-07-12T08:49:20.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 -- 172.21.15.61:0/842599713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc0ad3b0 msgr2=0x7f43fc142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:20.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 --2- 172.21.15.61:0/842599713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc0ad3b0 0x7f43fc142cd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f43f4002f70 tx=0x7f43f4006df0).stop 2022-07-12T08:49:20.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 -- 172.21.15.61:0/842599713 shutdown_connections 2022-07-12T08:49:20.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 --2- 172.21.15.61:0/842599713 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc0ad3b0 0x7f43fc142cd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:20.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 -- 172.21.15.61:0/842599713 >> 172.21.15.61:0/842599713 conn(0x7f43fc05f140 msgr2=0x7f43fc05f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:20.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 -- 172.21.15.61:0/842599713 shutdown_connections 2022-07-12T08:49:20.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.065+0000 7f4404587700 1 -- 172.21.15.61:0/842599713 wait complete. 2022-07-12T08:49:20.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 Processor -- start 2022-07-12T08:49:20.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 -- start start 2022-07-12T08:49:20.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc060630 0x7f43fc13b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:20.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 0x7f43fc13a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:20.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43fc0ad3b0 0x7f43fc13cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:20.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc1399e0 con 0x7f43fc061460 2022-07-12T08:49:20.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc13f310 con 0x7f43fc060630 2022-07-12T08:49:20.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4404587700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43fc139740 con 0x7f43fc0ad3b0 2022-07-12T08:49:20.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.069+0000 7f4401b22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 0x7f43fc13a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:20.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 0x7f43fc13a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59898/0 (socket says 172.21.15.61:59898) 2022-07-12T08:49:20.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 -- 172.21.15.61:0/1717854515 learned_addr learned my addr 172.21.15.61:0/1717854515 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:20.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4402323700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc060630 0x7f43fc13b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:20.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4402b24700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43fc0ad3b0 0x7f43fc13cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:20.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 -- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43fc0ad3b0 msgr2=0x7f43fc13cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:20.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43fc0ad3b0 0x7f43fc13cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:20.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 -- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc060630 msgr2=0x7f43fc13b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:20.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc060630 0x7f43fc13b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:20.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 -- 172.21.15.61:0/1717854515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f400b180 con 0x7f43fc061460 2022-07-12T08:49:20.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4402b24700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43fc0ad3b0 0x7f43fc13cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:20.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4401b22700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 0x7f43fc13a4c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f43fc062650 tx=0x7f43f0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:20.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f0012400 con 0x7f43fc061460 2022-07-12T08:49:20.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f0012560 con 0x7f43fc061460 2022-07-12T08:49:20.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43f00128c0 con 0x7f43fc061460 2022-07-12T08:49:20.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43fc13ee70 con 0x7f43fc061460 2022-07-12T08:49:20.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43fc13f560 con 0x7f43fc061460 2022-07-12T08:49:20.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f43f0019030 con 0x7f43fc061460 2022-07-12T08:49:20.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.073+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43fc13f000 con 0x7f43fc061460 2022-07-12T08:49:20.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.077+0000 7f43ef7fe700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43dc053a60 0x7f43dc055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:20.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.077+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f43f006b030 con 0x7f43fc061460 2022-07-12T08:49:20.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.077+0000 7f4402323700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43dc053a60 0x7f43dc055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:20.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.077+0000 7f4402323700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43dc053a60 0x7f43dc055f20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f43f800d000 tx=0x7f43f8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:20.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.081+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43f00421d0 con 0x7f43fc061460 2022-07-12T08:49:20.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:20.221+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f43fc057460 con 0x7f43fc061460 2022-07-12T08:49:21.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.213+0000 7f43ef7fe700 1 -- 172.21.15.61:0/1717854515 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7f43f00457f0 con 0x7f43fc061460 2022-07-12T08:49:21.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43dc053a60 msgr2=0x7f43dc055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43dc053a60 0x7f43dc055f20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f43f800d000 tx=0x7f43f8009040).stop 2022-07-12T08:49:21.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 msgr2=0x7f43fc13a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 0x7f43fc13a4c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 rx=0x7f43fc062650 tx=0x7f43f0007c50).stop 2022-07-12T08:49:21.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 shutdown_connections 2022-07-12T08:49:21.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f43fc0ad3b0 0x7f43fc13cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43fc061460 0x7f43fc13a4c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f43dc053a60 0x7f43dc055f20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 --2- 172.21.15.61:0/1717854515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f43fc060630 0x7f43fc13b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 >> 172.21.15.61:0/1717854515 conn(0x7f43fc05f140 msgr2=0x7f43fc0b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:21.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 shutdown_connections 2022-07-12T08:49:21.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.217+0000 7f4404587700 1 -- 172.21.15.61:0/1717854515 wait complete. 2022-07-12T08:49:21.228 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:49:21.235 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:49:21.236 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:49:21.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe7a1d7d700 1 Processor -- start 2022-07-12T08:49:21.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe7a1d7d700 1 -- start start 2022-07-12T08:49:21.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe7a1d7d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c140570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe7a1d7d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79c0b3e80 con 0x7fe79c0b6250 2022-07-12T08:49:21.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe7a1d7d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79c147390 con 0x7fe79c1431e0 2022-07-12T08:49:21.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe7a1d7d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c0b6630 con 0x7fe79c0b4390 2022-07-12T08:49:21.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe79b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe79c0b6250 0x7fe79c13fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48124/0 (socket says 172.21.15.61:48124) 2022-07-12T08:49:21.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.337+0000 7fe79b7fe700 1 -- 172.21.15.61:0/1167437738 learned_addr learned my addr 172.21.15.61:0/1167437738 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:21.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79affd700 1 --2- 172.21.15.61:0/1167437738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983927768 0 0) 0x7fe79c0b3e80 con 0x7fe79c0b6250 2022-07-12T08:49:21.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77c003260 con 0x7fe79c0b6250 2022-07-12T08:49:21.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259028781 0 0) 0x7fe79c147390 con 0x7fe79c1431e0 2022-07-12T08:49:21.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79c0b3e80 con 0x7fe79c1431e0 2022-07-12T08:49:21.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79affd700 1 -- 172.21.15.61:0/1167437738 >> v1:172.21.15.111:6789/0 conn(0x7fe79c1431e0 legacy=0x7fe79c143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79affd700 1 -- 172.21.15.61:0/1167437738 >> v1:172.21.15.61:6789/0 conn(0x7fe79c0b6250 legacy=0x7fe79c13fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79affd700 1 -- 172.21.15.61:0/1167437738 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe79c1467b0 con 0x7fe79c0b4390 2022-07-12T08:49:21.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79affd700 1 --2- 172.21.15.61:0/1167437738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c140570 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe784002f70 tx=0x7fe784006df0).ready entity=mon.2 client_cookie=d63f50fed4c352f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe78400d5c0 con 0x7fe79c0b4390 2022-07-12T08:49:21.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe784000e90 con 0x7fe79c0b4390 2022-07-12T08:49:21.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe79a7fc700 1 -- 172.21.15.61:0/1167437738 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe78400d910 con 0x7fe79c0b4390 2022-07-12T08:49:21.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/1167437738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 msgr2=0x7fe79c140570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/1167437738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c140570 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe784002f70 tx=0x7fe784006df0).stop 2022-07-12T08:49:21.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/1167437738 shutdown_connections 2022-07-12T08:49:21.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/1167437738 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c140570 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/1167437738 >> 172.21.15.61:0/1167437738 conn(0x7fe79c066f00 msgr2=0x7fe79c0672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:21.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/1167437738 shutdown_connections 2022-07-12T08:49:21.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.341+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/1167437738 wait complete. 2022-07-12T08:49:21.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 Processor -- start 2022-07-12T08:49:21.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 -- start start 2022-07-12T08:49:21.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c1446b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe79c0b6250 0x7fe79c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 0x7fe79c0ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c0ad7e0 con 0x7fe79c1431e0 2022-07-12T08:49:21.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c139c80 con 0x7fe79c0b6250 2022-07-12T08:49:21.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe7a1d7d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c139eb0 con 0x7fe79c0b4390 2022-07-12T08:49:21.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 0x7fe79c0ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 0x7fe79c0ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59916/0 (socket says 172.21.15.61:59916) 2022-07-12T08:49:21.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 -- 172.21.15.61:0/4026303211 learned_addr learned my addr 172.21.15.61:0/4026303211 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:21.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79b7fe700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c1446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79affd700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe79c0b6250 0x7fe79c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 -- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 msgr2=0x7fe79c1446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c1446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 -- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe79c0b6250 msgr2=0x7fe79c0ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe79c0b6250 0x7fe79c0ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.345+0000 7fe79bfff700 1 -- 172.21.15.61:0/4026303211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78400b180 con 0x7fe79c1431e0 2022-07-12T08:49:21.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe79bfff700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 0x7fe79c0ad2f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fe7940014e0 tx=0x7fe794007e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe794002310 con 0x7fe79c1431e0 2022-07-12T08:49:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe794002470 con 0x7fe79c1431e0 2022-07-12T08:49:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe794001810 con 0x7fe79c1431e0 2022-07-12T08:49:21.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c13a150 con 0x7fe79c1431e0 2022-07-12T08:49:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c147e70 con 0x7fe79c1431e0 2022-07-12T08:49:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79c142dc0 con 0x7fe79c1431e0 2022-07-12T08:49:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe794019020 con 0x7fe79c1431e0 2022-07-12T08:49:21.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe798ff9700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe780053920 0x7fe780055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe79406ba40 con 0x7fe79c1431e0 2022-07-12T08:49:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.349+0000 7fe79b7fe700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe780053920 0x7fe780055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.353+0000 7fe79b7fe700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe780053920 0x7fe780055de0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fe78c00d000 tx=0x7fe78c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.357+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe794042bc0 con 0x7fe79c1431e0 2022-07-12T08:49:21.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.521+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe79c148120 con 0x7fe79c1431e0 2022-07-12T08:49:21.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe798ff9700 1 -- 172.21.15.61:0/4026303211 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fe7940461e0 con 0x7fe79c1431e0 2022-07-12T08:49:21.529 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:21.529 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:21.530 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:49:21.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:49:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:49:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:49:21.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:49:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:49:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:49:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:49:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:49:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:49:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:49:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:49:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:49:21.542 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.542 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.543 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.543 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.544 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.544 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:49:21.546 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:49:21.546 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.548 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.548 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:49:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:49:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:49:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:49:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.554 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:49:21.554 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:49:21.554 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.555 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.555 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.555 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.556 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.556 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.556 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.556 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:49:21.557 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:49:21.557 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.557 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.557 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.558 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.558 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.559 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.559 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:49:21.559 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:49:21.559 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:49:21.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:49:21.560 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:49:21.560 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.561 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.561 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.561 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:49:21.562 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:49:21.562 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.563 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:49:21.563 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:49:21.565 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:49:21.565 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.566 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.566 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.567 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:49:21.567 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:49:21.568 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.568 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:49:21.568 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:49:21.569 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:49:21.569 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.569 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.570 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.570 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:49:21.571 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:49:21.571 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.571 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:49:21.572 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:49:21.572 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:49:21.572 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.573 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.573 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.573 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.574 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:49:21.575 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:49:21.575 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.575 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:49:21.576 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:49:21.576 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.576 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.577 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:21.577 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:21.578 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:49:21.578 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:49:21.578 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:49:21.579 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:49:21.579 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:49:21.580 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:49:21.580 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.580 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:21.581 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:21.581 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:21.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe780053920 msgr2=0x7fe780055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe780053920 0x7fe780055de0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fe78c00d000 tx=0x7fe78c009040).stop 2022-07-12T08:49:21.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 msgr2=0x7fe79c0ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 0x7fe79c0ad2f0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 rx=0x7fe7940014e0 tx=0x7fe794007e30).stop 2022-07-12T08:49:21.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 shutdown_connections 2022-07-12T08:49:21.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe79c1431e0 0x7fe79c0ad2f0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe780053920 0x7fe780055de0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe79c0b6250 0x7fe79c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 --2- 172.21.15.61:0/4026303211 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe79c0b4390 0x7fe79c1446b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 >> 172.21.15.61:0/4026303211 conn(0x7fe79c066f00 msgr2=0x7fe79c0b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:21.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 shutdown_connections 2022-07-12T08:49:21.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.525+0000 7fe7a1d7d700 1 -- 172.21.15.61:0/4026303211 wait complete. 2022-07-12T08:49:21.590 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:21.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af99db700 1 Processor -- start 2022-07-12T08:49:21.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af99db700 1 -- start start 2022-07-12T08:49:21.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af99db700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 0x7f5af405d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af99db700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af413fd50 con 0x7f5af40ad590 2022-07-12T08:49:21.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af99db700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af4067900 con 0x7f5af40ac710 2022-07-12T08:49:21.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af99db700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af4067ba0 con 0x7f5af40b1e40 2022-07-12T08:49:21.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5af40ad590 0x7f5af41401b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48142/0 (socket says 172.21.15.61:48142) 2022-07-12T08:49:21.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 -- 172.21.15.61:0/2072854206 learned_addr learned my addr 172.21.15.61:0/2072854206 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:21.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af2ffd700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 0x7f5af405d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265719922 0 0) 0x7f5af413fd50 con 0x7f5af40ad590 2022-07-12T08:49:21.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad4003260 con 0x7f5af40ad590 2022-07-12T08:49:21.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272530897 0 0) 0x7f5ad4003260 con 0x7f5af40ad590 2022-07-12T08:49:21.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 msgr2=0x7f5af405d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:21.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 0x7f5af405d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 >> v1:172.21.15.111:6789/0 conn(0x7f5af40ac710 legacy=0x7f5af40ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40675f0 con 0x7f5af40ad590 2022-07-12T08:49:21.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f5aec0019e0 con 0x7f5af40ad590 2022-07-12T08:49:21.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 >> v1:172.21.15.61:6789/0 conn(0x7f5af40ad590 legacy=0x7f5af41401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 0x7f5ad4003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ad4003c70 0x7f5ad4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ad40049f0 0x7f5ad4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af4067ba0 con 0x7f5ad40049f0 2022-07-12T08:49:21.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad400b750 con 0x7f5ad4003c70 2022-07-12T08:49:21.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad400b910 con 0x7f5af40b1e40 2022-07-12T08:49:21.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ad40049f0 0x7f5ad4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af2ffd700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 0x7f5ad4003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5aebfff700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ad4003c70 0x7f5ad4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 -- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 msgr2=0x7f5ad4003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:21.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af40b1e40 0x7f5ad4003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 -- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ad4003c70 msgr2=0x7f5ad4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:21.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ad4003c70 0x7f5ad4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 -- 172.21.15.61:0/2072854206 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4003260 con 0x7f5ad40049f0 2022-07-12T08:49:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5aebfff700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ad4003c70 0x7f5ad4004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af37fe700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ad40049f0 0x7f5ad4004e70 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7f5aec0016f0 tx=0x7f5aec0021b0).ready entity=mon.0 client_cookie=97aa67aa8a509de6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aec00cc70 con 0x7f5ad40049f0 2022-07-12T08:49:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec00cdd0 con 0x7f5ad40049f0 2022-07-12T08:49:21.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.697+0000 7f5af27fc700 1 -- 172.21.15.61:0/2072854206 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aec005b80 con 0x7f5ad40049f0 2022-07-12T08:49:21.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- 172.21.15.61:0/2072854206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ad40049f0 msgr2=0x7f5ad4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 --2- 172.21.15.61:0/2072854206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ad40049f0 0x7f5ad4004e70 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 rx=0x7f5aec0016f0 tx=0x7f5aec0021b0).stop 2022-07-12T08:49:21.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af37fe700 1 -- 172.21.15.61:0/2072854206 reap_dead start 2022-07-12T08:49:21.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- 172.21.15.61:0/2072854206 shutdown_connections 2022-07-12T08:49:21.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- 172.21.15.61:0/2072854206 >> 172.21.15.61:0/2072854206 conn(0x7f5af4062c00 msgr2=0x7f5af4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:21.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- 172.21.15.61:0/2072854206 shutdown_connections 2022-07-12T08:49:21.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- 172.21.15.61:0/2072854206 wait complete. 2022-07-12T08:49:21.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 Processor -- start 2022-07-12T08:49:21.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- start start 2022-07-12T08:49:21.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af4143330 0x7f5af414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 0x7f5af4143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5af413a330 0x7f5af4063cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41367e0 con 0x7f5af414dcc0 2022-07-12T08:49:21.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af414e160 con 0x7f5af413a330 2022-07-12T08:49:21.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af99db700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af414e400 con 0x7f5af4143330 2022-07-12T08:49:21.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 0x7f5af4143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 0x7f5af4143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59934/0 (socket says 172.21.15.61:59934) 2022-07-12T08:49:21.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 -- 172.21.15.61:0/1833693228 learned_addr learned my addr 172.21.15.61:0/1833693228 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:21.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af2ffd700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af4143330 0x7f5af414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af37fe700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5af413a330 0x7f5af4063cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 -- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af4143330 msgr2=0x7f5af414eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af4143330 0x7f5af414eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 -- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5af413a330 msgr2=0x7f5af4063cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5af413a330 0x7f5af4063cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 -- 172.21.15.61:0/1833693228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af414e6a0 con 0x7f5af414dcc0 2022-07-12T08:49:21.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5aebfff700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 0x7f5af4143710 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f5adc00d000 tx=0x7f5adc009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.701+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5adc011780 con 0x7f5af414dcc0 2022-07-12T08:49:21.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af413d160 con 0x7f5af414dcc0 2022-07-12T08:49:21.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc0118e0 con 0x7f5af414dcc0 2022-07-12T08:49:21.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af40643e0 con 0x7f5af414dcc0 2022-07-12T08:49:21.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5adc011c40 con 0x7f5af414dcc0 2022-07-12T08:49:21.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af414e830 con 0x7f5af414dcc0 2022-07-12T08:49:21.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5adc01a030 con 0x7f5af414dcc0 2022-07-12T08:49:21.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af0ff9700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ad8053a40 0x7f5ad8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5adc06b680 con 0x7f5af414dcc0 2022-07-12T08:49:21.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.705+0000 7f5af2ffd700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ad8053a40 0x7f5ad8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.709+0000 7f5af2ffd700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ad8053a40 0x7f5ad8055f00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f5ae400b000 tx=0x7f5ae4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.713+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5adc042920 con 0x7f5af414dcc0 2022-07-12T08:49:21.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.837+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5af4057460 con 0x7f5af414dcc0 2022-07-12T08:49:21.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af0ff9700 1 -- 172.21.15.61:0/1833693228 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+13231 (secure 0 0 0) 0x7f5adc0253c0 con 0x7f5af414dcc0 2022-07-12T08:49:21.846 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:21.846 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":106,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:21.208782+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:21.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ad8053a40 msgr2=0x7f5ad8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ad8053a40 0x7f5ad8055f00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f5ae400b000 tx=0x7f5ae4006df0).stop 2022-07-12T08:49:21.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 msgr2=0x7f5af4143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 0x7f5af4143710 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f5adc00d000 tx=0x7f5adc009040).stop 2022-07-12T08:49:21.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 shutdown_connections 2022-07-12T08:49:21.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ad8053a40 0x7f5ad8055f00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5af413a330 0x7f5af4063cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af414dcc0 0x7f5af4143710 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 --2- 172.21.15.61:0/1833693228 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5af4143330 0x7f5af414eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 >> 172.21.15.61:0/1833693228 conn(0x7f5af4062c00 msgr2=0x7f5af40acc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:21.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 shutdown_connections 2022-07-12T08:49:21.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.841+0000 7f5af99db700 1 -- 172.21.15.61:0/1833693228 wait complete. 2022-07-12T08:49:21.859 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:49:21.860 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:49:21.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 Processor -- start 2022-07-12T08:49:21.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 -- start start 2022-07-12T08:49:21.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd66805d060 0x7fd6680b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142ab0 0x7fd6680afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66805d440 con 0x7fd668142f10 2022-07-12T08:49:21.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd668143d80 con 0x7fd66805d060 2022-07-12T08:49:21.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66debb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd668143ab0 con 0x7fd668142ab0 2022-07-12T08:49:21.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd66805d060 0x7fd6680b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd667fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142ab0 0x7fd6680afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd66805d060 0x7fd6680b0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59940/0 (socket says 172.21.15.61:59940) 2022-07-12T08:49:21.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 -- 172.21.15.61:0/4096105002 learned_addr learned my addr 172.21.15.61:0/4096105002 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:21.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 -- 172.21.15.61:0/4096105002 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142ab0 msgr2=0x7fd6680afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:21.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 --2- 172.21.15.61:0/4096105002 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142ab0 0x7fd6680afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 -- 172.21.15.61:0/4096105002 >> v1:172.21.15.111:6789/0 conn(0x7fd668142f10 legacy=0x7fd6681433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd66ceb9700 1 -- 172.21.15.61:0/4096105002 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6680674b0 con 0x7fd66805d060 2022-07-12T08:49:21.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.965+0000 7fd667fff700 1 --2- 172.21.15.61:0/4096105002 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142ab0 0x7fd6680afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:21.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 --2- 172.21.15.61:0/4096105002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd66805d060 0x7fd6680b0130 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7fd66805d5d0 tx=0x7fd65c006eb0).ready entity=mon.0 client_cookie=dfdb84988a521557 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd6677fe700 1 -- 172.21.15.61:0/4096105002 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd65c00d5c0 con 0x7fd66805d060 2022-07-12T08:49:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd6677fe700 1 -- 172.21.15.61:0/4096105002 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c000e90 con 0x7fd66805d060 2022-07-12T08:49:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd6677fe700 1 -- 172.21.15.61:0/4096105002 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd65c00d950 con 0x7fd66805d060 2022-07-12T08:49:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- 172.21.15.61:0/4096105002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd66805d060 msgr2=0x7fd6680b0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 --2- 172.21.15.61:0/4096105002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd66805d060 0x7fd6680b0130 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7fd66805d5d0 tx=0x7fd65c006eb0).stop 2022-07-12T08:49:21.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- 172.21.15.61:0/4096105002 shutdown_connections 2022-07-12T08:49:21.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 --2- 172.21.15.61:0/4096105002 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142ab0 0x7fd6680afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 --2- 172.21.15.61:0/4096105002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd66805d060 0x7fd6680b0130 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- 172.21.15.61:0/4096105002 >> 172.21.15.61:0/4096105002 conn(0x7fd6680b6900 msgr2=0x7fd668066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:21.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- 172.21.15.61:0/4096105002 shutdown_connections 2022-07-12T08:49:21.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- 172.21.15.61:0/4096105002 wait complete. 2022-07-12T08:49:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 Processor -- start 2022-07-12T08:49:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- start start 2022-07-12T08:49:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 0x7fd6680b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142f10 0x7fd668145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd668149e50 0x7fd668147380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66809e0f0 con 0x7fd668149e50 2022-07-12T08:49:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66814a2f0 con 0x7fd66805d060 2022-07-12T08:49:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66debb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6680b35f0 con 0x7fd668142f10 2022-07-12T08:49:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 0x7fd6680b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66d6ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd668149e50 0x7fd668147380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 0x7fd6680b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:38520/0 (socket says 172.21.15.61:38520) 2022-07-12T08:49:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 -- 172.21.15.61:0/3679999614 learned_addr learned my addr 172.21.15.61:0/3679999614 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd667fff700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142f10 0x7fd668145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 -- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142f10 msgr2=0x7fd668145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142f10 0x7fd668145850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 -- 172.21.15.61:0/3679999614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd668149e50 msgr2=0x7fd668147380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:21.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd668149e50 0x7fd668147380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:21.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.969+0000 7fd66ceb9700 1 -- 172.21.15.61:0/3679999614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd658009470 con 0x7fd66805d060 2022-07-12T08:49:21.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd66ceb9700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 0x7fd6680b29f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fd66805d5d0 tx=0x7fd65c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:21.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd65c011760 con 0x7fd66805d060 2022-07-12T08:49:21.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c0118c0 con 0x7fd66805d060 2022-07-12T08:49:21.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65c00b1e0 con 0x7fd66805d060 2022-07-12T08:49:21.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd65c011c20 con 0x7fd66805d060 2022-07-12T08:49:21.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd668156040 con 0x7fd66805d060 2022-07-12T08:49:22.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd65c00f070 con 0x7fd66805d060 2022-07-12T08:49:22.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd668057460 con 0x7fd66805d060 2022-07-12T08:49:22.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd665ffb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd648053920 0x7fd648055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd65c06d410 con 0x7fd66805d060 2022-07-12T08:49:22.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd667fff700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd648053920 0x7fd648055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.973+0000 7fd667fff700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd648053920 0x7fd648055de0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fd65800b000 tx=0x7fd6580091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:21.985+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd65c044600 con 0x7fd66805d060 2022-07-12T08:49:22.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.113+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fd668068350 con 0x7fd66805d060 2022-07-12T08:49:22.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.113+0000 7fd665ffb700 1 -- 172.21.15.61:0/3679999614 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v106) v1 ==== 110+0+10 (secure 0 0 0) 0x7fd65c047c20 con 0x7fd66805d060 2022-07-12T08:49:22.121 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:49:22.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd648053920 msgr2=0x7fd648055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd648053920 0x7fd648055de0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7fd65800b000 tx=0x7fd6580091d0).stop 2022-07-12T08:49:22.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 msgr2=0x7fd6680b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 0x7fd6680b29f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fd66805d5d0 tx=0x7fd65c015000).stop 2022-07-12T08:49:22.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 shutdown_connections 2022-07-12T08:49:22.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd668149e50 0x7fd668147380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd668142f10 0x7fd668145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd648053920 0x7fd648055de0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 --2- 172.21.15.61:0/3679999614 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd66805d060 0x7fd6680b29f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 >> 172.21.15.61:0/3679999614 conn(0x7fd6680b6900 msgr2=0x7fd6680b3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:22.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 shutdown_connections 2022-07-12T08:49:22.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.117+0000 7fd66debb700 1 -- 172.21.15.61:0/3679999614 wait complete. 2022-07-12T08:49:22.139 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:49:22.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.237+0000 7f4864e78700 1 Processor -- start 2022-07-12T08:49:22.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.237+0000 7f4864e78700 1 -- start start 2022-07-12T08:49:22.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f48600aefb0 0x7f4860140a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48600b0160 0x7f48600b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486005c090 con 0x7f48600b13c0 2022-07-12T08:49:22.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48600b0790 con 0x7f48600b0160 2022-07-12T08:49:22.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48600b0a30 con 0x7f48600aefb0 2022-07-12T08:49:22.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48600b0160 0x7f48600b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48600b0160 0x7f48600b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59956/0 (socket says 172.21.15.61:59956) 2022-07-12T08:49:22.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f485ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f48600b13c0 0x7f48600abc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48176/0 (socket says 172.21.15.61:48176) 2022-07-12T08:49:22.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 -- 172.21.15.61:0/2554285702 learned_addr learned my addr 172.21.15.61:0/2554285702 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:22.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f485f7fe700 1 --2- 172.21.15.61:0/2554285702 >> v2:172.21.15.111:3300/0 conn(0x7f48600aefb0 0x7f4860140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 -- 172.21.15.61:0/2554285702 >> v2:172.21.15.111:3300/0 conn(0x7f48600aefb0 msgr2=0x7f4860140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:22.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 --2- 172.21.15.61:0/2554285702 >> v2:172.21.15.111:3300/0 conn(0x7f48600aefb0 0x7f4860140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f485effd700 1 -- 172.21.15.61:0/2554285702 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359633544 0 0) 0x7f486005c090 con 0x7f48600b13c0 2022-07-12T08:49:22.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 -- 172.21.15.61:0/2554285702 >> v1:172.21.15.61:6789/0 conn(0x7f48600b13c0 legacy=0x7f48600abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 -- 172.21.15.61:0/2554285702 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486005bd10 con 0x7f48600b0160 2022-07-12T08:49:22.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4857fff700 1 --2- 172.21.15.61:0/2554285702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b0160 0x7f48600b0ed0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f4848002f70 tx=0x7f4848006eb0).ready entity=mon.0 client_cookie=cdb4fe1514599490 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f485effd700 1 -- 172.21.15.61:0/2554285702 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f484800d610 con 0x7f48600b0160 2022-07-12T08:49:22.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f485effd700 1 -- 172.21.15.61:0/2554285702 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4848000e90 con 0x7f48600b0160 2022-07-12T08:49:22.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f485effd700 1 -- 172.21.15.61:0/2554285702 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f484800d960 con 0x7f48600b0160 2022-07-12T08:49:22.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- 172.21.15.61:0/2554285702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b0160 msgr2=0x7f48600b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 --2- 172.21.15.61:0/2554285702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b0160 0x7f48600b0ed0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 rx=0x7f4848002f70 tx=0x7f4848006eb0).stop 2022-07-12T08:49:22.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- 172.21.15.61:0/2554285702 shutdown_connections 2022-07-12T08:49:22.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 --2- 172.21.15.61:0/2554285702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b0160 0x7f48600b0ed0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 --2- 172.21.15.61:0/2554285702 >> v2:172.21.15.111:3300/0 conn(0x7f48600aefb0 0x7f4860140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- 172.21.15.61:0/2554285702 >> 172.21.15.61:0/2554285702 conn(0x7f48600b57c0 msgr2=0x7f48600b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:22.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- 172.21.15.61:0/2554285702 shutdown_connections 2022-07-12T08:49:22.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.241+0000 7f4864e78700 1 -- 172.21.15.61:0/2554285702 wait complete. 2022-07-12T08:49:22.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 Processor -- start 2022-07-12T08:49:22.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 -- start start 2022-07-12T08:49:22.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48600aefb0 0x7f4860142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 0x7f486013bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 0x7f486013c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860143550 con 0x7f48600b13c0 2022-07-12T08:49:22.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48601503a0 con 0x7f48600b0160 2022-07-12T08:49:22.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4864e78700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4860150640 con 0x7f48600aefb0 2022-07-12T08:49:22.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f485ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 0x7f486013c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 0x7f486013bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 0x7f486013bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:38536/0 (socket says 172.21.15.61:38536) 2022-07-12T08:49:22.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 -- 172.21.15.61:0/2800844167 learned_addr learned my addr 172.21.15.61:0/2800844167 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:22.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f485ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 0x7f486013c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59964/0 (socket says 172.21.15.61:59964) 2022-07-12T08:49:22.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f485f7fe700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48600aefb0 0x7f4860142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 -- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48600aefb0 msgr2=0x7f4860142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48600aefb0 0x7f4860142bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 -- 172.21.15.61:0/2800844167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 msgr2=0x7f486013c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 0x7f486013c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 -- 172.21.15.61:0/2800844167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484800b1e0 con 0x7f48600b0160 2022-07-12T08:49:22.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f485ffff700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 0x7f486013c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:22.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.245+0000 7f4857fff700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 0x7f486013bdb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f4848002f70 tx=0x7f4848015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4848006b60 con 0x7f48600b0160 2022-07-12T08:49:22.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484800d290 con 0x7f48600b0160 2022-07-12T08:49:22.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486013c790 con 0x7f48600b0160 2022-07-12T08:49:22.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4848011d70 con 0x7f48600b0160 2022-07-12T08:49:22.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4860066b70 con 0x7f48600b0160 2022-07-12T08:49:22.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48601501c0 con 0x7f48600b0160 2022-07-12T08:49:22.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4848025030 con 0x7f48600b0160 2022-07-12T08:49:22.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485d7fa700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4838053a40 0x7f4838055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f484806cd70 con 0x7f48600b0160 2022-07-12T08:49:22.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485f7fe700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4838053a40 0x7f4838055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.249+0000 7f485f7fe700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4838053a40 0x7f4838055f00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f485000c000 tx=0x7f4850008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.261+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4848043ef0 con 0x7f48600b0160 2022-07-12T08:49:22.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.389+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4860057460 con 0x7f48600b0160 2022-07-12T08:49:22.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f485d7fa700 1 -- 172.21.15.61:0/2800844167 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v106) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4848023070 con 0x7f48600b0160 2022-07-12T08:49:22.398 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4838053a40 msgr2=0x7f4838055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4838053a40 0x7f4838055f00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f485000c000 tx=0x7f4850008040).stop 2022-07-12T08:49:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 msgr2=0x7f486013bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 0x7f486013bdb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7f4848002f70 tx=0x7f4848015000).stop 2022-07-12T08:49:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 shutdown_connections 2022-07-12T08:49:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48600b13c0 0x7f486013c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4838053a40 0x7f4838055f00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48600b0160 0x7f486013bdb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 --2- 172.21.15.61:0/2800844167 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48600aefb0 0x7f4860142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 >> 172.21.15.61:0/2800844167 conn(0x7f48600b57c0 msgr2=0x7f4860060600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 shutdown_connections 2022-07-12T08:49:22.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.393+0000 7f4864e78700 1 -- 172.21.15.61:0/2800844167 wait complete. 2022-07-12T08:49:22.411 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:49:22.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.509+0000 7f88c126f700 1 Processor -- start 2022-07-12T08:49:22.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.509+0000 7f88c126f700 1 -- start start 2022-07-12T08:49:22.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.509+0000 7f88c126f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.509+0000 7f88c126f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc0abc70 con 0x7f88bc0ab290 2022-07-12T08:49:22.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.509+0000 7f88c126f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc05c720 con 0x7f88bc0ab670 2022-07-12T08:49:22.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.509+0000 7f88c126f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0aba90 con 0x7f88bc05f140 2022-07-12T08:49:22.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f88bc0ab290 0x7f88bc05ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48186/0 (socket says 172.21.15.61:48186) 2022-07-12T08:49:22.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bbfff700 1 -- 172.21.15.61:0/2616049873 learned_addr learned my addr 172.21.15.61:0/2616049873 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:22.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c0a6e700 1 --2- 172.21.15.61:0/2616049873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49253970 0 0) 0x7f88bc0abc70 con 0x7f88bc0ab290 2022-07-12T08:49:22.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889c003260 con 0x7f88bc0ab290 2022-07-12T08:49:22.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660729177 0 0) 0x7f88bc05c720 con 0x7f88bc0ab670 2022-07-12T08:49:22.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc0abc70 con 0x7f88bc0ab670 2022-07-12T08:49:22.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c0a6e700 1 -- 172.21.15.61:0/2616049873 >> v1:172.21.15.111:6789/0 conn(0x7f88bc0ab670 legacy=0x7f88bc0afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c0a6e700 1 -- 172.21.15.61:0/2616049873 >> v1:172.21.15.61:6789/0 conn(0x7f88bc0ab290 legacy=0x7f88bc05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c0a6e700 1 -- 172.21.15.61:0/2616049873 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc05c9c0 con 0x7f88bc05f140 2022-07-12T08:49:22.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c0a6e700 1 --2- 172.21.15.61:0/2616049873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc151bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f88b40032f0 tx=0x7f88b40091d0).ready entity=mon.2 client_cookie=84a53f3f6723d8d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400b640 con 0x7f88bc05f140 2022-07-12T08:49:22.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b400b7a0 con 0x7f88bc05f140 2022-07-12T08:49:22.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88bb7fe700 1 -- 172.21.15.61:0/2616049873 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400b910 con 0x7f88bc05f140 2022-07-12T08:49:22.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 -- 172.21.15.61:0/2616049873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 msgr2=0x7f88bc151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 --2- 172.21.15.61:0/2616049873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc151bf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f88b40032f0 tx=0x7f88b40091d0).stop 2022-07-12T08:49:22.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 -- 172.21.15.61:0/2616049873 shutdown_connections 2022-07-12T08:49:22.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 --2- 172.21.15.61:0/2616049873 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc151bf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 -- 172.21.15.61:0/2616049873 >> 172.21.15.61:0/2616049873 conn(0x7f88bc0b1950 msgr2=0x7f88bc0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:22.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 -- 172.21.15.61:0/2616049873 shutdown_connections 2022-07-12T08:49:22.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.513+0000 7f88c126f700 1 -- 172.21.15.61:0/2616049873 wait complete. 2022-07-12T08:49:22.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 Processor -- start 2022-07-12T08:49:22.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- start start 2022-07-12T08:49:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc0674d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc0ab290 0x7f88bc066b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 0x7f88bc143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc05f5b0 con 0x7f88bc0ab670 2022-07-12T08:49:22.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc099b60 con 0x7f88bc0ab290 2022-07-12T08:49:22.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc0ae640 con 0x7f88bc05f140 2022-07-12T08:49:22.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 0x7f88bc143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 0x7f88bc143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59978/0 (socket says 172.21.15.61:59978) 2022-07-12T08:49:22.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 -- 172.21.15.61:0/3983114474 learned_addr learned my addr 172.21.15.61:0/3983114474 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:22.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88bbfff700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc0674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88b3fff700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc0ab290 0x7f88bc066b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 -- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 msgr2=0x7f88bc0674d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc0674d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 -- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc0ab290 msgr2=0x7f88bc066b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc0ab290 0x7f88bc066b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 -- 172.21.15.61:0/3983114474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4009470 con 0x7f88bc0ab670 2022-07-12T08:49:22.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88bbfff700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc0674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:22.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c0a6e700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 0x7f88bc143bc0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f88bc065880 tx=0x7f88b4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400b910 con 0x7f88bc0ab670 2022-07-12T08:49:22.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b401f3f0 con 0x7f88bc0ab670 2022-07-12T08:49:22.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc063940 con 0x7f88bc0ab670 2022-07-12T08:49:22.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400b640 con 0x7f88bc0ab670 2022-07-12T08:49:22.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc0ae2c0 con 0x7f88bc0ab670 2022-07-12T08:49:22.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.517+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc099ca0 con 0x7f88bc0ab670 2022-07-12T08:49:22.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.521+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f88b4026030 con 0x7f88bc0ab670 2022-07-12T08:49:22.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.521+0000 7f88b9ffb700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88a0053940 0x7f88a0055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.521+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f88b406c8a0 con 0x7f88bc0ab670 2022-07-12T08:49:22.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.521+0000 7f88bbfff700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88a0053940 0x7f88a0055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.521+0000 7f88bbfff700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88a0053940 0x7f88a0055e00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f88ac00b000 tx=0x7f88ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.525+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88b40439a0 con 0x7f88bc0ab670 2022-07-12T08:49:22.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.649+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f88bc057460 con 0x7f88bc0ab670 2022-07-12T08:49:22.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88b9ffb700 1 -- 172.21.15.61:0/3983114474 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+143 (secure 0 0 0) 0x7f88b4024090 con 0x7f88bc0ab670 2022-07-12T08:49:22.658 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:22.659 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:49:22.659 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:22.660 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:49:22.660 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:49:22.660 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:22.661 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:22.661 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:49:22.662 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:49:22.662 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:22.663 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:49:22.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88a0053940 msgr2=0x7f88a0055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88a0053940 0x7f88a0055e00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f88ac00b000 tx=0x7f88ac009040).stop 2022-07-12T08:49:22.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 msgr2=0x7f88bc143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 0x7f88bc143bc0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7f88bc065880 tx=0x7f88b4016000).stop 2022-07-12T08:49:22.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 shutdown_connections 2022-07-12T08:49:22.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0ab670 0x7f88bc143bc0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88a0053940 0x7f88a0055e00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88bc0ab290 0x7f88bc066b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 --2- 172.21.15.61:0/3983114474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88bc05f140 0x7f88bc0674d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 >> 172.21.15.61:0/3983114474 conn(0x7f88bc0b1950 msgr2=0x7f88bc0b0310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:22.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 shutdown_connections 2022-07-12T08:49:22.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.653+0000 7f88c126f700 1 -- 172.21.15.61:0/3983114474 wait complete. 2022-07-12T08:49:22.670 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:49:22.671 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:49:22.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 Processor -- start 2022-07-12T08:49:22.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 -- start start 2022-07-12T08:49:22.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf980b5920 0x7faf9814a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf980aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9805c3e0 con 0x7faf980b0280 2022-07-12T08:49:22.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf980b5d00 con 0x7faf980b5920 2022-07-12T08:49:22.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9805c610 con 0x7faf980aa830 2022-07-12T08:49:22.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9c901700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faf980b5920 0x7faf9814a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf97fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7faf980b0280 0x7faf9814aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48202/0 (socket says 172.21.15.61:48202) 2022-07-12T08:49:22.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf97fff700 1 -- 172.21.15.61:0/3947766249 learned_addr learned my addr 172.21.15.61:0/3947766249 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:22.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d102700 1 --2- 172.21.15.61:0/3947766249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf980aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9c901700 1 -- 172.21.15.61:0/3947766249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 msgr2=0x7faf980aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:22.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9c901700 1 --2- 172.21.15.61:0/3947766249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf980aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9c901700 1 -- 172.21.15.61:0/3947766249 >> v1:172.21.15.61:6789/0 conn(0x7faf980b0280 legacy=0x7faf9814aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf977fe700 1 -- 172.21.15.61:0/3947766249 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676642441 0 0) 0x7faf9805c3e0 con 0x7faf980b0280 2022-07-12T08:49:22.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9c901700 1 -- 172.21.15.61:0/3947766249 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9805c8b0 con 0x7faf980b5920 2022-07-12T08:49:22.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9c901700 1 --2- 172.21.15.61:0/3947766249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf9814a9e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7faf80002f70 tx=0x7faf80006df0).ready entity=mon.0 client_cookie=67929e117746580b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf977fe700 1 -- 172.21.15.61:0/3947766249 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf8000d570 con 0x7faf980b5920 2022-07-12T08:49:22.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf977fe700 1 -- 172.21.15.61:0/3947766249 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf80000e90 con 0x7faf980b5920 2022-07-12T08:49:22.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf977fe700 1 -- 172.21.15.61:0/3947766249 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf8000d8c0 con 0x7faf980b5920 2022-07-12T08:49:22.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 -- 172.21.15.61:0/3947766249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 msgr2=0x7faf9814a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.781+0000 7faf9d903700 1 --2- 172.21.15.61:0/3947766249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf9814a9e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7faf80002f70 tx=0x7faf80006df0).stop 2022-07-12T08:49:22.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- 172.21.15.61:0/3947766249 shutdown_connections 2022-07-12T08:49:22.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 --2- 172.21.15.61:0/3947766249 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf980aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 --2- 172.21.15.61:0/3947766249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf9814a9e0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- 172.21.15.61:0/3947766249 >> 172.21.15.61:0/3947766249 conn(0x7faf98063340 msgr2=0x7faf980b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:22.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- 172.21.15.61:0/3947766249 shutdown_connections 2022-07-12T08:49:22.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- 172.21.15.61:0/3947766249 wait complete. 2022-07-12T08:49:22.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 Processor -- start 2022-07-12T08:49:22.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- start start 2022-07-12T08:49:22.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf98063bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf980b0280 0x7faf980640b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf98159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf980645a0 con 0x7faf980b5920 2022-07-12T08:49:22.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9814d270 con 0x7faf980b0280 2022-07-12T08:49:22.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d903700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9814d4d0 con 0x7faf980aa830 2022-07-12T08:49:22.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d102700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf98159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d102700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf98159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:59992/0 (socket says 172.21.15.61:59992) 2022-07-12T08:49:22.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9d102700 1 -- 172.21.15.61:0/2131787349 learned_addr learned my addr 172.21.15.61:0/2131787349 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:22.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.785+0000 7faf9c901700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf98063bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf97fff700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf980b0280 0x7faf980640b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d102700 1 -- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 msgr2=0x7faf98063bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d102700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf98063bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d102700 1 -- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf980b0280 msgr2=0x7faf980640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d102700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf980b0280 0x7faf980640b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d102700 1 -- 172.21.15.61:0/2131787349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8000b180 con 0x7faf980b5920 2022-07-12T08:49:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d102700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf98159420 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7faf9000a000 tx=0x7faf90008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf90011780 con 0x7faf980b5920 2022-07-12T08:49:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf900118e0 con 0x7faf980b5920 2022-07-12T08:49:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9814dfb0 con 0x7faf980b5920 2022-07-12T08:49:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf90011c40 con 0x7faf980b5920 2022-07-12T08:49:22.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf980666e0 con 0x7faf980b5920 2022-07-12T08:49:22.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9814e170 con 0x7faf980b5920 2022-07-12T08:49:22.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faf90012420 con 0x7faf980b5920 2022-07-12T08:49:22.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf95ffb700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf7c053ad0 0x7faf7c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:22.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.789+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5253+0+0 (secure 0 0 0) 0x7faf9006b950 con 0x7faf980b5920 2022-07-12T08:49:22.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.793+0000 7faf9c901700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf7c053ad0 0x7faf7c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:22.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.793+0000 7faf9c901700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf7c053ad0 0x7faf7c055f90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7faf8001c000 tx=0x7faf80015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:22.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.797+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faf9814e170 con 0x7faf980b5920 2022-07-12T08:49:22.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:22.921+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7faf9814e140 con 0x7faf980b5920 2022-07-12T08:49:23.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.305+0000 7faf95ffb700 1 -- 172.21.15.61:0/2131787349 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v107) v1 ==== 133+0+0 (secure 0 0 0) 0x7faf90042b80 con 0x7faf980b5920 2022-07-12T08:49:23.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf7c053ad0 msgr2=0x7faf7c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf7c053ad0 0x7faf7c055f90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7faf8001c000 tx=0x7faf80015000).stop 2022-07-12T08:49:23.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 msgr2=0x7faf98159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf98159420 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 rx=0x7faf9000a000 tx=0x7faf90008040).stop 2022-07-12T08:49:23.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 shutdown_connections 2022-07-12T08:49:23.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faf980b5920 0x7faf98159420 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faf7c053ad0 0x7faf7c055f90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf980b0280 0x7faf980640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 --2- 172.21.15.61:0/2131787349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faf980aa830 0x7faf98063bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 >> 172.21.15.61:0/2131787349 conn(0x7faf98063340 msgr2=0x7faf980b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:23.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 shutdown_connections 2022-07-12T08:49:23.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.309+0000 7faf9d903700 1 -- 172.21.15.61:0/2131787349 wait complete. 2022-07-12T08:49:23.319 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:49:23.333 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:49:23.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 Processor -- start 2022-07-12T08:49:23.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 -- start start 2022-07-12T08:49:23.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7858061e30 0x7f7858062210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:23.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7858143e80 con 0x7f78580b1a70 2022-07-12T08:49:23.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78581440b0 con 0x7f78580ad2b0 2022-07-12T08:49:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78580ac290 con 0x7f7858061e30 2022-07-12T08:49:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e140700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f78580b1a70 0x7f785813ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48218/0 (socket says 172.21.15.61:48218) 2022-07-12T08:49:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e140700 1 -- 172.21.15.61:0/1640363413 learned_addr learned my addr 172.21.15.61:0/1640363413 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785d93f700 1 --2- 172.21.15.61:0/1640363413 >> v2:172.21.15.111:3300/0 conn(0x7f7858061e30 0x7f7858062210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785c93d700 1 -- 172.21.15.61:0/1640363413 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880882652 0 0) 0x7f7858143e80 con 0x7f78580b1a70 2022-07-12T08:49:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785c93d700 1 -- 172.21.15.61:0/1640363413 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f783c003260 con 0x7f78580b1a70 2022-07-12T08:49:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785d93f700 1 -- 172.21.15.61:0/1640363413 >> v1:172.21.15.111:6789/0 conn(0x7f78580ad2b0 legacy=0x7f78581447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785d93f700 1 -- 172.21.15.61:0/1640363413 >> v1:172.21.15.61:6789/0 conn(0x7f78580b1a70 legacy=0x7f785813ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785d93f700 1 -- 172.21.15.61:0/1640363413 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858144350 con 0x7f7858061e30 2022-07-12T08:49:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785d93f700 1 --2- 172.21.15.61:0/1640363413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7858061e30 0x7f7858062210 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f7854002f70 tx=0x7f7854006df0).ready entity=mon.1 client_cookie=c05438a331571de3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785c93d700 1 -- 172.21.15.61:0/1640363413 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f785400d610 con 0x7f7858061e30 2022-07-12T08:49:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785c93d700 1 -- 172.21.15.61:0/1640363413 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854000e90 con 0x7f7858061e30 2022-07-12T08:49:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785c93d700 1 -- 172.21.15.61:0/1640363413 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f785400d960 con 0x7f7858061e30 2022-07-12T08:49:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 -- 172.21.15.61:0/1640363413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7858061e30 msgr2=0x7f7858062210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.445+0000 7f785e941700 1 --2- 172.21.15.61:0/1640363413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7858061e30 0x7f7858062210 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f7854002f70 tx=0x7f7854006df0).stop 2022-07-12T08:49:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- 172.21.15.61:0/1640363413 shutdown_connections 2022-07-12T08:49:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 --2- 172.21.15.61:0/1640363413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7858061e30 0x7f7858062210 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- 172.21.15.61:0/1640363413 >> 172.21.15.61:0/1640363413 conn(0x7f78580b70d0 msgr2=0x7f78580b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- 172.21.15.61:0/1640363413 shutdown_connections 2022-07-12T08:49:23.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- 172.21.15.61:0/1640363413 wait complete. 2022-07-12T08:49:23.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 Processor -- start 2022-07-12T08:49:23.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- start start 2022-07-12T08:49:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7858061e30 0x7f785809cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78580ad2b0 0x7f78580b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 0x7f785809a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78581430e0 con 0x7f78580b1a70 2022-07-12T08:49:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7858143e60 con 0x7f78580ad2b0 2022-07-12T08:49:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e941700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7858143310 con 0x7f7858061e30 2022-07-12T08:49:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 0x7f785809a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:23.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 0x7f785809a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60006/0 (socket says 172.21.15.61:60006) 2022-07-12T08:49:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 -- 172.21.15.61:0/3619795684 learned_addr learned my addr 172.21.15.61:0/3619795684 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785d93f700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7858061e30 0x7f785809cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785d13e700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78580ad2b0 0x7f78580b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 -- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7858061e30 msgr2=0x7f785809cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7858061e30 0x7f785809cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 -- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78580ad2b0 msgr2=0x7f78580b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78580ad2b0 0x7f78580b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 -- 172.21.15.61:0/3619795684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785400b180 con 0x7f78580b1a70 2022-07-12T08:49:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.449+0000 7f785e140700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 0x7f785809a520 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f7850001490 tx=0x7f7850007e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7850005ba0 con 0x7f78580b1a70 2022-07-12T08:49:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7850005d00 con 0x7f78580b1a70 2022-07-12T08:49:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f78500072c0 con 0x7f78580b1a70 2022-07-12T08:49:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7858151b20 con 0x7f78580b1a70 2022-07-12T08:49:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7858152200 con 0x7f78580b1a70 2022-07-12T08:49:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7858143690 con 0x7f78580b1a70 2022-07-12T08:49:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7850010070 con 0x7f78580b1a70 2022-07-12T08:49:23.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.453+0000 7f784affd700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7840053920 0x7f7840055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.457+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f785006b530 con 0x7f78580b1a70 2022-07-12T08:49:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.457+0000 7f785d93f700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7840053920 0x7f7840055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.457+0000 7f785d93f700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7840053920 0x7f7840055de0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f785401a000 tx=0x7f7854014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.461+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7850042660 con 0x7f78580b1a70 2022-07-12T08:49:23.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:23.585+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f7858099db0 con 0x7f78580b1a70 2022-07-12T08:49:24.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.321+0000 7f784affd700 1 -- 172.21.15.61:0/3619795684 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v108) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7850018020 con 0x7f78580b1a70 2022-07-12T08:49:24.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7840053920 msgr2=0x7f7840055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7840053920 0x7f7840055de0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f785401a000 tx=0x7f7854014000).stop 2022-07-12T08:49:24.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 msgr2=0x7f785809a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 0x7f785809a520 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f7850001490 tx=0x7f7850007e40).stop 2022-07-12T08:49:24.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 shutdown_connections 2022-07-12T08:49:24.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f78580b1a70 0x7f785809a520 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7840053920 0x7f7840055de0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f78580ad2b0 0x7f78580b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 --2- 172.21.15.61:0/3619795684 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7858061e30 0x7f785809cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 >> 172.21.15.61:0/3619795684 conn(0x7f78580b70d0 msgr2=0x7f78580b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:24.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 shutdown_connections 2022-07-12T08:49:24.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.325+0000 7f785e941700 1 -- 172.21.15.61:0/3619795684 wait complete. 2022-07-12T08:49:24.335 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:49:24.345 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:49:24.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.445+0000 7fc52b25e700 1 Processor -- start 2022-07-12T08:49:24.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.445+0000 7fc52b25e700 1 -- start start 2022-07-12T08:49:24.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc5240b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5240b2940 0x7fc52405c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240b2f20 con 0x7fc5240b45d0 2022-07-12T08:49:24.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5240b3170 con 0x7fc5240b2940 2022-07-12T08:49:24.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5240b4170 con 0x7fc524066fc0 2022-07-12T08:49:24.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5240b2940 0x7fc52405c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5240b2940 0x7fc52405c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60012/0 (socket says 172.21.15.61:60012) 2022-07-12T08:49:24.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 -- 172.21.15.61:0/2659545391 learned_addr learned my addr 172.21.15.61:0/2659545391 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:24.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc528ffa700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc5240b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 -- 172.21.15.61:0/2659545391 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 msgr2=0x7fc5240b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:24.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc5240b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 -- 172.21.15.61:0/2659545391 >> v1:172.21.15.111:6789/0 conn(0x7fc5240b45d0 legacy=0x7fc5240b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 -- 172.21.15.61:0/2659545391 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240673a0 con 0x7fc5240b2940 2022-07-12T08:49:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc528ffa700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc5240b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc5297fb700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc52405c090 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7fc524142d40 tx=0x7fc51c009290).ready entity=mon.0 client_cookie=2a1603c56e4bd60f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc51b7fe700 1 -- 172.21.15.61:0/2659545391 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc51c00b860 con 0x7fc5240b2940 2022-07-12T08:49:24.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc51b7fe700 1 -- 172.21.15.61:0/2659545391 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c00b9c0 con 0x7fc5240b2940 2022-07-12T08:49:24.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc51b7fe700 1 -- 172.21.15.61:0/2659545391 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc51c00bb30 con 0x7fc5240b2940 2022-07-12T08:49:24.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 -- 172.21.15.61:0/2659545391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 msgr2=0x7fc52405c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc52405c090 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 rx=0x7fc524142d40 tx=0x7fc51c009290).stop 2022-07-12T08:49:24.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.449+0000 7fc52b25e700 1 -- 172.21.15.61:0/2659545391 shutdown_connections 2022-07-12T08:49:24.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc52405c090 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 --2- 172.21.15.61:0/2659545391 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc5240b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- 172.21.15.61:0/2659545391 >> 172.21.15.61:0/2659545391 conn(0x7fc5240b94d0 msgr2=0x7fc5240b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:24.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- 172.21.15.61:0/2659545391 shutdown_connections 2022-07-12T08:49:24.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- 172.21.15.61:0/2659545391 wait complete. 2022-07-12T08:49:24.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 Processor -- start 2022-07-12T08:49:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- start start 2022-07-12T08:49:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc524152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc524145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5240b45d0 0x7fc52413f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5241462e0 con 0x7fc5240b2940 2022-07-12T08:49:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524145da0 con 0x7fc5240b45d0 2022-07-12T08:49:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.453+0000 7fc52b25e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc524146040 con 0x7fc524066fc0 2022-07-12T08:49:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc524145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc524145700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60018/0 (socket says 172.21.15.61:60018) 2022-07-12T08:49:24.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 -- 172.21.15.61:0/1604625730 learned_addr learned my addr 172.21.15.61:0/1604625730 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:24.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc528ffa700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc524152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc5297fb700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5240b45d0 0x7fc52413f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 -- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 msgr2=0x7fc524152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc524152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 -- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5240b45d0 msgr2=0x7fc52413f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5240b45d0 0x7fc52413f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 -- 172.21.15.61:0/1604625730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c009530 con 0x7fc5240b2940 2022-07-12T08:49:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc51bfff700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc524145700 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7fc52405d000 tx=0x7fc514007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc514007e40 con 0x7fc5240b2940 2022-07-12T08:49:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc514007030 con 0x7fc5240b2940 2022-07-12T08:49:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52413fbf0 con 0x7fc5240b2940 2022-07-12T08:49:24.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc514007390 con 0x7fc5240b2940 2022-07-12T08:49:24.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52413e980 con 0x7fc5240b2940 2022-07-12T08:49:24.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.457+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc524145bf0 con 0x7fc5240b2940 2022-07-12T08:49:24.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.461+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc514010030 con 0x7fc5240b2940 2022-07-12T08:49:24.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.461+0000 7fc519ffb700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc504053a40 0x7fc504055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.461+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fc51406b880 con 0x7fc5240b2940 2022-07-12T08:49:24.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.461+0000 7fc528ffa700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc504053a40 0x7fc504055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.461+0000 7fc528ffa700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc504053a40 0x7fc504055f00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7fc52000c000 tx=0x7fc520008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:24.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.465+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5140427f0 con 0x7fc5240b2940 2022-07-12T08:49:24.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.629+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fc524144160 con 0x7fc5240b2940 2022-07-12T08:49:24.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.637+0000 7fc519ffb700 1 -- 172.21.15.61:0/1604625730 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 21 and data pool 22 v61) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc514045e10 con 0x7fc5240b2940 2022-07-12T08:49:24.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc504053a40 msgr2=0x7fc504055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc504053a40 0x7fc504055f00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7fc52000c000 tx=0x7fc520008040).stop 2022-07-12T08:49:24.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 msgr2=0x7fc524145700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc524145700 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7fc52405d000 tx=0x7fc514007bd0).stop 2022-07-12T08:49:24.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 shutdown_connections 2022-07-12T08:49:24.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc504053a40 0x7fc504055f00 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc5240b45d0 0x7fc52413f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5240b2940 0x7fc524145700 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 --2- 172.21.15.61:0/1604625730 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc524066fc0 0x7fc524152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.641+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 >> 172.21.15.61:0/1604625730 conn(0x7fc5240b94d0 msgr2=0x7fc5240b7300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:24.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.645+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 shutdown_connections 2022-07-12T08:49:24.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.645+0000 7fc52b25e700 1 -- 172.21.15.61:0/1604625730 wait complete. 2022-07-12T08:49:24.654 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 21 and data pool 22 2022-07-12T08:49:24.663 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:24.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 Processor -- start 2022-07-12T08:49:24.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- start start 2022-07-12T08:49:24.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd41405c050 0x7fd4140ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41414a8f0 0x7fd4141443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4140ad2f0 con 0x7fd4140b01f0 2022-07-12T08:49:24.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4140b0a40 con 0x7fd41405c050 2022-07-12T08:49:24.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41405c430 con 0x7fd41414a8f0 2022-07-12T08:49:24.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd41405c050 0x7fd4140ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd41405c050 0x7fd4140ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60036/0 (socket says 172.21.15.61:60036) 2022-07-12T08:49:24.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd418b8a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd4140b01f0 0x7fd41405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48254/0 (socket says 172.21.15.61:48254) 2022-07-12T08:49:24.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 -- 172.21.15.61:0/919547956 learned_addr learned my addr 172.21.15.61:0/919547956 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:24.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd419b8c700 1 --2- 172.21.15.61:0/919547956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41414a8f0 0x7fd4141443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 -- 172.21.15.61:0/919547956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41414a8f0 msgr2=0x7fd4141443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 --2- 172.21.15.61:0/919547956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41414a8f0 0x7fd4141443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 -- 172.21.15.61:0/919547956 >> v1:172.21.15.61:6789/0 conn(0x7fd4140b01f0 legacy=0x7fd41405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd40bfff700 1 -- 172.21.15.61:0/919547956 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094249603 0 0) 0x7fd4140ad2f0 con 0x7fd4140b01f0 2022-07-12T08:49:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 -- 172.21.15.61:0/919547956 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4140b2420 con 0x7fd41405c050 2022-07-12T08:49:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41938b700 1 --2- 172.21.15.61:0/919547956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd41405c050 0x7fd4140ac710 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7fd4100032f0 tx=0x7fd410009290).ready entity=mon.0 client_cookie=6b7e99f1db610c76 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd40bfff700 1 -- 172.21.15.61:0/919547956 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd41000b8a0 con 0x7fd41405c050 2022-07-12T08:49:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd40bfff700 1 -- 172.21.15.61:0/919547956 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd41000ba00 con 0x7fd41405c050 2022-07-12T08:49:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd40bfff700 1 -- 172.21.15.61:0/919547956 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd41000bb70 con 0x7fd41405c050 2022-07-12T08:49:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- 172.21.15.61:0/919547956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd41405c050 msgr2=0x7fd4140ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/919547956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd41405c050 0x7fd4140ac710 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 rx=0x7fd4100032f0 tx=0x7fd410009290).stop 2022-07-12T08:49:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- 172.21.15.61:0/919547956 shutdown_connections 2022-07-12T08:49:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/919547956 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41414a8f0 0x7fd4141443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/919547956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd41405c050 0x7fd4140ac710 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- 172.21.15.61:0/919547956 >> 172.21.15.61:0/919547956 conn(0x7fd414062c00 msgr2=0x7fd414062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- 172.21.15.61:0/919547956 shutdown_connections 2022-07-12T08:49:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.753+0000 7fd41b5ef700 1 -- 172.21.15.61:0/919547956 wait complete. 2022-07-12T08:49:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 Processor -- start 2022-07-12T08:49:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- start start 2022-07-12T08:49:24.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41405c050 0x7fd4140641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 0x7fd414159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd41414a8f0 0x7fd41415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140a1fb0 con 0x7fd4140b01f0 2022-07-12T08:49:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140a1000 con 0x7fd41414a8f0 2022-07-12T08:49:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4140a0da0 con 0x7fd41405c050 2022-07-12T08:49:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 0x7fd414159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 0x7fd414159e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60044/0 (socket says 172.21.15.61:60044) 2022-07-12T08:49:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 -- 172.21.15.61:0/1598361452 learned_addr learned my addr 172.21.15.61:0/1598361452 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd419b8c700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd41414a8f0 0x7fd41415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41938b700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41405c050 0x7fd4140641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 -- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41405c050 msgr2=0x7fd4140641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41405c050 0x7fd4140641a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 -- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd41414a8f0 msgr2=0x7fd41415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd41414a8f0 0x7fd41415a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 -- 172.21.15.61:0/1598361452 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd410009530 con 0x7fd4140b01f0 2022-07-12T08:49:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd419b8c700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd41414a8f0 0x7fd41415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41938b700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41405c050 0x7fd4140641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd418b8a700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 0x7fd414159e60 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7fd4140a20f0 tx=0x7fd404007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4040072e0 con 0x7fd4140b01f0 2022-07-12T08:49:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404005550 con 0x7fd4140b01f0 2022-07-12T08:49:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4140a0aa0 con 0x7fd4140b01f0 2022-07-12T08:49:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4040058b0 con 0x7fd4140b01f0 2022-07-12T08:49:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4140a13d0 con 0x7fd4140b01f0 2022-07-12T08:49:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.757+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4140a1ca0 con 0x7fd4140b01f0 2022-07-12T08:49:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.761+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd40401a060 con 0x7fd4140b01f0 2022-07-12T08:49:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.761+0000 7fd40a7fc700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3fc0539b0 0x7fd3fc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.761+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd40406b770 con 0x7fd4140b01f0 2022-07-12T08:49:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.761+0000 7fd41938b700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3fc0539b0 0x7fd3fc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.761+0000 7fd41938b700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3fc0539b0 0x7fd3fc055e70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7fd410007000 tx=0x7fd410016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:24.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.765+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4040426d0 con 0x7fd4140b01f0 2022-07-12T08:49:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.893+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd414057460 con 0x7fd4140b01f0 2022-07-12T08:49:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd40a7fc700 1 -- 172.21.15.61:0/1598361452 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16112 (secure 0 0 0) 0x7fd404018020 con 0x7fd4140b01f0 2022-07-12T08:49:24.902 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:24.902 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":109,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:24.635088+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:49:22.980785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-12T08:49:23.643810+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3fc0539b0 msgr2=0x7fd3fc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3fc0539b0 0x7fd3fc055e70 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7fd410007000 tx=0x7fd410016000).stop 2022-07-12T08:49:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 msgr2=0x7fd414159e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 0x7fd414159e60 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 rx=0x7fd4140a20f0 tx=0x7fd404007b40).stop 2022-07-12T08:49:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 shutdown_connections 2022-07-12T08:49:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd3fc0539b0 0x7fd3fc055e70 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd41414a8f0 0x7fd41415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4140b01f0 0x7fd414159e60 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 --2- 172.21.15.61:0/1598361452 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd41405c050 0x7fd4140641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:24.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 >> 172.21.15.61:0/1598361452 conn(0x7fd414062c00 msgr2=0x7fd414145a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:24.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 shutdown_connections 2022-07-12T08:49:24.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:24.897+0000 7fd41b5ef700 1 -- 172.21.15.61:0/1598361452 wait complete. 2022-07-12T08:49:24.916 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:25.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.009+0000 7f030acf1700 1 Processor -- start 2022-07-12T08:49:25.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.009+0000 7f030acf1700 1 -- start start 2022-07-12T08:49:25.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f030413ce60 0x7f03040b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030413d240 con 0x7f03040b47f0 2022-07-12T08:49:25.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03041431f0 con 0x7f0304142d50 2022-07-12T08:49:25.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b24c0 con 0x7f030413ce60 2022-07-12T08:49:25.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030928e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f03040b47f0 0x7f03040b4ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48272/0 (socket says 172.21.15.61:48272) 2022-07-12T08:49:25.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030928e700 1 -- 172.21.15.61:0/3968128102 learned_addr learned my addr 172.21.15.61:0/3968128102 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:25.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f0308a8d700 1 --2- 172.21.15.61:0/3968128102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f030413ce60 0x7f03040b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f03037fe700 1 -- 172.21.15.61:0/3968128102 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079201404 0 0) 0x7f030413d240 con 0x7f03040b47f0 2022-07-12T08:49:25.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f03037fe700 1 -- 172.21.15.61:0/3968128102 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e4003260 con 0x7f03040b47f0 2022-07-12T08:49:25.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f03037fe700 1 -- 172.21.15.61:0/3968128102 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629929946 0 0) 0x7f03041431f0 con 0x7f0304142d50 2022-07-12T08:49:25.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f0308a8d700 1 -- 172.21.15.61:0/3968128102 >> v1:172.21.15.111:6789/0 conn(0x7f0304142d50 legacy=0x7f03040b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f0308a8d700 1 -- 172.21.15.61:0/3968128102 >> v1:172.21.15.61:6789/0 conn(0x7f03040b47f0 legacy=0x7f03040b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f0308a8d700 1 -- 172.21.15.61:0/3968128102 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0304143490 con 0x7f030413ce60 2022-07-12T08:49:25.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f0308a8d700 1 --2- 172.21.15.61:0/3968128102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f030413ce60 0x7f03040b15a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f02fc0032f0 tx=0x7f02fc0091d0).ready entity=mon.2 client_cookie=d514f75652a42d90 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f03037fe700 1 -- 172.21.15.61:0/3968128102 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02fc00b6c0 con 0x7f030413ce60 2022-07-12T08:49:25.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f03037fe700 1 -- 172.21.15.61:0/3968128102 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc00b820 con 0x7f030413ce60 2022-07-12T08:49:25.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f03037fe700 1 -- 172.21.15.61:0/3968128102 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02fc00b990 con 0x7f030413ce60 2022-07-12T08:49:25.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- 172.21.15.61:0/3968128102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f030413ce60 msgr2=0x7f03040b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 --2- 172.21.15.61:0/3968128102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f030413ce60 0x7f03040b15a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f02fc0032f0 tx=0x7f02fc0091d0).stop 2022-07-12T08:49:25.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- 172.21.15.61:0/3968128102 shutdown_connections 2022-07-12T08:49:25.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 --2- 172.21.15.61:0/3968128102 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f030413ce60 0x7f03040b15a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- 172.21.15.61:0/3968128102 >> 172.21.15.61:0/3968128102 conn(0x7f030405f9f0 msgr2=0x7f0304066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:25.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- 172.21.15.61:0/3968128102 shutdown_connections 2022-07-12T08:49:25.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.013+0000 7f030acf1700 1 -- 172.21.15.61:0/3968128102 wait complete. 2022-07-12T08:49:25.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 Processor -- start 2022-07-12T08:49:25.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 -- start start 2022-07-12T08:49:25.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 0x7f0304151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f030413ce60 0x7f03040b9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0304142d50 0x7f030413d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b84a0 con 0x7f03040b47f0 2022-07-12T08:49:25.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b8740 con 0x7f030413ce60 2022-07-12T08:49:25.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03040b89e0 con 0x7f0304142d50 2022-07-12T08:49:25.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 0x7f0304151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 0x7f0304151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60056/0 (socket says 172.21.15.61:60056) 2022-07-12T08:49:25.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 -- 172.21.15.61:0/303639418 learned_addr learned my addr 172.21.15.61:0/303639418 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:25.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030928e700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0304142d50 0x7f030413d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0303fff700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f030413ce60 0x7f03040b9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 -- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0304142d50 msgr2=0x7f030413d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0304142d50 0x7f030413d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 -- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f030413ce60 msgr2=0x7f03040b9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f030413ce60 0x7f03040b9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 -- 172.21.15.61:0/303639418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc009470 con 0x7f03040b47f0 2022-07-12T08:49:25.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030928e700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0304142d50 0x7f030413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:25.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0303fff700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f030413ce60 0x7f03040b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:25.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0308a8d700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 0x7f0304151e60 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f02fc0032f0 tx=0x7f02fc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02fc00b990 con 0x7f03040b47f0 2022-07-12T08:49:25.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc01f3f0 con 0x7f03040b47f0 2022-07-12T08:49:25.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03040b9eb0 con 0x7f03040b47f0 2022-07-12T08:49:25.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02fc00b6c0 con 0x7f03040b47f0 2022-07-12T08:49:25.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.017+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0304152620 con 0x7f03040b47f0 2022-07-12T08:49:25.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.021+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304057460 con 0x7f03040b47f0 2022-07-12T08:49:25.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.021+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f02fc026030 con 0x7f03040b47f0 2022-07-12T08:49:25.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.021+0000 7f0301ffb700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02e8053a60 0x7f02e8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.021+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f02fc06cab0 con 0x7f03040b47f0 2022-07-12T08:49:25.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.021+0000 7f0303fff700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02e8053a60 0x7f02e8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.021+0000 7f0303fff700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02e8053a60 0x7f02e8055f20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f02f400b000 tx=0x7f02f4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.029+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02fc043a10 con 0x7f03040b47f0 2022-07-12T08:49:25.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.157+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03040b9b50 con 0x7f03040b47f0 2022-07-12T08:49:25.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.157+0000 7f0301ffb700 1 -- 172.21.15.61:0/303639418 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16112 (secure 0 0 0) 0x7f02fc024090 con 0x7f03040b47f0 2022-07-12T08:49:25.165 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:25.166 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":109,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:24.635088+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:49:22.980785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-12T08:49:23.643810+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:25.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02e8053a60 msgr2=0x7f02e8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02e8053a60 0x7f02e8055f20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f02f400b000 tx=0x7f02f4009040).stop 2022-07-12T08:49:25.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 msgr2=0x7f0304151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 0x7f0304151e60 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f02fc0032f0 tx=0x7f02fc016000).stop 2022-07-12T08:49:25.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 shutdown_connections 2022-07-12T08:49:25.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0304142d50 0x7f030413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f02e8053a60 0x7f02e8055f20 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f030413ce60 0x7f03040b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 --2- 172.21.15.61:0/303639418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03040b47f0 0x7f0304151e60 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 >> 172.21.15.61:0/303639418 conn(0x7f030405f9f0 msgr2=0x7f0304144c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:25.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 shutdown_connections 2022-07-12T08:49:25.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.161+0000 7f030acf1700 1 -- 172.21.15.61:0/303639418 wait complete. 2022-07-12T08:49:25.182 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:49:25.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 Processor -- start 2022-07-12T08:49:25.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 -- start start 2022-07-12T08:49:25.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3f280b3ea0 0x7f3f280b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f2805c810 0x7f3f280b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f280b3120 con 0x7f3f280b9db0 2022-07-12T08:49:25.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2805bd10 con 0x7f3f280b3ea0 2022-07-12T08:49:25.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f2e115700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f280b33c0 con 0x7f3f2805c810 2022-07-12T08:49:25.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f1ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3f280b3ea0 0x7f3f280b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f277fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3f280b9db0 0x7f3f2805c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48282/0 (socket says 172.21.15.61:48282) 2022-07-12T08:49:25.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f277fe700 1 -- 172.21.15.61:0/1308015525 learned_addr learned my addr 172.21.15.61:0/1308015525 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:25.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f27fff700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f2805c810 0x7f3f280b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f26ffd700 1 -- 172.21.15.61:0/1308015525 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507145505 0 0) 0x7f3f280b3120 con 0x7f3f280b9db0 2022-07-12T08:49:25.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f26ffd700 1 -- 172.21.15.61:0/1308015525 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f08003260 con 0x7f3f280b9db0 2022-07-12T08:49:25.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f1ffff700 1 -- 172.21.15.61:0/1308015525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f2805c810 msgr2=0x7f3f280b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:25.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f1ffff700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f2805c810 0x7f3f280b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f1ffff700 1 -- 172.21.15.61:0/1308015525 >> v1:172.21.15.61:6789/0 conn(0x7f3f280b9db0 legacy=0x7f3f2805c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f1ffff700 1 -- 172.21.15.61:0/1308015525 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2805cc60 con 0x7f3f280b3ea0 2022-07-12T08:49:25.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f27fff700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f2805c810 0x7f3f280b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:25.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f1ffff700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f280b3ea0 0x7f3f280b2a40 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f3f280b1230 tx=0x7f3f10006df0).ready entity=mon.0 client_cookie=b3038f979184cd1b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f26ffd700 1 -- 172.21.15.61:0/1308015525 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f1000d570 con 0x7f3f280b3ea0 2022-07-12T08:49:25.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f26ffd700 1 -- 172.21.15.61:0/1308015525 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f10000e90 con 0x7f3f280b3ea0 2022-07-12T08:49:25.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.289+0000 7f3f26ffd700 1 -- 172.21.15.61:0/1308015525 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f1000d8c0 con 0x7f3f280b3ea0 2022-07-12T08:49:25.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- 172.21.15.61:0/1308015525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f280b3ea0 msgr2=0x7f3f280b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f280b3ea0 0x7f3f280b2a40 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f3f280b1230 tx=0x7f3f10006df0).stop 2022-07-12T08:49:25.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- 172.21.15.61:0/1308015525 shutdown_connections 2022-07-12T08:49:25.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f2805c810 0x7f3f280b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 --2- 172.21.15.61:0/1308015525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f280b3ea0 0x7f3f280b2a40 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- 172.21.15.61:0/1308015525 >> 172.21.15.61:0/1308015525 conn(0x7f3f28067960 msgr2=0x7f3f280b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:25.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- 172.21.15.61:0/1308015525 shutdown_connections 2022-07-12T08:49:25.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- 172.21.15.61:0/1308015525 wait complete. 2022-07-12T08:49:25.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 Processor -- start 2022-07-12T08:49:25.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- start start 2022-07-12T08:49:25.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 0x7f3f28151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f280b3ea0 0x7f3f280b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f280b9db0 0x7f3f28141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f28114060 con 0x7f3f2805c810 2022-07-12T08:49:25.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f28114530 con 0x7f3f280b9db0 2022-07-12T08:49:25.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f2e115700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f28114290 con 0x7f3f280b3ea0 2022-07-12T08:49:25.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 0x7f3f28151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 0x7f3f28151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60070/0 (socket says 172.21.15.61:60070) 2022-07-12T08:49:25.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f277fe700 1 -- 172.21.15.61:0/4159492374 learned_addr learned my addr 172.21.15.61:0/4159492374 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:25.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f1ffff700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f280b3ea0 0x7f3f280b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.293+0000 7f3f27fff700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f280b9db0 0x7f3f28141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f277fe700 1 -- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f280b3ea0 msgr2=0x7f3f280b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f277fe700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f280b3ea0 0x7f3f280b7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f277fe700 1 -- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f280b9db0 msgr2=0x7f3f28141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f277fe700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f280b9db0 0x7f3f28141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f277fe700 1 -- 172.21.15.61:0/4159492374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20009470 con 0x7f3f2805c810 2022-07-12T08:49:25.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f277fe700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 0x7f3f28151f10 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f3f18001110 tx=0x7f3f18007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f180070d0 con 0x7f3f2805c810 2022-07-12T08:49:25.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f18007230 con 0x7f3f2805c810 2022-07-12T08:49:25.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f1000b180 con 0x7f3f2805c810 2022-07-12T08:49:25.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f18005610 con 0x7f3f2805c810 2022-07-12T08:49:25.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f280af5e0 con 0x7f3f2805c810 2022-07-12T08:49:25.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f280af8f0 con 0x7f3f2805c810 2022-07-12T08:49:25.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3f18010070 con 0x7f3f2805c810 2022-07-12T08:49:25.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f257fa700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3f0c0539b0 0x7f3f0c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3f1806b520 con 0x7f3f2805c810 2022-07-12T08:49:25.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.297+0000 7f3f1ffff700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3f0c0539b0 0x7f3f0c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.301+0000 7f3f1ffff700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3f0c0539b0 0x7f3f0c055e70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f3f1001a000 tx=0x7f3f10014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.305+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f18042480 con 0x7f3f2805c810 2022-07-12T08:49:25.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.465+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3f28057460 con 0x7f3f2805c810 2022-07-12T08:49:25.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.649+0000 7f3f257fa700 1 -- 172.21.15.61:0/4159492374 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3f18045aa0 con 0x7f3f2805c810 2022-07-12T08:49:25.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.649+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3f0c0539b0 msgr2=0x7f3f0c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.649+0000 7f3f2e115700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3f0c0539b0 0x7f3f0c055e70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f3f1001a000 tx=0x7f3f10014000).stop 2022-07-12T08:49:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.649+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 msgr2=0x7f3f28151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.649+0000 7f3f2e115700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 0x7f3f28151f10 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 rx=0x7f3f18001110 tx=0x7f3f18007b20).stop 2022-07-12T08:49:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 shutdown_connections 2022-07-12T08:49:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3f0c0539b0 0x7f3f0c055e70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f280b9db0 0x7f3f28141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3f280b3ea0 0x7f3f280b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 --2- 172.21.15.61:0/4159492374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f2805c810 0x7f3f28151f10 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 >> 172.21.15.61:0/4159492374 conn(0x7f3f28067960 msgr2=0x7f3f280b8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 shutdown_connections 2022-07-12T08:49:25.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.653+0000 7f3f2e115700 1 -- 172.21.15.61:0/4159492374 wait complete. 2022-07-12T08:49:25.672 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:25.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 Processor -- start 2022-07-12T08:49:25.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- start start 2022-07-12T08:49:25.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab670 0x7f3a5005f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a500abc70 con 0x7f3a500ab290 2022-07-12T08:49:25.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5005c720 con 0x7f3a500afa60 2022-07-12T08:49:25.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a500aba90 con 0x7f3a500ab670 2022-07-12T08:49:25.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4e59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3a500ab290 0x7f3a5005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48296/0 (socket says 172.21.15.61:48296) 2022-07-12T08:49:25.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4e59c700 1 -- 172.21.15.61:0/3901169843 learned_addr learned my addr 172.21.15.61:0/3901169843 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:25.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4dd9b700 1 --2- 172.21.15.61:0/3901169843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab670 0x7f3a5005f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4d59a700 1 -- 172.21.15.61:0/3901169843 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710417262 0 0) 0x7f3a500abc70 con 0x7f3a500ab290 2022-07-12T08:49:25.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4d59a700 1 -- 172.21.15.61:0/3901169843 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30003260 con 0x7f3a500ab290 2022-07-12T08:49:25.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4dd9b700 1 -- 172.21.15.61:0/3901169843 >> v1:172.21.15.111:6789/0 conn(0x7f3a500afa60 legacy=0x7f3a50147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4dd9b700 1 -- 172.21.15.61:0/3901169843 >> v1:172.21.15.61:6789/0 conn(0x7f3a500ab290 legacy=0x7f3a5005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4d59a700 1 -- 172.21.15.61:0/3901169843 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912047090 0 0) 0x7f3a5005c720 con 0x7f3a500afa60 2022-07-12T08:49:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4dd9b700 1 -- 172.21.15.61:0/3901169843 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5005c9c0 con 0x7f3a500ab670 2022-07-12T08:49:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4dd9b700 1 --2- 172.21.15.61:0/3901169843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab670 0x7f3a5005f140 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f3a480032f0 tx=0x7f3a480091d0).ready entity=mon.2 client_cookie=f637b71f2ed3dc4d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4d59a700 1 -- 172.21.15.61:0/3901169843 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a4800b6c0 con 0x7f3a500ab670 2022-07-12T08:49:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4d59a700 1 -- 172.21.15.61:0/3901169843 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4800b820 con 0x7f3a500ab670 2022-07-12T08:49:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4d59a700 1 -- 172.21.15.61:0/3901169843 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a4800b990 con 0x7f3a500ab670 2022-07-12T08:49:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- 172.21.15.61:0/3901169843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab670 msgr2=0x7f3a5005f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/3901169843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab670 0x7f3a5005f140 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f3a480032f0 tx=0x7f3a480091d0).stop 2022-07-12T08:49:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- 172.21.15.61:0/3901169843 shutdown_connections 2022-07-12T08:49:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/3901169843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab670 0x7f3a5005f140 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- 172.21.15.61:0/3901169843 >> 172.21.15.61:0/3901169843 conn(0x7f3a500b1950 msgr2=0x7f3a500b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:25.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- 172.21.15.61:0/3901169843 shutdown_connections 2022-07-12T08:49:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.777+0000 7f3a4f59e700 1 -- 172.21.15.61:0/3901169843 wait complete. 2022-07-12T08:49:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 Processor -- start 2022-07-12T08:49:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 -- start start 2022-07-12T08:49:25.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab290 0x7f3a500660b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a500ab670 0x7f3a50148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a500afa60 0x7f3a50159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5014b2b0 con 0x7f3a500afa60 2022-07-12T08:49:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5014b010 con 0x7f3a500ab670 2022-07-12T08:49:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5014ad30 con 0x7f3a500ab290 2022-07-12T08:49:25.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4e59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab290 0x7f3a500660b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a500ab670 0x7f3a50148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a500afa60 0x7f3a50159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a500ab670 0x7f3a50148950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:38660/0 (socket says 172.21.15.61:38660) 2022-07-12T08:49:25.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4dd9b700 1 -- 172.21.15.61:0/4012197257 learned_addr learned my addr 172.21.15.61:0/4012197257 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:25.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 -- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab290 msgr2=0x7f3a500660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab290 0x7f3a500660b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 -- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a500ab670 msgr2=0x7f3a50148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a500ab670 0x7f3a50148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 -- 172.21.15.61:0/4012197257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a48009470 con 0x7f3a500afa60 2022-07-12T08:49:25.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4ed9d700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a500afa60 0x7f3a50159250 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f3a40000bc0 tx=0x7f3a40007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a4000cd60 con 0x7f3a500afa60 2022-07-12T08:49:25.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40012400 con 0x7f3a500afa60 2022-07-12T08:49:25.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a50066f50 con 0x7f3a500afa60 2022-07-12T08:49:25.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a400127a0 con 0x7f3a500afa60 2022-07-12T08:49:25.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.781+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a50148fe0 con 0x7f3a500afa60 2022-07-12T08:49:25.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.785+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a50065da0 con 0x7f3a500afa60 2022-07-12T08:49:25.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.785+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3a40019030 con 0x7f3a500afa60 2022-07-12T08:49:25.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.785+0000 7f3a477fe700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a340539b0 0x7f3a34055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:25.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.785+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3a4006af90 con 0x7f3a500afa60 2022-07-12T08:49:25.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.785+0000 7f3a4e59c700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a340539b0 0x7f3a34055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:25.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.785+0000 7f3a4e59c700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a340539b0 0x7f3a34055e70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f3a3800d000 tx=0x7f3a38009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:25.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.789+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a40041fa0 con 0x7f3a500afa60 2022-07-12T08:49:25.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a50057460 con 0x7f3a500afa60 2022-07-12T08:49:25.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a477fe700 1 -- 172.21.15.61:0/4012197257 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f3a400455c0 con 0x7f3a500afa60 2022-07-12T08:49:25.963 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:25.963 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2585981801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2585981801},{"type":"v1","addr":"172.21.15.111:6835","nonce":2585981801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:49:24.635457+0000","modified":"2022-07-12T08:49:25.639064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2867307616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2867307616},{"type":"v1","addr":"172.21.15.61:6833","nonce":2867307616}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-12T08:49:25.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a340539b0 msgr2=0x7f3a34055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a340539b0 0x7f3a34055e70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f3a3800d000 tx=0x7f3a38009040).stop 2022-07-12T08:49:25.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a500afa60 msgr2=0x7f3a50159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:25.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a500afa60 0x7f3a50159250 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f3a40000bc0 tx=0x7f3a40007ab0).stop 2022-07-12T08:49:25.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 shutdown_connections 2022-07-12T08:49:25.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a500afa60 0x7f3a50159250 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a340539b0 0x7f3a34055e70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a500ab670 0x7f3a50148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 --2- 172.21.15.61:0/4012197257 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a500ab290 0x7f3a500660b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:25.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 >> 172.21.15.61:0/4012197257 conn(0x7f3a500b1950 msgr2=0x7f3a5005bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:25.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 shutdown_connections 2022-07-12T08:49:25.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:25.957+0000 7f3a4f59e700 1 -- 172.21.15.61:0/4012197257 wait complete. 2022-07-12T08:49:25.972 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-12T08:49:25.979 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:26.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f7212700 1 Processor -- start 2022-07-12T08:49:26.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f7212700 1 -- start start 2022-07-12T08:49:26.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f7212700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 0x7f04f0151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f7212700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f00abc70 con 0x7f04f00ab670 2022-07-12T08:49:26.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f7212700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f005c720 con 0x7f04f00ab290 2022-07-12T08:49:26.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f7212700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00aba90 con 0x7f04f005f140 2022-07-12T08:49:26.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f5a0f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f04f00ab670 0x7f04f00afa60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48310/0 (socket says 172.21.15.61:48310) 2022-07-12T08:49:26.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f5a0f700 1 -- 172.21.15.61:0/1495077552 learned_addr learned my addr 172.21.15.61:0/1495077552 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:26.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6a11700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 0x7f04f0151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856271121 0 0) 0x7f04f00abc70 con 0x7f04f00ab670 2022-07-12T08:49:26.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d8003260 con 0x7f04f00ab670 2022-07-12T08:49:26.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573393826 0 0) 0x7f04d8003260 con 0x7f04f00ab670 2022-07-12T08:49:26.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 msgr2=0x7f04f0151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 0x7f04f0151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 >> v1:172.21.15.111:6789/0 conn(0x7f04f00ab290 legacy=0x7f04f005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f005c9c0 con 0x7f04f00ab670 2022-07-12T08:49:26.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f04e80019e0 con 0x7f04f00ab670 2022-07-12T08:49:26.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 >> v1:172.21.15.61:6789/0 conn(0x7f04f00ab670 legacy=0x7f04f00afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04d8003780 0x7f04d8003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d8004500 0x7f04d8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 0x7f04d8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f00aba90 con 0x7f04d8003780 2022-07-12T08:49:26.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d800b780 con 0x7f04d8004500 2022-07-12T08:49:26.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04d8003780 0x7f04d8003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d800b940 con 0x7f04f005f140 2022-07-12T08:49:26.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6a11700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 0x7f04d8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 -- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 msgr2=0x7f04d8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04f005f140 0x7f04d8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 -- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d8004500 msgr2=0x7f04d8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04d8004500 0x7f04d8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-12T08:49:26.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 -- 172.21.15.61:0/1495077552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d8003260 con 0x7f04d8003780 2022-07-12T08:49:26.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f6210700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04d8003780 0x7f04d8003ba0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f04f00af510 tx=0x7f04e0007a00).ready entity=mon.0 client_cookie=7badab639ee97e04 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04e0007d10 con 0x7f04d8003780 2022-07-12T08:49:26.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e0007030 con 0x7f04d8003780 2022-07-12T08:49:26.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.077+0000 7f04f520e700 1 -- 172.21.15.61:0/1495077552 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04e0007390 con 0x7f04d8003780 2022-07-12T08:49:26.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/1495077552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04d8003780 msgr2=0x7f04d8003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 --2- 172.21.15.61:0/1495077552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04d8003780 0x7f04d8003ba0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f04f00af510 tx=0x7f04e0007a00).stop 2022-07-12T08:49:26.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 -- 172.21.15.61:0/1495077552 reap_dead start 2022-07-12T08:49:26.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/1495077552 shutdown_connections 2022-07-12T08:49:26.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/1495077552 >> 172.21.15.61:0/1495077552 conn(0x7f04f00b1950 msgr2=0x7f04f00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:26.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/1495077552 shutdown_connections 2022-07-12T08:49:26.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/1495077552 wait complete. 2022-07-12T08:49:26.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 Processor -- start 2022-07-12T08:49:26.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- start start 2022-07-12T08:49:26.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04d8003780 0x7f04f015a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f00ac220 0x7f04f0144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 0x7f04f0156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f005f5b0 con 0x7f04f01454b0 2022-07-12T08:49:26.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0159770 con 0x7f04f00ac220 2022-07-12T08:49:26.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f0159a10 con 0x7f04d8003780 2022-07-12T08:49:26.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 0x7f04f0156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 0x7f04f0156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60108/0 (socket says 172.21.15.61:60108) 2022-07-12T08:49:26.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 -- 172.21.15.61:0/2306103233 learned_addr learned my addr 172.21.15.61:0/2306103233 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6210700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04d8003780 0x7f04f015a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f5a0f700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f00ac220 0x7f04f0144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 -- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04d8003780 msgr2=0x7f04f015a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04d8003780 0x7f04f015a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 -- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f00ac220 msgr2=0x7f04f0144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f00ac220 0x7f04f0144fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 -- 172.21.15.61:0/2306103233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0159cb0 con 0x7f04f01454b0 2022-07-12T08:49:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f6a11700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 0x7f04f0156110 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f04ec00b000 tx=0x7f04ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04ec014430 con 0x7f04f01454b0 2022-07-12T08:49:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec014590 con 0x7f04f01454b0 2022-07-12T08:49:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0156600 con 0x7f04f01454b0 2022-07-12T08:49:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04ec014700 con 0x7f04f01454b0 2022-07-12T08:49:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.081+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f0156ab0 con 0x7f04f01454b0 2022-07-12T08:49:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.085+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f0159e40 con 0x7f04f01454b0 2022-07-12T08:49:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.085+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f04ec014860 con 0x7f04f01454b0 2022-07-12T08:49:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.085+0000 7f04df7fe700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04d0053ad0 0x7f04d0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.085+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f04ec06bdc0 con 0x7f04f01454b0 2022-07-12T08:49:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.085+0000 7f04f6210700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04d0053ad0 0x7f04d0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.085+0000 7f04f6210700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04d0053ad0 0x7f04d0055f90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f04e001b000 tx=0x7f04e0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.093+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f04ec042d20 con 0x7f04f01454b0 2022-07-12T08:49:26.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.217+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f006b2b0 con 0x7f04f01454b0 2022-07-12T08:49:26.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.217+0000 7f04df7fe700 1 -- 172.21.15.61:0/2306103233 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f04ec046340 con 0x7f04f01454b0 2022-07-12T08:49:26.225 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:26.226 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":110,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:25.639162+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:49:22.980785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-12T08:49:23.643810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:26.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04d0053ad0 msgr2=0x7f04d0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04d0053ad0 0x7f04d0055f90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f04e001b000 tx=0x7f04e0015000).stop 2022-07-12T08:49:26.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 msgr2=0x7f04f0156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 0x7f04f0156110 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f04ec00b000 tx=0x7f04ec008040).stop 2022-07-12T08:49:26.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 shutdown_connections 2022-07-12T08:49:26.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04f01454b0 0x7f04f0156110 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04d0053ad0 0x7f04d0055f90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04f00ac220 0x7f04f0144fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 --2- 172.21.15.61:0/2306103233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f04d8003780 0x7f04f015a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 >> 172.21.15.61:0/2306103233 conn(0x7f04f00b1950 msgr2=0x7f04f0099c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:26.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 shutdown_connections 2022-07-12T08:49:26.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.221+0000 7f04f7212700 1 -- 172.21.15.61:0/2306103233 wait complete. 2022-07-12T08:49:26.239 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:49:26.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.353+0000 7f2fd0b61700 1 Processor -- start 2022-07-12T08:49:26.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.353+0000 7f2fd0b61700 1 -- start start 2022-07-12T08:49:26.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2fcc1427b0 0x7f2fcc0618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2fcc0b02f0 0x7f2fcc138830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc142b90 con 0x7f2fcc138d20 2022-07-12T08:49:26.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc060ef0 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc061160 con 0x7f2fcc1427b0 2022-07-12T08:49:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2fcc138d20 0x7f2fcc140370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48334/0 (socket says 172.21.15.61:48334) 2022-07-12T08:49:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcbfff700 1 -- 172.21.15.61:0/1326076596 learned_addr learned my addr 172.21.15.61:0/1326076596 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fc3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2fcc0b02f0 0x7f2fcc138830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcb7fe700 1 --2- 172.21.15.61:0/1326076596 >> v2:172.21.15.111:3300/0 conn(0x7f2fcc1427b0 0x7f2fcc0618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcaffd700 1 -- 172.21.15.61:0/1326076596 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817685359 0 0) 0x7f2fcc142b90 con 0x7f2fcc138d20 2022-07-12T08:49:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcaffd700 1 -- 172.21.15.61:0/1326076596 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac003260 con 0x7f2fcc138d20 2022-07-12T08:49:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fc3fff700 1 -- 172.21.15.61:0/1326076596 >> v2:172.21.15.111:3300/0 conn(0x7f2fcc1427b0 msgr2=0x7f2fcc0618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fc3fff700 1 --2- 172.21.15.61:0/1326076596 >> v2:172.21.15.111:3300/0 conn(0x7f2fcc1427b0 0x7f2fcc0618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fc3fff700 1 -- 172.21.15.61:0/1326076596 >> v1:172.21.15.61:6789/0 conn(0x7f2fcc138d20 legacy=0x7f2fcc140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fc3fff700 1 -- 172.21.15.61:0/1326076596 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc061400 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fc3fff700 1 --2- 172.21.15.61:0/1326076596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc138830 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f2fb4002f70 tx=0x7f2fb4006df0).ready entity=mon.0 client_cookie=4d66106999e810fc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcaffd700 1 -- 172.21.15.61:0/1326076596 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fb4014070 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcaffd700 1 -- 172.21.15.61:0/1326076596 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4000e90 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fcaffd700 1 -- 172.21.15.61:0/1326076596 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fb400e990 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 -- 172.21.15.61:0/1326076596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 msgr2=0x7f2fcc138830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/1326076596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc138830 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 rx=0x7f2fb4002f70 tx=0x7f2fb4006df0).stop 2022-07-12T08:49:26.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 -- 172.21.15.61:0/1326076596 shutdown_connections 2022-07-12T08:49:26.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/1326076596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc138830 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.357+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/1326076596 >> v2:172.21.15.111:3300/0 conn(0x7f2fcc1427b0 0x7f2fcc0618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- 172.21.15.61:0/1326076596 >> 172.21.15.61:0/1326076596 conn(0x7f2fcc0b5860 msgr2=0x7f2fcc0b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:26.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- 172.21.15.61:0/1326076596 shutdown_connections 2022-07-12T08:49:26.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- 172.21.15.61:0/1326076596 wait complete. 2022-07-12T08:49:26.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 Processor -- start 2022-07-12T08:49:26.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- start start 2022-07-12T08:49:26.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fcc138d20 0x7f2fcc154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fcc1427b0 0x7f2fcc143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc1435e0 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc13d240 con 0x7f2fcc138d20 2022-07-12T08:49:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fd0b61700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc13d4e0 con 0x7f2fcc1427b0 2022-07-12T08:49:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60118/0 (socket says 172.21.15.61:60118) 2022-07-12T08:49:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 -- 172.21.15.61:0/3838441475 learned_addr learned my addr 172.21.15.61:0/3838441475 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:26.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcbfff700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fcc1427b0 0x7f2fcc143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fc3fff700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fcc138d20 0x7f2fcc154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 -- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fcc1427b0 msgr2=0x7f2fcc143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fcc1427b0 0x7f2fcc143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 -- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fcc138d20 msgr2=0x7f2fcc154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fcc138d20 0x7f2fcc154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 -- 172.21.15.61:0/3838441475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb400c290 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fc3fff700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fcc138d20 0x7f2fcc154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.361+0000 7f2fcb7fe700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc0b4c20 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f2fbc00c000 tx=0x7f2fbc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fbc010780 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc0108e0 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc13bec0 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fbc010c40 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc13e3d0 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc142e00 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2fbc01c460 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc97fa700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fb0053920 0x7f2fb0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2fbc06c370 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.365+0000 7f2fc3fff700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fb0053920 0x7f2fb0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.369+0000 7f2fc3fff700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fb0053920 0x7f2fb0055de0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f2fb400b000 tx=0x7f2fb4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.373+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2fbc043270 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.537+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2fcc143720 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fc97fa700 1 -- 172.21.15.61:0/3838441475 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2fbc046890 con 0x7f2fcc0b02f0 2022-07-12T08:49:26.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fb0053920 msgr2=0x7f2fb0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fb0053920 0x7f2fb0055de0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f2fb400b000 tx=0x7f2fb4016000).stop 2022-07-12T08:49:26.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 msgr2=0x7f2fcc0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc0b4c20 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7f2fbc00c000 tx=0x7f2fbc006df0).stop 2022-07-12T08:49:26.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 shutdown_connections 2022-07-12T08:49:26.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2fcc1427b0 0x7f2fcc143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fb0053920 0x7f2fb0055de0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fcc138d20 0x7f2fcc154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 --2- 172.21.15.61:0/3838441475 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2fcc0b02f0 0x7f2fcc0b4c20 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 >> 172.21.15.61:0/3838441475 conn(0x7f2fcc0b5860 msgr2=0x7f2fcc141aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:26.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 shutdown_connections 2022-07-12T08:49:26.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.545+0000 7f2fd0b61700 1 -- 172.21.15.61:0/3838441475 wait complete. 2022-07-12T08:49:26.558 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:49:26.565 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:49:26.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.677+0000 7f4a1b744700 1 Processor -- start 2022-07-12T08:49:26.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.677+0000 7f4a1b744700 1 -- start start 2022-07-12T08:49:26.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a140b2810 0x7f4a14142a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a140ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a140accb0 con 0x7f4a1405f480 2022-07-12T08:49:26.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a140acf20 con 0x7f4a140b2810 2022-07-12T08:49:26.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140b36b0 con 0x7f4a1405f860 2022-07-12T08:49:26.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a194e0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4a1405f480 0x7f4a14142320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48344/0 (socket says 172.21.15.61:48344) 2022-07-12T08:49:26.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a194e0700 1 -- 172.21.15.61:0/943526966 learned_addr learned my addr 172.21.15.61:0/943526966 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:26.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/943526966 >> v2:172.21.15.111:3300/0 conn(0x7f4a140b2810 0x7f4a14142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a19ce1700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a140ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874403505 0 0) 0x7f4a140accb0 con 0x7f4a1405f480 2022-07-12T08:49:26.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a00003260 con 0x7f4a1405f480 2022-07-12T08:49:26.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2216466698 0 0) 0x7f4a00003260 con 0x7f4a1405f480 2022-07-12T08:49:26.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 msgr2=0x7f4a140ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a140ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 >> v2:172.21.15.111:3300/0 conn(0x7f4a140b2810 msgr2=0x7f4a14142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 --2- 172.21.15.61:0/943526966 >> v2:172.21.15.111:3300/0 conn(0x7f4a140b2810 0x7f4a14142a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1405fcf0 con 0x7f4a1405f480 2022-07-12T08:49:26.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f4a0c001660 con 0x7f4a1405f480 2022-07-12T08:49:26.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 >> v1:172.21.15.61:6789/0 conn(0x7f4a1405f480 legacy=0x7f4a14142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a00003750 0x7f4a00003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a00004500 0x7f4a00004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a00004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140acf20 con 0x7f4a00004500 2022-07-12T08:49:26.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140b36b0 con 0x7f4a00003750 2022-07-12T08:49:26.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0000b750 con 0x7f4a1405f860 2022-07-12T08:49:26.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a00004500 0x7f4a00004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a194e0700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a00003750 0x7f4a00003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a19ce1700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a00004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 -- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 msgr2=0x7f4a00004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a00004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 -- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a00003750 msgr2=0x7f4a00003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:26.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a00003750 0x7f4a00003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 -- 172.21.15.61:0/943526966 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a00003260 con 0x7f4a00004500 2022-07-12T08:49:26.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a19ce1700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1405f860 0x7f4a00004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:26.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a00004500 0x7f4a00004980 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f4a0800c000 tx=0x7f4a08006df0).ready entity=mon.0 client_cookie=2b89e9cba8816748 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0800e830 con 0x7f4a00004500 2022-07-12T08:49:26.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0800e990 con 0x7f4a00004500 2022-07-12T08:49:26.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a13fff700 1 -- 172.21.15.61:0/943526966 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0800ecf0 con 0x7f4a00004500 2022-07-12T08:49:26.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- 172.21.15.61:0/943526966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a00004500 msgr2=0x7f4a00004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 --2- 172.21.15.61:0/943526966 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a00004500 0x7f4a00004980 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f4a0800c000 tx=0x7f4a08006df0).stop 2022-07-12T08:49:26.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a19ce1700 1 -- 172.21.15.61:0/943526966 reap_dead start 2022-07-12T08:49:26.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- 172.21.15.61:0/943526966 shutdown_connections 2022-07-12T08:49:26.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- 172.21.15.61:0/943526966 >> 172.21.15.61:0/943526966 conn(0x7f4a140b47e0 msgr2=0x7f4a1405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:26.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- 172.21.15.61:0/943526966 shutdown_connections 2022-07-12T08:49:26.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.681+0000 7f4a1b744700 1 -- 172.21.15.61:0/943526966 wait complete. 2022-07-12T08:49:26.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.685+0000 7f4a1b744700 1 Processor -- start 2022-07-12T08:49:26.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.685+0000 7f4a1b744700 1 -- start start 2022-07-12T08:49:26.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a14144510 0x7f4a141448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 0x7f4a140af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a140afae0 0x7f4a1413f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14067400 con 0x7f4a140677c0 2022-07-12T08:49:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140b1540 con 0x7f4a140afae0 2022-07-12T08:49:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a14066420 con 0x7f4a14144510 2022-07-12T08:49:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 0x7f4a140af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 0x7f4a140af5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60140/0 (socket says 172.21.15.61:60140) 2022-07-12T08:49:26.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 -- 172.21.15.61:0/3506672256 learned_addr learned my addr 172.21.15.61:0/3506672256 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:26.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a194e0700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a14144510 0x7f4a141448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a19ce1700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a140afae0 0x7f4a1413f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 -- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a14144510 msgr2=0x7f4a141448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a14144510 0x7f4a141448f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 -- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a140afae0 msgr2=0x7f4a1413f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a140afae0 0x7f4a1413f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 -- 172.21.15.61:0/3506672256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14067160 con 0x7f4a140677c0 2022-07-12T08:49:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a18cdf700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 0x7f4a140af5f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f4a0800c000 tx=0x7f4a08019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0801cd60 con 0x7f4a140677c0 2022-07-12T08:49:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a08000e90 con 0x7f4a140677c0 2022-07-12T08:49:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a14153090 con 0x7f4a140677c0 2022-07-12T08:49:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a080245e0 con 0x7f4a140677c0 2022-07-12T08:49:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a140aff60 con 0x7f4a140677c0 2022-07-12T08:49:26.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a14066fa0 con 0x7f4a140677c0 2022-07-12T08:49:26.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.689+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4a0802a020 con 0x7f4a140677c0 2022-07-12T08:49:26.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.693+0000 7f4a127fc700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49f4053a40 0x7f49f4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:26.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.693+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4a0806d410 con 0x7f4a140677c0 2022-07-12T08:49:26.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.693+0000 7f4a194e0700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49f4053a40 0x7f49f4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:26.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.693+0000 7f4a194e0700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49f4053a40 0x7f49f4055f00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f4a0c005000 tx=0x7f4a0c001e10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:26.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.697+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a08044370 con 0x7f4a140677c0 2022-07-12T08:49:26.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.865+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4a14153220 con 0x7f4a140677c0 2022-07-12T08:49:26.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a127fc700 1 -- 172.21.15.61:0/3506672256 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4a0800be60 con 0x7f4a140677c0 2022-07-12T08:49:26.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49f4053a40 msgr2=0x7f49f4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49f4053a40 0x7f49f4055f00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f4a0c005000 tx=0x7f4a0c001e10).stop 2022-07-12T08:49:26.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 msgr2=0x7f4a140af5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:26.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 0x7f4a140af5f0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f4a0800c000 tx=0x7f4a08019000).stop 2022-07-12T08:49:26.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 shutdown_connections 2022-07-12T08:49:26.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f49f4053a40 0x7f49f4055f00 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a140afae0 0x7f4a1413f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a140677c0 0x7f4a140af5f0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 --2- 172.21.15.61:0/3506672256 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a14144510 0x7f4a141448f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:26.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 >> 172.21.15.61:0/3506672256 conn(0x7f4a140b47e0 msgr2=0x7f4a1405c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:26.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 shutdown_connections 2022-07-12T08:49:26.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:26.873+0000 7f4a1b744700 1 -- 172.21.15.61:0/3506672256 wait complete. 2022-07-12T08:49:26.891 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:49:26.893 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:49:27.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.005+0000 7f2b22274700 1 Processor -- start 2022-07-12T08:49:27.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.005+0000 7f2b22274700 1 -- start start 2022-07-12T08:49:27.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c05f480 0x7f2b1c146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b1c0b0ef0 0x7f2b1c060050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2b1c064ae0 0x7f2b1c060540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c0655a0 con 0x7f2b1c0b0ef0 2022-07-12T08:49:27.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1c142a60 con 0x7f2b1c064ae0 2022-07-12T08:49:27.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0b29f0 con 0x7f2b1c05f480 2022-07-12T08:49:27.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b1c0b0ef0 0x7f2b1c060050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b21a73700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2b1c064ae0 0x7f2b1c060540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b1c0b0ef0 0x7f2b1c060050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60150/0 (socket says 172.21.15.61:60150) 2022-07-12T08:49:27.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 -- 172.21.15.61:0/1483993535 learned_addr learned my addr 172.21.15.61:0/1483993535 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:27.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b21272700 1 --2- 172.21.15.61:0/1483993535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c05f480 0x7f2b1c146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 -- 172.21.15.61:0/1483993535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c05f480 msgr2=0x7f2b1c146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:27.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 --2- 172.21.15.61:0/1483993535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c05f480 0x7f2b1c146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 -- 172.21.15.61:0/1483993535 >> v2:172.21.15.111:3300/0 conn(0x7f2b1c064ae0 msgr2=0x7f2b1c060540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:27.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 --2- 172.21.15.61:0/1483993535 >> v2:172.21.15.111:3300/0 conn(0x7f2b1c064ae0 0x7f2b1c060540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 -- 172.21.15.61:0/1483993535 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c05f860 con 0x7f2b1c0b0ef0 2022-07-12T08:49:27.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b21a73700 1 --2- 172.21.15.61:0/1483993535 >> v2:172.21.15.111:3300/0 conn(0x7f2b1c064ae0 0x7f2b1c060540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:27.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b20a71700 1 --2- 172.21.15.61:0/1483993535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c0b0ef0 0x7f2b1c060050 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f2b10002f70 tx=0x7f2b10006eb0).ready entity=mon.0 client_cookie=532ad175f3914a04 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b1bfff700 1 -- 172.21.15.61:0/1483993535 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b1000d570 con 0x7f2b1c0b0ef0 2022-07-12T08:49:27.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b1bfff700 1 -- 172.21.15.61:0/1483993535 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10000e90 con 0x7f2b1c0b0ef0 2022-07-12T08:49:27.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b1bfff700 1 -- 172.21.15.61:0/1483993535 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b1000d900 con 0x7f2b1c0b0ef0 2022-07-12T08:49:27.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- 172.21.15.61:0/1483993535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c0b0ef0 msgr2=0x7f2b1c060050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- 172.21.15.61:0/1483993535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c0b0ef0 0x7f2b1c060050 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f2b10002f70 tx=0x7f2b10006eb0).stop 2022-07-12T08:49:27.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- 172.21.15.61:0/1483993535 shutdown_connections 2022-07-12T08:49:27.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- 172.21.15.61:0/1483993535 >> v2:172.21.15.111:3300/0 conn(0x7f2b1c064ae0 0x7f2b1c060540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- 172.21.15.61:0/1483993535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c0b0ef0 0x7f2b1c060050 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 --2- 172.21.15.61:0/1483993535 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c05f480 0x7f2b1c146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- 172.21.15.61:0/1483993535 >> 172.21.15.61:0/1483993535 conn(0x7f2b1c063ad0 msgr2=0x7f2b1c063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:27.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- 172.21.15.61:0/1483993535 shutdown_connections 2022-07-12T08:49:27.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.009+0000 7f2b22274700 1 -- 172.21.15.61:0/1483993535 wait complete. 2022-07-12T08:49:27.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 Processor -- start 2022-07-12T08:49:27.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 -- start start 2022-07-12T08:49:27.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1c05f480 0x7f2b1c154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 0x7f2b1c0b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c0b0ef0 0x7f2b1c115ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c114090 con 0x7f2b1c064ae0 2022-07-12T08:49:27.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c114310 con 0x7f2b1c05f480 2022-07-12T08:49:27.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b22274700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c0aef30 con 0x7f2b1c0b0ef0 2022-07-12T08:49:27.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 0x7f2b1c0b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 0x7f2b1c0b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60154/0 (socket says 172.21.15.61:60154) 2022-07-12T08:49:27.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 -- 172.21.15.61:0/1369688001 learned_addr learned my addr 172.21.15.61:0/1369688001 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:27.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b21272700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1c05f480 0x7f2b1c154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b21a73700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c0b0ef0 0x7f2b1c115ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 -- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c0b0ef0 msgr2=0x7f2b1c115ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c0b0ef0 0x7f2b1c115ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 -- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1c05f480 msgr2=0x7f2b1c154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1c05f480 0x7f2b1c154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.013+0000 7f2b20a71700 1 -- 172.21.15.61:0/1369688001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b14009470 con 0x7f2b1c064ae0 2022-07-12T08:49:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b20a71700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 0x7f2b1c0b51d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f2b10002f70 tx=0x7f2b10015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b10011950 con 0x7f2b1c064ae0 2022-07-12T08:49:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10011ab0 con 0x7f2b1c064ae0 2022-07-12T08:49:27.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1000b1e0 con 0x7f2b1c064ae0 2022-07-12T08:49:27.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b10011e10 con 0x7f2b1c064ae0 2022-07-12T08:49:27.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c0af380 con 0x7f2b1c064ae0 2022-07-12T08:49:27.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b10025030 con 0x7f2b1c064ae0 2022-07-12T08:49:27.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c057460 con 0x7f2b1c064ae0 2022-07-12T08:49:27.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b1a7fc700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2afc053af0 0x7f2afc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2b1006d0a0 con 0x7f2b1c064ae0 2022-07-12T08:49:27.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b21272700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2afc053af0 0x7f2afc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.017+0000 7f2b21272700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2afc053af0 0x7f2afc055fb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f2b0c00b000 tx=0x7f2b0c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.021+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b10043ef0 con 0x7f2b1c064ae0 2022-07-12T08:49:27.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.185+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2b1c1144d0 con 0x7f2b1c064ae0 2022-07-12T08:49:27.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.193+0000 7f2b1a7fc700 1 -- 172.21.15.61:0/1369688001 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7f2b10023070 con 0x7f2b1c064ae0 2022-07-12T08:49:27.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.193+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2afc053af0 msgr2=0x7f2afc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2afc053af0 0x7f2afc055fb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f2b0c00b000 tx=0x7f2b0c006df0).stop 2022-07-12T08:49:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 msgr2=0x7f2b1c0b51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 0x7f2b1c0b51d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 rx=0x7f2b10002f70 tx=0x7f2b10015000).stop 2022-07-12T08:49:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 shutdown_connections 2022-07-12T08:49:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2b1c0b0ef0 0x7f2b1c115ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b1c064ae0 0x7f2b1c0b51d0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2afc053af0 0x7f2afc055fb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 --2- 172.21.15.61:0/1369688001 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2b1c05f480 0x7f2b1c154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 >> 172.21.15.61:0/1369688001 conn(0x7f2b1c063ad0 msgr2=0x7f2b1c140d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:27.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 shutdown_connections 2022-07-12T08:49:27.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.197+0000 7f2b22274700 1 -- 172.21.15.61:0/1369688001 wait complete. 2022-07-12T08:49:27.206 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:49:27.215 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:27.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd888f700 1 Processor -- start 2022-07-12T08:49:27.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd888f700 1 -- start start 2022-07-12T08:49:27.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd888f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd414aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd888f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd40b1cf0 con 0x7fabd40b35c0 2022-07-12T08:49:27.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd888f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd405d440 con 0x7fabd405f140 2022-07-12T08:49:27.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd888f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd4148150 con 0x7fabd405d060 2022-07-12T08:49:27.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd359e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fabd40b35c0 0x7fabd41517a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48382/0 (socket says 172.21.15.61:48382) 2022-07-12T08:49:27.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd359e700 1 -- 172.21.15.61:0/1036910103 learned_addr learned my addr 172.21.15.61:0/1036910103 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:27.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd2d9d700 1 --2- 172.21.15.61:0/1036910103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd414aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.309+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127578295 0 0) 0x7fabd40b1cf0 con 0x7fabd40b35c0 2022-07-12T08:49:27.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8003260 con 0x7fabd40b35c0 2022-07-12T08:49:27.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070468732 0 0) 0x7fabd405d440 con 0x7fabd405f140 2022-07-12T08:49:27.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd40b1cf0 con 0x7fabd405f140 2022-07-12T08:49:27.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467730635 0 0) 0x7fabb8003260 con 0x7fabd40b35c0 2022-07-12T08:49:27.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd2d9d700 1 -- 172.21.15.61:0/1036910103 >> v1:172.21.15.111:6789/0 conn(0x7fabd405f140 legacy=0x7fabd40b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd2d9d700 1 -- 172.21.15.61:0/1036910103 >> v1:172.21.15.61:6789/0 conn(0x7fabd40b35c0 legacy=0x7fabd41517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd2d9d700 1 -- 172.21.15.61:0/1036910103 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd40b1650 con 0x7fabd405d060 2022-07-12T08:49:27.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd2d9d700 1 --2- 172.21.15.61:0/1036910103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd414aa10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fabcc001490 tx=0x7fabc8006df0).ready entity=mon.2 client_cookie=b6a4c95a1eae0a8d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabcc0019e0 con 0x7fabd405d060 2022-07-12T08:49:27.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8000e90 con 0x7fabd405d060 2022-07-12T08:49:27.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd1d9b700 1 -- 172.21.15.61:0/1036910103 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc8012460 con 0x7fabd405d060 2022-07-12T08:49:27.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 -- 172.21.15.61:0/1036910103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 msgr2=0x7fabd414aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 --2- 172.21.15.61:0/1036910103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd414aa10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fabcc001490 tx=0x7fabc8006df0).stop 2022-07-12T08:49:27.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 -- 172.21.15.61:0/1036910103 shutdown_connections 2022-07-12T08:49:27.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 --2- 172.21.15.61:0/1036910103 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd414aa10 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 -- 172.21.15.61:0/1036910103 >> 172.21.15.61:0/1036910103 conn(0x7fabd40b3b50 msgr2=0x7fabd40b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:27.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 -- 172.21.15.61:0/1036910103 shutdown_connections 2022-07-12T08:49:27.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 -- 172.21.15.61:0/1036910103 wait complete. 2022-07-12T08:49:27.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 Processor -- start 2022-07-12T08:49:27.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.313+0000 7fabd888f700 1 -- start start 2022-07-12T08:49:27.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd4144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 0x7fabd414a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd40b35c0 0x7fabd4146160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd4061990 con 0x7fabd405f140 2022-07-12T08:49:27.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd414b750 con 0x7fabd40b35c0 2022-07-12T08:49:27.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd414bfb0 con 0x7fabd405d060 2022-07-12T08:49:27.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 0x7fabd414a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 0x7fabd414a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60166/0 (socket says 172.21.15.61:60166) 2022-07-12T08:49:27.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 -- 172.21.15.61:0/2524072481 learned_addr learned my addr 172.21.15.61:0/2524072481 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:27.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd359e700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd40b35c0 0x7fabd4146160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd2d9d700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd4144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 -- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 msgr2=0x7fabd4144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd4144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 -- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd40b35c0 msgr2=0x7fabd4146160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd40b35c0 0x7fabd4146160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 -- 172.21.15.61:0/2524072481 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc800b180 con 0x7fabd405f140 2022-07-12T08:49:27.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd259c700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 0x7fabd414a450 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7fabc4001110 tx=0x7fabc4007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc400ce50 con 0x7fabd405f140 2022-07-12T08:49:27.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc400f470 con 0x7fabd405f140 2022-07-12T08:49:27.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc400f7d0 con 0x7fabd405f140 2022-07-12T08:49:27.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd40658b0 con 0x7fabd405f140 2022-07-12T08:49:27.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.317+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd414a940 con 0x7fabd405f140 2022-07-12T08:49:27.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.321+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fabc4019030 con 0x7fabd405f140 2022-07-12T08:49:27.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.321+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd40656e0 con 0x7fabd405f140 2022-07-12T08:49:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.321+0000 7fabc3fff700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabb0053a40 0x7fabb0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.321+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fabc406b3d0 con 0x7fabd405f140 2022-07-12T08:49:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.321+0000 7fabd2d9d700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabb0053a40 0x7fabb0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.321+0000 7fabd2d9d700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabb0053a40 0x7fabb0055f00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fabc8018000 tx=0x7fabc8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.329+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabc4042290 con 0x7fabd405f140 2022-07-12T08:49:27.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd4065a40 con 0x7fabd405f140 2022-07-12T08:49:27.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabc3fff700 1 -- 172.21.15.61:0/2524072481 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7fabc40458b0 con 0x7fabd405f140 2022-07-12T08:49:27.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:27.501 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2585981801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2585981801},{"type":"v1","addr":"172.21.15.111:6835","nonce":2585981801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:49:24.635457+0000","modified":"2022-07-12T08:49:25.639064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2867307616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2867307616},{"type":"v1","addr":"172.21.15.61:6833","nonce":2867307616}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-12T08:49:27.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabb0053a40 msgr2=0x7fabb0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabb0053a40 0x7fabb0055f00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7fabc8018000 tx=0x7fabc8014000).stop 2022-07-12T08:49:27.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 msgr2=0x7fabd414a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 0x7fabd414a450 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7fabc4001110 tx=0x7fabc4007b60).stop 2022-07-12T08:49:27.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 shutdown_connections 2022-07-12T08:49:27.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabb0053a40 0x7fabb0055f00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd40b35c0 0x7fabd4146160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd405f140 0x7fabd414a450 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.493+0000 7fabd888f700 1 --2- 172.21.15.61:0/2524072481 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd405d060 0x7fabd4144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.497+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 >> 172.21.15.61:0/2524072481 conn(0x7fabd40b3b50 msgr2=0x7fabd41470a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.497+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 shutdown_connections 2022-07-12T08:49:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.497+0000 7fabd888f700 1 -- 172.21.15.61:0/2524072481 wait complete. 2022-07-12T08:49:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-12T08:49:27.715 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:49:24.635457+0000', 'modified': '2022-07-12T08:49:25.639064+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2867307616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2867307616}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2867307616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:49:27.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:49:27.716 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:49:27.837 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":7671,"want_state":"up:active","state":"up:active","rank_uptime":3.1874914269999999,"mdsmap_epoch":62,"osdmap_epoch":110,"osdmap_epoch_barrier":109,"uptime":7.780260577} 2022-07-12T08:49:27.838 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 7671, "mdsmap_epoch": 62, "osdmap_epoch": 110, "osdmap_epoch_barrier": 109, "rank_uptime": 3.187491427, "state": "up:active", "uptime": 7.780260577, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:49:27.839 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:27.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 Processor -- start 2022-07-12T08:49:27.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- start start 2022-07-12T08:49:27.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff4540ab290 0x7ff4540ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4540afa60 0x7ff4540afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4540abc70 con 0x7ff45405f140 2022-07-12T08:49:27.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45405f580 con 0x7ff4540afa60 2022-07-12T08:49:27.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45405c720 con 0x7ff4540ab290 2022-07-12T08:49:27.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff44affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff45405f140 0x7ff45405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48392/0 (socket says 172.21.15.61:48392) 2022-07-12T08:49:27.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff453fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4540afa60 0x7ff4540afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff44affd700 1 -- 172.21.15.61:0/917268736 learned_addr learned my addr 172.21.15.61:0/917268736 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:27.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff4537fe700 1 --2- 172.21.15.61:0/917268736 >> v2:172.21.15.111:3300/0 conn(0x7ff4540ab290 0x7ff4540ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff453fff700 1 -- 172.21.15.61:0/917268736 >> v2:172.21.15.111:3300/0 conn(0x7ff4540ab290 msgr2=0x7ff4540ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:27.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff453fff700 1 --2- 172.21.15.61:0/917268736 >> v2:172.21.15.111:3300/0 conn(0x7ff4540ab290 0x7ff4540ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff453fff700 1 -- 172.21.15.61:0/917268736 >> v1:172.21.15.61:6789/0 conn(0x7ff45405f140 legacy=0x7ff45405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff453fff700 1 -- 172.21.15.61:0/917268736 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45405c9c0 con 0x7ff4540afa60 2022-07-12T08:49:27.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff452ffd700 1 -- 172.21.15.61:0/917268736 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086742130 0 0) 0x7ff4540abc70 con 0x7ff45405f140 2022-07-12T08:49:27.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff453fff700 1 --2- 172.21.15.61:0/917268736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff4540afed0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7ff44c0032f0 tx=0x7ff44c0091d0).ready entity=mon.0 client_cookie=6626d37b6572c394 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff452ffd700 1 -- 172.21.15.61:0/917268736 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c00b680 con 0x7ff4540afa60 2022-07-12T08:49:27.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff452ffd700 1 -- 172.21.15.61:0/917268736 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c00b7e0 con 0x7ff4540afa60 2022-07-12T08:49:27.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff452ffd700 1 -- 172.21.15.61:0/917268736 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c00b950 con 0x7ff4540afa60 2022-07-12T08:49:27.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- 172.21.15.61:0/917268736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 msgr2=0x7ff4540afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 --2- 172.21.15.61:0/917268736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff4540afed0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 rx=0x7ff44c0032f0 tx=0x7ff44c0091d0).stop 2022-07-12T08:49:27.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- 172.21.15.61:0/917268736 shutdown_connections 2022-07-12T08:49:27.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 --2- 172.21.15.61:0/917268736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff4540afed0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 --2- 172.21.15.61:0/917268736 >> v2:172.21.15.111:3300/0 conn(0x7ff4540ab290 0x7ff4540ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- 172.21.15.61:0/917268736 >> 172.21.15.61:0/917268736 conn(0x7ff4540b1950 msgr2=0x7ff4540b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:27.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- 172.21.15.61:0/917268736 shutdown_connections 2022-07-12T08:49:27.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.945+0000 7ff45892b700 1 -- 172.21.15.61:0/917268736 wait complete. 2022-07-12T08:49:27.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 Processor -- start 2022-07-12T08:49:27.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 -- start start 2022-07-12T08:49:27.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45405f140 0x7ff4541595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4540ab290 0x7ff454159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff454149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454065cc0 con 0x7ff4540afa60 2022-07-12T08:49:27.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4540666f0 con 0x7ff45405f140 2022-07-12T08:49:27.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff45892b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff454066990 con 0x7ff4540ab290 2022-07-12T08:49:27.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff454149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff454149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60184/0 (socket says 172.21.15.61:60184) 2022-07-12T08:49:27.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 -- 172.21.15.61:0/3206305473 learned_addr learned my addr 172.21.15.61:0/3206305473 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:27.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff4537fe700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45405f140 0x7ff4541595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff44affd700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4540ab290 0x7ff454159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 -- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4540ab290 msgr2=0x7ff454159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4540ab290 0x7ff454159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 -- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45405f140 msgr2=0x7ff4541595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:27.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45405f140 0x7ff4541595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:27.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 -- 172.21.15.61:0/3206305473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c009470 con 0x7ff4540afa60 2022-07-12T08:49:27.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.949+0000 7ff453fff700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff454149020 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7ff44c0032f0 tx=0x7ff44c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c00b680 con 0x7ff4540afa60 2022-07-12T08:49:27.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c021630 con 0x7ff4540afa60 2022-07-12T08:49:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44c021990 con 0x7ff4540afa60 2022-07-12T08:49:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45415a0f0 con 0x7ff4540afa60 2022-07-12T08:49:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff454063df0 con 0x7ff4540afa60 2022-07-12T08:49:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff454067550 con 0x7ff4540afa60 2022-07-12T08:49:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff44c00f070 con 0x7ff4540afa60 2022-07-12T08:49:27.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4517fa700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff438053a60 0x7ff438055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:27.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff44c06ca50 con 0x7ff4540afa60 2022-07-12T08:49:27.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.953+0000 7ff4537fe700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff438053a60 0x7ff438055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:27.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.957+0000 7ff4537fe700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff438053a60 0x7ff438055f20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7ff44400b000 tx=0x7ff444006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:27.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:27.961+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff454067550 con 0x7ff4540afa60 2022-07-12T08:49:28.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.089+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff454067550 con 0x7ff4540afa60 2022-07-12T08:49:28.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff4517fa700 1 -- 172.21.15.61:0/3206305473 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16099 (secure 0 0 0) 0x7ff454067550 con 0x7ff4540afa60 2022-07-12T08:49:28.099 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:28.099 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":110,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:25.639162+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:49:22.980785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-12T08:49:23.643810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:28.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff438053a60 msgr2=0x7ff438055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff438053a60 0x7ff438055f20 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7ff44400b000 tx=0x7ff444006df0).stop 2022-07-12T08:49:28.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 msgr2=0x7ff454149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff454149020 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7ff44c0032f0 tx=0x7ff44c016000).stop 2022-07-12T08:49:28.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 shutdown_connections 2022-07-12T08:49:28.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4540afa60 0x7ff454149020 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4540ab290 0x7ff454159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff438053a60 0x7ff438055f20 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 --2- 172.21.15.61:0/3206305473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45405f140 0x7ff4541595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 >> 172.21.15.61:0/3206305473 conn(0x7ff4540b1950 msgr2=0x7ff4540b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 shutdown_connections 2022-07-12T08:49:28.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.093+0000 7ff45892b700 1 -- 172.21.15.61:0/3206305473 wait complete. 2022-07-12T08:49:28.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:49:22.980785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:49:23.643810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:49:28.117 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:49:28.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f480b6a8700 1 Processor -- start 2022-07-12T08:49:28.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f480b6a8700 1 -- start start 2022-07-12T08:49:28.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f480b6a8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48040af9e0 0x7f4804140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f480b6a8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48041440a0 con 0x7f480413ada0 2022-07-12T08:49:28.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f480b6a8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48040b23a0 con 0x7f480405c5a0 2022-07-12T08:49:28.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f480b6a8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4804141b10 con 0x7f48040af9e0 2022-07-12T08:49:28.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f4809c45700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f480413ada0 0x7f48041444d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48410/0 (socket says 172.21.15.61:48410) 2022-07-12T08:49:28.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f4809c45700 1 -- 172.21.15.61:0/1187409430 learned_addr learned my addr 172.21.15.61:0/1187409430 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:28.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f4809444700 1 --2- 172.21.15.61:0/1187409430 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48040af9e0 0x7f4804140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.225+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088397017 0 0) 0x7f48041440a0 con 0x7f480413ada0 2022-07-12T08:49:28.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47ec003260 con 0x7f480413ada0 2022-07-12T08:49:28.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464129594 0 0) 0x7f48040b23a0 con 0x7f480405c5a0 2022-07-12T08:49:28.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48041440a0 con 0x7f480405c5a0 2022-07-12T08:49:28.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4809444700 1 -- 172.21.15.61:0/1187409430 >> v1:172.21.15.111:6789/0 conn(0x7f480405c5a0 legacy=0x7f480413a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4809444700 1 -- 172.21.15.61:0/1187409430 >> v1:172.21.15.61:6789/0 conn(0x7f480413ada0 legacy=0x7f48041444d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3117906277 0 0) 0x7f47ec003260 con 0x7f480413ada0 2022-07-12T08:49:28.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4809444700 1 -- 172.21.15.61:0/1187409430 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804141ca0 con 0x7f48040af9e0 2022-07-12T08:49:28.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4809444700 1 --2- 172.21.15.61:0/1187409430 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48040af9e0 0x7f4804140ad0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f47fc001490 tx=0x7f4800006df0).ready entity=mon.2 client_cookie=1bc03807e23ca86e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47fc0019e0 con 0x7f48040af9e0 2022-07-12T08:49:28.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4800000e90 con 0x7f48040af9e0 2022-07-12T08:49:28.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f47f3fff700 1 -- 172.21.15.61:0/1187409430 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4800012460 con 0x7f48040af9e0 2022-07-12T08:49:28.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- 172.21.15.61:0/1187409430 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48040af9e0 msgr2=0x7f4804140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 --2- 172.21.15.61:0/1187409430 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48040af9e0 0x7f4804140ad0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f47fc001490 tx=0x7f4800006df0).stop 2022-07-12T08:49:28.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- 172.21.15.61:0/1187409430 shutdown_connections 2022-07-12T08:49:28.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 --2- 172.21.15.61:0/1187409430 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f48040af9e0 0x7f4804140ad0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- 172.21.15.61:0/1187409430 >> 172.21.15.61:0/1187409430 conn(0x7f48040621d0 msgr2=0x7f4804063bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- 172.21.15.61:0/1187409430 shutdown_connections 2022-07-12T08:49:28.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- 172.21.15.61:0/1187409430 wait complete. 2022-07-12T08:49:28.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 Processor -- start 2022-07-12T08:49:28.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- start start 2022-07-12T08:49:28.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f480405c5a0 0x7f48040677e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 0x7f4804139ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4804140750 0x7f4804064bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48040670a0 con 0x7f480413ada0 2022-07-12T08:49:28.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4804067340 con 0x7f480405c5a0 2022-07-12T08:49:28.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f480b6a8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4804066b20 con 0x7f4804140750 2022-07-12T08:49:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4808c43700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 0x7f4804139ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4808c43700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 0x7f4804139ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60196/0 (socket says 172.21.15.61:60196) 2022-07-12T08:49:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4808c43700 1 -- 172.21.15.61:0/2956048892 learned_addr learned my addr 172.21.15.61:0/2956048892 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4809444700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f480405c5a0 0x7f48040677e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.229+0000 7f4809c45700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4804140750 0x7f4804064bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4808c43700 1 -- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4804140750 msgr2=0x7f4804064bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4808c43700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4804140750 0x7f4804064bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4808c43700 1 -- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f480405c5a0 msgr2=0x7f48040677e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4808c43700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f480405c5a0 0x7f48040677e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4808c43700 1 -- 172.21.15.61:0/2956048892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480000b180 con 0x7f480413ada0 2022-07-12T08:49:28.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4808c43700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 0x7f4804139ed0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f47f4001110 tx=0x7f47f4007b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47f4012400 con 0x7f480413ada0 2022-07-12T08:49:28.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f4012560 con 0x7f480413ada0 2022-07-12T08:49:28.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804066160 con 0x7f480413ada0 2022-07-12T08:49:28.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47f40128c0 con 0x7f480413ada0 2022-07-12T08:49:28.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4804065250 con 0x7f480413ada0 2022-07-12T08:49:28.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4804067500 con 0x7f480413ada0 2022-07-12T08:49:28.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f47f4019030 con 0x7f480413ada0 2022-07-12T08:49:28.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f47f27fc700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f47e40539b0 0x7f47e4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f47f406b2c0 con 0x7f480413ada0 2022-07-12T08:49:28.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.233+0000 7f4809444700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f47e40539b0 0x7f47e4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.237+0000 7f4809444700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f47e40539b0 0x7f47e4055e70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f480001a000 tx=0x7f4800014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.241+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47f40421c0 con 0x7f480413ada0 2022-07-12T08:49:28.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4804057460 con 0x7f480413ada0 2022-07-12T08:49:28.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f47f27fc700 1 -- 172.21.15.61:0/2956048892 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v110) v1 ==== 110+0+10 (secure 0 0 0) 0x7f47f40457e0 con 0x7f480413ada0 2022-07-12T08:49:28.377 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:49:28.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f47e40539b0 msgr2=0x7f47e4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f47e40539b0 0x7f47e4055e70 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f480001a000 tx=0x7f4800014000).stop 2022-07-12T08:49:28.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 msgr2=0x7f4804139ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 0x7f4804139ed0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7f47f4001110 tx=0x7f47f4007b60).stop 2022-07-12T08:49:28.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 shutdown_connections 2022-07-12T08:49:28.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4804140750 0x7f4804064bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f480413ada0 0x7f4804139ed0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f47e40539b0 0x7f47e4055e70 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 --2- 172.21.15.61:0/2956048892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f480405c5a0 0x7f48040677e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 >> 172.21.15.61:0/2956048892 conn(0x7f48040621d0 msgr2=0x7f4804143520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 shutdown_connections 2022-07-12T08:49:28.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.369+0000 7f480b6a8700 1 -- 172.21.15.61:0/2956048892 wait complete. 2022-07-12T08:49:28.397 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:49:28.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.501+0000 7fb82bfff700 1 Processor -- start 2022-07-12T08:49:28.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.501+0000 7fb82bfff700 1 -- start start 2022-07-12T08:49:28.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb82c0b5920 0x7fb82c14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c05c3e0 con 0x7fb82c0b0280 2022-07-12T08:49:28.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82c0b5d00 con 0x7fb82c0aa830 2022-07-12T08:49:28.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82c05c610 con 0x7fb82c0b5920 2022-07-12T08:49:28.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82a7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb82c0b0280 0x7fb82c14aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48420/0 (socket says 172.21.15.61:48420) 2022-07-12T08:49:28.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82a7fc700 1 -- 172.21.15.61:0/280359693 learned_addr learned my addr 172.21.15.61:0/280359693 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:28.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82affd700 1 --2- 172.21.15.61:0/280359693 >> v2:172.21.15.61:3300/0 conn(0x7fb82c0b5920 0x7fb82c14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944648162 0 0) 0x7fb82c05c3e0 con 0x7fb82c0b0280 2022-07-12T08:49:28.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810003260 con 0x7fb82c0b0280 2022-07-12T08:49:28.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883529578 0 0) 0x7fb82c0b5d00 con 0x7fb82c0aa830 2022-07-12T08:49:28.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c05c3e0 con 0x7fb82c0aa830 2022-07-12T08:49:28.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283466370 0 0) 0x7fb810003260 con 0x7fb82c0b0280 2022-07-12T08:49:28.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82affd700 1 -- 172.21.15.61:0/280359693 >> v1:172.21.15.111:6789/0 conn(0x7fb82c0aa830 legacy=0x7fb82c0aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82affd700 1 -- 172.21.15.61:0/280359693 >> v1:172.21.15.61:6789/0 conn(0x7fb82c0b0280 legacy=0x7fb82c14aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82affd700 1 -- 172.21.15.61:0/280359693 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c05c8b0 con 0x7fb82c0b5920 2022-07-12T08:49:28.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82affd700 1 --2- 172.21.15.61:0/280359693 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c14a9e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fb81c001110 tx=0x7fb820006df0).ready entity=mon.0 client_cookie=bb5b8a8d805f42c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb81c001660 con 0x7fb82c0b5920 2022-07-12T08:49:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb820000e90 con 0x7fb82c0b5920 2022-07-12T08:49:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb829ffb700 1 -- 172.21.15.61:0/280359693 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb820012460 con 0x7fb82c0b5920 2022-07-12T08:49:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- 172.21.15.61:0/280359693 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 msgr2=0x7fb82c14a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 --2- 172.21.15.61:0/280359693 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c14a9e0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7fb81c001110 tx=0x7fb820006df0).stop 2022-07-12T08:49:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- 172.21.15.61:0/280359693 shutdown_connections 2022-07-12T08:49:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 --2- 172.21.15.61:0/280359693 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c14a9e0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- 172.21.15.61:0/280359693 >> 172.21.15.61:0/280359693 conn(0x7fb82c063340 msgr2=0x7fb82c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- 172.21.15.61:0/280359693 shutdown_connections 2022-07-12T08:49:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.505+0000 7fb82bfff700 1 -- 172.21.15.61:0/280359693 wait complete. 2022-07-12T08:49:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 Processor -- start 2022-07-12T08:49:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 -- start start 2022-07-12T08:49:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb82c0aa830 0x7fb82c098fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 0x7fb82c148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c064450 con 0x7fb82c0b5920 2022-07-12T08:49:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0adff0 con 0x7fb82c0b0280 2022-07-12T08:49:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c0adab0 con 0x7fb82c0aa830 2022-07-12T08:49:28.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 0x7fb82c148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c148a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60212/0 (socket says 172.21.15.61:60212) 2022-07-12T08:49:28.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 0x7fb82c148560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:38782/0 (socket says 172.21.15.61:38782) 2022-07-12T08:49:28.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82b7fe700 1 -- 172.21.15.61:0/1248313113 learned_addr learned my addr 172.21.15.61:0/1248313113 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:28.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82affd700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb82c0aa830 0x7fb82c098fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 -- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb82c0aa830 msgr2=0x7fb82c098fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb82c0aa830 0x7fb82c098fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 -- 172.21.15.61:0/1248313113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 msgr2=0x7fb82c148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 -- 172.21.15.61:0/1248313113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82000b180 con 0x7fb82c0b0280 2022-07-12T08:49:28.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82b7fe700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:28.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82a7fc700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 0x7fb82c148560 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fb81c001840 tx=0x7fb81c001660).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb81c007170 con 0x7fb82c0b0280 2022-07-12T08:49:28.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb82c0add50 con 0x7fb82c0b0280 2022-07-12T08:49:28.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c0072d0 con 0x7fb82c0b0280 2022-07-12T08:49:28.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb81c004710 con 0x7fb82c0b0280 2022-07-12T08:49:28.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.509+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82c0996c0 con 0x7fb82c0b0280 2022-07-12T08:49:28.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.513+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82c065840 con 0x7fb82c0b0280 2022-07-12T08:49:28.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.513+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb81c01b070 con 0x7fb82c0b0280 2022-07-12T08:49:28.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.513+0000 7fb81bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb808053ad0 0x7fb808055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.513+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb81c06b450 con 0x7fb82c0b0280 2022-07-12T08:49:28.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.513+0000 7fb82affd700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb808053ad0 0x7fb808055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.513+0000 7fb82affd700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb808053ad0 0x7fb808055f90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fb82001a000 tx=0x7fb820014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.521+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb81c042220 con 0x7fb82c0b0280 2022-07-12T08:49:28.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.653+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb82c06b2b0 con 0x7fb82c0b0280 2022-07-12T08:49:28.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.653+0000 7fb81bfff700 1 -- 172.21.15.61:0/1248313113 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v110) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb81c045840 con 0x7fb82c0b0280 2022-07-12T08:49:28.667 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:28.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb808053ad0 msgr2=0x7fb808055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb808053ad0 0x7fb808055f90 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fb82001a000 tx=0x7fb820014000).stop 2022-07-12T08:49:28.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 msgr2=0x7fb82c148560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 0x7fb82c148560 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7fb81c001840 tx=0x7fb81c001660).stop 2022-07-12T08:49:28.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 shutdown_connections 2022-07-12T08:49:28.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb82c0b5920 0x7fb82c148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb808053ad0 0x7fb808055f90 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb82c0b0280 0x7fb82c148560 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 --2- 172.21.15.61:0/1248313113 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb82c0aa830 0x7fb82c098fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 >> 172.21.15.61:0/1248313113 conn(0x7fb82c063340 msgr2=0x7fb82c0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 shutdown_connections 2022-07-12T08:49:28.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.657+0000 7fb82bfff700 1 -- 172.21.15.61:0/1248313113 wait complete. 2022-07-12T08:49:28.675 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:49:28.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 Processor -- start 2022-07-12T08:49:28.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 -- start start 2022-07-12T08:49:28.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d44098c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4d44143580 0x7f4d44143a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d44097c80 con 0x7f4d440b5920 2022-07-12T08:49:28.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440b5d00 con 0x7f4d44143580 2022-07-12T08:49:28.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d4a7d0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4405c7f0 con 0x7f4d440987f0 2022-07-12T08:49:28.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d497ce700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4d440b5920 0x7f4d440b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48434/0 (socket says 172.21.15.61:48434) 2022-07-12T08:49:28.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d497ce700 1 -- 172.21.15.61:0/1272675526 learned_addr learned my addr 172.21.15.61:0/1272675526 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:28.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d49fcf700 1 --2- 172.21.15.61:0/1272675526 >> v2:172.21.15.61:3300/0 conn(0x7f4d44143580 0x7f4d44143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d48fcd700 1 --2- 172.21.15.61:0/1272675526 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d44098c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d43fff700 1 -- 172.21.15.61:0/1272675526 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500538513 0 0) 0x7f4d44097c80 con 0x7f4d440b5920 2022-07-12T08:49:28.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d43fff700 1 -- 172.21.15.61:0/1272675526 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003260 con 0x7f4d440b5920 2022-07-12T08:49:28.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d49fcf700 1 -- 172.21.15.61:0/1272675526 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 msgr2=0x7f4d44098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:28.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d49fcf700 1 --2- 172.21.15.61:0/1272675526 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d44098c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d49fcf700 1 -- 172.21.15.61:0/1272675526 >> v1:172.21.15.61:6789/0 conn(0x7f4d440b5920 legacy=0x7f4d440b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d49fcf700 1 -- 172.21.15.61:0/1272675526 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4405cbf0 con 0x7f4d44143580 2022-07-12T08:49:28.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d49fcf700 1 --2- 172.21.15.61:0/1272675526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d44143a10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f4d340032f0 tx=0x7f4d340091d0).ready entity=mon.0 client_cookie=a4833e7d36ecbd12 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d43fff700 1 -- 172.21.15.61:0/1272675526 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3400b6c0 con 0x7f4d44143580 2022-07-12T08:49:28.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d43fff700 1 -- 172.21.15.61:0/1272675526 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3400b820 con 0x7f4d44143580 2022-07-12T08:49:28.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.785+0000 7f4d43fff700 1 -- 172.21.15.61:0/1272675526 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3400b990 con 0x7f4d44143580 2022-07-12T08:49:28.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/1272675526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 msgr2=0x7f4d44143a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/1272675526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d44143a10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 rx=0x7f4d340032f0 tx=0x7f4d340091d0).stop 2022-07-12T08:49:28.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/1272675526 shutdown_connections 2022-07-12T08:49:28.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/1272675526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d44143a10 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/1272675526 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d44098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/1272675526 >> 172.21.15.61:0/1272675526 conn(0x7f4d44063340 msgr2=0x7f4d440b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/1272675526 shutdown_connections 2022-07-12T08:49:28.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/1272675526 wait complete. 2022-07-12T08:49:28.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 Processor -- start 2022-07-12T08:49:28.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- start start 2022-07-12T08:49:28.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d440aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b5920 0x7f4d440b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d4413a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44152750 con 0x7f4d44143580 2022-07-12T08:49:28.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4413b1e0 con 0x7f4d440b5920 2022-07-12T08:49:28.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d4a7d0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d44152470 con 0x7f4d440987f0 2022-07-12T08:49:28.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d49fcf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d4413a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d49fcf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d4413a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60224/0 (socket says 172.21.15.61:60224) 2022-07-12T08:49:28.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d49fcf700 1 -- 172.21.15.61:0/2105758089 learned_addr learned my addr 172.21.15.61:0/2105758089 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:28.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d497ce700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d440aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.789+0000 7f4d48fcd700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b5920 0x7f4d440b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d49fcf700 1 -- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 msgr2=0x7f4d440aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d49fcf700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d440aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d49fcf700 1 -- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b5920 msgr2=0x7f4d440b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d49fcf700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b5920 0x7f4d440b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d49fcf700 1 -- 172.21.15.61:0/2105758089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d34009470 con 0x7f4d44143580 2022-07-12T08:49:28.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d49fcf700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d4413a050 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f4d340032f0 tx=0x7f4d34016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d34011d70 con 0x7f4d44143580 2022-07-12T08:49:28.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d34021440 con 0x7f4d44143580 2022-07-12T08:49:28.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4413b680 con 0x7f4d44143580 2022-07-12T08:49:28.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d340217a0 con 0x7f4d44143580 2022-07-12T08:49:28.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440abed0 con 0x7f4d44143580 2022-07-12T08:49:28.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44152950 con 0x7f4d44143580 2022-07-12T08:49:28.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4d3400f070 con 0x7f4d44143580 2022-07-12T08:49:28.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d427fc700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053a40 0x7f4d24055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:28.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d3406c540 con 0x7f4d44143580 2022-07-12T08:49:28.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.793+0000 7f4d497ce700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053a40 0x7f4d24055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:28.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.797+0000 7f4d497ce700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053a40 0x7f4d24055f00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f4d3c00d000 tx=0x7f4d3c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:28.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.801+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4d340434a0 con 0x7f4d44143580 2022-07-12T08:49:28.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.925+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4d4413b3a0 con 0x7f4d44143580 2022-07-12T08:49:28.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.925+0000 7f4d427fc700 1 -- 172.21.15.61:0/2105758089 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4d340150a0 con 0x7f4d44143580 2022-07-12T08:49:28.934 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:28.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053a40 msgr2=0x7f4d24055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053a40 0x7f4d24055f00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f4d3c00d000 tx=0x7f4d3c009040).stop 2022-07-12T08:49:28.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 msgr2=0x7f4d4413a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:28.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d4413a050 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7f4d340032f0 tx=0x7f4d34016000).stop 2022-07-12T08:49:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 shutdown_connections 2022-07-12T08:49:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d44143580 0x7f4d4413a050 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4d24053a40 0x7f4d24055f00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4d440b5920 0x7f4d440b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 --2- 172.21.15.61:0/2105758089 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4d440987f0 0x7f4d440aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:28.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 >> 172.21.15.61:0/2105758089 conn(0x7f4d44063340 msgr2=0x7f4d44141e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:28.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 shutdown_connections 2022-07-12T08:49:28.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:28.929+0000 7f4d4a7d0700 1 -- 172.21.15.61:0/2105758089 wait complete. 2022-07-12T08:49:28.951 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:49:29.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.057+0000 7f3749f92700 1 Processor -- start 2022-07-12T08:49:29.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.057+0000 7f3749f92700 1 -- start start 2022-07-12T08:49:29.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b5920 0x7f37440aa500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 0x7f3744060200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37440abfa0 con 0x7f37440b0240 2022-07-12T08:49:29.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374405c870 con 0x7f37440b0680 2022-07-12T08:49:29.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37440b5d00 con 0x7f37440b5920 2022-07-12T08:49:29.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749791700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 0x7f3744060200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3748f90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b5920 0x7f37440aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749791700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 0x7f3744060200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60234/0 (socket says 172.21.15.61:60234) 2022-07-12T08:49:29.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749791700 1 -- 172.21.15.61:0/2495577914 learned_addr learned my addr 172.21.15.61:0/2495577914 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:29.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f373bfff700 1 --1- 172.21.15.61:0/2495577914 >> v1:172.21.15.111:6789/0 conn(0x7f37440b0240 0x7f374405faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:33820/0 (socket says 172.21.15.61:33820) 2022-07-12T08:49:29.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3748f90700 1 -- 172.21.15.61:0/2495577914 >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 msgr2=0x7f3744060200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:29.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3748f90700 1 --2- 172.21.15.61:0/2495577914 >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 0x7f3744060200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3748f90700 1 -- 172.21.15.61:0/2495577914 >> v1:172.21.15.111:6789/0 conn(0x7f37440b0240 legacy=0x7f374405faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3748f90700 1 -- 172.21.15.61:0/2495577914 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374405cd00 con 0x7f37440b5920 2022-07-12T08:49:29.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749791700 1 --2- 172.21.15.61:0/2495577914 >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 0x7f3744060200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:29.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f373b7fe700 1 -- 172.21.15.61:0/2495577914 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627717582 0 0) 0x7f37440abfa0 con 0x7f37440b0240 2022-07-12T08:49:29.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3748f90700 1 --2- 172.21.15.61:0/2495577914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b5920 0x7f37440aa500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3740002f70 tx=0x7f3740006df0).ready entity=mon.2 client_cookie=fdfc5ae68d7a5270 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:29.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f373b7fe700 1 -- 172.21.15.61:0/2495577914 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374000d570 con 0x7f37440b5920 2022-07-12T08:49:29.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f373b7fe700 1 -- 172.21.15.61:0/2495577914 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3740000e90 con 0x7f37440b5920 2022-07-12T08:49:29.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f373b7fe700 1 -- 172.21.15.61:0/2495577914 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374000d8c0 con 0x7f37440b5920 2022-07-12T08:49:29.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- 172.21.15.61:0/2495577914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b5920 msgr2=0x7f37440aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 --2- 172.21.15.61:0/2495577914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b5920 0x7f37440aa500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3740002f70 tx=0x7f3740006df0).stop 2022-07-12T08:49:29.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- 172.21.15.61:0/2495577914 shutdown_connections 2022-07-12T08:49:29.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 --2- 172.21.15.61:0/2495577914 >> v2:172.21.15.61:3300/0 conn(0x7f37440b0680 0x7f3744060200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 --2- 172.21.15.61:0/2495577914 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b5920 0x7f37440aa500 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- 172.21.15.61:0/2495577914 >> 172.21.15.61:0/2495577914 conn(0x7f3744063340 msgr2=0x7f37440b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:29.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- 172.21.15.61:0/2495577914 shutdown_connections 2022-07-12T08:49:29.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.061+0000 7f3749f92700 1 -- 172.21.15.61:0/2495577914 wait complete. 2022-07-12T08:49:29.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 Processor -- start 2022-07-12T08:49:29.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 -- start start 2022-07-12T08:49:29.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b0240 0x7f37441432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 0x7f37440970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37440b5920 0x7f37440975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374408fa10 con 0x7f37440b0680 2022-07-12T08:49:29.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374408fcb0 con 0x7f37440b5920 2022-07-12T08:49:29.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374408ff50 con 0x7f37440b0240 2022-07-12T08:49:29.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 0x7f37440970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 0x7f37440970f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60238/0 (socket says 172.21.15.61:60238) 2022-07-12T08:49:29.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 -- 172.21.15.61:0/1460261158 learned_addr learned my addr 172.21.15.61:0/1460261158 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:29.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3748f90700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b0240 0x7f37441432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749791700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37440b5920 0x7f37440975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 -- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b0240 msgr2=0x7f37441432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b0240 0x7f37441432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 -- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37440b5920 msgr2=0x7f37440975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37440b5920 0x7f37440975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 -- 172.21.15.61:0/1460261158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c009530 con 0x7f37440b0680 2022-07-12T08:49:29.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3748f90700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b0240 0x7f37441432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:29.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749791700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37440b5920 0x7f37440975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:49:29.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f373bfff700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 0x7f37440970f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f3734000fe0 tx=0x7f3734007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:29.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37340072e0 con 0x7f37440b0680 2022-07-12T08:49:29.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3734005550 con 0x7f37440b0680 2022-07-12T08:49:29.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37340058b0 con 0x7f37440b0680 2022-07-12T08:49:29.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374000b180 con 0x7f37440b0680 2022-07-12T08:49:29.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.065+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374405cad0 con 0x7f37440b0680 2022-07-12T08:49:29.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.069+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3734010070 con 0x7f37440b0680 2022-07-12T08:49:29.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.069+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37440ad920 con 0x7f37440b0680 2022-07-12T08:49:29.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.069+0000 7f3739ffb700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3724053ad0 0x7f3724055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.069+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f373406b750 con 0x7f37440b0680 2022-07-12T08:49:29.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.069+0000 7f3748f90700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3724053ad0 0x7f3724055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.069+0000 7f3748f90700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3724053ad0 0x7f3724055f90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f3740018000 tx=0x7f3740014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:29.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.073+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3734042610 con 0x7f37440b0680 2022-07-12T08:49:29.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.201+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3744057460 con 0x7f37440b0680 2022-07-12T08:49:29.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.201+0000 7f3739ffb700 1 -- 172.21.15.61:0/1460261158 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3734045c30 con 0x7f37440b0680 2022-07-12T08:49:29.210 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:29.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3724053ad0 msgr2=0x7f3724055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3724053ad0 0x7f3724055f90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f3740018000 tx=0x7f3740014000).stop 2022-07-12T08:49:29.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 msgr2=0x7f37440970f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 0x7f37440970f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7f3734000fe0 tx=0x7f3734007b40).stop 2022-07-12T08:49:29.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 shutdown_connections 2022-07-12T08:49:29.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3724053ad0 0x7f3724055f90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37440b5920 0x7f37440975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37440b0680 0x7f37440970f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 --2- 172.21.15.61:0/1460261158 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f37440b0240 0x7f37441432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 >> 172.21.15.61:0/1460261158 conn(0x7f3744063340 msgr2=0x7f3744060240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:29.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 shutdown_connections 2022-07-12T08:49:29.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.205+0000 7f3749f92700 1 -- 172.21.15.61:0/1460261158 wait complete. 2022-07-12T08:49:29.229 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:49:29.230 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:29.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 Processor -- start 2022-07-12T08:49:29.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- start start 2022-07-12T08:49:29.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f5ae405c050 0x7f5ae40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ae414a8f0 0x7f5ae41443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae40ad2f0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae40b0a40 con 0x7f5ae414a8f0 2022-07-12T08:49:29.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae405c430 con 0x7f5ae405c050 2022-07-12T08:49:29.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ae414a8f0 0x7f5ae41443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ae414a8f0 0x7f5ae41443c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60246/0 (socket says 172.21.15.61:60246) 2022-07-12T08:49:29.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 -- 172.21.15.61:0/1903416512 learned_addr learned my addr 172.21.15.61:0/1903416512 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:29.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae8be7700 1 --2- 172.21.15.61:0/1903416512 >> v2:172.21.15.111:3300/0 conn(0x7f5ae405c050 0x7f5ae40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 -- 172.21.15.61:0/1903416512 >> v2:172.21.15.111:3300/0 conn(0x7f5ae405c050 msgr2=0x7f5ae40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:29.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 --2- 172.21.15.61:0/1903416512 >> v2:172.21.15.111:3300/0 conn(0x7f5ae405c050 0x7f5ae40ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 -- 172.21.15.61:0/1903416512 >> v1:172.21.15.111:6789/0 conn(0x7f5ae40b01f0 legacy=0x7f5ae405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 -- 172.21.15.61:0/1903416512 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40b2420 con 0x7f5ae414a8f0 2022-07-12T08:49:29.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae93e8700 1 --2- 172.21.15.61:0/1903416512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae414a8f0 0x7f5ae41443c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f5adc0032f0 tx=0x7f5adc009290).ready entity=mon.0 client_cookie=e7868aaa45dcd2c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:29.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae37fe700 1 -- 172.21.15.61:0/1903416512 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5adc00b870 con 0x7f5ae414a8f0 2022-07-12T08:49:29.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae37fe700 1 -- 172.21.15.61:0/1903416512 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc00b9d0 con 0x7f5ae414a8f0 2022-07-12T08:49:29.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5ae37fe700 1 -- 172.21.15.61:0/1903416512 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5adc00bb40 con 0x7f5ae414a8f0 2022-07-12T08:49:29.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- 172.21.15.61:0/1903416512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae414a8f0 msgr2=0x7f5ae41443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/1903416512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae414a8f0 0x7f5ae41443c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f5adc0032f0 tx=0x7f5adc009290).stop 2022-07-12T08:49:29.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- 172.21.15.61:0/1903416512 shutdown_connections 2022-07-12T08:49:29.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/1903416512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae414a8f0 0x7f5ae41443c0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/1903416512 >> v2:172.21.15.111:3300/0 conn(0x7f5ae405c050 0x7f5ae40ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- 172.21.15.61:0/1903416512 >> 172.21.15.61:0/1903416512 conn(0x7f5ae4062c00 msgr2=0x7f5ae4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:29.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- 172.21.15.61:0/1903416512 shutdown_connections 2022-07-12T08:49:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.349+0000 7f5aeae4b700 1 -- 172.21.15.61:0/1903416512 wait complete. 2022-07-12T08:49:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 Processor -- start 2022-07-12T08:49:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 -- start start 2022-07-12T08:49:29.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ae405c050 0x7f5ae414cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 0x7f5ae40643c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae414a8f0 0x7f5ae4065b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae414a710 con 0x7f5ae40b01f0 2022-07-12T08:49:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae4149690 con 0x7f5ae414a8f0 2022-07-12T08:49:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5aeae4b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae4063c80 con 0x7f5ae405c050 2022-07-12T08:49:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 0x7f5ae40643c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 0x7f5ae40643c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60252/0 (socket says 172.21.15.61:60252) 2022-07-12T08:49:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 -- 172.21.15.61:0/2668531511 learned_addr learned my addr 172.21.15.61:0/2668531511 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae93e8700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae414a8f0 0x7f5ae4065b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae8be7700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ae405c050 0x7f5ae414cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 -- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ae405c050 msgr2=0x7f5ae414cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ae405c050 0x7f5ae414cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 -- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae414a8f0 msgr2=0x7f5ae4065b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae414a8f0 0x7f5ae4065b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 -- 172.21.15.61:0/2668531511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc009530 con 0x7f5ae40b01f0 2022-07-12T08:49:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.353+0000 7f5ae3fff700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 0x7f5ae40643c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f5ae414a850 tx=0x7f5ad0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad0012400 con 0x7f5ae40b01f0 2022-07-12T08:49:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0012560 con 0x7f5ae40b01f0 2022-07-12T08:49:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae4063f20 con 0x7f5ae40b01f0 2022-07-12T08:49:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad00128c0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae4149330 con 0x7f5ae40b01f0 2022-07-12T08:49:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae4144ae0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5ad0019030 con 0x7f5ae40b01f0 2022-07-12T08:49:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae1ffb700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ac4053ad0 0x7f5ac4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ad006b1f0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.357+0000 7f5ae8be7700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ac4053ad0 0x7f5ac4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:29.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.361+0000 7f5ae8be7700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ac4053ad0 0x7f5ac4055f90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f5ad800c000 tx=0x7f5ad8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:29.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.365+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ae4144ae0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ae4144ae0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5ae1ffb700 1 -- 172.21.15.61:0/2668531511 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f5ae4144ae0 con 0x7f5ae40b01f0 2022-07-12T08:49:29.535 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:29.536 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2585981801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2585981801},{"type":"v1","addr":"172.21.15.111:6835","nonce":2585981801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:49:24.635457+0000","modified":"2022-07-12T08:49:25.639064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2867307616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2867307616},{"type":"v1","addr":"172.21.15.61:6833","nonce":2867307616}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-12T08:49:29.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ac4053ad0 msgr2=0x7f5ac4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ac4053ad0 0x7f5ac4055f90 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f5ad800c000 tx=0x7f5ad8008040).stop 2022-07-12T08:49:29.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 msgr2=0x7f5ae40643c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:29.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 0x7f5ae40643c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f5ae414a850 tx=0x7f5ad0007c50).stop 2022-07-12T08:49:29.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 shutdown_connections 2022-07-12T08:49:29.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5ac4053ad0 0x7f5ac4055f90 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ae414a8f0 0x7f5ae4065b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae40b01f0 0x7f5ae40643c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 --2- 172.21.15.61:0/2668531511 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ae405c050 0x7f5ae414cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:29.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 >> 172.21.15.61:0/2668531511 conn(0x7f5ae4062c00 msgr2=0x7f5ae40b44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:29.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 shutdown_connections 2022-07-12T08:49:29.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:29.525+0000 7f5aeae4b700 1 -- 172.21.15.61:0/2668531511 wait complete. 2022-07-12T08:49:29.546 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-12T08:49:29.750 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:49:24.635457+0000', 'modified': '2022-07-12T08:49:25.639064+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2867307616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2867307616}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2867307616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:49:29.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:49:29.753 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:49:29.891 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":7671,"want_state":"up:active","state":"up:active","rank_uptime":5.240331587,"mdsmap_epoch":62,"osdmap_epoch":110,"osdmap_epoch_barrier":109,"uptime":9.8330919540000004} 2022-07-12T08:49:29.893 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 7671, "mdsmap_epoch": 62, "osdmap_epoch": 110, "osdmap_epoch_barrier": 109, "rank_uptime": 5.240331587, "state": "up:active", "uptime": 9.833091954, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:49:29.894 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:49:29.895 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:49:29.896 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:49:29.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:29.897 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:29.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:49:29.945 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:29.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:30.002 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:30.006 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:30.007 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:49:30.020 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:49:30.021 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:30.065 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:49:30.087 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:30.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:30.114 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:30.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:30.116 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:49:30.138 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:49:30.131+0000 7fd7010aa080 -1 init, newargv = 0x560d09271570 newargc=9ceph-fuse[ 2022-07-12T08:49:30.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:22026]: starting ceph client 2022-07-12T08:49:30.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22026]: starting fuse 2022-07-12T08:49:31.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:49:31.266 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:31.267 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:31.287 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:31.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:31.289 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:49:31.344 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:49:31.345 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:49:31.346 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:31.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:49:31.347 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:49:31.348 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:49:31.349 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:49:31.350 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:49:31.351 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:31.351 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:49:31.352 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:49:31.353 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:49:31.354 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:31.355 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:49:31.356 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:49:31.356 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:49:31.357 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:49:31.358 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:31.359 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:49:31.360 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:49:31.360 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:49:31.361 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:49:31.362 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:49:31.363 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:49:31.364 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:49:31.364 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:49:31.432 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.22026.asok 2022-07-12T08:49:31.439 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22026.asok 2022-07-12T08:49:31.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:31.440 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22026.asok status 2022-07-12T08:49:31.530 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:49:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:49:31.530 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:49:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:49:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:49:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:49:31.531 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:49:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "22026", 2022-07-12T08:49:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:49:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:31.532 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:49:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:49:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 7657, 2022-07-12T08:49:31.533 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:49:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:49:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:49:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7657 2022-07-12T08:49:31.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:49:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:49:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:49:31.536 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1823768445 2022-07-12T08:49:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:49:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:31.537 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:49:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:49:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:49:31.538 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1823768445 2022-07-12T08:49:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:49:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.7657 172.21.15.121:0/1823768445", 2022-07-12T08:49:31.539 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/1823768445", 2022-07-12T08:49:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:49:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 62, 2022-07-12T08:49:31.540 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 110, 2022-07-12T08:49:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:49:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:49:31.541 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:49:31.541 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:49:31.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:31.544 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:31.545 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:31.556 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:49:31.557 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:31.558 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:31.559 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:31.618 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:49:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 Processor -- start 2022-07-12T08:49:31.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- start start 2022-07-12T08:49:31.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f805d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:31.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f813fd50 con 0x7f45f80ac710 2022-07-12T08:49:31.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f8067900 con 0x7f45f80b1e40 2022-07-12T08:49:31.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8067ba0 con 0x7f45f80ad590 2022-07-12T08:49:31.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fe71a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f45f80ac710 0x7f45f81401b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48480/0 (socket says 172.21.15.61:48480) 2022-07-12T08:49:31.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fe71a700 1 -- 172.21.15.61:0/1952667680 learned_addr learned my addr 172.21.15.61:0/1952667680 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:31.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fd718700 1 --2- 172.21.15.61:0/1952667680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f805d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:31.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fcf17700 1 -- 172.21.15.61:0/1952667680 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684674189 0 0) 0x7f45f813fd50 con 0x7f45f80ac710 2022-07-12T08:49:31.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fcf17700 1 -- 172.21.15.61:0/1952667680 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e0003260 con 0x7f45f80ac710 2022-07-12T08:49:31.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fd718700 1 -- 172.21.15.61:0/1952667680 >> v1:172.21.15.111:6789/0 conn(0x7f45f80b1e40 legacy=0x7f45f80ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fd718700 1 -- 172.21.15.61:0/1952667680 >> v1:172.21.15.61:6789/0 conn(0x7f45f80ac710 legacy=0x7f45f81401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fd718700 1 -- 172.21.15.61:0/1952667680 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f80675f0 con 0x7f45f80ad590 2022-07-12T08:49:31.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fd718700 1 --2- 172.21.15.61:0/1952667680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f805d0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f45e8002f70 tx=0x7f45e8006df0).ready entity=mon.2 client_cookie=687a970e6ebdc91b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:31.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fcf17700 1 -- 172.21.15.61:0/1952667680 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e800d570 con 0x7f45f80ad590 2022-07-12T08:49:31.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fcf17700 1 -- 172.21.15.61:0/1952667680 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e8000e90 con 0x7f45f80ad590 2022-07-12T08:49:31.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f45fcf17700 1 -- 172.21.15.61:0/1952667680 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45e800d8c0 con 0x7f45f80ad590 2022-07-12T08:49:31.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- 172.21.15.61:0/1952667680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 msgr2=0x7f45f805d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 --2- 172.21.15.61:0/1952667680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f805d0d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f45e8002f70 tx=0x7f45e8006df0).stop 2022-07-12T08:49:31.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- 172.21.15.61:0/1952667680 shutdown_connections 2022-07-12T08:49:31.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 --2- 172.21.15.61:0/1952667680 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f805d0d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- 172.21.15.61:0/1952667680 >> 172.21.15.61:0/1952667680 conn(0x7f45f8062c00 msgr2=0x7f45f8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:31.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- 172.21.15.61:0/1952667680 shutdown_connections 2022-07-12T08:49:31.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.729+0000 7f460017d700 1 -- 172.21.15.61:0/1952667680 wait complete. 2022-07-12T08:49:31.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 Processor -- start 2022-07-12T08:49:31.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 -- start start 2022-07-12T08:49:31.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f80ac710 0x7f45f813c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:31.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f80ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:31.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 0x7f45f80abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:31.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8139b40 con 0x7f45f80b1e40 2022-07-12T08:49:31.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f813d830 con 0x7f45f80ac710 2022-07-12T08:49:31.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f813dad0 con 0x7f45f80ad590 2022-07-12T08:49:31.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 0x7f45f80abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:31.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 0x7f45f80abab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60268/0 (socket says 172.21.15.61:60268) 2022-07-12T08:49:31.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 -- 172.21.15.61:0/1000987473 learned_addr learned my addr 172.21.15.61:0/1000987473 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:31.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fd718700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f80ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:31.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fdf19700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f80ac710 0x7f45f813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:31.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 -- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 msgr2=0x7f45f80ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f80ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 -- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f80ac710 msgr2=0x7f45f813c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f80ac710 0x7f45f813c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 -- 172.21.15.61:0/1000987473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e800b180 con 0x7f45f80b1e40 2022-07-12T08:49:31.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45fe71a700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 0x7f45f80abab0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f45f0001490 tx=0x7f45f0007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:31.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45f0005ba0 con 0x7f45f80b1e40 2022-07-12T08:49:31.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f0005d00 con 0x7f45f80b1e40 2022-07-12T08:49:31.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f813dd70 con 0x7f45f80b1e40 2022-07-12T08:49:31.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.733+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45f00072c0 con 0x7f45f80b1e40 2022-07-12T08:49:31.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f80abfa0 con 0x7f45f80b1e40 2022-07-12T08:49:31.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f813df00 con 0x7f45f80b1e40 2022-07-12T08:49:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f45f001a070 con 0x7f45f80b1e40 2022-07-12T08:49:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f45e6ffd700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d8053940 0x7f45d8055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f45f006b1b0 con 0x7f45f80b1e40 2022-07-12T08:49:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f45fdf19700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d8053940 0x7f45d8055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.737+0000 7f45fdf19700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d8053940 0x7f45d8055e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f45f400d000 tx=0x7f45f4007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:31.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.745+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45f0042110 con 0x7f45f80b1e40 2022-07-12T08:49:31.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.873+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45f8057460 con 0x7f45f80b1e40 2022-07-12T08:49:31.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f45e6ffd700 1 -- 172.21.15.61:0/1000987473 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f45f0045730 con 0x7f45f80b1e40 2022-07-12T08:49:31.882 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:31.883 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":110,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:49:25.639162+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:49:22.980785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-12T08:49:23.643810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:49:31.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d8053940 msgr2=0x7f45d8055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d8053940 0x7f45d8055e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f45f400d000 tx=0x7f45f4007b70).stop 2022-07-12T08:49:31.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 msgr2=0x7f45f80abab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:31.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 0x7f45f80abab0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 rx=0x7f45f0001490 tx=0x7f45f0007e70).stop 2022-07-12T08:49:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 shutdown_connections 2022-07-12T08:49:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45f80b1e40 0x7f45f80abab0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f45f80ad590 0x7f45f80ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f45d8053940 0x7f45d8055e00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 --2- 172.21.15.61:0/1000987473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f45f80ac710 0x7f45f813c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 >> 172.21.15.61:0/1000987473 conn(0x7f45f8062c00 msgr2=0x7f45f8065b80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 shutdown_connections 2022-07-12T08:49:31.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:31.877+0000 7f460017d700 1 -- 172.21.15.61:0/1000987473 wait complete. 2022-07-12T08:49:31.897 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:49:22.980785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:49:23.643810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:49:31.898 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:49:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 Processor -- start 2022-07-12T08:49:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 -- start start 2022-07-12T08:49:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b5f20 0x7f84cc0b6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84cc0b3e90 0x7f84cc1438b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84cc143da0 con 0x7f84cc05bd10 2022-07-12T08:49:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84cc142ed0 con 0x7f84cc0b3e90 2022-07-12T08:49:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d39f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc143170 con 0x7f84cc0b5f20 2022-07-12T08:49:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d0f93700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84cc0b3e90 0x7f84cc1438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d1f95700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f84cc05bd10 0x7f84cc146390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48494/0 (socket says 172.21.15.61:48494) 2022-07-12T08:49:32.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d1f95700 1 -- 172.21.15.61:0/627143715 learned_addr learned my addr 172.21.15.61:0/627143715 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:32.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d1794700 1 --2- 172.21.15.61:0/627143715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b5f20 0x7f84cc0b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d0f93700 1 -- 172.21.15.61:0/627143715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b5f20 msgr2=0x7f84cc0b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d0f93700 1 --2- 172.21.15.61:0/627143715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b5f20 0x7f84cc0b6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d0f93700 1 -- 172.21.15.61:0/627143715 >> v1:172.21.15.61:6789/0 conn(0x7f84cc05bd10 legacy=0x7f84cc146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d0f93700 1 -- 172.21.15.61:0/627143715 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc143410 con 0x7f84cc0b3e90 2022-07-12T08:49:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84d0f93700 1 --2- 172.21.15.61:0/627143715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc0b3e90 0x7f84cc1438b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f84c4001490 tx=0x7f84bc006df0).ready entity=mon.0 client_cookie=d8453a2cf0cb4656 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84c3fff700 1 -- 172.21.15.61:0/627143715 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84bc00d4a0 con 0x7f84cc0b3e90 2022-07-12T08:49:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84c3fff700 1 -- 172.21.15.61:0/627143715 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc000e90 con 0x7f84cc0b3e90 2022-07-12T08:49:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.009+0000 7f84c3fff700 1 -- 172.21.15.61:0/627143715 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84bc00d7f0 con 0x7f84cc0b3e90 2022-07-12T08:49:32.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- 172.21.15.61:0/627143715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc0b3e90 msgr2=0x7f84cc1438b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 --2- 172.21.15.61:0/627143715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc0b3e90 0x7f84cc1438b0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f84c4001490 tx=0x7f84bc006df0).stop 2022-07-12T08:49:32.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- 172.21.15.61:0/627143715 shutdown_connections 2022-07-12T08:49:32.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 --2- 172.21.15.61:0/627143715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc0b3e90 0x7f84cc1438b0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 --2- 172.21.15.61:0/627143715 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b5f20 0x7f84cc0b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- 172.21.15.61:0/627143715 >> 172.21.15.61:0/627143715 conn(0x7f84cc066e90 msgr2=0x7f84cc0672a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- 172.21.15.61:0/627143715 shutdown_connections 2022-07-12T08:49:32.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- 172.21.15.61:0/627143715 wait complete. 2022-07-12T08:49:32.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 Processor -- start 2022-07-12T08:49:32.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- start start 2022-07-12T08:49:32.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 0x7f84cc140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b3e90 0x7f84cc155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84cc0b5f20 0x7f84cc155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc140f00 con 0x7f84cc05bd10 2022-07-12T08:49:32.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc1411a0 con 0x7f84cc0b5f20 2022-07-12T08:49:32.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d39f8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc13f170 con 0x7f84cc0b3e90 2022-07-12T08:49:32.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d1794700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 0x7f84cc140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d1794700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 0x7f84cc140540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60278/0 (socket says 172.21.15.61:60278) 2022-07-12T08:49:32.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.013+0000 7f84d1794700 1 -- 172.21.15.61:0/2795021179 learned_addr learned my addr 172.21.15.61:0/2795021179 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:32.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d0f93700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b3e90 0x7f84cc155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1f95700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84cc0b5f20 0x7f84cc155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1794700 1 -- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b3e90 msgr2=0x7f84cc155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1794700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b3e90 0x7f84cc155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1794700 1 -- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84cc0b5f20 msgr2=0x7f84cc155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1794700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84cc0b5f20 0x7f84cc155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1794700 1 -- 172.21.15.61:0/2795021179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc00b180 con 0x7f84cc05bd10 2022-07-12T08:49:32.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d0f93700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b3e90 0x7f84cc155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:32.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d1794700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 0x7f84cc140540 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f84c800c000 tx=0x7f84c8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84c8011750 con 0x7f84cc05bd10 2022-07-12T08:49:32.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c80118b0 con 0x7f84cc05bd10 2022-07-12T08:49:32.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84cc145ec0 con 0x7f84cc05bd10 2022-07-12T08:49:32.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84c8011c10 con 0x7f84cc05bd10 2022-07-12T08:49:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc155eb0 con 0x7f84cc05bd10 2022-07-12T08:49:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc140d70 con 0x7f84cc05bd10 2022-07-12T08:49:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.017+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f84c8011d70 con 0x7f84cc05bd10 2022-07-12T08:49:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.021+0000 7f84c27fc700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84b4053ad0 0x7f84b4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.021+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f84c806c280 con 0x7f84cc05bd10 2022-07-12T08:49:32.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.021+0000 7f84d0f93700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84b4053ad0 0x7f84b4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.021+0000 7f84d0f93700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84b4053ad0 0x7f84b4055f90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f84bc01a000 tx=0x7f84bc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.025+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84c8043180 con 0x7f84cc05bd10 2022-07-12T08:49:32.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f84cc057460 con 0x7f84cc05bd10 2022-07-12T08:49:32.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84c27fc700 1 -- 172.21.15.61:0/2795021179 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v110) v1 ==== 110+0+10 (secure 0 0 0) 0x7f84c80467a0 con 0x7f84cc05bd10 2022-07-12T08:49:32.159 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:49:32.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84b4053ad0 msgr2=0x7f84b4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84b4053ad0 0x7f84b4055f90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7f84bc01a000 tx=0x7f84bc014000).stop 2022-07-12T08:49:32.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 msgr2=0x7f84cc140540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 0x7f84cc140540 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 rx=0x7f84c800c000 tx=0x7f84c8006df0).stop 2022-07-12T08:49:32.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 shutdown_connections 2022-07-12T08:49:32.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84b4053ad0 0x7f84b4055f90 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.153+0000 7f84d39f8700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84cc0b5f20 0x7f84cc155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.157+0000 7f84d39f8700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84cc0b3e90 0x7f84cc155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.157+0000 7f84d39f8700 1 --2- 172.21.15.61:0/2795021179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84cc05bd10 0x7f84cc140540 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.157+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 >> 172.21.15.61:0/2795021179 conn(0x7f84cc066e90 msgr2=0x7f84cc05c950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.157+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 shutdown_connections 2022-07-12T08:49:32.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.157+0000 7f84d39f8700 1 -- 172.21.15.61:0/2795021179 wait complete. 2022-07-12T08:49:32.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:49:32.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.273+0000 7f0785bed700 1 Processor -- start 2022-07-12T08:49:32.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.273+0000 7f0785bed700 1 -- start start 2022-07-12T08:49:32.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005c6d0 0x7f07800b6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0780143500 con 0x7f0780142eb0 2022-07-12T08:49:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0780143290 con 0x7f078005d130 2022-07-12T08:49:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0780143760 con 0x7f078005c6d0 2022-07-12T08:49:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0784beb700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0780142eb0 0x7f07800b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48504/0 (socket says 172.21.15.61:48504) 2022-07-12T08:49:32.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0784beb700 1 -- 172.21.15.61:0/541777253 learned_addr learned my addr 172.21.15.61:0/541777253 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:32.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f07853ec700 1 --2- 172.21.15.61:0/541777253 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005c6d0 0x7f07800b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f077f7fe700 1 -- 172.21.15.61:0/541777253 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213807789 0 0) 0x7f0780143500 con 0x7f0780142eb0 2022-07-12T08:49:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f077f7fe700 1 -- 172.21.15.61:0/541777253 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0760003260 con 0x7f0780142eb0 2022-07-12T08:49:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f077f7fe700 1 -- 172.21.15.61:0/541777253 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677285368 0 0) 0x7f0780143290 con 0x7f078005d130 2022-07-12T08:49:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f07853ec700 1 -- 172.21.15.61:0/541777253 >> v1:172.21.15.111:6789/0 conn(0x7f078005d130 legacy=0x7f07800b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f07853ec700 1 -- 172.21.15.61:0/541777253 >> v1:172.21.15.61:6789/0 conn(0x7f0780142eb0 legacy=0x7f07800b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f07853ec700 1 -- 172.21.15.61:0/541777253 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07800b4030 con 0x7f078005c6d0 2022-07-12T08:49:32.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f07853ec700 1 --2- 172.21.15.61:0/541777253 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005c6d0 0x7f07800b6130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f07780032f0 tx=0x7f07780091d0).ready entity=mon.2 client_cookie=71abd4f3c9a6ec6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f077f7fe700 1 -- 172.21.15.61:0/541777253 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f077800b6c0 con 0x7f078005c6d0 2022-07-12T08:49:32.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f077f7fe700 1 -- 172.21.15.61:0/541777253 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077800b820 con 0x7f078005c6d0 2022-07-12T08:49:32.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f077f7fe700 1 -- 172.21.15.61:0/541777253 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f077800b990 con 0x7f078005c6d0 2022-07-12T08:49:32.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- 172.21.15.61:0/541777253 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005c6d0 msgr2=0x7f07800b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 --2- 172.21.15.61:0/541777253 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005c6d0 0x7f07800b6130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f07780032f0 tx=0x7f07780091d0).stop 2022-07-12T08:49:32.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- 172.21.15.61:0/541777253 shutdown_connections 2022-07-12T08:49:32.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 --2- 172.21.15.61:0/541777253 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005c6d0 0x7f07800b6130 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- 172.21.15.61:0/541777253 >> 172.21.15.61:0/541777253 conn(0x7f0780066d70 msgr2=0x7f0780067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- 172.21.15.61:0/541777253 shutdown_connections 2022-07-12T08:49:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.277+0000 7f0785bed700 1 -- 172.21.15.61:0/541777253 wait complete. 2022-07-12T08:49:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 Processor -- start 2022-07-12T08:49:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 -- start start 2022-07-12T08:49:32.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 0x7f0780151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005d130 0x7f07800b2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0780142eb0 0x7f078013ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07800ae1d0 con 0x7f078005c6d0 2022-07-12T08:49:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0780149040 con 0x7f0780142eb0 2022-07-12T08:49:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0780149ce0 con 0x7f078005d130 2022-07-12T08:49:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 0x7f0780151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 0x7f0780151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60292/0 (socket says 172.21.15.61:60292) 2022-07-12T08:49:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 -- 172.21.15.61:0/2170154931 learned_addr learned my addr 172.21.15.61:0/2170154931 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:32.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f077ffff700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005d130 0x7f07800b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f07853ec700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0780142eb0 0x7f078013ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 -- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005d130 msgr2=0x7f07800b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005d130 0x7f07800b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 -- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0780142eb0 msgr2=0x7f078013ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0780142eb0 0x7f078013ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 -- 172.21.15.61:0/2170154931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0778009470 con 0x7f078005c6d0 2022-07-12T08:49:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0784beb700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 0x7f0780151850 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7f0768001110 tx=0x7f0768007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0768007e10 con 0x7f078005c6d0 2022-07-12T08:49:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07800adaf0 con 0x7f078005c6d0 2022-07-12T08:49:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07680072e0 con 0x7f078005c6d0 2022-07-12T08:49:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.281+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0780149fc0 con 0x7f078005c6d0 2022-07-12T08:49:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0768007450 con 0x7f078005c6d0 2022-07-12T08:49:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07800aef90 con 0x7f078005c6d0 2022-07-12T08:49:32.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0768010070 con 0x7f078005c6d0 2022-07-12T08:49:32.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f077dffb700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0764053a40 0x7f0764055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f076806b3d0 con 0x7f078005c6d0 2022-07-12T08:49:32.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f077ffff700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0764053a40 0x7f0764055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.285+0000 7f077ffff700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0764053a40 0x7f0764055f00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f077000b000 tx=0x7f0770009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.293+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07800aef90 con 0x7f078005c6d0 2022-07-12T08:49:32.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.421+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f07800aef90 con 0x7f078005c6d0 2022-07-12T08:49:32.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.421+0000 7f077dffb700 1 -- 172.21.15.61:0/2170154931 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v110) v1 ==== 92+0+10 (secure 0 0 0) 0x7f07800aef90 con 0x7f078005c6d0 2022-07-12T08:49:32.428 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:32.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0764053a40 msgr2=0x7f0764055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0764053a40 0x7f0764055f00 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f077000b000 tx=0x7f0770009040).stop 2022-07-12T08:49:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 msgr2=0x7f0780151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 0x7f0780151850 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 rx=0x7f0768001110 tx=0x7f0768007ad0).stop 2022-07-12T08:49:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 shutdown_connections 2022-07-12T08:49:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0764053a40 0x7f0764055f00 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0780142eb0 0x7f078013ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f078005d130 0x7f07800b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 --2- 172.21.15.61:0/2170154931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f078005c6d0 0x7f0780151850 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 >> 172.21.15.61:0/2170154931 conn(0x7f0780066d70 msgr2=0x7f07800b8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 shutdown_connections 2022-07-12T08:49:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.425+0000 7f0785bed700 1 -- 172.21.15.61:0/2170154931 wait complete. 2022-07-12T08:49:32.444 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:49:32.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.549+0000 7fdd09b70700 1 Processor -- start 2022-07-12T08:49:32.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.549+0000 7fdd09b70700 1 -- start start 2022-07-12T08:49:32.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.549+0000 7fdd09b70700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd040b0280 0x7fdd0414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.549+0000 7fdd09b70700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fdd040aa830 0x7fdd040aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd0405c3e0 con 0x7fdd040b5920 2022-07-12T08:49:32.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd040b5d00 con 0x7fdd040b0280 2022-07-12T08:49:32.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0405c610 con 0x7fdd040aa830 2022-07-12T08:49:32.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd040b0280 0x7fdd0414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd040b0280 0x7fdd0414b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60302/0 (socket says 172.21.15.61:60302) 2022-07-12T08:49:32.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd08b6e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdd040b5920 0x7fdd0414a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48518/0 (socket says 172.21.15.61:48518) 2022-07-12T08:49:32.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 -- 172.21.15.61:0/3593159636 learned_addr learned my addr 172.21.15.61:0/3593159636 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:32.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd0936f700 1 --2- 172.21.15.61:0/3593159636 >> v2:172.21.15.111:3300/0 conn(0x7fdd040aa830 0x7fdd040aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 -- 172.21.15.61:0/3593159636 >> v2:172.21.15.111:3300/0 conn(0x7fdd040aa830 msgr2=0x7fdd040aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:32.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 --2- 172.21.15.61:0/3593159636 >> v2:172.21.15.111:3300/0 conn(0x7fdd040aa830 0x7fdd040aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 -- 172.21.15.61:0/3593159636 >> v1:172.21.15.61:6789/0 conn(0x7fdd040b5920 legacy=0x7fdd0414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 -- 172.21.15.61:0/3593159636 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0405c8b0 con 0x7fdd040b0280 2022-07-12T08:49:32.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd037fe700 1 -- 172.21.15.61:0/3593159636 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060563244 0 0) 0x7fdd0405c3e0 con 0x7fdd040b5920 2022-07-12T08:49:32.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd03fff700 1 --2- 172.21.15.61:0/3593159636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040b0280 0x7fdd0414b0f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7fdcf40032f0 tx=0x7fdcf4009290).ready entity=mon.0 client_cookie=e188696bff6476b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd037fe700 1 -- 172.21.15.61:0/3593159636 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcf400b870 con 0x7fdd040b0280 2022-07-12T08:49:32.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd037fe700 1 -- 172.21.15.61:0/3593159636 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf400b9d0 con 0x7fdd040b0280 2022-07-12T08:49:32.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd037fe700 1 -- 172.21.15.61:0/3593159636 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcf400bb40 con 0x7fdd040b0280 2022-07-12T08:49:32.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- 172.21.15.61:0/3593159636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040b0280 msgr2=0x7fdd0414b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 --2- 172.21.15.61:0/3593159636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040b0280 0x7fdd0414b0f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7fdcf40032f0 tx=0x7fdcf4009290).stop 2022-07-12T08:49:32.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- 172.21.15.61:0/3593159636 shutdown_connections 2022-07-12T08:49:32.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 --2- 172.21.15.61:0/3593159636 >> v2:172.21.15.111:3300/0 conn(0x7fdd040aa830 0x7fdd040aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 --2- 172.21.15.61:0/3593159636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040b0280 0x7fdd0414b0f0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- 172.21.15.61:0/3593159636 >> 172.21.15.61:0/3593159636 conn(0x7fdd04063340 msgr2=0x7fdd040b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- 172.21.15.61:0/3593159636 shutdown_connections 2022-07-12T08:49:32.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- 172.21.15.61:0/3593159636 wait complete. 2022-07-12T08:49:32.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 Processor -- start 2022-07-12T08:49:32.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.553+0000 7fdd09b70700 1 -- start start 2022-07-12T08:49:32.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 0x7fdd040ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd040b0280 0x7fdd04065d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd040b5920 0x7fdd04066260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04066750 con 0x7fdd040aa830 2022-07-12T08:49:32.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd040aeac0 con 0x7fdd040b0280 2022-07-12T08:49:32.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd04065910 con 0x7fdd040b5920 2022-07-12T08:49:32.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 0x7fdd040ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 0x7fdd040ac390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60306/0 (socket says 172.21.15.61:60306) 2022-07-12T08:49:32.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 -- 172.21.15.61:0/687814977 learned_addr learned my addr 172.21.15.61:0/687814977 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:32.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd0936f700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd040b5920 0x7fdd04066260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd03fff700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd040b0280 0x7fdd04065d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 -- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd040b5920 msgr2=0x7fdd04066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd040b5920 0x7fdd04066260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 -- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd040b0280 msgr2=0x7fdd04065d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd040b0280 0x7fdd04065d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 -- 172.21.15.61:0/687814977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf4009530 con 0x7fdd040aa830 2022-07-12T08:49:32.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd03fff700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd040b0280 0x7fdd04065d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:32.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd08b6e700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 0x7fdd040ac390 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7fdd04066890 tx=0x7fdcf8007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcf80072e0 con 0x7fdd040aa830 2022-07-12T08:49:32.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf8007440 con 0x7fdd040aa830 2022-07-12T08:49:32.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd040aecf0 con 0x7fdd040aa830 2022-07-12T08:49:32.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcf8005740 con 0x7fdd040aa830 2022-07-12T08:49:32.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd040adeb0 con 0x7fdd040aa830 2022-07-12T08:49:32.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd040ac880 con 0x7fdd040aa830 2022-07-12T08:49:32.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdcf801a060 con 0x7fdd040aa830 2022-07-12T08:49:32.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.557+0000 7fdd01ffb700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdce4053a40 0x7fdce4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.561+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdcf806b770 con 0x7fdd040aa830 2022-07-12T08:49:32.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.561+0000 7fdd03fff700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdce4053a40 0x7fdce4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.561+0000 7fdd03fff700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdce4053a40 0x7fdce4055f00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7fdcf4019000 tx=0x7fdcf4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.565+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd040ac880 con 0x7fdd040aa830 2022-07-12T08:49:32.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.685+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd040ac880 con 0x7fdd040aa830 2022-07-12T08:49:32.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.685+0000 7fdd01ffb700 1 -- 172.21.15.61:0/687814977 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdd040ac880 con 0x7fdd040aa830 2022-07-12T08:49:32.693 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:32.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdce4053a40 msgr2=0x7fdce4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdce4053a40 0x7fdce4055f00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7fdcf4019000 tx=0x7fdcf4016000).stop 2022-07-12T08:49:32.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 msgr2=0x7fdd040ac390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 0x7fdd040ac390 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 rx=0x7fdd04066890 tx=0x7fdcf8007c40).stop 2022-07-12T08:49:32.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 shutdown_connections 2022-07-12T08:49:32.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdd040b5920 0x7fdd04066260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdce4053a40 0x7fdce4055f00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd040b0280 0x7fdd04065d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 --2- 172.21.15.61:0/687814977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd040aa830 0x7fdd040ac390 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 >> 172.21.15.61:0/687814977 conn(0x7fdd04063340 msgr2=0x7fdd040ab740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 shutdown_connections 2022-07-12T08:49:32.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.689+0000 7fdd09b70700 1 -- 172.21.15.61:0/687814977 wait complete. 2022-07-12T08:49:32.706 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:49:32.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 Processor -- start 2022-07-12T08:49:32.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 -- start start 2022-07-12T08:49:32.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08e8142eb0 0x7f08e805c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e80b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e8143500 con 0x7f08e80b5f10 2022-07-12T08:49:32.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e8143290 con 0x7f08e8142eb0 2022-07-12T08:49:32.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ef438700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8143760 con 0x7f08e805d130 2022-07-12T08:49:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08eec37700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f08e80b5f10 0x7f08e8146090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48534/0 (socket says 172.21.15.61:48534) 2022-07-12T08:49:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08eec37700 1 -- 172.21.15.61:0/2878316717 learned_addr learned my addr 172.21.15.61:0/2878316717 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ee436700 1 --2- 172.21.15.61:0/2878316717 >> v2:172.21.15.61:3300/0 conn(0x7f08e8142eb0 0x7f08e805c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08edc35700 1 --2- 172.21.15.61:0/2878316717 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e80b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.817+0000 7f08ed434700 1 -- 172.21.15.61:0/2878316717 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913201875 0 0) 0x7f08e8143500 con 0x7f08e80b5f10 2022-07-12T08:49:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ed434700 1 -- 172.21.15.61:0/2878316717 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d0003260 con 0x7f08e80b5f10 2022-07-12T08:49:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ed434700 1 -- 172.21.15.61:0/2878316717 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621645892 0 0) 0x7f08d0003260 con 0x7f08e80b5f10 2022-07-12T08:49:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ee436700 1 -- 172.21.15.61:0/2878316717 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 msgr2=0x7f08e80b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:32.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ee436700 1 --2- 172.21.15.61:0/2878316717 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e80b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ee436700 1 -- 172.21.15.61:0/2878316717 >> v1:172.21.15.61:6789/0 conn(0x7f08e80b5f10 legacy=0x7f08e8146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ee436700 1 -- 172.21.15.61:0/2878316717 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e80b4030 con 0x7f08e8142eb0 2022-07-12T08:49:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ee436700 1 --2- 172.21.15.61:0/2878316717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e8142eb0 0x7f08e805c6d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f08e0001b80 tx=0x7f08d8006df0).ready entity=mon.0 client_cookie=3d643ebc8b159be8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ed434700 1 -- 172.21.15.61:0/2878316717 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e00019e0 con 0x7f08e8142eb0 2022-07-12T08:49:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ed434700 1 -- 172.21.15.61:0/2878316717 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8000e90 con 0x7f08e8142eb0 2022-07-12T08:49:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ed434700 1 -- 172.21.15.61:0/2878316717 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08d8012460 con 0x7f08e8142eb0 2022-07-12T08:49:32.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 -- 172.21.15.61:0/2878316717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e8142eb0 msgr2=0x7f08e805c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 --2- 172.21.15.61:0/2878316717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e8142eb0 0x7f08e805c6d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f08e0001b80 tx=0x7f08d8006df0).stop 2022-07-12T08:49:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 -- 172.21.15.61:0/2878316717 shutdown_connections 2022-07-12T08:49:32.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 --2- 172.21.15.61:0/2878316717 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e80b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 --2- 172.21.15.61:0/2878316717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e8142eb0 0x7f08e805c6d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 -- 172.21.15.61:0/2878316717 >> 172.21.15.61:0/2878316717 conn(0x7f08e8066d70 msgr2=0x7f08e8067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 -- 172.21.15.61:0/2878316717 shutdown_connections 2022-07-12T08:49:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.821+0000 7f08ef438700 1 -- 172.21.15.61:0/2878316717 wait complete. 2022-07-12T08:49:32.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 Processor -- start 2022-07-12T08:49:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 -- start start 2022-07-12T08:49:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e8112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 0x7f08e813eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e8142eb0 0x7f08e813f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e81528f0 con 0x7f08e80b5f10 2022-07-12T08:49:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e81130c0 con 0x7f08e8142eb0 2022-07-12T08:49:32.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ef438700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e8152650 con 0x7f08e805d130 2022-07-12T08:49:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 0x7f08e813eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 0x7f08e813eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60322/0 (socket says 172.21.15.61:60322) 2022-07-12T08:49:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 -- 172.21.15.61:0/1754332293 learned_addr learned my addr 172.21.15.61:0/1754332293 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08ee436700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e8112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08eec37700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e8142eb0 0x7f08e813f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 -- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 msgr2=0x7f08e8112bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e8112bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 -- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e8142eb0 msgr2=0x7f08e813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e8142eb0 0x7f08e813f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 -- 172.21.15.61:0/1754332293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d800b180 con 0x7f08e80b5f10 2022-07-12T08:49:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.825+0000 7f08edc35700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 0x7f08e813eee0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f08e400c000 tx=0x7f08e4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e4010780 con 0x7f08e80b5f10 2022-07-12T08:49:32.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e40108e0 con 0x7f08e80b5f10 2022-07-12T08:49:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e8152370 con 0x7f08e80b5f10 2022-07-12T08:49:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08e4010c40 con 0x7f08e80b5f10 2022-07-12T08:49:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e8146de0 con 0x7f08e80b5f10 2022-07-12T08:49:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8057460 con 0x7f08e80b5f10 2022-07-12T08:49:32.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f08e401c460 con 0x7f08e80b5f10 2022-07-12T08:49:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08d77fe700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08c8053a60 0x7f08c8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f08e406c170 con 0x7f08e80b5f10 2022-07-12T08:49:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08ee436700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08c8053a60 0x7f08c8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:32.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.829+0000 7f08ee436700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08c8053a60 0x7f08c8055f20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f08d801a000 tx=0x7f08d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:32.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.837+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08e4043070 con 0x7f08e80b5f10 2022-07-12T08:49:32.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f08e8113230 con 0x7f08e80b5f10 2022-07-12T08:49:32.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08d77fe700 1 -- 172.21.15.61:0/1754332293 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+10 (secure 0 0 0) 0x7f08e4046690 con 0x7f08e80b5f10 2022-07-12T08:49:32.976 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:49:32.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08c8053a60 msgr2=0x7f08c8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08c8053a60 0x7f08c8055f20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f08d801a000 tx=0x7f08d8014000).stop 2022-07-12T08:49:32.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 msgr2=0x7f08e813eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:32.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 0x7f08e813eee0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f08e400c000 tx=0x7f08e4006df0).stop 2022-07-12T08:49:32.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 shutdown_connections 2022-07-12T08:49:32.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f08c8053a60 0x7f08c8055f20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08e8142eb0 0x7f08e813f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08e80b5f10 0x7f08e813eee0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 --2- 172.21.15.61:0/1754332293 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08e805d130 0x7f08e8112bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:32.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 >> 172.21.15.61:0/1754332293 conn(0x7f08e8066d70 msgr2=0x7f08e8141be0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:32.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 shutdown_connections 2022-07-12T08:49:32.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:32.965+0000 7f08ef438700 1 -- 172.21.15.61:0/1754332293 wait complete. 2022-07-12T08:49:32.989 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:49:32.989 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:33.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 Processor -- start 2022-07-12T08:49:33.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 -- start start 2022-07-12T08:49:33.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980aca50 0x7f0b980aee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0b980b1200 0x7f0b98142970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0b980b1660 0x7f0b98142e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b980ace30 con 0x7f0b980b1660 2022-07-12T08:49:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b980ae420 con 0x7f0b980b1200 2022-07-12T08:49:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.093+0000 7f0b9da5c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b980ae6c0 con 0x7f0b980aca50 2022-07-12T08:49:33.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0b980b1660 0x7f0b98142e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0b980b1660 0x7f0b98142e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60332/0 (socket says 172.21.15.61:60332) 2022-07-12T08:49:33.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 -- 172.21.15.61:0/3792738474 learned_addr learned my addr 172.21.15.61:0/3792738474 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:33.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b96ffd700 1 --2- 172.21.15.61:0/3792738474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980aca50 0x7f0b980aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b8ffff700 1 --2- 172.21.15.61:0/3792738474 >> v2:172.21.15.111:3300/0 conn(0x7f0b980b1200 0x7f0b98142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 -- 172.21.15.61:0/3792738474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980aca50 msgr2=0x7f0b980aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:33.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 --2- 172.21.15.61:0/3792738474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980aca50 0x7f0b980aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 -- 172.21.15.61:0/3792738474 >> v2:172.21.15.111:3300/0 conn(0x7f0b980b1200 msgr2=0x7f0b98142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:33.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 --2- 172.21.15.61:0/3792738474 >> v2:172.21.15.111:3300/0 conn(0x7f0b980b1200 0x7f0b98142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 -- 172.21.15.61:0/3792738474 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b980ae960 con 0x7f0b980b1660 2022-07-12T08:49:33.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b8ffff700 1 --2- 172.21.15.61:0/3792738474 >> v2:172.21.15.111:3300/0 conn(0x7f0b980b1200 0x7f0b98142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:33.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b977fe700 1 --2- 172.21.15.61:0/3792738474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980b1660 0x7f0b98142e60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7f0b900032f0 tx=0x7f0b90009290).ready entity=mon.0 client_cookie=fbe86d66c12f294e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:33.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b967fc700 1 -- 172.21.15.61:0/3792738474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b9000b870 con 0x7f0b980b1660 2022-07-12T08:49:33.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b967fc700 1 -- 172.21.15.61:0/3792738474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b9000b9d0 con 0x7f0b980b1660 2022-07-12T08:49:33.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b967fc700 1 -- 172.21.15.61:0/3792738474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b9000bb40 con 0x7f0b980b1660 2022-07-12T08:49:33.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 -- 172.21.15.61:0/3792738474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980b1660 msgr2=0x7f0b98142e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:33.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/3792738474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980b1660 0x7f0b98142e60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7f0b900032f0 tx=0x7f0b90009290).stop 2022-07-12T08:49:33.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 -- 172.21.15.61:0/3792738474 shutdown_connections 2022-07-12T08:49:33.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/3792738474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980b1660 0x7f0b98142e60 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/3792738474 >> v2:172.21.15.111:3300/0 conn(0x7f0b980b1200 0x7f0b98142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/3792738474 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980aca50 0x7f0b980aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 -- 172.21.15.61:0/3792738474 >> 172.21.15.61:0/3792738474 conn(0x7f0b98062c00 msgr2=0x7f0b98062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:33.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 -- 172.21.15.61:0/3792738474 shutdown_connections 2022-07-12T08:49:33.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 -- 172.21.15.61:0/3792738474 wait complete. 2022-07-12T08:49:33.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 Processor -- start 2022-07-12T08:49:33.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.097+0000 7f0b9da5c700 1 -- start start 2022-07-12T08:49:33.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 0x7f0b98130f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b980b1200 0x7f0b98137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980b1660 0x7f0b98153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9813dca0 con 0x7f0b980aca50 2022-07-12T08:49:33.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9813ded0 con 0x7f0b980b1200 2022-07-12T08:49:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9813e170 con 0x7f0b980b1660 2022-07-12T08:49:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 0x7f0b98130f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 0x7f0b98130f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60334/0 (socket says 172.21.15.61:60334) 2022-07-12T08:49:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 -- 172.21.15.61:0/2966647063 learned_addr learned my addr 172.21.15.61:0/2966647063 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b8ffff700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b980b1200 0x7f0b98137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b977fe700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980b1660 0x7f0b98153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 -- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980b1660 msgr2=0x7f0b98153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980b1660 0x7f0b98153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 -- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b980b1200 msgr2=0x7f0b98137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b980b1200 0x7f0b98137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 -- 172.21.15.61:0/2966647063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b90009530 con 0x7f0b980aca50 2022-07-12T08:49:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b96ffd700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 0x7f0b98130f20 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7f0b880030b0 tx=0x7f0b88008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b88011780 con 0x7f0b980aca50 2022-07-12T08:49:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b880118e0 con 0x7f0b980aca50 2022-07-12T08:49:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b98130420 con 0x7f0b980aca50 2022-07-12T08:49:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b88011c40 con 0x7f0b980aca50 2022-07-12T08:49:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b98153ad0 con 0x7f0b980aca50 2022-07-12T08:49:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.101+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b98057460 con 0x7f0b980aca50 2022-07-12T08:49:33.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.105+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0b8801c460 con 0x7f0b980aca50 2022-07-12T08:49:33.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.105+0000 7f0b94ff9700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0b7c053920 0x7f0b7c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:33.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.105+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0b8806bf90 con 0x7f0b980aca50 2022-07-12T08:49:33.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.105+0000 7f0b8ffff700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0b7c053920 0x7f0b7c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:33.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.105+0000 7f0b8ffff700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0b7c053920 0x7f0b7c055de0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f0b8000c000 tx=0x7f0b80008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:33.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.109+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0b88042e90 con 0x7f0b980aca50 2022-07-12T08:49:33.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b980447a0 con 0x7f0b980aca50 2022-07-12T08:49:33.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b94ff9700 1 -- 172.21.15.61:0/2966647063 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f0b880464b0 con 0x7f0b980aca50 2022-07-12T08:49:33.275 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:33.276 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6668,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2585981801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2585981801},{"type":"v1","addr":"172.21.15.111:6835","nonce":2585981801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:49:24.635457+0000","modified":"2022-07-12T08:49:25.639064+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7671},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7671":{"gid":7671,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2867307616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2867307616},{"type":"v1","addr":"172.21.15.61:6833","nonce":2867307616}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-12T08:49:33.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0b7c053920 msgr2=0x7f0b7c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:33.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0b7c053920 0x7f0b7c055de0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f0b8000c000 tx=0x7f0b80008040).stop 2022-07-12T08:49:33.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 msgr2=0x7f0b98130f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:33.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 0x7f0b98130f20 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 rx=0x7f0b880030b0 tx=0x7f0b88008040).stop 2022-07-12T08:49:33.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 shutdown_connections 2022-07-12T08:49:33.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0b980b1660 0x7f0b98153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0b7c053920 0x7f0b7c055de0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0b980b1200 0x7f0b98137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 --2- 172.21.15.61:0/2966647063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b980aca50 0x7f0b98130f20 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:33.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 >> 172.21.15.61:0/2966647063 conn(0x7f0b98062c00 msgr2=0x7f0b98135250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:33.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 shutdown_connections 2022-07-12T08:49:33.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:33.269+0000 7f0b9da5c700 1 -- 172.21.15.61:0/2966647063 wait complete. 2022-07-12T08:49:33.281 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-12T08:49:33.489 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:33.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:49:24.635457+0000', 'modified': '2022-07-12T08:49:25.639064+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7671}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7671': {'gid': 7671, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2867307616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2867307616}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2867307616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:49:33.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:49:33.491 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:49:33.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:33.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:33.501 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:33.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:49:33.609 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":7671,"want_state":"up:active","state":"up:active","rank_uptime":8.9611003839999999,"mdsmap_epoch":62,"osdmap_epoch":110,"osdmap_epoch_barrier":109,"uptime":13.553861401000001} 2022-07-12T08:49:33.610 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 7671, "mdsmap_epoch": 62, "osdmap_epoch": 110, "osdmap_epoch_barrier": 109, "rank_uptime": 8.961100384, "state": "up:active", "uptime": 13.553861401, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:49:33.611 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:49:33.612 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:49:33.613 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:49:33.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:33.614 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:33.623 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:49:33.642 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:33.642 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:33.696 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:33.699 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:33.700 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:49:33.709 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:49:33.709 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:33.753 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:49:33.769 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:33.770 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:33.780 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:33.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:33.783 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:49:33.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21483]: starting ceph client 2022-07-12T08:49:33.830 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:49:33.825+0000 7fd5b9925080 -1 init, newargv = 0x5555fa8145c0 newargc=9 2022-07-12T08:49:33.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21483]: starting fuse 2022-07-12T08:49:34.940 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:49:34.959 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:49:34.960 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:49:35.018 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:49:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:49:35.020 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.022 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:49:35.029 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:49:35.030 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:49:35.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.032 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:49:35.033 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:49:35.034 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:49:35.034 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:49:35.035 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:49:35.036 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:35.037 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:49:35.037 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:49:35.039 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:49:35.039 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:35.040 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:49:35.040 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:49:35.042 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:49:35.042 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:49:35.042 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:35.043 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:49:35.043 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:49:35.044 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:49:35.044 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:49:35.045 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:49:35.045 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:49:35.045 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:49:35.046 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:49:35.094 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21483.asok 2022-07-12T08:49:35.099 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21483.asok 2022-07-12T08:49:35.100 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.101 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21483.asok status 2022-07-12T08:49:35.231 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:49:35.232 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:49:35.232 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:49:35.232 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:49:35.233 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:49:35.233 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:49:35.233 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:49:35.233 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21483", 2022-07-12T08:49:35.234 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:49:35.234 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:35.234 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:49:35.234 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:49:35.235 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 7896, 2022-07-12T08:49:35.235 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:49:35.235 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:49:35.235 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:49:35.236 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 7896 2022-07-12T08:49:35.236 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:35.236 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:49:35.236 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:49:35.236 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:49:35.237 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 961240154 2022-07-12T08:49:35.237 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:49:35.237 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:35.238 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:49:35.239 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:49:35.239 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:49:35.239 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 961240154 2022-07-12T08:49:35.240 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:49:35.240 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.7896 172.21.15.130:0/961240154", 2022-07-12T08:49:35.241 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/961240154", 2022-07-12T08:49:35.241 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:49:35.241 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 62, 2022-07-12T08:49:35.242 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 110, 2022-07-12T08:49:35.242 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:49:35.243 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:49:35.243 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:49:35.244 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:49:35.246 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.247 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.248 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:35.255 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:49:35.258 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:35.260 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.261 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:35.284 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.285 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.285 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:35.298 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:49:35.299 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:35.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.300 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.301 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:35.350 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:49:35.353 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:35.359 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:49:35.360 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:49:35.361 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:35.382 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22026]: fuse finished with error 0 and tester_r 0 2022-07-12T08:49:35.383 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.384 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.384 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:35.442 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:49:35.444 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:49:35.445 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:49:35.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:35.540 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:35.551 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.552 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.553 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:35.566 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:49:35.567 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:35.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.569 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.570 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:35.581 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:49:35.582 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:35.582 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:49:35.583 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:49:35.592 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:35.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21483]: fuse finished with error 0 and tester_r 0 2022-07-12T08:49:35.612 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:35.613 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:35.614 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:35.662 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:49:35.663 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:49:35.663 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:49:41.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:41.766 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:41.776 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:49:41.777 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:49:41.777 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:49:41.777 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:49:41.793 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:49:41.793 DEBUG:teuthology.parallel:result is None 2022-07-12T08:49:41.794 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:49:41.795 DEBUG:teuthology.parallel:result is None 2022-07-12T08:49:41.795 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2022-07-12T08:49:41.796 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2022-07-12T08:49:41.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.893+0000 7f91e9300700 1 Processor -- start 2022-07-12T08:49:41.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.893+0000 7f91e9300700 1 -- start start 2022-07-12T08:49:41.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.893+0000 7fcda3dd9700 1 Processor -- start 2022-07-12T08:49:41.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.893+0000 7fcda3dd9700 1 -- start start 2022-07-12T08:49:41.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e40b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f91e40aeb30 0x7f91e413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e40afbd0 con 0x7f91e40b0120 2022-07-12T08:49:41.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e40aefc0 con 0x7f91e40aeb30 2022-07-12T08:49:41.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40acb60 con 0x7f91e40b1180 2022-07-12T08:49:41.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f91e40b0120 0x7f91e40ae420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48564/0 (socket says 172.21.15.61:48564) 2022-07-12T08:49:41.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e37fe700 1 -- 172.21.15.61:0/3547879686 learned_addr learned my addr 172.21.15.61:0/3547879686 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e3fff700 1 --2- 172.21.15.61:0/3547879686 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e40b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e8aff700 1 --2- 172.21.15.61:0/3547879686 >> v2:172.21.15.111:3300/0 conn(0x7f91e40aeb30 0x7f91e413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e2ffd700 1 -- 172.21.15.61:0/3547879686 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346070128 0 0) 0x7f91e40afbd0 con 0x7f91e40b0120 2022-07-12T08:49:41.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e2ffd700 1 -- 172.21.15.61:0/3547879686 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d0003260 con 0x7f91e40b0120 2022-07-12T08:49:41.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e3fff700 1 -- 172.21.15.61:0/3547879686 >> v2:172.21.15.111:3300/0 conn(0x7f91e40aeb30 msgr2=0x7f91e413ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:41.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e3fff700 1 --2- 172.21.15.61:0/3547879686 >> v2:172.21.15.111:3300/0 conn(0x7f91e40aeb30 0x7f91e413ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e3fff700 1 -- 172.21.15.61:0/3547879686 >> v1:172.21.15.61:6789/0 conn(0x7f91e40b0120 legacy=0x7f91e40ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcd9c0ae080 0x7fcd9c05bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e3fff700 1 -- 172.21.15.61:0/3547879686 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40ad340 con 0x7f91e40b1180 2022-07-12T08:49:41.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c141f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c0b45f0 con 0x7fcd9c0b4210 2022-07-12T08:49:41.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c0ad1c0 con 0x7fcd9c0ae080 2022-07-12T08:49:41.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0b2490 con 0x7fcd9c0acb60 2022-07-12T08:49:41.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e8aff700 1 --2- 172.21.15.61:0/3547879686 >> v2:172.21.15.111:3300/0 conn(0x7f91e40aeb30 0x7f91e413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:41.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1374700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcd9c0ae080 0x7fcd9c05bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1b75700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcd9c0b4210 0x7fcd9c05f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48568/0 (socket says 172.21.15.61:48568) 2022-07-12T08:49:41.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1b75700 1 -- 172.21.15.61:0/2022137999 learned_addr learned my addr 172.21.15.61:0/2022137999 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:41.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e3fff700 1 --2- 172.21.15.61:0/3547879686 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e40b1560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f91d4002f70 tx=0x7f91d4006df0).ready entity=mon.2 client_cookie=70217dd88d15e558 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:41.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda2376700 1 --2- 172.21.15.61:0/2022137999 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c141f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda0b73700 1 -- 172.21.15.61:0/2022137999 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490149916 0 0) 0x7fcd9c0b45f0 con 0x7fcd9c0b4210 2022-07-12T08:49:41.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda0b73700 1 -- 172.21.15.61:0/2022137999 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd84003260 con 0x7fcd9c0b4210 2022-07-12T08:49:41.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e2ffd700 1 -- 172.21.15.61:0/3547879686 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d4014070 con 0x7f91e40b1180 2022-07-12T08:49:41.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e2ffd700 1 -- 172.21.15.61:0/3547879686 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4000e90 con 0x7f91e40b1180 2022-07-12T08:49:41.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e2ffd700 1 -- 172.21.15.61:0/3547879686 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d400fa90 con 0x7f91e40b1180 2022-07-12T08:49:41.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1374700 1 -- 172.21.15.61:0/2022137999 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 msgr2=0x7fcd9c141f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:41.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1374700 1 --2- 172.21.15.61:0/2022137999 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c141f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1374700 1 -- 172.21.15.61:0/2022137999 >> v1:172.21.15.61:6789/0 conn(0x7fcd9c0b4210 legacy=0x7fcd9c05f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1374700 1 -- 172.21.15.61:0/2022137999 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0b3c20 con 0x7fcd9c0ae080 2022-07-12T08:49:41.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda1374700 1 --2- 172.21.15.61:0/2022137999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c05bd10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7fcd8c002f70 tx=0x7fcd8c006df0).ready entity=mon.0 client_cookie=56df1eb349361a18 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda0b73700 1 -- 172.21.15.61:0/2022137999 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c00d4a0 con 0x7fcd9c0ae080 2022-07-12T08:49:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda0b73700 1 -- 172.21.15.61:0/2022137999 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c000e90 con 0x7fcd9c0ae080 2022-07-12T08:49:41.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda0b73700 1 -- 172.21.15.61:0/2022137999 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c00d7f0 con 0x7fcd9c0ae080 2022-07-12T08:49:41.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- 172.21.15.61:0/3547879686 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 msgr2=0x7f91e40b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 --2- 172.21.15.61:0/3547879686 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e40b1560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f91d4002f70 tx=0x7f91d4006df0).stop 2022-07-12T08:49:41.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- 172.21.15.61:0/2022137999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 msgr2=0x7fcd9c05bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- 172.21.15.61:0/3547879686 shutdown_connections 2022-07-12T08:49:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/2022137999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c05bd10 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 rx=0x7fcd8c002f70 tx=0x7fcd8c006df0).stop 2022-07-12T08:49:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 --2- 172.21.15.61:0/3547879686 >> v2:172.21.15.111:3300/0 conn(0x7f91e40aeb30 0x7f91e413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 --2- 172.21.15.61:0/3547879686 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e40b1560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- 172.21.15.61:0/3547879686 >> 172.21.15.61:0/3547879686 conn(0x7f91e40b5780 msgr2=0x7f91e40b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- 172.21.15.61:0/2022137999 shutdown_connections 2022-07-12T08:49:41.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/2022137999 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c141f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/2022137999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c05bd10 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- 172.21.15.61:0/2022137999 >> 172.21.15.61:0/2022137999 conn(0x7fcd9c064e10 msgr2=0x7fcd9c0651f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:41.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- 172.21.15.61:0/2022137999 shutdown_connections 2022-07-12T08:49:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7fcda3dd9700 1 -- 172.21.15.61:0/2022137999 wait complete. 2022-07-12T08:49:41.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- 172.21.15.61:0/3547879686 shutdown_connections 2022-07-12T08:49:41.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.897+0000 7f91e9300700 1 -- 172.21.15.61:0/3547879686 wait complete. 2022-07-12T08:49:41.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 Processor -- start 2022-07-12T08:49:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 Processor -- start 2022-07-12T08:49:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 -- start start 2022-07-12T08:49:41.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 -- start start 2022-07-12T08:49:41.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c0b6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c1405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd9c0b4210 0x7fcd9c140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c146ae0 con 0x7fcd9c0ae080 2022-07-12T08:49:41.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c147ac0 con 0x7fcd9c0b4210 2022-07-12T08:49:41.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda3dd9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c147820 con 0x7fcd9c0acb60 2022-07-12T08:49:41.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c1405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 0x7f91e4060720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e40b0120 0x7f91e4064fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c1405e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60356/0 (socket says 172.21.15.61:60356) 2022-07-12T08:49:41.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 -- 172.21.15.61:0/3653216638 learned_addr learned my addr 172.21.15.61:0/3653216638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:41.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e414d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e413efc0 con 0x7f91e40aeb30 2022-07-12T08:49:41.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e4065d70 con 0x7f91e40b0120 2022-07-12T08:49:41.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e9300700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e40654b0 con 0x7f91e40b1180 2022-07-12T08:49:41.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 0x7f91e4060720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda2376700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd9c0b4210 0x7fcd9c140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1b75700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c0b6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 0x7f91e4060720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60360/0 (socket says 172.21.15.61:60360) 2022-07-12T08:49:41.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 -- 172.21.15.61:0/3587948703 learned_addr learned my addr 172.21.15.61:0/3587948703 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:41.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e8aff700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e414d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e37fe700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e40b0120 0x7f91e4064fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 -- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 msgr2=0x7fcd9c0b6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c0b6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 -- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd9c0b4210 msgr2=0x7fcd9c140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd9c0b4210 0x7fcd9c140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 -- 172.21.15.61:0/3653216638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c00b180 con 0x7fcd9c0ae080 2022-07-12T08:49:41.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 -- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 msgr2=0x7f91e414d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e414d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcda1374700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c1405e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7fcd8c002f70 tx=0x7fcd8c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:41.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 -- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e40b0120 msgr2=0x7f91e4064fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:41.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e40b0120 0x7f91e4064fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:41.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 -- 172.21.15.61:0/3587948703 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc009470 con 0x7f91e40aeb30 2022-07-12T08:49:41.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7f91e3fff700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 0x7f91e4060720 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f91e4065640 tx=0x7f91d4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:41.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.901+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c012930 con 0x7fcd9c0ae080 2022-07-12T08:49:41.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c012a90 con 0x7fcd9c0ae080 2022-07-12T08:49:41.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd8c012df0 con 0x7fcd9c0ae080 2022-07-12T08:49:41.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c144d70 con 0x7fcd9c0ae080 2022-07-12T08:49:41.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c144860 con 0x7fcd9c0ae080 2022-07-12T08:49:41.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d4010b70 con 0x7f91e40aeb30 2022-07-12T08:49:41.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4010cd0 con 0x7f91e40aeb30 2022-07-12T08:49:41.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d401d640 con 0x7f91e40aeb30 2022-07-12T08:49:41.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d400b180 con 0x7f91e40aeb30 2022-07-12T08:49:41.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e4064810 con 0x7f91e40aeb30 2022-07-12T08:49:41.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c057460 con 0x7fcd9c0ae080 2022-07-12T08:49:41.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcd8c01c070 con 0x7fcd9c0ae080 2022-07-12T08:49:41.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcd8affd700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd7c053a60 0x7fcd7c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e4057460 con 0x7f91e40aeb30 2022-07-12T08:49:41.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcd8c06c9c0 con 0x7fcd9c0ae080 2022-07-12T08:49:41.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcda1b75700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd7c053a60 0x7fcd7c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f91d401d7a0 con 0x7f91e40aeb30 2022-07-12T08:49:41.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e17fa700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f91c4053ad0 0x7f91c4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:41.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7fcda1b75700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd7c053a60 0x7fcd7c055f20 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fcd9800d000 tx=0x7fcd98009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:41.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.905+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f91d406fde0 con 0x7f91e40aeb30 2022-07-12T08:49:41.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.909+0000 7f91e37fe700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f91c4053ad0 0x7f91c4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:41.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.909+0000 7f91e37fe700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f91c4053ad0 0x7f91c4055f90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f91d800d000 tx=0x7f91d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:41.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.909+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd8c043880 con 0x7fcd9c0ae080 2022-07-12T08:49:41.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:41.913+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f91d4046d20 con 0x7f91e40aeb30 2022-07-12T08:49:42.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.077+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fcd9c0447a0 con 0x7fcd9c0ae080 2022-07-12T08:49:42.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.089+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f91e4062b20 con 0x7f91e40aeb30 2022-07-12T08:49:42.299 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:49:42.289+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:49:42.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7fcd8affd700 1 -- 172.21.15.61:0/3653216638 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7671 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7fcd8c019030 con 0x7fcd9c0ae080 2022-07-12T08:49:42.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e17fa700 1 -- 172.21.15.61:0/3587948703 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6668 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f91d404a300 con 0x7f91e40aeb30 2022-07-12T08:49:42.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f91c4053ad0 msgr2=0x7f91c4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f91c4053ad0 0x7f91c4055f90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7f91d800d000 tx=0x7f91d8009040).stop 2022-07-12T08:49:42.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 msgr2=0x7f91e4060720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 0x7f91e4060720 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f91e4065640 tx=0x7f91d4016000).stop 2022-07-12T08:49:42.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 shutdown_connections 2022-07-12T08:49:42.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f91e40b1180 0x7f91e414d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f91c4053ad0 0x7f91c4055f90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f91e40b0120 0x7f91e4064fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 --2- 172.21.15.61:0/3587948703 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f91e40aeb30 0x7f91e4060720 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 >> 172.21.15.61:0/3587948703 conn(0x7f91e40b5780 msgr2=0x7f91e40b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:42.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 shutdown_connections 2022-07-12T08:49:42.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.297+0000 7f91e9300700 1 -- 172.21.15.61:0/3587948703 wait complete. 2022-07-12T08:49:42.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd7c053a60 msgr2=0x7fcd7c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd7c053a60 0x7fcd7c055f20 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7fcd9800d000 tx=0x7fcd98009040).stop 2022-07-12T08:49:42.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 msgr2=0x7fcd9c1405e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c1405e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 rx=0x7fcd8c002f70 tx=0x7fcd8c014000).stop 2022-07-12T08:49:42.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 shutdown_connections 2022-07-12T08:49:42.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd7c053a60 0x7fcd7c055f20 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd9c0b4210 0x7fcd9c140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd9c0ae080 0x7fcd9c1405e0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 --2- 172.21.15.61:0/3653216638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd9c0acb60 0x7fcd9c0b6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 >> 172.21.15.61:0/3653216638 conn(0x7fcd9c064e10 msgr2=0x7fcd9c060300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:42.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 shutdown_connections 2022-07-12T08:49:42.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.301+0000 7fcda3dd9700 1 -- 172.21.15.61:0/3653216638 wait complete. 2022-07-12T08:49:42.313 INFO:teuthology.orchestra.run.smithi061.stderr:failed mds gid 6668 2022-07-12T08:49:42.315 INFO:teuthology.orchestra.run.smithi061.stderr:failed mds gid 7671 2022-07-12T08:49:42.329 DEBUG:teuthology.parallel:result is '' 2022-07-12T08:49:42.334 DEBUG:teuthology.parallel:result is '' 2022-07-12T08:49:42.336 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:49:42.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd778e700 1 Processor -- start 2022-07-12T08:49:42.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd778e700 1 -- start start 2022-07-12T08:49:42.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd778e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd014baa0 0x7fecd014de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:42.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd778e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd00b56c0 con 0x7fecd00b52e0 2022-07-12T08:49:42.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd778e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd00b4bd0 con 0x7fecd00b4050 2022-07-12T08:49:42.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd778e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd0142f00 con 0x7fecd014baa0 2022-07-12T08:49:42.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd552a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fecd00b52e0 0x7fecd00b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48588/0 (socket says 172.21.15.61:48588) 2022-07-12T08:49:42.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.449+0000 7fecd552a700 1 -- 172.21.15.61:0/3472510787 learned_addr learned my addr 172.21.15.61:0/3472510787 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd5d2b700 1 --2- 172.21.15.61:0/3472510787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd014baa0 0x7fecd014de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561271886 0 0) 0x7fecd00b56c0 con 0x7fecd00b52e0 2022-07-12T08:49:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb4003260 con 0x7fecd00b52e0 2022-07-12T08:49:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991014724 0 0) 0x7fecd00b4bd0 con 0x7fecd00b4050 2022-07-12T08:49:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd00b56c0 con 0x7fecd00b4050 2022-07-12T08:49:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd5d2b700 1 -- 172.21.15.61:0/3472510787 >> v1:172.21.15.111:6789/0 conn(0x7fecd00b4050 legacy=0x7fecd00b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd5d2b700 1 -- 172.21.15.61:0/3472510787 >> v1:172.21.15.61:6789/0 conn(0x7fecd00b52e0 legacy=0x7fecd00b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd5d2b700 1 -- 172.21.15.61:0/3472510787 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd00b0e80 con 0x7fecd014baa0 2022-07-12T08:49:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd5d2b700 1 --2- 172.21.15.61:0/3472510787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd014baa0 0x7fecd014de90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fecc80032f0 tx=0x7fecc80091d0).ready entity=mon.2 client_cookie=536dd79353122750 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:42.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc800b640 con 0x7fecd014baa0 2022-07-12T08:49:42.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc800b7a0 con 0x7fecd014baa0 2022-07-12T08:49:42.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecc3fff700 1 -- 172.21.15.61:0/3472510787 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc800b910 con 0x7fecd014baa0 2022-07-12T08:49:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 -- 172.21.15.61:0/3472510787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd014baa0 msgr2=0x7fecd014de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 --2- 172.21.15.61:0/3472510787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd014baa0 0x7fecd014de90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fecc80032f0 tx=0x7fecc80091d0).stop 2022-07-12T08:49:42.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 -- 172.21.15.61:0/3472510787 shutdown_connections 2022-07-12T08:49:42.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 --2- 172.21.15.61:0/3472510787 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd014baa0 0x7fecd014de90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 -- 172.21.15.61:0/3472510787 >> 172.21.15.61:0/3472510787 conn(0x7fecd0067020 msgr2=0x7fecd0067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:42.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 -- 172.21.15.61:0/3472510787 shutdown_connections 2022-07-12T08:49:42.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.453+0000 7fecd778e700 1 -- 172.21.15.61:0/3472510787 wait complete. 2022-07-12T08:49:42.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 Processor -- start 2022-07-12T08:49:42.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 -- start start 2022-07-12T08:49:42.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd00b4050 0x7fecd00af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:42.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd00b52e0 0x7fecd0141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:42.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 0x7fecd0152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:42.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd013bd60 con 0x7fecd014baa0 2022-07-12T08:49:42.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd00aed70 con 0x7fecd00b52e0 2022-07-12T08:49:42.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd0144550 con 0x7fecd00b4050 2022-07-12T08:49:42.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 0x7fecd0152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:42.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 0x7fecd0152430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60380/0 (socket says 172.21.15.61:60380) 2022-07-12T08:49:42.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 -- 172.21.15.61:0/1653664278 learned_addr learned my addr 172.21.15.61:0/1653664278 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:42.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd552a700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd00b4050 0x7fecd00af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:42.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd4d29700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd00b52e0 0x7fecd0141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:42.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 -- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd00b4050 msgr2=0x7fecd00af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd00b4050 0x7fecd00af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 -- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd00b52e0 msgr2=0x7fecd0141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:42.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd00b52e0 0x7fecd0141930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:42.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 -- 172.21.15.61:0/1653664278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8009470 con 0x7fecd014baa0 2022-07-12T08:49:42.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd5d2b700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 0x7fecd0152430 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7fecc80032f0 tx=0x7fecc8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:42.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc800b640 con 0x7fecd014baa0 2022-07-12T08:49:42.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd01465b0 con 0x7fecd014baa0 2022-07-12T08:49:42.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc80205f0 con 0x7fecd014baa0 2022-07-12T08:49:42.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecc8020950 con 0x7fecd014baa0 2022-07-12T08:49:42.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.457+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd0141e20 con 0x7fecd014baa0 2022-07-12T08:49:42.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.461+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd0057460 con 0x7fecd014baa0 2022-07-12T08:49:42.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.461+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fecc801e070 con 0x7fecd014baa0 2022-07-12T08:49:42.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.461+0000 7fecc27fc700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539b0 0x7fecb8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:42.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.461+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fecc806ca30 con 0x7fecd014baa0 2022-07-12T08:49:42.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.461+0000 7fecd552a700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539b0 0x7fecb8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:42.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.461+0000 7fecd552a700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539b0 0x7fecb8055e70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fecc400d000 tx=0x7fecc4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:42.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.469+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fecc80438a0 con 0x7fecd014baa0 2022-07-12T08:49:42.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:42.633+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fecd00b0320 con 0x7fecd014baa0 2022-07-12T08:49:43.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.313+0000 7fecc27fc700 1 -- 172.21.15.61:0/1653664278 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7fecc80150a0 con 0x7fecd014baa0 2022-07-12T08:49:43.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539b0 msgr2=0x7fecb8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539b0 0x7fecb8055e70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fecc400d000 tx=0x7fecc4009040).stop 2022-07-12T08:49:43.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 msgr2=0x7fecd0152430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 0x7fecd0152430 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7fecc80032f0 tx=0x7fecc8016000).stop 2022-07-12T08:49:43.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 shutdown_connections 2022-07-12T08:49:43.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fecd014baa0 0x7fecd0152430 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fecb80539b0 0x7fecb8055e70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fecd00b52e0 0x7fecd0141930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 --2- 172.21.15.61:0/1653664278 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fecd00b4050 0x7fecd00af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 >> 172.21.15.61:0/1653664278 conn(0x7fecd0067020 msgr2=0x7fecd013f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:43.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 shutdown_connections 2022-07-12T08:49:43.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.317+0000 7fecd778e700 1 -- 172.21.15.61:0/1653664278 wait complete. 2022-07-12T08:49:43.339 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:49:43.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.453+0000 7fcf89106700 1 Processor -- start 2022-07-12T08:49:43.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.453+0000 7fcf89106700 1 -- start start 2022-07-12T08:49:43.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf840a9cd0 0x7fcf840b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:43.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf840b1ea0 con 0x7fcf840b36c0 2022-07-12T08:49:43.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8405d000 con 0x7fcf84099e60 2022-07-12T08:49:43.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8405d2a0 con 0x7fcf840a9cd0 2022-07-12T08:49:43.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf82d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcf840b36c0 0x7fcf84140ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48602/0 (socket says 172.21.15.61:48602) 2022-07-12T08:49:43.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf82d9d700 1 -- 172.21.15.61:0/890675820 learned_addr learned my addr 172.21.15.61:0/890675820 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:43.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8359e700 1 --2- 172.21.15.61:0/890675820 >> v2:172.21.15.111:3300/0 conn(0x7fcf840a9cd0 0x7fcf840b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:43.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680303359 0 0) 0x7fcf840b1ea0 con 0x7fcf840b36c0 2022-07-12T08:49:43.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf64003260 con 0x7fcf840b36c0 2022-07-12T08:49:43.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665735472 0 0) 0x7fcf8405d000 con 0x7fcf84099e60 2022-07-12T08:49:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf840b1ea0 con 0x7fcf84099e60 2022-07-12T08:49:43.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8359e700 1 -- 172.21.15.61:0/890675820 >> v1:172.21.15.111:6789/0 conn(0x7fcf84099e60 legacy=0x7fcf841418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8359e700 1 -- 172.21.15.61:0/890675820 >> v1:172.21.15.61:6789/0 conn(0x7fcf840b36c0 legacy=0x7fcf84140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8359e700 1 -- 172.21.15.61:0/890675820 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf84141fb0 con 0x7fcf840a9cd0 2022-07-12T08:49:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8359e700 1 --2- 172.21.15.61:0/890675820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840a9cd0 0x7fcf840b2300 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fcf7c0032f0 tx=0x7fcf7c0091d0).ready entity=mon.1 client_cookie=5ee93182c3bfc0d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:43.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf7c00b600 con 0x7fcf840a9cd0 2022-07-12T08:49:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c00b760 con 0x7fcf840a9cd0 2022-07-12T08:49:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf8259c700 1 -- 172.21.15.61:0/890675820 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf7c00b8d0 con 0x7fcf840a9cd0 2022-07-12T08:49:43.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- 172.21.15.61:0/890675820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840a9cd0 msgr2=0x7fcf840b2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 --2- 172.21.15.61:0/890675820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840a9cd0 0x7fcf840b2300 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fcf7c0032f0 tx=0x7fcf7c0091d0).stop 2022-07-12T08:49:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- 172.21.15.61:0/890675820 shutdown_connections 2022-07-12T08:49:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 --2- 172.21.15.61:0/890675820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840a9cd0 0x7fcf840b2300 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- 172.21.15.61:0/890675820 >> 172.21.15.61:0/890675820 conn(0x7fcf840b6e40 msgr2=0x7fcf840b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- 172.21.15.61:0/890675820 shutdown_connections 2022-07-12T08:49:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.457+0000 7fcf89106700 1 -- 172.21.15.61:0/890675820 wait complete. 2022-07-12T08:49:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 Processor -- start 2022-07-12T08:49:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 -- start start 2022-07-12T08:49:43.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 0x7fcf840ada20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:43.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf840a9cd0 0x7fcf8413eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:43.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840b36c0 0x7fcf8413f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:43.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf841541d0 con 0x7fcf84099e60 2022-07-12T08:49:43.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf840acd40 con 0x7fcf840b36c0 2022-07-12T08:49:43.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf89106700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf84153d70 con 0x7fcf840a9cd0 2022-07-12T08:49:43.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf82d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 0x7fcf840ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:43.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf82d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 0x7fcf840ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60388/0 (socket says 172.21.15.61:60388) 2022-07-12T08:49:43.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf82d9d700 1 -- 172.21.15.61:0/926026692 learned_addr learned my addr 172.21.15.61:0/926026692 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:43.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf7bfff700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf840a9cd0 0x7fcf8413eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:43.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.461+0000 7fcf8359e700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840b36c0 0x7fcf8413f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:43.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf82d9d700 1 -- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf840a9cd0 msgr2=0x7fcf8413eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf82d9d700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf840a9cd0 0x7fcf8413eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf82d9d700 1 -- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840b36c0 msgr2=0x7fcf8413f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:43.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf82d9d700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840b36c0 0x7fcf8413f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:43.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf82d9d700 1 -- 172.21.15.61:0/926026692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c009470 con 0x7fcf84099e60 2022-07-12T08:49:43.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf8359e700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840b36c0 0x7fcf8413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:43.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf82d9d700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 0x7fcf840ada20 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fcf74001110 tx=0x7fcf74007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:43.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf740070d0 con 0x7fcf84099e60 2022-07-12T08:49:43.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf74007230 con 0x7fcf84099e60 2022-07-12T08:49:43.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf840acaa0 con 0x7fcf84099e60 2022-07-12T08:49:43.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf74005610 con 0x7fcf84099e60 2022-07-12T08:49:43.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8413f530 con 0x7fcf84099e60 2022-07-12T08:49:43.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf84057460 con 0x7fcf84099e60 2022-07-12T08:49:43.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcf7401a060 con 0x7fcf84099e60 2022-07-12T08:49:43.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.465+0000 7fcf80d99700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf68053920 0x7fcf68055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:43.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.469+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcf7406b1b0 con 0x7fcf84099e60 2022-07-12T08:49:43.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.469+0000 7fcf7bfff700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf68053920 0x7fcf68055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:43.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.469+0000 7fcf7bfff700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf68053920 0x7fcf68055de0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7fcf6c00d000 tx=0x7fcf6c007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:43.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.477+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf740420c0 con 0x7fcf84099e60 2022-07-12T08:49:43.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:43.609+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcf840447a0 con 0x7fcf84099e60 2022-07-12T08:49:44.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.329+0000 7fcf80d99700 1 -- 172.21.15.61:0/926026692 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v113) v1 ==== 185+0+0 (secure 0 0 0) 0x7fcf74018020 con 0x7fcf84099e60 2022-07-12T08:49:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf68053920 msgr2=0x7fcf68055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf68053920 0x7fcf68055de0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7fcf6c00d000 tx=0x7fcf6c007ad0).stop 2022-07-12T08:49:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 msgr2=0x7fcf840ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 0x7fcf840ada20 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fcf74001110 tx=0x7fcf74007ad0).stop 2022-07-12T08:49:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 shutdown_connections 2022-07-12T08:49:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcf68053920 0x7fcf68055de0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf840b36c0 0x7fcf8413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf840a9cd0 0x7fcf8413eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 --2- 172.21.15.61:0/926026692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf84099e60 0x7fcf840ada20 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 >> 172.21.15.61:0/926026692 conn(0x7fcf840b6e40 msgr2=0x7fcf840b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:44.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 shutdown_connections 2022-07-12T08:49:44.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.333+0000 7fcf89106700 1 -- 172.21.15.61:0/926026692 wait complete. 2022-07-12T08:49:44.342 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:49:44.352 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:49:44.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 Processor -- start 2022-07-12T08:49:44.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 -- start start 2022-07-12T08:49:44.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65e4142eb0 0x7f65e405c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:44.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e40b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:44.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e4143500 con 0x7f65e405d130 2022-07-12T08:49:44.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e40b65c0 con 0x7f65e4142eb0 2022-07-12T08:49:44.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e4143290 con 0x7f65e40b6130 2022-07-12T08:49:44.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e1d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f65e405d130 0x7f65e40b5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48614/0 (socket says 172.21.15.61:48614) 2022-07-12T08:49:44.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e1d9b700 1 -- 172.21.15.61:0/1147228819 learned_addr learned my addr 172.21.15.61:0/1147228819 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:44.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e259c700 1 --2- 172.21.15.61:0/1147228819 >> v2:172.21.15.61:3300/0 conn(0x7f65e4142eb0 0x7f65e405c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:44.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e2d9d700 1 --2- 172.21.15.61:0/1147228819 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e40b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:44.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.457+0000 7f65e159a700 1 -- 172.21.15.61:0/1147228819 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115507062 0 0) 0x7f65e4143500 con 0x7f65e405d130 2022-07-12T08:49:44.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e159a700 1 -- 172.21.15.61:0/1147228819 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c4003260 con 0x7f65e405d130 2022-07-12T08:49:44.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e259c700 1 -- 172.21.15.61:0/1147228819 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 msgr2=0x7f65e40b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:44.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e159a700 1 -- 172.21.15.61:0/1147228819 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557293273 0 0) 0x7f65c4003260 con 0x7f65e405d130 2022-07-12T08:49:44.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e259c700 1 --2- 172.21.15.61:0/1147228819 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e40b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e259c700 1 -- 172.21.15.61:0/1147228819 >> v1:172.21.15.61:6789/0 conn(0x7f65e405d130 legacy=0x7f65e40b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:44.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e259c700 1 -- 172.21.15.61:0/1147228819 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e4143760 con 0x7f65e4142eb0 2022-07-12T08:49:44.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e259c700 1 --2- 172.21.15.61:0/1147228819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e4142eb0 0x7f65e405c6d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f65d4001800 tx=0x7f65cc006df0).ready entity=mon.0 client_cookie=1b2c3980fc35ffd0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:44.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e159a700 1 -- 172.21.15.61:0/1147228819 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d4001660 con 0x7f65e4142eb0 2022-07-12T08:49:44.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e159a700 1 -- 172.21.15.61:0/1147228819 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc000e90 con 0x7f65e4142eb0 2022-07-12T08:49:44.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e159a700 1 -- 172.21.15.61:0/1147228819 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65cc012460 con 0x7f65e4142eb0 2022-07-12T08:49:44.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 -- 172.21.15.61:0/1147228819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e4142eb0 msgr2=0x7f65e405c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:44.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 --2- 172.21.15.61:0/1147228819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e4142eb0 0x7f65e405c6d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f65d4001800 tx=0x7f65cc006df0).stop 2022-07-12T08:49:44.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 -- 172.21.15.61:0/1147228819 shutdown_connections 2022-07-12T08:49:44.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 --2- 172.21.15.61:0/1147228819 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e40b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 --2- 172.21.15.61:0/1147228819 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e4142eb0 0x7f65e405c6d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 -- 172.21.15.61:0/1147228819 >> 172.21.15.61:0/1147228819 conn(0x7f65e4066d70 msgr2=0x7f65e4067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:44.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 -- 172.21.15.61:0/1147228819 shutdown_connections 2022-07-12T08:49:44.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.461+0000 7f65e359e700 1 -- 172.21.15.61:0/1147228819 wait complete. 2022-07-12T08:49:44.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 Processor -- start 2022-07-12T08:49:44.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 -- start start 2022-07-12T08:49:44.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 0x7f65e4145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:44.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e4154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:44.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65e4142eb0 0x7f65e4155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:44.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40af510 con 0x7f65e405d130 2022-07-12T08:49:44.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40af7b0 con 0x7f65e4142eb0 2022-07-12T08:49:44.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e40afa50 con 0x7f65e40b6130 2022-07-12T08:49:44.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 0x7f65e4145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:44.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 0x7f65e4145020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60402/0 (socket says 172.21.15.61:60402) 2022-07-12T08:49:44.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 -- 172.21.15.61:0/263464473 learned_addr learned my addr 172.21.15.61:0/263464473 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:44.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e1d9b700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e4154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:44.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e2d9d700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65e4142eb0 0x7f65e4155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:44.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 -- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 msgr2=0x7f65e4154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:44.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e4154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 -- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65e4142eb0 msgr2=0x7f65e4155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:44.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65e4142eb0 0x7f65e4155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:44.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 -- 172.21.15.61:0/263464473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc00b180 con 0x7f65e405d130 2022-07-12T08:49:44.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e2d9d700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65e4142eb0 0x7f65e4155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:44.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.465+0000 7f65e259c700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 0x7f65e4145020 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f65d4001800 tx=0x7f65cc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:44.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65cc00d9a0 con 0x7f65e405d130 2022-07-12T08:49:44.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00db00 con 0x7f65e405d130 2022-07-12T08:49:44.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e41518e0 con 0x7f65e405d130 2022-07-12T08:49:44.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65cc00de60 con 0x7f65e405d130 2022-07-12T08:49:44.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e41468d0 con 0x7f65e405d130 2022-07-12T08:49:44.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e40afc10 con 0x7f65e405d130 2022-07-12T08:49:44.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f65cc025030 con 0x7f65e405d130 2022-07-12T08:49:44.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65db7fe700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65c8053a40 0x7f65c8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:44.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f65cc06c8b0 con 0x7f65e405d130 2022-07-12T08:49:44.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65e1d9b700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65c8053a40 0x7f65c8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:44.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.469+0000 7f65e1d9b700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65c8053a40 0x7f65c8055f00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f65d400d000 tx=0x7f65d4001230).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:44.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.477+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65cc043970 con 0x7f65e405d130 2022-07-12T08:49:44.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:44.609+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f65e4057460 con 0x7f65e405d130 2022-07-12T08:49:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.341+0000 7f65db7fe700 1 -- 172.21.15.61:0/263464473 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v114) v1 ==== 133+0+0 (secure 0 0 0) 0x7f65cc023020 con 0x7f65e405d130 2022-07-12T08:49:45.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65c8053a40 msgr2=0x7f65c8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:45.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65c8053a40 0x7f65c8055f00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f65d400d000 tx=0x7f65d4001230).stop 2022-07-12T08:49:45.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 msgr2=0x7f65e4145020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:45.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 0x7f65e4145020 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f65d4001800 tx=0x7f65cc014000).stop 2022-07-12T08:49:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 shutdown_connections 2022-07-12T08:49:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65c8053a40 0x7f65c8055f00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65e4142eb0 0x7f65e4155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65e40b6130 0x7f65e4154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 --2- 172.21.15.61:0/263464473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e405d130 0x7f65e4145020 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 >> 172.21.15.61:0/263464473 conn(0x7f65e4066d70 msgr2=0x7f65e40b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 shutdown_connections 2022-07-12T08:49:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:45.345+0000 7f65e359e700 1 -- 172.21.15.61:0/263464473 wait complete. 2022-07-12T08:49:45.360 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:49:45.365 DEBUG:teuthology.orchestra.run.smithi121:> sudo rados -p cephfs_metadata put key /etc/fstab 2022-07-12T08:49:45.445 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.415+0000 7fd79541cb80 1 Processor -- start 2022-07-12T08:49:45.446 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.415+0000 7fd79541cb80 1 -- start start 2022-07-12T08:49:45.446 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.446 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.446 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1bc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.447 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15a0b57d0 con 0x55e15a1bc990 2022-07-12T08:49:45.447 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15a0a56f0 con 0x55e15a1c04a0 2022-07-12T08:49:45.447 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e15a1bed80 con 0x55e15a1b4bf0 2022-07-12T08:49:45.447 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.448 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79405a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1bc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.448 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793859700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.448 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1bbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.121:36398/0 (socket says 172.21.15.121:36398) 2022-07-12T08:49:45.449 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 -- 172.21.15.121:0/2358482648 learned_addr learned my addr 172.21.15.121:0/2358482648 (peer_addr_for_me v2:172.21.15.121:0/0) 2022-07-12T08:49:45.449 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 -- 172.21.15.121:0/2358482648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 msgr2=0x55e15a1b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:45.449 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.450 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 -- 172.21.15.121:0/2358482648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 msgr2=0x55e15a1bc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:45.450 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1bc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.451 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 -- 172.21.15.121:0/2358482648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c00a0e0 con 0x55e15a1bc990 2022-07-12T08:49:45.451 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1bbce0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7fd77c0011d0 tx=0x7fd77c001880).ready entity=mon.0 client_cookie=81ece7dd49236db2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:45.451 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd791855700 1 -- 172.21.15.121:0/2358482648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd77c00bb40 con 0x55e15a1bc990 2022-07-12T08:49:45.452 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd791855700 1 -- 172.21.15.121:0/2358482648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77c00bca0 con 0x55e15a1bc990 2022-07-12T08:49:45.452 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd791855700 1 -- 172.21.15.121:0/2358482648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd77c00be10 con 0x55e15a1bc990 2022-07-12T08:49:45.453 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- 172.21.15.121:0/2358482648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 msgr2=0x55e15a1bbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:45.453 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1bbce0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7fd77c0011d0 tx=0x7fd77c001880).stop 2022-07-12T08:49:45.454 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- 172.21.15.121:0/2358482648 shutdown_connections 2022-07-12T08:49:45.454 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1bc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.455 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1bbce0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.456 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- 172.21.15.121:0/2358482648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.458 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- 172.21.15.121:0/2358482648 >> 172.21.15.121:0/2358482648 conn(0x55e15a1ae400 msgr2=0x55e15a1b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:45.458 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- 172.21.15.121:0/2358482648 shutdown_connections 2022-07-12T08:49:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- 172.21.15.121:0/2358482648 wait complete. 2022-07-12T08:49:45.459 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 Processor -- start 2022-07-12T08:49:45.460 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- start start 2022-07-12T08:49:45.460 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.460 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.461 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.461 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15a1b3cb0 con 0x55e15a1bc990 2022-07-12T08:49:45.462 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e15a1c6590 con 0x55e15a1c04a0 2022-07-12T08:49:45.462 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79541cb80 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e15a1c6750 con 0x55e15a1b4bf0 2022-07-12T08:49:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793859700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793859700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.121:43344/0 (socket says 172.21.15.121:43344) 2022-07-12T08:49:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793859700 1 -- 172.21.15.121:0/1861050506 learned_addr learned my addr 172.21.15.121:0/1861050506 (peer_addr_for_me v2:172.21.15.121:0/0) 2022-07-12T08:49:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd79405a700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.419+0000 7fd793058700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793859700 1 -- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 msgr2=0x55e15a1c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793859700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x55e15a1c04a0 0x55e15a1c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.465 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793859700 1 -- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 msgr2=0x55e15a1cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:45.465 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793859700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793859700 1 -- 172.21.15.121:0/1861050506 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd780001e10 con 0x55e15a1b4bf0 2022-07-12T08:49:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793058700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55e15a1bc990 0x55e15a1cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd793859700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1cbf10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x55e15a1c76a0 tx=0x7fd784000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd791054700 1 -- 172.21.15.121:0/1861050506 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd784015070 con 0x55e15a1b4bf0 2022-07-12T08:49:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd791054700 1 -- 172.21.15.121:0/1861050506 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd784005030 con 0x55e15a1b4bf0 2022-07-12T08:49:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e15a09e270 con 0x55e15a1b4bf0 2022-07-12T08:49:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd791054700 1 -- 172.21.15.121:0/1861050506 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7840107e0 con 0x55e15a1b4bf0 2022-07-12T08:49:45.468 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.423+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e15a1c6ca0 con 0x55e15a1b4bf0 2022-07-12T08:49:45.468 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd791054700 1 -- 172.21.15.121:0/1861050506 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd784010940 con 0x55e15a1b4bf0 2022-07-12T08:49:45.468 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd791054700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7740525c0 0x7fd774054a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.469 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd791054700 1 -- 172.21.15.121:0/1861050506 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6075+0+0 (secure 0 0 0) 0x7fd784008070 con 0x55e15a1b4bf0 2022-07-12T08:49:45.469 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd793058700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7740525c0 0x7fd774054a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.469 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd79541cb80 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x55e15a1d3880 0x55e15a1d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:45.469 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 --> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] -- osd_op(unknown.0.0:1 23.0 23:0620e6f9:::key:head [writefull 0~617 in=617b] snapc 0=[] ondisk+write+known_if_redirected e114) v8 -- 0x55e15a1d6160 con 0x55e15a1d3880 2022-07-12T08:49:45.469 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd79405a700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x55e15a1d3880 0x55e15a1d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:45.470 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd793058700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7740525c0 0x7fd774054a80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fd77c01a000 tx=0x7fd77c001880).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:45.470 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:45.427+0000 7fd79405a700 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x55e15a1d3880 0x55e15a1d5c70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:46.467 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.459+0000 7fd791054700 1 -- 172.21.15.121:0/1861050506 <== osd.0 v2:172.21.15.61:6800/21668 1 ==== osd_map(115..115 src has 1..115) v4 ==== 320+0+0 (crc 0 0 0) 0x7fd78c00dae0 con 0x55e15a1d3880 2022-07-12T08:49:46.470 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79405a700 1 -- 172.21.15.121:0/1861050506 <== osd.0 v2:172.21.15.61:6800/21668 2 ==== osd_op_reply(1 key [writefull 0~617] v115'1 uv1 ondisk = 0) v8 ==== 147+0+0 (crc 0 0 0) 0x7fd78c00dca0 con 0x55e15a1d3880 2022-07-12T08:49:46.471 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x55e15a1d3880 msgr2=0x55e15a1d5c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.471 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x55e15a1d3880 0x55e15a1d5c70 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.471 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7740525c0 msgr2=0x7fd774054a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.471 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd7740525c0 0x7fd774054a80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7fd77c01a000 tx=0x7fd77c001880).stop 2022-07-12T08:49:46.472 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 msgr2=0x55e15a1cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.472 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 --2- 172.21.15.121:0/1861050506 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x55e15a1b4bf0 0x55e15a1cbf10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x55e15a1c76a0 tx=0x7fd784000bc0).stop 2022-07-12T08:49:46.472 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79405a700 1 -- 172.21.15.121:0/1861050506 reap_dead start 2022-07-12T08:49:46.472 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 shutdown_connections 2022-07-12T08:49:46.473 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 >> 172.21.15.121:0/1861050506 conn(0x55e15a1ae400 msgr2=0x55e15a1bf430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:46.473 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 shutdown_connections 2022-07-12T08:49:46.473 INFO:teuthology.orchestra.run.smithi121.stderr:2022-07-12T08:49:46.463+0000 7fd79541cb80 1 -- 172.21.15.121:0/1861050506 wait complete. 2022-07-12T08:49:46.475 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-12T08:49:46.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 Processor -- start 2022-07-12T08:49:46.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 -- start start 2022-07-12T08:49:46.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb740b4250 0x7fbb74066440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 0x7fbb74144350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb740ad1c0 0x7fbb74144840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740b4630 con 0x7fbb740b4250 2022-07-12T08:49:46.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb74066930 con 0x7fbb740ad1c0 2022-07-12T08:49:46.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7c40d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740acce0 con 0x7fbb740b2480 2022-07-12T08:49:46.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb740b4250 0x7fbb74066440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb799a8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 0x7fbb74144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb799a8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 0x7fbb74144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42606/0 (socket says 172.21.15.61:42606) 2022-07-12T08:49:46.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb799a8700 1 -- 172.21.15.61:0/1796171415 learned_addr learned my addr 172.21.15.61:0/1796171415 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:46.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a9aa700 1 --2- 172.21.15.61:0/1796171415 >> v2:172.21.15.111:3300/0 conn(0x7fbb740ad1c0 0x7fbb74144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb740b4250 0x7fbb74066440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60412/0 (socket says 172.21.15.61:60412) 2022-07-12T08:49:46.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 -- 172.21.15.61:0/1796171415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 msgr2=0x7fbb74144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:46.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 --2- 172.21.15.61:0/1796171415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 0x7fbb74144350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 -- 172.21.15.61:0/1796171415 >> v2:172.21.15.111:3300/0 conn(0x7fbb740ad1c0 msgr2=0x7fbb74144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:46.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 --2- 172.21.15.61:0/1796171415 >> v2:172.21.15.111:3300/0 conn(0x7fbb740ad1c0 0x7fbb74144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 -- 172.21.15.61:0/1796171415 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb74065fa0 con 0x7fbb740b4250 2022-07-12T08:49:46.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb799a8700 1 --2- 172.21.15.61:0/1796171415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 0x7fbb74144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:46.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a9aa700 1 --2- 172.21.15.61:0/1796171415 >> v2:172.21.15.111:3300/0 conn(0x7fbb740ad1c0 0x7fbb74144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:46.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb7a1a9700 1 --2- 172.21.15.61:0/1796171415 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74066440 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7fbb68002f70 tx=0x7fbb68006df0).ready entity=mon.0 client_cookie=94db156ea34e21ac server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:46.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb791a7700 1 -- 172.21.15.61:0/1796171415 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb6800d4a0 con 0x7fbb740b4250 2022-07-12T08:49:46.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb791a7700 1 -- 172.21.15.61:0/1796171415 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68000e90 con 0x7fbb740b4250 2022-07-12T08:49:46.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.577+0000 7fbb791a7700 1 -- 172.21.15.61:0/1796171415 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb6800d7f0 con 0x7fbb740b4250 2022-07-12T08:49:46.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- 172.21.15.61:0/1796171415 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 msgr2=0x7fbb74066440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/1796171415 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74066440 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7fbb68002f70 tx=0x7fbb68006df0).stop 2022-07-12T08:49:46.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- 172.21.15.61:0/1796171415 shutdown_connections 2022-07-12T08:49:46.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/1796171415 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740b2480 0x7fbb74144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/1796171415 >> v2:172.21.15.111:3300/0 conn(0x7fbb740ad1c0 0x7fbb74144840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/1796171415 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74066440 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- 172.21.15.61:0/1796171415 >> 172.21.15.61:0/1796171415 conn(0x7fbb74064e10 msgr2=0x7fbb74065220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:46.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- 172.21.15.61:0/1796171415 shutdown_connections 2022-07-12T08:49:46.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- 172.21.15.61:0/1796171415 wait complete. 2022-07-12T08:49:46.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 Processor -- start 2022-07-12T08:49:46.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- start start 2022-07-12T08:49:46.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740ad1c0 0x7fbb740b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb740b2480 0x7fbb740b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74113330 con 0x7fbb740b4250 2022-07-12T08:49:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740b81e0 con 0x7fbb740b2480 2022-07-12T08:49:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7c40d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb74113560 con 0x7fbb740ad1c0 2022-07-12T08:49:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7a9aa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb799a8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb740b2480 0x7fbb740b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7a1a9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740ad1c0 0x7fbb740b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7a9aa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74110ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60420/0 (socket says 172.21.15.61:60420) 2022-07-12T08:49:46.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.581+0000 7fbb7a9aa700 1 -- 172.21.15.61:0/461605499 learned_addr learned my addr 172.21.15.61:0/461605499 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:46.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7a9aa700 1 -- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740ad1c0 msgr2=0x7fbb740b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7a9aa700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740ad1c0 0x7fbb740b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7a9aa700 1 -- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb740b2480 msgr2=0x7fbb740b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7a9aa700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb740b2480 0x7fbb740b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7a9aa700 1 -- 172.21.15.61:0/461605499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c009470 con 0x7fbb740b4250 2022-07-12T08:49:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7a9aa700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74110ad0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7fbb6c00c000 tx=0x7fbb6c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb6c00c850 con 0x7fbb740b4250 2022-07-12T08:49:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c00c9b0 con 0x7fbb740b4250 2022-07-12T08:49:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7000b1e0 con 0x7fbb740b4250 2022-07-12T08:49:46.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb6c00cd10 con 0x7fbb740b4250 2022-07-12T08:49:46.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6800b180 con 0x7fbb740b4250 2022-07-12T08:49:46.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb74057460 con 0x7fbb740b4250 2022-07-12T08:49:46.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.585+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbb6c022070 con 0x7fbb740b4250 2022-07-12T08:49:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.589+0000 7fbb677fe700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb54053af0 0x7fbb54055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.589+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6075+0+0 (secure 0 0 0) 0x7fbb6c06c6a0 con 0x7fbb740b4250 2022-07-12T08:49:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.589+0000 7fbb7a1a9700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb54053af0 0x7fbb54055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.589+0000 7fbb7a1a9700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb54053af0 0x7fbb54055fb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fbb6801a000 tx=0x7fbb68014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:46.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.597+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb6c0434f0 con 0x7fbb740b4250 2022-07-12T08:49:46.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fbb7405cec0 con 0x7fbb740b4250 2022-07-12T08:49:46.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb677fe700 1 -- 172.21.15.61:0/461605499 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1594 (secure 0 0 0) 0x7fbb6c016070 con 0x7fbb740b4250 2022-07-12T08:49:46.776 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:46.777 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:46.777 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:46.778 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-12T08:49:46.779 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761052000256, 2022-07-12T08:49:46.779 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6533103616, 2022-07-12T08:49:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6533320704, 2022-07-12T08:49:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085115237161517143, 2022-07-12T08:49:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: "num_osds": 8, 2022-07-12T08:49:46.782 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_osds": 0, 2022-07-12T08:49:46.782 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_omap_osds": 0 2022-07-12T08:49:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: "stats_by_class": { 2022-07-12T08:49:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: "ssd": { 2022-07-12T08:49:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-12T08:49:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761052000256, 2022-07-12T08:49:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6533103616, 2022-07-12T08:49:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6533320704, 2022-07-12T08:49:46.786 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085115237161517143 2022-07-12T08:49:46.786 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:49:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "device_health_metrics", 2022-07-12T08:49:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 1, 2022-07-12T08:49:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335422976 2022-07-12T08:49:46.793 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:46.793 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:46.794 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:46.794 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "rbd", 2022-07-12T08:49:46.794 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 2, 2022-07-12T08:49:46.795 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:46.795 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335422976 2022-07-12T08:49:46.798 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:46.798 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_data", 2022-07-12T08:49:46.800 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 22, 2022-07-12T08:49:46.801 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:46.801 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:46.802 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:46.802 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:46.803 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:46.803 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:46.804 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335422976 2022-07-12T08:49:46.804 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:46.805 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:46.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb54053af0 msgr2=0x7fbb54055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb54053af0 0x7fbb54055fb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7fbb6801a000 tx=0x7fbb68014000).stop 2022-07-12T08:49:46.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 msgr2=0x7fbb74110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:46.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74110ad0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 rx=0x7fbb6c00c000 tx=0x7fbb6c0091d0).stop 2022-07-12T08:49:46.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 shutdown_connections 2022-07-12T08:49:46.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb740b4250 0x7fbb74110ad0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb54053af0 0x7fbb54055fb0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb740b2480 0x7fbb740b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 --2- 172.21.15.61:0/461605499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb740ad1c0 0x7fbb740b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 >> 172.21.15.61:0/461605499 conn(0x7fbb74064e10 msgr2=0x7fbb7413dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:46.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.769+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 shutdown_connections 2022-07-12T08:49:46.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:46.773+0000 7fbb7c40d700 1 -- 172.21.15.61:0/461605499 wait complete. 2022-07-12T08:49:46.816 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30)... 2022-07-12T08:49:51.818 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-12T08:49:51.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.920+0000 7f1a3ffff700 1 Processor -- start 2022-07-12T08:49:51.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.920+0000 7f1a3ffff700 1 -- start start 2022-07-12T08:49:51.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a400b23c0 0x7f1a40129520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:51.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40129a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:51.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a400b27a0 con 0x7f1a400ae110 2022-07-12T08:49:51.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a40140b20 con 0x7f1a400b23c0 2022-07-12T08:49:51.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40140dc0 con 0x7f1a40140640 2022-07-12T08:49:51.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a400b23c0 0x7f1a40129520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:51.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1a400ae110 0x7f1a4005bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48646/0 (socket says 172.21.15.61:48646) 2022-07-12T08:49:51.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3f7fe700 1 -- 172.21.15.61:0/1657103390 learned_addr learned my addr 172.21.15.61:0/1657103390 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:51.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3e7fc700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40129a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:51.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3dffb700 1 -- 172.21.15.61:0/1657103390 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369963908 0 0) 0x7f1a400b27a0 con 0x7f1a400ae110 2022-07-12T08:49:51.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3dffb700 1 -- 172.21.15.61:0/1657103390 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a20003260 con 0x7f1a400ae110 2022-07-12T08:49:51.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3dffb700 1 -- 172.21.15.61:0/1657103390 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148877149 0 0) 0x7f1a20003260 con 0x7f1a400ae110 2022-07-12T08:49:51.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3effd700 1 -- 172.21.15.61:0/1657103390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 msgr2=0x7f1a40129a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:51.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3effd700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40129a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:51.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3effd700 1 -- 172.21.15.61:0/1657103390 >> v1:172.21.15.61:6789/0 conn(0x7f1a400ae110 legacy=0x7f1a4005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:51.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3effd700 1 -- 172.21.15.61:0/1657103390 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a40141060 con 0x7f1a400b23c0 2022-07-12T08:49:51.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3e7fc700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40129a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:51.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3effd700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400b23c0 0x7f1a40129520 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f1a38001b80 tx=0x7f1a30006df0).ready entity=mon.0 client_cookie=2d3d02364bc538f5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:51.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3dffb700 1 -- 172.21.15.61:0/1657103390 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a380019e0 con 0x7f1a400b23c0 2022-07-12T08:49:51.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3dffb700 1 -- 172.21.15.61:0/1657103390 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30000e90 con 0x7f1a400b23c0 2022-07-12T08:49:51.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3dffb700 1 -- 172.21.15.61:0/1657103390 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a30012460 con 0x7f1a400b23c0 2022-07-12T08:49:51.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- 172.21.15.61:0/1657103390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400b23c0 msgr2=0x7f1a40129520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:51.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400b23c0 0x7f1a40129520 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7f1a38001b80 tx=0x7f1a30006df0).stop 2022-07-12T08:49:51.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- 172.21.15.61:0/1657103390 shutdown_connections 2022-07-12T08:49:51.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40129a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:51.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/1657103390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400b23c0 0x7f1a40129520 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:51.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- 172.21.15.61:0/1657103390 >> 172.21.15.61:0/1657103390 conn(0x7f1a40063390 msgr2=0x7f1a400b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:51.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- 172.21.15.61:0/1657103390 shutdown_connections 2022-07-12T08:49:51.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.924+0000 7f1a3ffff700 1 -- 172.21.15.61:0/1657103390 wait complete. 2022-07-12T08:49:51.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.928+0000 7f1a3ffff700 1 Processor -- start 2022-07-12T08:49:51.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.928+0000 7f1a3ffff700 1 -- start start 2022-07-12T08:49:51.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 0x7f1a401421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:51.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a400b23c0 0x7f1a40125210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:51.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40125700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:51.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40128f80 con 0x7f1a400ae110 2022-07-12T08:49:51.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40066430 con 0x7f1a400b23c0 2022-07-12T08:49:51.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a400666d0 con 0x7f1a40140640 2022-07-12T08:49:51.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 0x7f1a401421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:51.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 0x7f1a401421b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60430/0 (socket says 172.21.15.61:60430) 2022-07-12T08:49:51.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 -- 172.21.15.61:0/862720005 learned_addr learned my addr 172.21.15.61:0/862720005 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:51.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3e7fc700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a400b23c0 0x7f1a40125210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:51.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3f7fe700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40125700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:51.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 -- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 msgr2=0x7f1a40125700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:51.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40125700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:51.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 -- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a400b23c0 msgr2=0x7f1a40125210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:51.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a400b23c0 0x7f1a40125210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:51.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 -- 172.21.15.61:0/862720005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2800b180 con 0x7f1a400ae110 2022-07-12T08:49:51.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3effd700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 0x7f1a401421b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f1a38001b80 tx=0x7f1a30014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:51.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a30016070 con 0x7f1a400ae110 2022-07-12T08:49:51.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3000bec0 con 0x7f1a400ae110 2022-07-12T08:49:51.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3000b180 con 0x7f1a400ae110 2022-07-12T08:49:51.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3000ddf0 con 0x7f1a400ae110 2022-07-12T08:49:51.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a401426a0 con 0x7f1a400ae110 2022-07-12T08:49:51.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a40129e50 con 0x7f1a400ae110 2022-07-12T08:49:51.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.932+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1a30025030 con 0x7f1a400ae110 2022-07-12T08:49:51.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.936+0000 7f1a37fff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a24053920 0x7f1a24055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:51.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.936+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f1a3006cdd0 con 0x7f1a400ae110 2022-07-12T08:49:51.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.936+0000 7f1a3e7fc700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a24053920 0x7f1a24055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:51.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.936+0000 7f1a3e7fc700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a24053920 0x7f1a24055de0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f1a2800b000 tx=0x7f1a28006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:51.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:51.944+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a30043c90 con 0x7f1a400ae110 2022-07-12T08:49:52.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f1a40065c30 con 0x7f1a400ae110 2022-07-12T08:49:52.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a37fff700 1 -- 172.21.15.61:0/862720005 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1937 (secure 0 0 0) 0x7f1a30023070 con 0x7f1a400ae110 2022-07-12T08:49:52.123 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:52.123 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:52.123 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:52.123 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-12T08:49:52.124 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761050505216, 2022-07-12T08:49:52.124 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6534381568, 2022-07-12T08:49:52.124 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6534815744, 2022-07-12T08:49:52.124 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085134720429778099, 2022-07-12T08:49:52.125 INFO:teuthology.orchestra.run.smithi061.stdout: "num_osds": 8, 2022-07-12T08:49:52.125 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_osds": 0, 2022-07-12T08:49:52.125 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_omap_osds": 0 2022-07-12T08:49:52.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:52.125 INFO:teuthology.orchestra.run.smithi061.stdout: "stats_by_class": { 2022-07-12T08:49:52.126 INFO:teuthology.orchestra.run.smithi061.stdout: "ssd": { 2022-07-12T08:49:52.126 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-12T08:49:52.126 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761050505216, 2022-07-12T08:49:52.127 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6534381568, 2022-07-12T08:49:52.127 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6534815744, 2022-07-12T08:49:52.127 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085134720429778099 2022-07-12T08:49:52.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:52.128 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:52.128 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:49:52.128 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:52.128 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "device_health_metrics", 2022-07-12T08:49:52.129 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 1, 2022-07-12T08:49:52.129 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:52.129 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:52.129 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:52.129 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:52.130 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:52.130 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:52.130 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335488512 2022-07-12T08:49:52.131 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:52.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:52.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:52.131 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "rbd", 2022-07-12T08:49:52.131 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 2, 2022-07-12T08:49:52.132 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:52.132 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:52.132 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:52.132 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:52.132 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:52.133 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:52.133 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335488512 2022-07-12T08:49:52.133 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:52.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:52.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:52.134 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_data", 2022-07-12T08:49:52.134 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 22, 2022-07-12T08:49:52.134 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:52.134 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-12T08:49:52.135 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-12T08:49:52.135 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-12T08:49:52.135 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-12T08:49:52.135 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-12T08:49:52.136 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335488512 2022-07-12T08:49:52.136 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:52.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:52.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:52.137 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_metadata", 2022-07-12T08:49:52.137 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 23, 2022-07-12T08:49:52.137 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-12T08:49:52.137 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 617, 2022-07-12T08:49:52.138 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 1, 2022-07-12T08:49:52.138 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 1, 2022-07-12T08:49:52.138 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 617, 2022-07-12T08:49:52.138 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 8.5377721559254383e-10, 2022-07-12T08:49:52.139 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335488512 2022-07-12T08:49:52.139 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:52.139 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:52.139 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:52.139 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:52.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a24053920 msgr2=0x7f1a24055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a24053920 0x7f1a24055de0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f1a2800b000 tx=0x7f1a28006df0).stop 2022-07-12T08:49:52.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 msgr2=0x7f1a401421b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 0x7f1a401421b0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 rx=0x7f1a38001b80 tx=0x7f1a30014000).stop 2022-07-12T08:49:52.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 shutdown_connections 2022-07-12T08:49:52.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a40140640 0x7f1a40125700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a24053920 0x7f1a24055de0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a400b23c0 0x7f1a40125210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/862720005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ae110 0x7f1a401421b0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 >> 172.21.15.61:0/862720005 conn(0x7f1a40063390 msgr2=0x7f1a400af120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:52.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 shutdown_connections 2022-07-12T08:49:52.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.116+0000 7f1a3ffff700 1 -- 172.21.15.61:0/862720005 wait complete. 2022-07-12T08:49:52.145 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2022-07-12T08:49:52.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-12T08:49:52.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 Processor -- start 2022-07-12T08:49:52.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 -- start start 2022-07-12T08:49:52.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88d0064580 0x7f88d01460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0060580 0x7f88d0141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00b1170 con 0x7f88d00b0d90 2022-07-12T08:49:52.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d005f1e0 con 0x7f88d0064580 2022-07-12T08:49:52.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88d4874700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00b3180 con 0x7f88d0060580 2022-07-12T08:49:52.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88cffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0060580 0x7f88d0141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88ceffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88d0064580 0x7f88d01460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88ceffd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88d0064580 0x7f88d01460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39014/0 (socket says 172.21.15.61:39014) 2022-07-12T08:49:52.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88cffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0060580 0x7f88d0141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42638/0 (socket says 172.21.15.61:42638) 2022-07-12T08:49:52.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88ceffd700 1 -- 172.21.15.61:0/1929974412 learned_addr learned my addr 172.21.15.61:0/1929974412 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:52.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.248+0000 7f88cf7fe700 1 --1- 172.21.15.61:0/1929974412 >> v1:172.21.15.111:6789/0 conn(0x7f88d00b0d90 0x7f88d00ad5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34026/0 (socket says 172.21.15.61:34026) 2022-07-12T08:49:52.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ceffd700 1 -- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0060580 msgr2=0x7f88d0141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:52.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ceffd700 1 --2- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0060580 0x7f88d0141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ceffd700 1 -- 172.21.15.61:0/1929974412 >> v1:172.21.15.111:6789/0 conn(0x7f88d00b0d90 legacy=0x7f88d00ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ceffd700 1 -- 172.21.15.61:0/1929974412 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d00adce0 con 0x7f88d0064580 2022-07-12T08:49:52.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ceffd700 1 --2- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0064580 0x7f88d01460c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f88c4002f70 tx=0x7f88c4006eb0).ready entity=mon.1 client_cookie=c889c1f476158c5e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ce7fc700 1 -- 172.21.15.61:0/1929974412 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c400d570 con 0x7f88d0064580 2022-07-12T08:49:52.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ce7fc700 1 -- 172.21.15.61:0/1929974412 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c4000e90 con 0x7f88d0064580 2022-07-12T08:49:52.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88ce7fc700 1 -- 172.21.15.61:0/1929974412 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c400d900 con 0x7f88d0064580 2022-07-12T08:49:52.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 -- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0064580 msgr2=0x7f88d01460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 --2- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0064580 0x7f88d01460c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f88c4002f70 tx=0x7f88c4006eb0).stop 2022-07-12T08:49:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 -- 172.21.15.61:0/1929974412 shutdown_connections 2022-07-12T08:49:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 --2- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0060580 0x7f88d0141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 --2- 172.21.15.61:0/1929974412 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0064580 0x7f88d01460c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 -- 172.21.15.61:0/1929974412 >> 172.21.15.61:0/1929974412 conn(0x7f88d0061f50 msgr2=0x7f88d0063a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:52.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 -- 172.21.15.61:0/1929974412 shutdown_connections 2022-07-12T08:49:52.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 -- 172.21.15.61:0/1929974412 wait complete. 2022-07-12T08:49:52.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 Processor -- start 2022-07-12T08:49:52.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.252+0000 7f88d4874700 1 -- start start 2022-07-12T08:49:52.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0060580 0x7f88d01433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0064580 0x7f88d013f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88d00b0d90 0x7f88d01152b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00660a0 con 0x7f88d00b0d90 2022-07-12T08:49:52.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d01136b0 con 0x7f88d0060580 2022-07-12T08:49:52.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d0113950 con 0x7f88d0064580 2022-07-12T08:49:52.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88d00b0d90 0x7f88d01152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88d00b0d90 0x7f88d01152b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60448/0 (socket says 172.21.15.61:60448) 2022-07-12T08:49:52.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 -- 172.21.15.61:0/4023950285 learned_addr learned my addr 172.21.15.61:0/4023950285 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:52.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cf7fe700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0060580 0x7f88d01433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88ceffd700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0064580 0x7f88d013f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 -- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0064580 msgr2=0x7f88d013f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f88d0064580 0x7f88d013f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 -- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0060580 msgr2=0x7f88d01433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88d0060580 0x7f88d01433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 -- 172.21.15.61:0/4023950285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c400b1e0 con 0x7f88d00b0d90 2022-07-12T08:49:52.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88cffff700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88d00b0d90 0x7f88d01152b0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7f88c8006e00 tx=0x7f88c80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c800c640 con 0x7f88d00b0d90 2022-07-12T08:49:52.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c800c7a0 con 0x7f88d00b0d90 2022-07-12T08:49:52.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 -- 172.21.15.61:0/4023950285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d0113bf0 con 0x7f88d00b0d90 2022-07-12T08:49:52.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c800cb00 con 0x7f88d00b0d90 2022-07-12T08:49:52.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.256+0000 7f88d4874700 1 -- 172.21.15.61:0/4023950285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d0065ae0 con 0x7f88d00b0d90 2022-07-12T08:49:52.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.260+0000 7f88d4874700 1 -- 172.21.15.61:0/4023950285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d0114090 con 0x7f88d00b0d90 2022-07-12T08:49:52.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.260+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f88c800cc60 con 0x7f88d00b0d90 2022-07-12T08:49:52.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.260+0000 7f88ccff9700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88b0053ad0 0x7f88b0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.260+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f88c806c280 con 0x7f88d00b0d90 2022-07-12T08:49:52.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.260+0000 7f88cf7fe700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88b0053ad0 0x7f88b0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.260+0000 7f88cf7fe700 1 --2- 172.21.15.61:0/4023950285 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f88b0053ad0 0x7f88b0055f90 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f88c000d000 tx=0x7f88c0007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.264+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88c8043070 con 0x7f88d00b0d90 2022-07-12T08:49:52.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.432+0000 7f88d4874700 1 -- 172.21.15.61:0/4023950285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f88d0057460 con 0x7f88d00b0d90 2022-07-12T08:49:52.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.432+0000 7f88ccff9700 1 -- 172.21.15.61:0/4023950285 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=-22 pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. v64) v1 ==== 209+0+0 (secure 0 0 0) 0x7f88c8046690 con 0x7f88d00b0d90 2022-07-12T08:49:52.439 INFO:teuthology.orchestra.run.smithi061.stderr:Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. 2022-07-12T08:49:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-07-12T08:49:52.446 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:49:52.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 Processor -- start 2022-07-12T08:49:52.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 -- start start 2022-07-12T08:49:52.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c005cf30 0x7f14c013ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c0060f90 con 0x7f14c0098c50 2022-07-12T08:49:52.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c00b14e0 con 0x7f14c0060bb0 2022-07-12T08:49:52.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00b1780 con 0x7f14c005cf30 2022-07-12T08:49:52.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bf59e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f14c0098c50 0x7f14c0140490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48674/0 (socket says 172.21.15.61:48674) 2022-07-12T08:49:52.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bf59e700 1 -- 172.21.15.61:0/686381883 learned_addr learned my addr 172.21.15.61:0/686381883 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:52.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14be59c700 1 --2- 172.21.15.61:0/686381883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c005cf30 0x7f14c013ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870364143 0 0) 0x7f14c0060f90 con 0x7f14c0098c50 2022-07-12T08:49:52.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a4003260 con 0x7f14c0098c50 2022-07-12T08:49:52.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351725036 0 0) 0x7f14c00b14e0 con 0x7f14c0060bb0 2022-07-12T08:49:52.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0060f90 con 0x7f14c0060bb0 2022-07-12T08:49:52.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14be59c700 1 -- 172.21.15.61:0/686381883 >> v1:172.21.15.111:6789/0 conn(0x7f14c0060bb0 legacy=0x7f14c009a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14be59c700 1 -- 172.21.15.61:0/686381883 >> v1:172.21.15.61:6789/0 conn(0x7f14c0098c50 legacy=0x7f14c0140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14be59c700 1 -- 172.21.15.61:0/686381883 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c009a380 con 0x7f14c005cf30 2022-07-12T08:49:52.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14be59c700 1 --2- 172.21.15.61:0/686381883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c005cf30 0x7f14c013ffa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f14c009a180 tx=0x7f14b4006df0).ready entity=mon.2 client_cookie=8b6a72f0459863ec server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b400d5c0 con 0x7f14c005cf30 2022-07-12T08:49:52.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b4000e90 con 0x7f14c005cf30 2022-07-12T08:49:52.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14bdd9b700 1 -- 172.21.15.61:0/686381883 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b400d910 con 0x7f14c005cf30 2022-07-12T08:49:52.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 -- 172.21.15.61:0/686381883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c005cf30 msgr2=0x7f14c013ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.548+0000 7f14c49c3700 1 --2- 172.21.15.61:0/686381883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c005cf30 0x7f14c013ffa0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f14c009a180 tx=0x7f14b4006df0).stop 2022-07-12T08:49:52.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- 172.21.15.61:0/686381883 shutdown_connections 2022-07-12T08:49:52.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 --2- 172.21.15.61:0/686381883 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c005cf30 0x7f14c013ffa0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- 172.21.15.61:0/686381883 >> 172.21.15.61:0/686381883 conn(0x7f14c005bd10 msgr2=0x7f14c005c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:52.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- 172.21.15.61:0/686381883 shutdown_connections 2022-07-12T08:49:52.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- 172.21.15.61:0/686381883 wait complete. 2022-07-12T08:49:52.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 Processor -- start 2022-07-12T08:49:52.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- start start 2022-07-12T08:49:52.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 0x7f14c01464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c0060bb0 0x7f14c0062e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c0098c50 0x7f14c0143650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c0144f50 con 0x7f14c005cf30 2022-07-12T08:49:52.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c0063330 con 0x7f14c0098c50 2022-07-12T08:49:52.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14c49c3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c00652e0 con 0x7f14c0060bb0 2022-07-12T08:49:52.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 0x7f14c01464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 0x7f14c01464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60458/0 (socket says 172.21.15.61:60458) 2022-07-12T08:49:52.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 -- 172.21.15.61:0/2833425722 learned_addr learned my addr 172.21.15.61:0/2833425722 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:52.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bf59e700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c0098c50 0x7f14c0143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14be59c700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c0060bb0 0x7f14c0062e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 -- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c0060bb0 msgr2=0x7f14c0062e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c0060bb0 0x7f14c0062e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 -- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c0098c50 msgr2=0x7f14c0143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c0098c50 0x7f14c0143650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bed9d700 1 -- 172.21.15.61:0/2833425722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b400b180 con 0x7f14c005cf30 2022-07-12T08:49:52.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.552+0000 7f14bf59e700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c0098c50 0x7f14c0143650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:52.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14bed9d700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 0x7f14c01464a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f14b0001160 tx=0x7f14b0007b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b000f470 con 0x7f14c005cf30 2022-07-12T08:49:52.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b000f5d0 con 0x7f14c005cf30 2022-07-12T08:49:52.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c0065580 con 0x7f14c005cf30 2022-07-12T08:49:52.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14b000f930 con 0x7f14c005cf30 2022-07-12T08:49:52.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c0146ea0 con 0x7f14c005cf30 2022-07-12T08:49:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c0144cb0 con 0x7f14c005cf30 2022-07-12T08:49:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f14b0010070 con 0x7f14c005cf30 2022-07-12T08:49:52.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14affff700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f149c053a40 0x7f149c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.556+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f14b006b130 con 0x7f14c005cf30 2022-07-12T08:49:52.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.560+0000 7f14be59c700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f149c053a40 0x7f149c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.560+0000 7f14be59c700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f149c053a40 0x7f149c055f00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f14b4018000 tx=0x7f14b4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.564+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14c0144cb0 con 0x7f14c005cf30 2022-07-12T08:49:52.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.728+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f14c0144cb0 con 0x7f14c005cf30 2022-07-12T08:49:52.735 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:49:52.728+0000 7fec541a1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:49:52.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.736+0000 7f14affff700 1 -- 172.21.15.61:0/2833425722 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 23 and data pool 22 v65) v1 ==== 189+0+0 (secure 0 0 0) 0x7f14c0144cb0 con 0x7f14c005cf30 2022-07-12T08:49:52.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f149c053a40 msgr2=0x7f149c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f149c053a40 0x7f149c055f00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f14b4018000 tx=0x7f14b4014000).stop 2022-07-12T08:49:52.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 msgr2=0x7f14c01464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 0x7f14c01464a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f14b0001160 tx=0x7f14b0007b70).stop 2022-07-12T08:49:52.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 shutdown_connections 2022-07-12T08:49:52.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f149c053a40 0x7f149c055f00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c0098c50 0x7f14c0143650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14c0060bb0 0x7f14c0062e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 --2- 172.21.15.61:0/2833425722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14c005cf30 0x7f14c01464a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 >> 172.21.15.61:0/2833425722 conn(0x7f14c005bd10 msgr2=0x7f14c00ac3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:52.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 shutdown_connections 2022-07-12T08:49:52.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.740+0000 7f14c49c3700 1 -- 172.21.15.61:0/2833425722 wait complete. 2022-07-12T08:49:52.775 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 23 and data pool 22 2022-07-12T08:49:52.811 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:49:52.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 Processor -- start 2022-07-12T08:49:52.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 -- start start 2022-07-12T08:49:52.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f06a005f730 0x7f06a0143870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0142060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a005fb10 con 0x7f06a0142550 2022-07-12T08:49:52.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a00b1c60 con 0x7f06a005f730 2022-07-12T08:49:52.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f06a5d90700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a00602a0 con 0x7f06a00b3380 2022-07-12T08:49:52.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f069f7fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f06a005f730 0x7f06a0143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0142060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42664/0 (socket says 172.21.15.61:42664) 2022-07-12T08:49:52.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 -- 172.21.15.61:0/2829506184 learned_addr learned my addr 172.21.15.61:0/2829506184 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:52.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 -- 172.21.15.61:0/2829506184 >> v2:172.21.15.111:3300/0 conn(0x7f06a005f730 msgr2=0x7f06a0143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 --2- 172.21.15.61:0/2829506184 >> v2:172.21.15.111:3300/0 conn(0x7f06a005f730 0x7f06a0143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 -- 172.21.15.61:0/2829506184 >> v1:172.21.15.111:6789/0 conn(0x7f06a0142550 legacy=0x7f06a01464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 -- 172.21.15.61:0/2829506184 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0060540 con 0x7f06a00b3380 2022-07-12T08:49:52.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f069f7fe700 1 --2- 172.21.15.61:0/2829506184 >> v2:172.21.15.111:3300/0 conn(0x7f06a005f730 0x7f06a0143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f0697fff700 1 --2- 172.21.15.61:0/2829506184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0142060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0688002f70 tx=0x7f0688006eb0).ready entity=mon.2 client_cookie=5c220b378d9609a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.924+0000 7f069effd700 1 -- 172.21.15.61:0/2829506184 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f068800d570 con 0x7f06a00b3380 2022-07-12T08:49:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f069effd700 1 -- 172.21.15.61:0/2829506184 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688000e90 con 0x7f06a00b3380 2022-07-12T08:49:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f069effd700 1 -- 172.21.15.61:0/2829506184 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f068800d900 con 0x7f06a00b3380 2022-07-12T08:49:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- 172.21.15.61:0/2829506184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 msgr2=0x7f06a0142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 --2- 172.21.15.61:0/2829506184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0142060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0688002f70 tx=0x7f0688006eb0).stop 2022-07-12T08:49:52.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- 172.21.15.61:0/2829506184 shutdown_connections 2022-07-12T08:49:52.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 --2- 172.21.15.61:0/2829506184 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0142060 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 --2- 172.21.15.61:0/2829506184 >> v2:172.21.15.111:3300/0 conn(0x7f06a005f730 0x7f06a0143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- 172.21.15.61:0/2829506184 >> 172.21.15.61:0/2829506184 conn(0x7f06a00b6ca0 msgr2=0x7f06a00b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:52.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- 172.21.15.61:0/2829506184 shutdown_connections 2022-07-12T08:49:52.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- 172.21.15.61:0/2829506184 wait complete. 2022-07-12T08:49:52.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 Processor -- start 2022-07-12T08:49:52.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- start start 2022-07-12T08:49:52.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06a005f730 0x7f06a00b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0142550 0x7f06a01557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0152690 con 0x7f06a0142550 2022-07-12T08:49:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0152930 con 0x7f06a005f730 2022-07-12T08:49:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f06a5d90700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0155340 con 0x7f06a00b3380 2022-07-12T08:49:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f069ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0142550 0x7f06a01557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0099240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42668/0 (socket says 172.21.15.61:42668) 2022-07-12T08:49:52.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 -- 172.21.15.61:0/602427234 learned_addr learned my addr 172.21.15.61:0/602427234 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:52.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f069f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06a005f730 0x7f06a00b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 -- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06a005f730 msgr2=0x7f06a00b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06a005f730 0x7f06a00b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 -- 172.21.15.61:0/602427234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0142550 msgr2=0x7f06a01557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0142550 0x7f06a01557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 -- 172.21.15.61:0/602427234 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069000b180 con 0x7f06a00b3380 2022-07-12T08:49:52.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.928+0000 7f0697fff700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0099240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0688002f70 tx=0x7f0688015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0688011610 con 0x7f06a00b3380 2022-07-12T08:49:52.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688011770 con 0x7f06a00b3380 2022-07-12T08:49:52.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068800b1e0 con 0x7f06a00b3380 2022-07-12T08:49:52.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0688011ad0 con 0x7f06a00b3380 2022-07-12T08:49:52.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a0145ea0 con 0x7f06a00b3380 2022-07-12T08:49:52.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a0155500 con 0x7f06a00b3380 2022-07-12T08:49:52.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0688025030 con 0x7f06a00b3380 2022-07-12T08:49:52.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.932+0000 7f069d7fa700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f06840539d0 0x7f0684055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:52.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.936+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f068806d080 con 0x7f06a00b3380 2022-07-12T08:49:52.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.936+0000 7f069f7fe700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f06840539d0 0x7f0684055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:52.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.936+0000 7f069f7fe700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f06840539d0 0x7f0684055e90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f069000b000 tx=0x7f0690006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:52.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:52.944+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0688043e80 con 0x7f06a00b3380 2022-07-12T08:49:53.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.108+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f06a0152af0 con 0x7f06a00b3380 2022-07-12T08:49:53.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.744+0000 7f069d7fa700 1 -- 172.21.15.61:0/602427234 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v66) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0688023020 con 0x7f06a00b3380 2022-07-12T08:49:53.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f06840539d0 msgr2=0x7f0684055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:53.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f06840539d0 0x7f0684055e90 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f069000b000 tx=0x7f0690006df0).stop 2022-07-12T08:49:53.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 msgr2=0x7f06a0099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:53.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0099240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0688002f70 tx=0x7f0688015000).stop 2022-07-12T08:49:53.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 shutdown_connections 2022-07-12T08:49:53.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0142550 0x7f06a01557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06a00b3380 0x7f06a0099240 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f06840539d0 0x7f0684055e90 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 --2- 172.21.15.61:0/602427234 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06a005f730 0x7f06a00b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 >> 172.21.15.61:0/602427234 conn(0x7f06a00b6ca0 msgr2=0x7f06a0060c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:53.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 shutdown_connections 2022-07-12T08:49:53.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.748+0000 7f06a5d90700 1 -- 172.21.15.61:0/602427234 wait complete. 2022-07-12T08:49:53.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:49:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 Processor -- start 2022-07-12T08:49:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- start start 2022-07-12T08:49:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa6c40ab290 0x7fa6c40ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa6c40afa60 0x7fa6c40afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40abc70 con 0x7fa6c405f140 2022-07-12T08:49:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c405f580 con 0x7fa6c40afa60 2022-07-12T08:49:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c405c720 con 0x7fa6c40ab290 2022-07-12T08:49:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa6c40afa60 0x7fa6c40afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c2ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa6c405f140 0x7fa6c405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48698/0 (socket says 172.21.15.61:48698) 2022-07-12T08:49:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c2ffd700 1 -- 172.21.15.61:0/2058823461 learned_addr learned my addr 172.21.15.61:0/2058823461 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:53.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c37fe700 1 --2- 172.21.15.61:0/2058823461 >> v2:172.21.15.111:3300/0 conn(0x7fa6c40ab290 0x7fa6c40ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:53.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c3fff700 1 -- 172.21.15.61:0/2058823461 >> v2:172.21.15.111:3300/0 conn(0x7fa6c40ab290 msgr2=0x7fa6c40ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:53.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c3fff700 1 --2- 172.21.15.61:0/2058823461 >> v2:172.21.15.111:3300/0 conn(0x7fa6c40ab290 0x7fa6c40ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c3fff700 1 -- 172.21.15.61:0/2058823461 >> v1:172.21.15.61:6789/0 conn(0x7fa6c405f140 legacy=0x7fa6c405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:53.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c3fff700 1 -- 172.21.15.61:0/2058823461 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c405c9c0 con 0x7fa6c40afa60 2022-07-12T08:49:53.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c3fff700 1 --2- 172.21.15.61:0/2058823461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c40afa60 0x7fa6c40afed0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7fa6bc0032f0 tx=0x7fa6bc0091d0).ready entity=mon.0 client_cookie=458b00843b730016 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:53.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c27fc700 1 -- 172.21.15.61:0/2058823461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6bc00b680 con 0x7fa6c40afa60 2022-07-12T08:49:53.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c27fc700 1 -- 172.21.15.61:0/2058823461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00b7e0 con 0x7fa6c40afa60 2022-07-12T08:49:53.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c27fc700 1 -- 172.21.15.61:0/2058823461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6bc00b950 con 0x7fa6c40afa60 2022-07-12T08:49:53.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2058823461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c40afa60 msgr2=0x7fa6c40afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:53.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2058823461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c40afa60 0x7fa6c40afed0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7fa6bc0032f0 tx=0x7fa6bc0091d0).stop 2022-07-12T08:49:53.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2058823461 shutdown_connections 2022-07-12T08:49:53.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2058823461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c40afa60 0x7fa6c40afed0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2058823461 >> v2:172.21.15.111:3300/0 conn(0x7fa6c40ab290 0x7fa6c40ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2058823461 >> 172.21.15.61:0/2058823461 conn(0x7fa6c40b1950 msgr2=0x7fa6c40b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:53.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2058823461 shutdown_connections 2022-07-12T08:49:53.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.888+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2058823461 wait complete. 2022-07-12T08:49:53.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 Processor -- start 2022-07-12T08:49:53.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 -- start start 2022-07-12T08:49:53.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 0x7fa6c4099e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6c40ab290 0x7fa6c40ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6c40afa60 0x7fa6c40ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4155ed0 con 0x7fa6c405f140 2022-07-12T08:49:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40af220 con 0x7fa6c40ab290 2022-07-12T08:49:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c4064a40 con 0x7fa6c40afa60 2022-07-12T08:49:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 0x7fa6c4099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 0x7fa6c4099e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60486/0 (socket says 172.21.15.61:60486) 2022-07-12T08:49:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 -- 172.21.15.61:0/2271965757 learned_addr learned my addr 172.21.15.61:0/2271965757 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:53.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c2ffd700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6c40ab290 0x7fa6c40ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:53.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c3fff700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6c40afa60 0x7fa6c40ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:53.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 -- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6c40afa60 msgr2=0x7fa6c40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:53.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6c40afa60 0x7fa6c40ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 -- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6c40ab290 msgr2=0x7fa6c40ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6c40ab290 0x7fa6c40ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 -- 172.21.15.61:0/2271965757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc009470 con 0x7fa6c405f140 2022-07-12T08:49:53.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c37fe700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 0x7fa6c4099e30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7fa6c40af360 tx=0x7fa6ac006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:53.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6ac010780 con 0x7fa6c405f140 2022-07-12T08:49:53.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac0108e0 con 0x7fa6c405f140 2022-07-12T08:49:53.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c40ae300 con 0x7fa6c405f140 2022-07-12T08:49:53.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6ac010c40 con 0x7fa6c405f140 2022-07-12T08:49:53.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.892+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c40638e0 con 0x7fa6c405f140 2022-07-12T08:49:53.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.896+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c4057460 con 0x7fa6c405f140 2022-07-12T08:49:53.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.896+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa6ac01c460 con 0x7fa6c405f140 2022-07-12T08:49:53.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.896+0000 7fa6c0ff9700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a80539b0 0x7fa6a8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:53.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.896+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa6ac06c420 con 0x7fa6c405f140 2022-07-12T08:49:53.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.896+0000 7fa6c2ffd700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a80539b0 0x7fa6a8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:53.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.896+0000 7fa6c2ffd700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a80539b0 0x7fa6a8055e70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fa6b400d000 tx=0x7fa6b4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:53.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:53.904+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa6ac043140 con 0x7fa6c405f140 2022-07-12T08:49:54.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.036+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa6c406b2b0 con 0x7fa6c405f140 2022-07-12T08:49:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.756+0000 7fa6c0ff9700 1 -- 172.21.15.61:0/2271965757 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 185+0+0 (secure 0 0 0) 0x7fa6ac046760 con 0x7fa6c405f140 2022-07-12T08:49:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.756+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a80539b0 msgr2=0x7fa6a8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.756+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a80539b0 0x7fa6a8055e70 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7fa6b400d000 tx=0x7fa6b4009040).stop 2022-07-12T08:49:54.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.756+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 msgr2=0x7fa6c4099e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:54.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.756+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 0x7fa6c4099e30 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7fa6c40af360 tx=0x7fa6ac006df0).stop 2022-07-12T08:49:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 shutdown_connections 2022-07-12T08:49:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa6c40afa60 0x7fa6c40ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa6a80539b0 0x7fa6a8055e70 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6c40ab290 0x7fa6c40ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 --2- 172.21.15.61:0/2271965757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6c405f140 0x7fa6c4099e30 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 >> 172.21.15.61:0/2271965757 conn(0x7fa6c40b1950 msgr2=0x7fa6c40b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:54.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 shutdown_connections 2022-07-12T08:49:54.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.760+0000 7fa6c8cf6700 1 -- 172.21.15.61:0/2271965757 wait complete. 2022-07-12T08:49:54.782 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:49:54.785 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:49:54.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.896+0000 7f539d8cc700 1 Processor -- start 2022-07-12T08:49:54.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.896+0000 7f539d8cc700 1 -- start start 2022-07-12T08:49:54.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d8cc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53980aefb0 0x7f5398140a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:54.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d8cc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f53980ac100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:54.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d8cc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539805c090 con 0x7f53980b0160 2022-07-12T08:49:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d8cc700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53980b0790 con 0x7f53980aefb0 2022-07-12T08:49:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d8cc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53980b0a30 con 0x7f53980abc70 2022-07-12T08:49:54.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d0cb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f53980ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:54.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53980aefb0 0x7f5398140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538ffff700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f53980b0160 0x7f53980b0ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34082/0 (socket says 172.21.15.61:34082) 2022-07-12T08:49:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f53980aefb0 0x7f5398140a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39068/0 (socket says 172.21.15.61:39068) 2022-07-12T08:49:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d0cb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f53980ac100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42694/0 (socket says 172.21.15.61:42694) 2022-07-12T08:49:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538ffff700 1 -- 172.21.15.61:0/2826177739 learned_addr learned my addr 172.21.15.61:0/2826177739 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538f7fe700 1 -- 172.21.15.61:0/2826177739 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176441371 0 0) 0x7f539805c090 con 0x7f53980b0160 2022-07-12T08:49:54.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538f7fe700 1 -- 172.21.15.61:0/2826177739 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5384003260 con 0x7f53980b0160 2022-07-12T08:49:54.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 -- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 msgr2=0x7f53980ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:54.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 --2- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f53980ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 -- 172.21.15.61:0/2826177739 >> v1:172.21.15.111:6789/0 conn(0x7f53980b0160 legacy=0x7f53980b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 -- 172.21.15.61:0/2826177739 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53980b15e0 con 0x7f53980aefb0 2022-07-12T08:49:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539d0cb700 1 --2- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f53980ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:49:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f539c8ca700 1 --2- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f5398140a10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f5394002f70 tx=0x7f5394006df0).ready entity=mon.1 client_cookie=2383315d662f839b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:54.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538f7fe700 1 -- 172.21.15.61:0/2826177739 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f539400d4a0 con 0x7f53980aefb0 2022-07-12T08:49:54.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538f7fe700 1 -- 172.21.15.61:0/2826177739 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394000e90 con 0x7f53980aefb0 2022-07-12T08:49:54.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.900+0000 7f538f7fe700 1 -- 172.21.15.61:0/2826177739 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f539400d7f0 con 0x7f53980aefb0 2022-07-12T08:49:54.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 -- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 msgr2=0x7f5398140a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:54.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 --2- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f5398140a10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f5394002f70 tx=0x7f5394006df0).stop 2022-07-12T08:49:54.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 -- 172.21.15.61:0/2826177739 shutdown_connections 2022-07-12T08:49:54.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 --2- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f53980ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 --2- 172.21.15.61:0/2826177739 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f5398140a10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 -- 172.21.15.61:0/2826177739 >> 172.21.15.61:0/2826177739 conn(0x7f53980b57c0 msgr2=0x7f53980b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 -- 172.21.15.61:0/2826177739 shutdown_connections 2022-07-12T08:49:54.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 -- 172.21.15.61:0/2826177739 wait complete. 2022-07-12T08:49:54.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 Processor -- start 2022-07-12T08:49:54.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.904+0000 7f539d8cc700 1 -- start start 2022-07-12T08:49:54.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f5398064330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:54.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f539813c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:54.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 0x7f53981448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:54.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398153cc0 con 0x7f53980b0160 2022-07-12T08:49:54.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398066ac0 con 0x7f53980aefb0 2022-07-12T08:49:54.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5398141f00 con 0x7f53980abc70 2022-07-12T08:49:54.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 0x7f53981448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:54.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 0x7f53981448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60504/0 (socket says 172.21.15.61:60504) 2022-07-12T08:49:54.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 -- 172.21.15.61:0/1902418274 learned_addr learned my addr 172.21.15.61:0/1902418274 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:54.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f538ffff700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f539813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:54.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539c8ca700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f5398064330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:54.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 -- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 msgr2=0x7f5398064330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:54.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f5398064330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 -- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 msgr2=0x7f539813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:54.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f539813c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:54.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 -- 172.21.15.61:0/1902418274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390009470 con 0x7f53980b0160 2022-07-12T08:49:54.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539c8ca700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f5398064330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:54.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d0cb700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 0x7f53981448b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f539000c000 tx=0x7f53900091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:54.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f539000c920 con 0x7f53980b0160 2022-07-12T08:49:54.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539000ca80 con 0x7f53980b0160 2022-07-12T08:49:54.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539400b180 con 0x7f53980b0160 2022-07-12T08:49:54.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f539000cde0 con 0x7f53980b0160 2022-07-12T08:49:54.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53980648a0 con 0x7f53980b0160 2022-07-12T08:49:54.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.908+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5398057460 con 0x7f53980b0160 2022-07-12T08:49:54.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.912+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5390022070 con 0x7f53980b0160 2022-07-12T08:49:54.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.912+0000 7f538dffb700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5378053a60 0x7f5378055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:54.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.912+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f539006c560 con 0x7f53980b0160 2022-07-12T08:49:54.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.912+0000 7f539c8ca700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5378053a60 0x7f5378055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:54.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.912+0000 7f539c8ca700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5378053a60 0x7f5378055f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f5394018000 tx=0x7f5394014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:54.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:54.916+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5390043620 con 0x7f53980b0160 2022-07-12T08:49:55.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.040+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f53981423a0 con 0x7f53980b0160 2022-07-12T08:49:55.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.768+0000 7f538dffb700 1 -- 172.21.15.61:0/1902418274 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v119) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5390046c40 con 0x7f53980b0160 2022-07-12T08:49:55.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5378053a60 msgr2=0x7f5378055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:55.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5378053a60 0x7f5378055f20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f5394018000 tx=0x7f5394014000).stop 2022-07-12T08:49:55.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 msgr2=0x7f53981448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 0x7f53981448b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f539000c000 tx=0x7f53900091d0).stop 2022-07-12T08:49:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 shutdown_connections 2022-07-12T08:49:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f53980b0160 0x7f53981448b0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5378053a60 0x7f5378055f20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f53980aefb0 0x7f539813c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 --2- 172.21.15.61:0/1902418274 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f53980abc70 0x7f5398064330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 >> 172.21.15.61:0/1902418274 conn(0x7f53980b57c0 msgr2=0x7f5398140a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:55.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 shutdown_connections 2022-07-12T08:49:55.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.772+0000 7f539d8cc700 1 -- 172.21.15.61:0/1902418274 wait complete. 2022-07-12T08:49:55.782 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:49:55.798 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-12T08:49:55.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.908+0000 7f1bed390700 1 Processor -- start 2022-07-12T08:49:55.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.908+0000 7f1bed390700 1 -- start start 2022-07-12T08:49:55.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be80af3d0 0x7f1be81442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1be805bd10 0x7f1be80b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1be80b08d0 0x7f1be80aba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be805fce0 con 0x7f1be805bd10 2022-07-12T08:49:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be80af7b0 con 0x7f1be80b08d0 2022-07-12T08:49:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80afea0 con 0x7f1be80af3d0 2022-07-12T08:49:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1be805bd10 0x7f1be80b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1be805bd10 0x7f1be80b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60512/0 (socket says 172.21.15.61:60512) 2022-07-12T08:49:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 -- 172.21.15.61:0/3374649498 learned_addr learned my addr 172.21.15.61:0/3374649498 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be6ffd700 1 --2- 172.21.15.61:0/3374649498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be80af3d0 0x7f1be81442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be77fe700 1 --2- 172.21.15.61:0/3374649498 >> v2:172.21.15.111:3300/0 conn(0x7f1be80b08d0 0x7f1be80aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 -- 172.21.15.61:0/3374649498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be80af3d0 msgr2=0x7f1be81442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 --2- 172.21.15.61:0/3374649498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be80af3d0 0x7f1be81442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 -- 172.21.15.61:0/3374649498 >> v2:172.21.15.111:3300/0 conn(0x7f1be80b08d0 msgr2=0x7f1be80aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 --2- 172.21.15.61:0/3374649498 >> v2:172.21.15.111:3300/0 conn(0x7f1be80b08d0 0x7f1be80aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 -- 172.21.15.61:0/3374649498 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be80b1cd0 con 0x7f1be805bd10 2022-07-12T08:49:55.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be67fc700 1 --2- 172.21.15.61:0/3374649498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be805bd10 0x7f1be80b03e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f1bdc002f70 tx=0x7f1bdc006eb0).ready entity=mon.0 client_cookie=2dd0c56e93614fbc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:55.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be5ffb700 1 -- 172.21.15.61:0/3374649498 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bdc00d570 con 0x7f1be805bd10 2022-07-12T08:49:55.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be5ffb700 1 -- 172.21.15.61:0/3374649498 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bdc000e90 con 0x7f1be805bd10 2022-07-12T08:49:55.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1be5ffb700 1 -- 172.21.15.61:0/3374649498 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bdc00d900 con 0x7f1be805bd10 2022-07-12T08:49:55.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- 172.21.15.61:0/3374649498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be805bd10 msgr2=0x7f1be80b03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:55.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- 172.21.15.61:0/3374649498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be805bd10 0x7f1be80b03e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 rx=0x7f1bdc002f70 tx=0x7f1bdc006eb0).stop 2022-07-12T08:49:55.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- 172.21.15.61:0/3374649498 shutdown_connections 2022-07-12T08:49:55.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- 172.21.15.61:0/3374649498 >> v2:172.21.15.111:3300/0 conn(0x7f1be80b08d0 0x7f1be80aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- 172.21.15.61:0/3374649498 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be805bd10 0x7f1be80b03e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 --2- 172.21.15.61:0/3374649498 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be80af3d0 0x7f1be81442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- 172.21.15.61:0/3374649498 >> 172.21.15.61:0/3374649498 conn(0x7f1be80b5a40 msgr2=0x7f1be80b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:55.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- 172.21.15.61:0/3374649498 shutdown_connections 2022-07-12T08:49:55.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.912+0000 7f1bed390700 1 -- 172.21.15.61:0/3374649498 wait complete. 2022-07-12T08:49:55.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 Processor -- start 2022-07-12T08:49:55.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 -- start start 2022-07-12T08:49:55.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be805bd10 0x7f1be80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be80af3d0 0x7f1be813bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 0x7f1be813c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80a80f0 con 0x7f1be80b08d0 2022-07-12T08:49:55.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8155fc0 con 0x7f1be80af3d0 2022-07-12T08:49:55.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be80a8d40 con 0x7f1be805bd10 2022-07-12T08:49:55.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 0x7f1be813c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 0x7f1be813c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60516/0 (socket says 172.21.15.61:60516) 2022-07-12T08:49:55.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 -- 172.21.15.61:0/449201722 learned_addr learned my addr 172.21.15.61:0/449201722 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:55.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be6ffd700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be805bd10 0x7f1be80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be67fc700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be80af3d0 0x7f1be813bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 -- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be805bd10 msgr2=0x7f1be80aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:55.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be805bd10 0x7f1be80aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 -- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be80af3d0 msgr2=0x7f1be813bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:55.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be80af3d0 0x7f1be813bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:55.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 -- 172.21.15.61:0/449201722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc00b1e0 con 0x7f1be80b08d0 2022-07-12T08:49:55.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be6ffd700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be805bd10 0x7f1be80aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:55.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1be77fe700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 0x7f1be813c220 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f1be000c000 tx=0x7f1be0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:55.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be0010780 con 0x7f1be80b08d0 2022-07-12T08:49:55.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be00108e0 con 0x7f1be80b08d0 2022-07-12T08:49:55.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be8064500 con 0x7f1be80b08d0 2022-07-12T08:49:55.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be0010c40 con 0x7f1be80b08d0 2022-07-12T08:49:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.916+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be80aedb0 con 0x7f1be80b08d0 2022-07-12T08:49:55.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.920+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1be001c460 con 0x7f1be80b08d0 2022-07-12T08:49:55.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.920+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be8064b40 con 0x7f1be80b08d0 2022-07-12T08:49:55.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.920+0000 7f1bd7fff700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bc4053a40 0x7f1bc4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:55.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.920+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f1be006c3f0 con 0x7f1be80b08d0 2022-07-12T08:49:55.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.920+0000 7f1be6ffd700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bc4053a40 0x7f1bc4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.920+0000 7f1be6ffd700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bc4053a40 0x7f1bc4055f00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f1bd800b000 tx=0x7f1bd8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:55.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:55.928+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1be00431e0 con 0x7f1be80b08d0 2022-07-12T08:49:56.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.088+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1be8064690 con 0x7f1be80b08d0 2022-07-12T08:49:56.095 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:49:56.088+0000 7fec541a1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:49:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.096+0000 7f1bd7fff700 1 -- 172.21.15.61:0/449201722 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 22 v67) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1be0046800 con 0x7f1be80b08d0 2022-07-12T08:49:56.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bc4053a40 msgr2=0x7f1bc4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:56.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bc4053a40 0x7f1bc4055f00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f1bd800b000 tx=0x7f1bd8008040).stop 2022-07-12T08:49:56.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 msgr2=0x7f1be813c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:56.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 0x7f1be813c220 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7f1be000c000 tx=0x7f1be0006df0).stop 2022-07-12T08:49:56.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 shutdown_connections 2022-07-12T08:49:56.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1be80b08d0 0x7f1be813c220 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1bc4053a40 0x7f1bc4055f00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1be80af3d0 0x7f1be813bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 --2- 172.21.15.61:0/449201722 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1be805bd10 0x7f1be80aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.100+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 >> 172.21.15.61:0/449201722 conn(0x7f1be80b5a40 msgr2=0x7f1be80abfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:56.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.104+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 shutdown_connections 2022-07-12T08:49:56.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.104+0000 7f1bed390700 1 -- 172.21.15.61:0/449201722 wait complete. 2022-07-12T08:49:56.121 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 24 and data pool 22 2022-07-12T08:49:56.123 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:49:56.124 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:56.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:56.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:56.204 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:56.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:49:56.251 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:49:56.252 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:56.252 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:56.253 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:56.263 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:49:56.264 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:49:56.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:56.281 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:49:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:56.302 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:49:56.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:56.303 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:56.303 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:56.320 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:49:56.320 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:49:56.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:56.324 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:49:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:56.375 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:49:56.376 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:49:56.377 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:49:56.377 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:49:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:56.397 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:49:56.400 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:49:56.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:49:56.403 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:49:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:49:56.456 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:49:56.456 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:49:56.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 Processor -- start 2022-07-12T08:49:56.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 -- start start 2022-07-12T08:49:56.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa33405fb90 0x7fa334146b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa3340677b0 0x7fa334141810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa334141d00 0x7fa334142190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa334142680 con 0x7fa33405fb90 2022-07-12T08:49:56.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa33405d230 con 0x7fa3340677b0 2022-07-12T08:49:56.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa333fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33405ff70 con 0x7fa334141d00 2022-07-12T08:49:56.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa33405fb90 0x7fa334146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa33405fb90 0x7fa334146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60526/0 (socket says 172.21.15.61:60526) 2022-07-12T08:49:56.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 -- 172.21.15.61:0/2756008558 learned_addr learned my addr 172.21.15.61:0/2756008558 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:56.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa3337fe700 1 --2- 172.21.15.61:0/2756008558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa334141d00 0x7fa334142190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa3327fc700 1 --2- 172.21.15.61:0/2756008558 >> v2:172.21.15.111:3300/0 conn(0x7fa3340677b0 0x7fa334141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 -- 172.21.15.61:0/2756008558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa334141d00 msgr2=0x7fa334142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:56.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 --2- 172.21.15.61:0/2756008558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa334141d00 0x7fa334142190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 -- 172.21.15.61:0/2756008558 >> v2:172.21.15.111:3300/0 conn(0x7fa3340677b0 msgr2=0x7fa334141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:56.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 --2- 172.21.15.61:0/2756008558 >> v2:172.21.15.111:3300/0 conn(0x7fa3340677b0 0x7fa334141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 -- 172.21.15.61:0/2756008558 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33405ccf0 con 0x7fa33405fb90 2022-07-12T08:49:56.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa332ffd700 1 --2- 172.21.15.61:0/2756008558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa33405fb90 0x7fa334146b30 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7fa32c002f70 tx=0x7fa32c006eb0).ready entity=mon.0 client_cookie=33054dc5f65584de server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:56.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa331ffb700 1 -- 172.21.15.61:0/2756008558 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa32c00d570 con 0x7fa33405fb90 2022-07-12T08:49:56.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa331ffb700 1 -- 172.21.15.61:0/2756008558 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c000e90 con 0x7fa33405fb90 2022-07-12T08:49:56.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.564+0000 7fa331ffb700 1 -- 172.21.15.61:0/2756008558 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa32c00d900 con 0x7fa33405fb90 2022-07-12T08:49:56.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- 172.21.15.61:0/2756008558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa33405fb90 msgr2=0x7fa334146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:56.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- 172.21.15.61:0/2756008558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa33405fb90 0x7fa334146b30 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 rx=0x7fa32c002f70 tx=0x7fa32c006eb0).stop 2022-07-12T08:49:56.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- 172.21.15.61:0/2756008558 shutdown_connections 2022-07-12T08:49:56.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- 172.21.15.61:0/2756008558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa334141d00 0x7fa334142190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- 172.21.15.61:0/2756008558 >> v2:172.21.15.111:3300/0 conn(0x7fa3340677b0 0x7fa334141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- 172.21.15.61:0/2756008558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa33405fb90 0x7fa334146b30 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- 172.21.15.61:0/2756008558 >> 172.21.15.61:0/2756008558 conn(0x7fa334063cd0 msgr2=0x7fa33405f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:56.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- 172.21.15.61:0/2756008558 shutdown_connections 2022-07-12T08:49:56.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- 172.21.15.61:0/2756008558 wait complete. 2022-07-12T08:49:56.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 Processor -- start 2022-07-12T08:49:56.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- start start 2022-07-12T08:49:56.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340677b0 0x7fa33413d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa334141d00 0x7fa3340b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 0x7fa334139000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33413af30 con 0x7fa3340677b0 2022-07-12T08:49:56.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa334152860 con 0x7fa334141d00 2022-07-12T08:49:56.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa333fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33413b160 con 0x7fa3340b5140 2022-07-12T08:49:56.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa332ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340677b0 0x7fa33413d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 0x7fa334139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 0x7fa334139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42724/0 (socket says 172.21.15.61:42724) 2022-07-12T08:49:56.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa332ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340677b0 0x7fa33413d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60532/0 (socket says 172.21.15.61:60532) 2022-07-12T08:49:56.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3327fc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa334141d00 0x7fa3340b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 -- 172.21.15.61:0/1994256994 learned_addr learned my addr 172.21.15.61:0/1994256994 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:56.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 -- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa334141d00 msgr2=0x7fa3340b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:56.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa334141d00 0x7fa3340b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 -- 172.21.15.61:0/1994256994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340677b0 msgr2=0x7fa33413d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:56.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340677b0 0x7fa33413d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:56.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 -- 172.21.15.61:0/1994256994 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c00b1e0 con 0x7fa3340b5140 2022-07-12T08:49:56.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa3337fe700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 0x7fa334139000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa32400c000 tx=0x7fa324006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:56.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.568+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa324011750 con 0x7fa3340b5140 2022-07-12T08:49:56.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3240118b0 con 0x7fa3340b5140 2022-07-12T08:49:56.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3341524e0 con 0x7fa3340b5140 2022-07-12T08:49:56.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa324011c10 con 0x7fa3340b5140 2022-07-12T08:49:56.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33413cb10 con 0x7fa3340b5140 2022-07-12T08:49:56.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33413d450 con 0x7fa3340b5140 2022-07-12T08:49:56.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa324011d70 con 0x7fa3340b5140 2022-07-12T08:49:56.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa32bfff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa30c053af0 0x7fa30c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:56.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa32406c200 con 0x7fa3340b5140 2022-07-12T08:49:56.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa332ffd700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa30c053af0 0x7fa30c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:56.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.572+0000 7fa332ffd700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa30c053af0 0x7fa30c055fb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fa32c01c000 tx=0x7fa32c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:56.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.580+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa33413d450 con 0x7fa3340b5140 2022-07-12T08:49:56.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.708+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa33413d450 con 0x7fa3340b5140 2022-07-12T08:49:56.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.712+0000 7fa32bfff700 1 -- 172.21.15.61:0/1994256994 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v120) v1 ==== 81+0+33874 (secure 0 0 0) 0x7fa33413d450 con 0x7fa3340b5140 2022-07-12T08:49:56.721 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:56.722 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:49:56.722 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 120, 2022-07-12T08:49:56.722 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:49:56.723 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:49:56.723 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:49:56.094438+0000", 2022-07-12T08:49:56.724 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:49:56.724 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:49:56.724 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:49:56.725 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:49:56.725 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:49:56.725 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:49:56.726 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:49:56.726 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:49:56.740 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:49:56.740 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:56.741 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:49:56.741 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:49:56.741 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:49:56.742 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:49:56.742 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:49:56.742 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 24, 2022-07-12T08:49:56.742 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:49:56.743 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:49:56.743 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:49:56.743 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:49:56.743 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:49:56.743 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.744 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:49:56.744 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:49:56.744 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:49:56.744 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:56.744 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:56.745 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:56.745 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:56.745 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:56.745 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:56.746 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:56.746 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:56.746 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:49:56.746 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:49:56.746 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:49:56.747 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:49:56.755 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:49:56.755 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:56.755 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:56.756 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:56.756 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:56.756 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:56.756 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:56.757 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:56.757 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.757 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:49:56.757 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:56.757 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:56.758 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:56.758 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:56.758 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:56.758 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:56.759 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:56.759 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:56.759 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:56.759 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:56.760 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:56.760 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:56.760 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:56.760 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:56.760 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:56.761 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:56.762 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:56.763 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:56.763 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:56.763 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:56.764 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:56.764 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:56.765 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:56.765 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:56.765 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:56.766 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:56.766 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.767 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:56.767 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:56.768 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:56.770 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:56.774 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:56.776 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:56.777 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:56.777 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:56.778 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:56.778 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:56.779 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:56.779 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:49:56.780 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:49:56.780 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.781 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:56.781 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:49:56.785 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.786 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.787 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:49:56.787 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:49:56.787 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:49:56.788 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:56.788 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:56.789 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:56.789 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:56.790 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:56.791 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:56.791 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:56.792 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:56.793 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:56.793 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:56.794 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:56.794 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:56.795 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:56.795 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:56.796 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:56.796 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:56.803 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:56.804 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:56.804 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:56.805 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:56.805 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.806 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:49:56.806 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:56.807 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:56.807 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:56.807 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:56.808 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:56.808 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:56.808 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:56.809 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:56.809 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:56.810 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:56.810 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:56.810 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:56.811 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:56.813 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:56.813 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:56.813 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:56.814 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:56.814 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:56.815 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:56.815 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:56.815 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:56.816 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:56.816 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:56.817 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:56.817 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:56.818 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:56.818 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.819 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:56.819 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:56.820 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:56.820 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:56.820 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:56.821 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:56.821 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:56.822 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:56.822 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:56.823 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:56.823 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:56.824 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:49:56.824 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:56.824 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:49:56.825 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.826 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.827 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 22, 2022-07-12T08:49:56.827 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:49:56.828 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:49:23.643810+0000", 2022-07-12T08:49:56.828 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:49:56.829 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:49:56.829 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:56.830 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:56.830 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:56.830 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:56.831 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:56.831 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:56.832 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:56.832 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:56.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:56.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:56.834 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:56.834 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:56.835 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:56.835 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:56.836 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:56.836 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:56.837 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:56.837 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:56.837 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "120", 2022-07-12T08:49:56.839 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:56.839 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:56.839 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:56.840 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:56.841 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:56.841 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:56.841 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:56.842 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:56.842 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:56.842 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:56.843 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:56.843 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:56.843 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:56.843 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:56.843 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:56.844 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:56.844 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:56.844 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:56.844 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:56.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:56.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:56.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:56.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:56.845 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:56.846 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:56.846 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:56.846 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.846 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:56.847 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:56.847 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:56.847 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:56.847 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:56.847 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:56.848 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:56.848 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:56.848 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:56.848 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:56.849 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:56.849 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:49:56.849 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:56.849 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:49:56.849 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:49:56.850 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.850 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.850 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.850 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.851 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 24, 2022-07-12T08:49:56.851 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:49:56.851 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:49:55.099876+0000", 2022-07-12T08:49:56.851 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 32769, 2022-07-12T08:49:56.852 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool,creating", 2022-07-12T08:49:56.852 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:49:56.852 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:49:56.852 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:49:56.853 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:49:56.853 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:49:56.853 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:49:56.853 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:49:56.853 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:49:56.854 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:49:56.854 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:49:56.854 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:49:56.854 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:49:56.855 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:49:56.855 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:49:56.855 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:49:56.855 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:49:56.855 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:49:56.856 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:49:56.856 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.856 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "120", 2022-07-12T08:49:56.856 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:49:56.857 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:49:56.857 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:49:56.857 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:49:56.857 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:49:56.857 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:49:56.858 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:49:56.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:49:56.858 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:49:56.858 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:49:56.859 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:49:56.859 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:49:56.859 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:49:56.859 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:49:56.860 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:49:56.860 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:49:56.860 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:49:56.860 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:49:56.860 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:49:56.861 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:49:56.861 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:49:56.861 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:49:56.861 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:49:56.862 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:49:56.862 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:49:56.862 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:49:56.863 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.863 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:49:56.863 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:49:56.864 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:49:56.864 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:49:56.865 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:49:56.865 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:49:56.865 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:49:56.866 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:49:56.866 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:49:56.867 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:49:56.867 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:49:56.867 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:49:56.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:49:56.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:49:56.869 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:49:56.869 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.870 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:49:56.870 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:49:56.870 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:49:56.871 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.871 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.872 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.872 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:56.873 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:49:56.874 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.874 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:49:56.875 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:49:56.875 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:56.876 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:56.877 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:56.877 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:56.878 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:56.878 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:56.879 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:56.879 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:49:56.879 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:56.880 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:56.880 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:56.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.881 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.882 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.882 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:49:56.883 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.883 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.883 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.884 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:49:56.885 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.891 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.900 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.901 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.901 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:56.902 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.902 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.903 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.903 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:49:56.904 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.905 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.905 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.906 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.906 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:49:56.907 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.907 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.908 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.908 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.908 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:56.909 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.909 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.910 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.910 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:49:56.911 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.911 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.912 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.912 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.913 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:49:56.914 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.914 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.915 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.915 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.916 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:56.916 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:49:56.919 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.920 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.920 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.921 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:49:56.921 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:49:56.935 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.935 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.936 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.936 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:49:56.937 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:49:56.937 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:49:56.937 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:49:56.938 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:56.939 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:56.940 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:56.941 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.941 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.942 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.943 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:49:56.943 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:49:56.944 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:56.945 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:56.946 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:56.946 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:56.946 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:56.947 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:56.947 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:56.948 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:49:56.948 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:56.948 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:56.949 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:56.950 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.950 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.951 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.951 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:49:56.952 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.952 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.952 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.953 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.953 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:49:56.954 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.954 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.955 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.955 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.958 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:56.959 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.959 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.959 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.961 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:49:56.962 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.962 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.963 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.963 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.964 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:49:56.964 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.965 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.965 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.966 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.966 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:56.967 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.967 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.967 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.970 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:49:56.971 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.972 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.973 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.973 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.974 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:49:56.974 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.975 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.975 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.976 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.976 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:56.977 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.977 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.978 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:56.978 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:49:56.979 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.980 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.981 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.981 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:56.983 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:49:56.983 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:49:56.983 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:56.984 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.985 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:49:56.985 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:49:56.985 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:49:56.986 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:49:56.986 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:56.987 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:56.987 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:56.988 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:56.988 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:56.988 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.989 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:49:56.989 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:49:56.990 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:56.991 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:56.991 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:56.992 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:56.992 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:56.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:56.995 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:56.995 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 114, 2022-07-12T08:49:56.996 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:56.996 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:56.999 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:56.999 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:56.999 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:56.999 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.000 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:49:57.000 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.000 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.000 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.001 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.001 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:49:57.001 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.001 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.002 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.002 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.002 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:57.002 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.003 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.003 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:49:57.003 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.003 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.004 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:49:57.004 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.005 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.005 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.005 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.005 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:57.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.006 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.006 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:49:57.007 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.007 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.007 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:49:57.008 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.008 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.008 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.009 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.009 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:57.009 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.010 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:49:57.010 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.010 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.011 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.011 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.011 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:49:57.011 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:49:57.011 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.012 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.012 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:49:57.012 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:49:57.013 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:49:57.013 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:49:57.013 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:57.013 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:57.013 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:57.014 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.014 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.014 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.014 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:49:57.015 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:49:57.015 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:57.015 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:57.015 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:57.016 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:57.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:57.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:57.016 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:57.017 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:49:57.017 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:57.017 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:57.017 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:57.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.018 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.018 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:49:57.019 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.019 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.019 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.020 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:49:57.020 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.020 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.020 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.021 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.021 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:57.021 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.021 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.022 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.022 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:49:57.022 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.022 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.022 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.023 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:49:57.023 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.023 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.024 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.024 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.024 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:57.024 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:49:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.026 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.026 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.026 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:49:57.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.027 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.027 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.028 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:57.028 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.028 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.028 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.029 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:49:57.029 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.030 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.030 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:49:57.031 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:49:57.031 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.032 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.032 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.032 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:49:57.032 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:49:57.033 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:49:57.033 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:49:57.033 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:57.033 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.035 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:49:57.035 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:49:57.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:57.036 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:57.036 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:57.036 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:57.036 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:57.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:57.037 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:57.037 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 114, 2022-07-12T08:49:57.037 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:57.038 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:57.038 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:57.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.039 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.039 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:49:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:49:57.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.041 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.041 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.042 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:57.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:49:57.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.043 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:49:57.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.044 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.044 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:49:57.046 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:49:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.049 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:49:57.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.049 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.050 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:49:57.050 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:49:57.051 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.051 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.052 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:49:57.052 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:49:57.052 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:49:57.053 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:49:57.053 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:57.053 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:57.054 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:57.054 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.054 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.054 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.055 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:49:57.055 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:49:57.055 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:57.056 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:57.056 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:57.056 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:57.056 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:57.057 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:57.057 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:57.057 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:49:57.057 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:57.058 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:57.058 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:57.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:49:57.059 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.060 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:49:57.061 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.061 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.061 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.062 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:57.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.062 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.062 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:49:57.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.063 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.064 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.064 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:49:57.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.064 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:57.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:49:57.066 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.067 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.068 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:49:57.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.069 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.069 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.070 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:57.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.072 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:49:57.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.073 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.074 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.074 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:49:57.075 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:49:57.075 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.075 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.076 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:49:57.077 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:49:57.077 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:49:57.078 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:49:57.078 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:57.078 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:57.079 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:57.079 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.080 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.080 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.081 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:49:57.081 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:49:57.081 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:57.082 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:57.082 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:57.083 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:57.083 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:57.084 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:57.084 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:57.084 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:49:57.085 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:57.085 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:57.086 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:57.086 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:49:57.088 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.088 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.090 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.090 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:49:57.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.091 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.091 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.091 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:57.091 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.092 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.092 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:49:57.092 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.093 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.093 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:49:57.093 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.094 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.094 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.094 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.095 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:57.095 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.095 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.095 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.096 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:49:57.096 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:49:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.098 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.098 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:57.098 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.099 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.099 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:49:57.099 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:49:57.100 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:49:57.101 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.101 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.102 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:49:57.102 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:49:57.102 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:49:57.102 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:49:57.103 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:57.103 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:57.103 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:57.103 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.104 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.104 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:49:57.104 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:49:57.105 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:49:57.105 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:49:57.105 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:49:57.105 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:49:57.106 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:49:57.106 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:49:57.106 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:49:57.106 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:49:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:49:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:49:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:49:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.108 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:49:57.108 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.109 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:49:57.109 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.110 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:49:57.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.111 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:49:57.111 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.112 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.112 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:49:57.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.113 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.113 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.114 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.114 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:49:57.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.114 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.115 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.115 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:49:57.115 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.115 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.116 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.116 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:49:57.116 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.116 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.117 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.117 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:49:57.117 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:49:57.118 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.118 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:49:57.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:49:57.118 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:49:57.119 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:49:57.120 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:49:57.120 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.120 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.121 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:49:57.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:49:57.121 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:49:57.122 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:49:57.122 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:49:57.122 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:49:57.123 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:49:57.123 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:49:57.123 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.123 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:57.124 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:49:57.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.124 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:49:57.124 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.125 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.125 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.125 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.125 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.126 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:49:57.126 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.126 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.127 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.127 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:49:57.127 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.128 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.128 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.128 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.128 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.128 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:49:57.129 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.130 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:49:57.130 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.130 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.131 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.131 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.131 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.132 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:49:57.132 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.132 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.133 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.133 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:49:57.134 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.134 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.135 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.135 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.136 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.136 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:49:57.137 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.138 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:49:57.138 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:49:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:49:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:49:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:49:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.145 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:49:57.145 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.145 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:49:57.145 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:49:57.146 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:49:57.146 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:49:57.147 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:49:57.147 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:49:57.147 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:49:57.148 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:49:57.148 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:49:57.149 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.149 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:49:57.149 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:49:57.150 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:49:57.150 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:49:57.151 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:49:57.151 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:49:57.151 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2867307616": "2022-07-13T08:49:42.084294+0000", 2022-07-12T08:49:57.151 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2867307616": "2022-07-13T08:49:42.084294+0000" 2022-07-12T08:49:57.152 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.152 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:49:57.153 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:49:57.153 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:49:57.153 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:49:57.153 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:49:57.154 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:49:57.154 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:49:57.154 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:49:57.154 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:49:57.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:49:57.155 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:49:57.155 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:49:57.155 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:49:57.156 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:49:57.156 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:49:57.156 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:49:57.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa30c053af0 msgr2=0x7fa30c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa30c053af0 0x7fa30c055fb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fa32c01c000 tx=0x7fa32c015000).stop 2022-07-12T08:49:57.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 msgr2=0x7fa334139000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 0x7fa334139000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fa32400c000 tx=0x7fa324006df0).stop 2022-07-12T08:49:57.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 shutdown_connections 2022-07-12T08:49:57.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3340b5140 0x7fa334139000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa30c053af0 0x7fa30c055fb0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa334141d00 0x7fa3340b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 --2- 172.21.15.61:0/1994256994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3340677b0 0x7fa33413d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 >> 172.21.15.61:0/1994256994 conn(0x7fa334063cd0 msgr2=0x7fa3340b0410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:57.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 shutdown_connections 2022-07-12T08:49:57.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:56.716+0000 7fa333fff700 1 -- 172.21.15.61:0/1994256994 wait complete. 2022-07-12T08:49:57.167 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:57.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.276+0000 7f7571a5f700 1 Processor -- start 2022-07-12T08:49:57.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.276+0000 7f7571a5f700 1 -- start start 2022-07-12T08:49:57.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7571a5f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f756c05c8e0 0x7f756c0ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7571a5f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c13f350 con 0x7f756c13e120 2022-07-12T08:49:57.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7571a5f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c13f5f0 con 0x7f756c0ad8b0 2022-07-12T08:49:57.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7571a5f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f756c0b1c60 con 0x7f756c05c8e0 2022-07-12T08:49:57.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7570a5d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f756c05c8e0 0x7f756c0ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f757125e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f756c13e120 0x7f756c13e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48758/0 (socket says 172.21.15.61:48758) 2022-07-12T08:49:57.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f757125e700 1 -- 172.21.15.61:0/1944074083 learned_addr learned my addr 172.21.15.61:0/1944074083 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:57.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f756b7fe700 1 -- 172.21.15.61:0/1944074083 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089947375 0 0) 0x7f756c13f350 con 0x7f756c13e120 2022-07-12T08:49:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f756b7fe700 1 -- 172.21.15.61:0/1944074083 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550003260 con 0x7f756c13e120 2022-07-12T08:49:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7570a5d700 1 -- 172.21.15.61:0/1944074083 >> v1:172.21.15.111:6789/0 conn(0x7f756c0ad8b0 legacy=0x7f756c0aed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7570a5d700 1 -- 172.21.15.61:0/1944074083 >> v1:172.21.15.61:6789/0 conn(0x7f756c13e120 legacy=0x7f756c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7570a5d700 1 -- 172.21.15.61:0/1944074083 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c13f890 con 0x7f756c05c8e0 2022-07-12T08:49:57.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f7570a5d700 1 --2- 172.21.15.61:0/1944074083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c05c8e0 0x7f756c0ae860 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f7560002f70 tx=0x7f7560006df0).ready entity=mon.0 client_cookie=452e81a04bc7a5cf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:57.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f756b7fe700 1 -- 172.21.15.61:0/1944074083 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f756000d610 con 0x7f756c05c8e0 2022-07-12T08:49:57.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f756b7fe700 1 -- 172.21.15.61:0/1944074083 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7560000e90 con 0x7f756c05c8e0 2022-07-12T08:49:57.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.280+0000 7f756b7fe700 1 -- 172.21.15.61:0/1944074083 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f756000d960 con 0x7f756c05c8e0 2022-07-12T08:49:57.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 -- 172.21.15.61:0/1944074083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c05c8e0 msgr2=0x7f756c0ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 --2- 172.21.15.61:0/1944074083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c05c8e0 0x7f756c0ae860 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f7560002f70 tx=0x7f7560006df0).stop 2022-07-12T08:49:57.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 -- 172.21.15.61:0/1944074083 shutdown_connections 2022-07-12T08:49:57.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 --2- 172.21.15.61:0/1944074083 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c05c8e0 0x7f756c0ae860 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 -- 172.21.15.61:0/1944074083 >> 172.21.15.61:0/1944074083 conn(0x7f756c0b5780 msgr2=0x7f756c0b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:57.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 -- 172.21.15.61:0/1944074083 shutdown_connections 2022-07-12T08:49:57.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 -- 172.21.15.61:0/1944074083 wait complete. 2022-07-12T08:49:57.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 Processor -- start 2022-07-12T08:49:57.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.284+0000 7f7571a5f700 1 -- start start 2022-07-12T08:49:57.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f756c05c8e0 0x7f756c060d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 0x7f756c140690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f756c13e120 0x7f756c140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c0651b0 con 0x7f756c0ad8b0 2022-07-12T08:49:57.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756c065460 con 0x7f756c13e120 2022-07-12T08:49:57.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756c065700 con 0x7f756c05c8e0 2022-07-12T08:49:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 0x7f756c140690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 0x7f756c140690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60544/0 (socket says 172.21.15.61:60544) 2022-07-12T08:49:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 -- 172.21.15.61:0/3413238011 learned_addr learned my addr 172.21.15.61:0/3413238011 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7570a5d700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f756c05c8e0 0x7f756c060d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f757125e700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f756c13e120 0x7f756c140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 -- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f756c05c8e0 msgr2=0x7f756c060d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f756c05c8e0 0x7f756c060d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 -- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f756c13e120 msgr2=0x7f756c140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f756c13e120 0x7f756c140b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 -- 172.21.15.61:0/3413238011 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756000b180 con 0x7f756c0ad8b0 2022-07-12T08:49:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7570a5d700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f756c05c8e0 0x7f756c060d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f756bfff700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 0x7f756c140690 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f756c0b3400 tx=0x7f7558007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:57.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7558012400 con 0x7f756c0ad8b0 2022-07-12T08:49:57.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7558012560 con 0x7f756c0ad8b0 2022-07-12T08:49:57.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756c0659a0 con 0x7f756c0ad8b0 2022-07-12T08:49:57.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75580128c0 con 0x7f756c0ad8b0 2022-07-12T08:49:57.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756c0b5030 con 0x7f756c0ad8b0 2022-07-12T08:49:57.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.288+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756c061d40 con 0x7f756c0ad8b0 2022-07-12T08:49:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.292+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7558019030 con 0x7f756c0ad8b0 2022-07-12T08:49:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.292+0000 7f7569ffb700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f754c0539b0 0x7f754c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.292+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f755806b330 con 0x7f756c0ad8b0 2022-07-12T08:49:57.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.292+0000 7f7570a5d700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f754c0539b0 0x7f754c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.292+0000 7f7570a5d700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f754c0539b0 0x7f754c055e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f7560018000 tx=0x7f7560014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:57.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.296+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7558042160 con 0x7f756c0ad8b0 2022-07-12T08:49:57.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f756c065b30 con 0x7f756c0ad8b0 2022-07-12T08:49:57.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7569ffb700 1 -- 172.21.15.61:0/3413238011 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1426 (secure 0 0 0) 0x7f7558045780 con 0x7f756c0ad8b0 2022-07-12T08:49:57.483 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:57.483 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":67,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:49:56.094596+0000","modified":"2022-07-12T08:49:56.094664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-12T08:49:57.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f754c0539b0 msgr2=0x7f754c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f754c0539b0 0x7f754c055e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7f7560018000 tx=0x7f7560014000).stop 2022-07-12T08:49:57.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 msgr2=0x7f756c140690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 0x7f756c140690 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 rx=0x7f756c0b3400 tx=0x7f7558007c50).stop 2022-07-12T08:49:57.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 shutdown_connections 2022-07-12T08:49:57.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f754c0539b0 0x7f754c055e70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f756c13e120 0x7f756c140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f756c0ad8b0 0x7f756c140690 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 --2- 172.21.15.61:0/3413238011 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f756c05c8e0 0x7f756c060d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 >> 172.21.15.61:0/3413238011 conn(0x7f756c0b5780 msgr2=0x7f756c0ae390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:57.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 shutdown_connections 2022-07-12T08:49:57.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.464+0000 7f7571a5f700 1 -- 172.21.15.61:0/3413238011 wait complete. 2022-07-12T08:49:57.489 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 67 2022-07-12T08:49:57.490 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:49:57.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0f2d3700 1 Processor -- start 2022-07-12T08:49:57.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0f2d3700 1 -- start start 2022-07-12T08:49:57.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0f2d3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fcf080b04d0 0x7fcf08097030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0f2d3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0805d440 con 0x7fcf0813e600 2022-07-12T08:49:57.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0f2d3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf08097690 con 0x7fcf0805d060 2022-07-12T08:49:57.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0f2d3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf08096050 con 0x7fcf080b04d0 2022-07-12T08:49:57.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0ead2700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcf0813e600 0x7fcf08142010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48772/0 (socket says 172.21.15.61:48772) 2022-07-12T08:49:57.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0ead2700 1 -- 172.21.15.61:0/4109486410 learned_addr learned my addr 172.21.15.61:0/4109486410 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:57.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0dad0700 1 --2- 172.21.15.61:0/4109486410 >> v2:172.21.15.111:3300/0 conn(0x7fcf080b04d0 0x7fcf08097030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961846907 0 0) 0x7fcf08097690 con 0x7fcf0805d060 2022-07-12T08:49:57.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef4003260 con 0x7fcf0805d060 2022-07-12T08:49:57.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302592945 0 0) 0x7fcf0805d440 con 0x7fcf0813e600 2022-07-12T08:49:57.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf08097690 con 0x7fcf0813e600 2022-07-12T08:49:57.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0dad0700 1 -- 172.21.15.61:0/4109486410 >> v1:172.21.15.111:6789/0 conn(0x7fcf0805d060 legacy=0x7fcf0813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 <== mon.1 v1:172.21.15.111:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498455218 0 0) 0x7fcef4003260 con 0x7fcf0805d060 2022-07-12T08:49:57.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0dad0700 1 -- 172.21.15.61:0/4109486410 >> v1:172.21.15.61:6789/0 conn(0x7fcf0813e600 legacy=0x7fcf08142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.592+0000 7fcf0dad0700 1 -- 172.21.15.61:0/4109486410 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08097930 con 0x7fcf080b04d0 2022-07-12T08:49:57.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0dad0700 1 --2- 172.21.15.61:0/4109486410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf080b04d0 0x7fcf08097030 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fcf00001110 tx=0x7fcefc006df0).ready entity=mon.1 client_cookie=b4441c40f62833e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:57.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf00001660 con 0x7fcf080b04d0 2022-07-12T08:49:57.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc000e90 con 0x7fcf080b04d0 2022-07-12T08:49:57.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0d2cf700 1 -- 172.21.15.61:0/4109486410 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef80019e0 con 0x7fcf080b04d0 2022-07-12T08:49:57.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/4109486410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf080b04d0 msgr2=0x7fcf08097030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/4109486410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf080b04d0 0x7fcf08097030 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fcf00001110 tx=0x7fcefc006df0).stop 2022-07-12T08:49:57.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/4109486410 shutdown_connections 2022-07-12T08:49:57.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/4109486410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf080b04d0 0x7fcf08097030 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/4109486410 >> 172.21.15.61:0/4109486410 conn(0x7fcf080b1d80 msgr2=0x7fcf080b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:57.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/4109486410 shutdown_connections 2022-07-12T08:49:57.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/4109486410 wait complete. 2022-07-12T08:49:57.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 Processor -- start 2022-07-12T08:49:57.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.596+0000 7fcf0f2d3700 1 -- start start 2022-07-12T08:49:57.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 0x7fcf0809a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b04d0 0x7fcf08139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0813e600 0x7fcf08139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf08145a90 con 0x7fcf080b04d0 2022-07-12T08:49:57.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf08141f30 con 0x7fcf0813e600 2022-07-12T08:49:57.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf08142160 con 0x7fcf0805d060 2022-07-12T08:49:57.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0dad0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b04d0 0x7fcf08139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 0x7fcf0809a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 0x7fcf0809a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42752/0 (socket says 172.21.15.61:42752) 2022-07-12T08:49:57.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 -- 172.21.15.61:0/3966497689 learned_addr learned my addr 172.21.15.61:0/3966497689 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:57.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0ead2700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0813e600 0x7fcf08139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 -- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0813e600 msgr2=0x7fcf08139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0813e600 0x7fcf08139940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 -- 172.21.15.61:0/3966497689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b04d0 msgr2=0x7fcf08139450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:57.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b04d0 0x7fcf08139450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:57.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 -- 172.21.15.61:0/3966497689 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc00b180 con 0x7fcf0805d060 2022-07-12T08:49:57.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0e2d1700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 0x7fcf0809a6c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcf08145db0 tx=0x7fcef800a550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:57.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef800ce30 con 0x7fcf0805d060 2022-07-12T08:49:57.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef8011070 con 0x7fcf0805d060 2022-07-12T08:49:57.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcef8003300 con 0x7fcf0805d060 2022-07-12T08:49:57.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf08142400 con 0x7fcf0805d060 2022-07-12T08:49:57.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf08139e30 con 0x7fcf0805d060 2022-07-12T08:49:57.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.600+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf08057460 con 0x7fcf0805d060 2022-07-12T08:49:57.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.604+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcef801d070 con 0x7fcf0805d060 2022-07-12T08:49:57.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.604+0000 7fcf077fe700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcee8053a40 0x7fcee8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:57.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.604+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcef806c950 con 0x7fcf0805d060 2022-07-12T08:49:57.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.604+0000 7fcf0dad0700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcee8053a40 0x7fcee8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:57.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.604+0000 7fcf0dad0700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcee8053a40 0x7fcee8055f00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fcefc019000 tx=0x7fcf08097820).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:57.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.612+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcef8043750 con 0x7fcf0805d060 2022-07-12T08:49:57.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:57.780+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcf08095760 con 0x7fcf0805d060 2022-07-12T08:49:58.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.108+0000 7fcf077fe700 1 -- 172.21.15.61:0/3966497689 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcef8033090 con 0x7fcf0805d060 2022-07-12T08:49:58.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.112+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcee8053a40 msgr2=0x7fcee8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.112+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcee8053a40 0x7fcee8055f00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fcefc019000 tx=0x7fcf08097820).stop 2022-07-12T08:49:58.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.112+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 msgr2=0x7fcf0809a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.112+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 0x7fcf0809a6c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fcf08145db0 tx=0x7fcef800a550).stop 2022-07-12T08:49:58.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 shutdown_connections 2022-07-12T08:49:58.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcee8053a40 0x7fcee8055f00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcf0813e600 0x7fcf08139940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b04d0 0x7fcf08139450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 --2- 172.21.15.61:0/3966497689 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcf0805d060 0x7fcf0809a6c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 >> 172.21.15.61:0/3966497689 conn(0x7fcf080b1d80 msgr2=0x7fcf080b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:58.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 shutdown_connections 2022-07-12T08:49:58.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.116+0000 7fcf0f2d3700 1 -- 172.21.15.61:0/3966497689 wait complete. 2022-07-12T08:49:58.129 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:49:58.138 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:49:58.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 Processor -- start 2022-07-12T08:49:58.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- start start 2022-07-12T08:49:58.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f982813ce60 0x7f98280b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98280b47f0 0x7f98280b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982813d240 con 0x7f9828142d50 2022-07-12T08:49:58.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98280b24c0 con 0x7f98280b47f0 2022-07-12T08:49:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98281431f0 con 0x7f982813ce60 2022-07-12T08:49:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f981dd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9828142d50 0x7f98280b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48784/0 (socket says 172.21.15.61:48784) 2022-07-12T08:49:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f981dd9b700 1 -- 172.21.15.61:0/815351518 learned_addr learned my addr 172.21.15.61:0/815351518 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982659c700 1 --2- 172.21.15.61:0/815351518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f982813ce60 0x7f98280b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9826d9d700 1 --2- 172.21.15.61:0/815351518 >> v2:172.21.15.111:3300/0 conn(0x7f98280b47f0 0x7f98280b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9825d9b700 1 -- 172.21.15.61:0/815351518 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102084561 0 0) 0x7f982813d240 con 0x7f9828142d50 2022-07-12T08:49:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9825d9b700 1 -- 172.21.15.61:0/815351518 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808003260 con 0x7f9828142d50 2022-07-12T08:49:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982659c700 1 -- 172.21.15.61:0/815351518 >> v2:172.21.15.111:3300/0 conn(0x7f98280b47f0 msgr2=0x7f98280b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982659c700 1 --2- 172.21.15.61:0/815351518 >> v2:172.21.15.111:3300/0 conn(0x7f98280b47f0 0x7f98280b4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982659c700 1 -- 172.21.15.61:0/815351518 >> v1:172.21.15.61:6789/0 conn(0x7f9828142d50 legacy=0x7f98280b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982659c700 1 -- 172.21.15.61:0/815351518 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828143490 con 0x7f982813ce60 2022-07-12T08:49:58.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9825d9b700 1 -- 172.21.15.61:0/815351518 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046617284 0 0) 0x7f9808003260 con 0x7f9828142d50 2022-07-12T08:49:58.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982659c700 1 --2- 172.21.15.61:0/815351518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f982813ce60 0x7f98280b15a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9818002f70 tx=0x7f9818006df0).ready entity=mon.2 client_cookie=3cc939f18ad215f5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:58.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9825d9b700 1 -- 172.21.15.61:0/815351518 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f981800d4a0 con 0x7f982813ce60 2022-07-12T08:49:58.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9825d9b700 1 -- 172.21.15.61:0/815351518 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818000e90 con 0x7f982813ce60 2022-07-12T08:49:58.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f9825d9b700 1 -- 172.21.15.61:0/815351518 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f981800d7f0 con 0x7f982813ce60 2022-07-12T08:49:58.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- 172.21.15.61:0/815351518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f982813ce60 msgr2=0x7f98280b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 --2- 172.21.15.61:0/815351518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f982813ce60 0x7f98280b15a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9818002f70 tx=0x7f9818006df0).stop 2022-07-12T08:49:58.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- 172.21.15.61:0/815351518 shutdown_connections 2022-07-12T08:49:58.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 --2- 172.21.15.61:0/815351518 >> v2:172.21.15.111:3300/0 conn(0x7f98280b47f0 0x7f98280b4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 --2- 172.21.15.61:0/815351518 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f982813ce60 0x7f98280b15a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.244+0000 7f982c9a3700 1 -- 172.21.15.61:0/815351518 >> 172.21.15.61:0/815351518 conn(0x7f982805f9f0 msgr2=0x7f9828066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:58.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 -- 172.21.15.61:0/815351518 shutdown_connections 2022-07-12T08:49:58.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 -- 172.21.15.61:0/815351518 wait complete. 2022-07-12T08:49:58.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 Processor -- start 2022-07-12T08:49:58.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 -- start start 2022-07-12T08:49:58.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98280b47f0 0x7f98280afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f982813ce60 0x7f982813a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 0x7f9828152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828147200 con 0x7f9828142d50 2022-07-12T08:49:58.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98281443a0 con 0x7f982813ce60 2022-07-12T08:49:58.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982c9a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9828144600 con 0x7f98280b47f0 2022-07-12T08:49:58.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f9826d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 0x7f9828152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f981dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f982813ce60 0x7f982813a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f9826d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 0x7f9828152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60574/0 (socket says 172.21.15.61:60574) 2022-07-12T08:49:58.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f9826d9d700 1 -- 172.21.15.61:0/3613297638 learned_addr learned my addr 172.21.15.61:0/3613297638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:58.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f982659c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98280b47f0 0x7f98280afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f9826d9d700 1 -- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98280b47f0 msgr2=0x7f98280afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.248+0000 7f9826d9d700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98280b47f0 0x7f98280afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f9826d9d700 1 -- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f982813ce60 msgr2=0x7f982813a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f9826d9d700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f982813ce60 0x7f982813a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f9826d9d700 1 -- 172.21.15.61:0/3613297638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981800b180 con 0x7f9828142d50 2022-07-12T08:49:58.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f9826d9d700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 0x7f9828152210 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f9820006e00 tx=0x7f98200091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:58.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9820012600 con 0x7f9828142d50 2022-07-12T08:49:58.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9820012760 con 0x7f9828142d50 2022-07-12T08:49:58.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98281448a0 con 0x7f9828142d50 2022-07-12T08:49:58.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9820012ac0 con 0x7f9828142d50 2022-07-12T08:49:58.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982813adf0 con 0x7f9828142d50 2022-07-12T08:49:58.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9828057460 con 0x7f9828142d50 2022-07-12T08:49:58.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9820022070 con 0x7f9828142d50 2022-07-12T08:49:58.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.252+0000 7f981ffff700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f980c053ad0 0x7f980c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.256+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f982006c480 con 0x7f9828142d50 2022-07-12T08:49:58.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.256+0000 7f982659c700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f980c053ad0 0x7f980c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.256+0000 7f982659c700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f980c053ad0 0x7f980c055f90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f9818018000 tx=0x7f9818014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:58.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.260+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f98200432f0 con 0x7f9828142d50 2022-07-12T08:49:58.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98280b3e20 con 0x7f9828142d50 2022-07-12T08:49:58.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f981ffff700 1 -- 172.21.15.61:0/3613297638 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+1426 (secure 0 0 0) 0x7f9820016090 con 0x7f9828142d50 2022-07-12T08:49:58.447 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:49:58.448 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":68,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:49:56.094596+0000","modified":"2022-07-12T08:49:58.101913+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-12T08:49:58.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f980c053ad0 msgr2=0x7f980c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f980c053ad0 0x7f980c055f90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f9818018000 tx=0x7f9818014000).stop 2022-07-12T08:49:58.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 msgr2=0x7f9828152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 0x7f9828152210 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f9820006e00 tx=0x7f98200091d0).stop 2022-07-12T08:49:58.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 shutdown_connections 2022-07-12T08:49:58.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9828142d50 0x7f9828152210 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f980c053ad0 0x7f980c055f90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f982813ce60 0x7f982813a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 --2- 172.21.15.61:0/3613297638 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98280b47f0 0x7f98280afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 >> 172.21.15.61:0/3613297638 conn(0x7f982805f9f0 msgr2=0x7f98280b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:58.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 shutdown_connections 2022-07-12T08:49:58.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.424+0000 7f982c9a3700 1 -- 172.21.15.61:0/3613297638 wait complete. 2022-07-12T08:49:58.467 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 68 2022-07-12T08:49:58.468 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:49:58.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 Processor -- start 2022-07-12T08:49:58.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- start start 2022-07-12T08:49:58.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805d100 0x7f4fc8060e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4fc81465a0 0x7f4fc81469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fc814a560 con 0x7f4fc805cd20 2022-07-12T08:49:58.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fc8097d40 con 0x7f4fc81465a0 2022-07-12T08:49:58.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc814b340 con 0x7f4fc805d100 2022-07-12T08:49:58.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4fc81465a0 0x7f4fc81469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4fc81465a0 0x7f4fc81469c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60582/0 (socket says 172.21.15.61:60582) 2022-07-12T08:49:58.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 -- 172.21.15.61:0/771545665 learned_addr learned my addr 172.21.15.61:0/771545665 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:58.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fc7fff700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805d100 0x7f4fc8060e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 -- 172.21.15.61:0/771545665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805d100 msgr2=0x7f4fc8060e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:49:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805d100 0x7f4fc8060e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 -- 172.21.15.61:0/771545665 >> v1:172.21.15.111:6789/0 conn(0x7f4fc805cd20 legacy=0x7f4fc814ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 -- 172.21.15.61:0/771545665 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc80aafa0 con 0x7f4fc81465a0 2022-07-12T08:49:58.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fc7fff700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805d100 0x7f4fc8060e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:49:58.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcd21c700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc81469c0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7f4fc00032f0 tx=0x7f4fc0009290).ready entity=mon.0 client_cookie=d709983c91b0640d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:58.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fc77fe700 1 -- 172.21.15.61:0/771545665 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fc000b8a0 con 0x7f4fc81465a0 2022-07-12T08:49:58.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fc77fe700 1 -- 172.21.15.61:0/771545665 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc000ba00 con 0x7f4fc81465a0 2022-07-12T08:49:58.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fc77fe700 1 -- 172.21.15.61:0/771545665 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fc000bb70 con 0x7f4fc81465a0 2022-07-12T08:49:58.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- 172.21.15.61:0/771545665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 msgr2=0x7f4fc81469c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc81469c0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7f4fc00032f0 tx=0x7f4fc0009290).stop 2022-07-12T08:49:58.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- 172.21.15.61:0/771545665 shutdown_connections 2022-07-12T08:49:58.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc81469c0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/771545665 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805d100 0x7f4fc8060e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- 172.21.15.61:0/771545665 >> 172.21.15.61:0/771545665 conn(0x7f4fc80b1a50 msgr2=0x7f4fc80b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:58.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- 172.21.15.61:0/771545665 shutdown_connections 2022-07-12T08:49:58.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- 172.21.15.61:0/771545665 wait complete. 2022-07-12T08:49:58.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 Processor -- start 2022-07-12T08:49:58.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.580+0000 7f4fcda1d700 1 -- start start 2022-07-12T08:49:58.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805cd20 0x7f4fc81442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fc805d100 0x7f4fc8142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc8159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc814d450 con 0x7f4fc81465a0 2022-07-12T08:49:58.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc80aed90 con 0x7f4fc805d100 2022-07-12T08:49:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc814bb30 con 0x7f4fc805cd20 2022-07-12T08:49:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc8159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc8159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60588/0 (socket says 172.21.15.61:60588) 2022-07-12T08:49:58.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 -- 172.21.15.61:0/2747266237 learned_addr learned my addr 172.21.15.61:0/2747266237 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:58.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fc7fff700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fc805d100 0x7f4fc8142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcca1b700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805cd20 0x7f4fc81442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 -- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805cd20 msgr2=0x7f4fc81442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805cd20 0x7f4fc81442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 -- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fc805d100 msgr2=0x7f4fc8142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:58.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fc805d100 0x7f4fc8142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:58.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 -- 172.21.15.61:0/2747266237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc0009530 con 0x7f4fc81465a0 2022-07-12T08:49:58.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcd21c700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc8159580 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f4fc00032f0 tx=0x7f4fc0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:58.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fc000b8a0 con 0x7f4fc81465a0 2022-07-12T08:49:58.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc0021480 con 0x7f4fc81465a0 2022-07-12T08:49:58.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc814bdd0 con 0x7f4fc81465a0 2022-07-12T08:49:58.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fc00217e0 con 0x7f4fc81465a0 2022-07-12T08:49:58.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc814c4a0 con 0x7f4fc81465a0 2022-07-12T08:49:58.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fc8057460 con 0x7f4fc81465a0 2022-07-12T08:49:58.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.584+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4fc000f070 con 0x7f4fc81465a0 2022-07-12T08:49:58.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.588+0000 7f4fc5ffb700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4fa8053af0 0x7f4fa8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.588+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4fc006d140 con 0x7f4fc81465a0 2022-07-12T08:49:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.588+0000 7f4fcca1b700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4fa8053af0 0x7f4fa8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.588+0000 7f4fcca1b700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4fa8053af0 0x7f4fa8055fb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f4fb800d000 tx=0x7f4fb8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:58.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.588+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4fc0044100 con 0x7f4fc81465a0 2022-07-12T08:49:58.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:58.752+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4fc806b2b0 con 0x7f4fc81465a0 2022-07-12T08:49:59.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.116+0000 7f4fc5ffb700 1 -- 172.21.15.61:0/2747266237 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4fc0015020 con 0x7f4fc81465a0 2022-07-12T08:49:59.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4fa8053af0 msgr2=0x7f4fa8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4fa8053af0 0x7f4fa8055fb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f4fb800d000 tx=0x7f4fb8007b90).stop 2022-07-12T08:49:59.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 msgr2=0x7f4fc8159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc8159580 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7f4fc00032f0 tx=0x7f4fc0016000).stop 2022-07-12T08:49:59.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 shutdown_connections 2022-07-12T08:49:59.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4fc81465a0 0x7f4fc8159580 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4fa8053af0 0x7f4fa8055fb0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4fc805d100 0x7f4fc8142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 --2- 172.21.15.61:0/2747266237 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4fc805cd20 0x7f4fc81442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 >> 172.21.15.61:0/2747266237 conn(0x7f4fc80b1a50 msgr2=0x7f4fc80afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:59.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 shutdown_connections 2022-07-12T08:49:59.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.120+0000 7f4fcda1d700 1 -- 172.21.15.61:0/2747266237 wait complete. 2022-07-12T08:49:59.150 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:49:59.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.280+0000 7f3fe4f8b700 1 Processor -- start 2022-07-12T08:49:59.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.280+0000 7f3fe4f8b700 1 -- start start 2022-07-12T08:49:59.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3fe0060b60 0x7f3fe0141f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:59.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00ab2c0 con 0x7f3fe00aaee0 2022-07-12T08:49:59.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00ab530 con 0x7f3fe0142470 2022-07-12T08:49:59.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe00ab7d0 con 0x7f3fe0060b60 2022-07-12T08:49:59.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3fe00aaee0 0x7f3fe00b0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48810/0 (socket says 172.21.15.61:48810) 2022-07-12T08:49:59.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdf7fe700 1 -- 172.21.15.61:0/3259436404 learned_addr learned my addr 172.21.15.61:0/3259436404 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:49:59.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdeffd700 1 --2- 172.21.15.61:0/3259436404 >> v2:172.21.15.111:3300/0 conn(0x7f3fe0060b60 0x7f3fe0141f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:59.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fde7fc700 1 -- 172.21.15.61:0/3259436404 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807750025 0 0) 0x7f3fe00ab2c0 con 0x7f3fe00aaee0 2022-07-12T08:49:59.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fde7fc700 1 -- 172.21.15.61:0/3259436404 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc0003260 con 0x7f3fe00aaee0 2022-07-12T08:49:59.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdeffd700 1 -- 172.21.15.61:0/3259436404 >> v1:172.21.15.111:6789/0 conn(0x7f3fe0142470 legacy=0x7f3fe01428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdeffd700 1 -- 172.21.15.61:0/3259436404 >> v1:172.21.15.61:6789/0 conn(0x7f3fe00aaee0 legacy=0x7f3fe00b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdeffd700 1 -- 172.21.15.61:0/3259436404 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe00b10d0 con 0x7f3fe0060b60 2022-07-12T08:49:59.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fdeffd700 1 --2- 172.21.15.61:0/3259436404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0060b60 0x7f3fe0141f80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f3fc8002f70 tx=0x7f3fc8006df0).ready entity=mon.1 client_cookie=f58d08f1a65bf851 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:59.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fde7fc700 1 -- 172.21.15.61:0/3259436404 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc800d610 con 0x7f3fe0060b60 2022-07-12T08:49:59.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fde7fc700 1 -- 172.21.15.61:0/3259436404 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc8000e90 con 0x7f3fe0060b60 2022-07-12T08:49:59.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fde7fc700 1 -- 172.21.15.61:0/3259436404 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc800d960 con 0x7f3fe0060b60 2022-07-12T08:49:59.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/3259436404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0060b60 msgr2=0x7f3fe0141f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/3259436404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0060b60 0x7f3fe0141f80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f3fc8002f70 tx=0x7f3fc8006df0).stop 2022-07-12T08:49:59.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/3259436404 shutdown_connections 2022-07-12T08:49:59.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/3259436404 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0060b60 0x7f3fe0141f80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/3259436404 >> 172.21.15.61:0/3259436404 conn(0x7f3fe00b1950 msgr2=0x7f3fe00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:49:59.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/3259436404 shutdown_connections 2022-07-12T08:49:59.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.284+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/3259436404 wait complete. 2022-07-12T08:49:59.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 Processor -- start 2022-07-12T08:49:59.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 -- start start 2022-07-12T08:49:59.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3fe0060b60 0x7f3fe0144f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:59.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 0x7f3fe013dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:59.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0142470 0x7f3fe013e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:59.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01528e0 con 0x7f3fe00aaee0 2022-07-12T08:49:59.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0145700 con 0x7f3fe0142470 2022-07-12T08:49:59.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fe4f8b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0145410 con 0x7f3fe0060b60 2022-07-12T08:49:59.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.288+0000 7f3fdeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 0x7f3fe013dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:59.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 0x7f3fe013dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60600/0 (socket says 172.21.15.61:60600) 2022-07-12T08:49:59.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 -- 172.21.15.61:0/1102789308 learned_addr learned my addr 172.21.15.61:0/1102789308 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:49:59.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdf7fe700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3fe0060b60 0x7f3fe0144f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:59.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdffff700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0142470 0x7f3fe013e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:59.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 -- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3fe0060b60 msgr2=0x7f3fe0144f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3fe0060b60 0x7f3fe0144f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 -- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0142470 msgr2=0x7f3fe013e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:49:59.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0142470 0x7f3fe013e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:49:59.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 -- 172.21.15.61:0/1102789308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc800b180 con 0x7f3fe00aaee0 2022-07-12T08:49:59.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdeffd700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 0x7f3fe013dd10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f3fc8002f70 tx=0x7f3fc8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:59.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc800d290 con 0x7f3fe00aaee0 2022-07-12T08:49:59.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc8013e90 con 0x7f3fe00aaee0 2022-07-12T08:49:59.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe013e6f0 con 0x7f3fe00aaee0 2022-07-12T08:49:59.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc801e5f0 con 0x7f3fe00aaee0 2022-07-12T08:49:59.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe0152430 con 0x7f3fe00aaee0 2022-07-12T08:49:59.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe00acc50 con 0x7f3fe00aaee0 2022-07-12T08:49:59.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.292+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3fc8025030 con 0x7f3fe00aaee0 2022-07-12T08:49:59.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.296+0000 7f3fdcff9700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3fc4053ad0 0x7f3fc4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:49:59.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.296+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3fc806cd80 con 0x7f3fe00aaee0 2022-07-12T08:49:59.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.296+0000 7f3fdf7fe700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3fc4053ad0 0x7f3fc4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:49:59.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.296+0000 7f3fdf7fe700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3fc4053ad0 0x7f3fc4055f90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f3fd800d000 tx=0x7f3fd8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:49:59.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.300+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3fe00acc50 con 0x7f3fe00aaee0 2022-07-12T08:49:59.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:49:59.428+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3fe00acc50 con 0x7f3fe00aaee0 2022-07-12T08:50:00.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.240+0000 7f3fdcff9700 1 -- 172.21.15.61:0/1102789308 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v122) v1 ==== 185+0+0 (secure 0 0 0) 0x7f3fe00acc50 con 0x7f3fe00aaee0 2022-07-12T08:50:00.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3fc4053ad0 msgr2=0x7f3fc4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:00.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3fc4053ad0 0x7f3fc4055f90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f3fd800d000 tx=0x7f3fd8009040).stop 2022-07-12T08:50:00.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 msgr2=0x7f3fe013dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:00.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 0x7f3fe013dd10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f3fc8002f70 tx=0x7f3fc8014000).stop 2022-07-12T08:50:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 shutdown_connections 2022-07-12T08:50:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3fc4053ad0 0x7f3fc4055f90 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3fe0142470 0x7f3fe013e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3fe00aaee0 0x7f3fe013dd10 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 --2- 172.21.15.61:0/1102789308 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3fe0060b60 0x7f3fe0144f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 >> 172.21.15.61:0/1102789308 conn(0x7f3fe00b1950 msgr2=0x7f3fe00b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:00.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 shutdown_connections 2022-07-12T08:50:00.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.244+0000 7f3fe4f8b700 1 -- 172.21.15.61:0/1102789308 wait complete. 2022-07-12T08:50:00.259 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:50:00.265 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:50:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.380+0000 7f46ec974700 1 Processor -- start 2022-07-12T08:50:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.380+0000 7f46ec974700 1 -- start start 2022-07-12T08:50:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46ec974700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 0x7f46e80aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46ec974700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 0x7f46e813f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:00.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46ec974700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e80acee0 con 0x7f46e80b1320 2022-07-12T08:50:00.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46ec974700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e80ad180 con 0x7f46e80b0120 2022-07-12T08:50:00.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46ec974700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e80acb60 con 0x7f46e813ec70 2022-07-12T08:50:00.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 0x7f46e813f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:00.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 0x7f46e80aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:00.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 0x7f46e813f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42806/0 (socket says 172.21.15.61:42806) 2022-07-12T08:50:00.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e659c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 0x7f46e80aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39182/0 (socket says 172.21.15.61:39182) 2022-07-12T08:50:00.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e6d9d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f46e80b1320 0x7f46e80ae500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34194/0 (socket says 172.21.15.61:34194) 2022-07-12T08:50:00.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 -- 172.21.15.61:0/4226360234 learned_addr learned my addr 172.21.15.61:0/4226360234 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:00.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 -- 172.21.15.61:0/4226360234 >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 msgr2=0x7f46e80aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:00.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 --2- 172.21.15.61:0/4226360234 >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 0x7f46e80aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 -- 172.21.15.61:0/4226360234 >> v1:172.21.15.111:6789/0 conn(0x7f46e80b1320 legacy=0x7f46e80ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:00.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 -- 172.21.15.61:0/4226360234 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e80b0630 con 0x7f46e813ec70 2022-07-12T08:50:00.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e5d9b700 1 -- 172.21.15.61:0/4226360234 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647823436 0 0) 0x7f46e80acee0 con 0x7f46e80b1320 2022-07-12T08:50:00.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e659c700 1 --2- 172.21.15.61:0/4226360234 >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 0x7f46e80aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:00.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e759e700 1 --2- 172.21.15.61:0/4226360234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 0x7f46e813f120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f46e0002f70 tx=0x7f46e0006eb0).ready entity=mon.2 client_cookie=f2e3846422873c2f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:00.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e5d9b700 1 -- 172.21.15.61:0/4226360234 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e000d570 con 0x7f46e813ec70 2022-07-12T08:50:00.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e5d9b700 1 -- 172.21.15.61:0/4226360234 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e0000e90 con 0x7f46e813ec70 2022-07-12T08:50:00.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.384+0000 7f46e5d9b700 1 -- 172.21.15.61:0/4226360234 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e000d8c0 con 0x7f46e813ec70 2022-07-12T08:50:00.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- 172.21.15.61:0/4226360234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 msgr2=0x7f46e813f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:00.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 --2- 172.21.15.61:0/4226360234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 0x7f46e813f120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f46e0002f70 tx=0x7f46e0006eb0).stop 2022-07-12T08:50:00.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- 172.21.15.61:0/4226360234 shutdown_connections 2022-07-12T08:50:00.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 --2- 172.21.15.61:0/4226360234 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e813ec70 0x7f46e813f120 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 --2- 172.21.15.61:0/4226360234 >> v2:172.21.15.111:3300/0 conn(0x7f46e80b0120 0x7f46e80aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- 172.21.15.61:0/4226360234 >> 172.21.15.61:0/4226360234 conn(0x7f46e80b5780 msgr2=0x7f46e80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:00.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- 172.21.15.61:0/4226360234 shutdown_connections 2022-07-12T08:50:00.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- 172.21.15.61:0/4226360234 wait complete. 2022-07-12T08:50:00.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 Processor -- start 2022-07-12T08:50:00.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- start start 2022-07-12T08:50:00.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 0x7f46e8064510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:00.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46e80b1320 0x7f46e8140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:00.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 0x7f46e8141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:00.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8150d50 con 0x7f46e813ec70 2022-07-12T08:50:00.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8150ff0 con 0x7f46e80b1320 2022-07-12T08:50:00.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.388+0000 7f46ec974700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8151290 con 0x7f46e80b0120 2022-07-12T08:50:00.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 0x7f46e8064510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:00.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 0x7f46e8141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:00.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46e80b1320 0x7f46e8140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:00.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 0x7f46e8141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60616/0 (socket says 172.21.15.61:60616) 2022-07-12T08:50:00.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 -- 172.21.15.61:0/2698489637 learned_addr learned my addr 172.21.15.61:0/2698489637 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:00.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 0x7f46e8064510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:42808/0 (socket says 172.21.15.61:42808) 2022-07-12T08:50:00.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 -- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 msgr2=0x7f46e8064510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:00.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 0x7f46e8064510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 -- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46e80b1320 msgr2=0x7f46e8140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:00.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46e80b1320 0x7f46e8140b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:00.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 -- 172.21.15.61:0/2698489637 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc00d410 con 0x7f46e813ec70 2022-07-12T08:50:00.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e6d9d700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 0x7f46e8064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:00.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e659c700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46e80b1320 0x7f46e8140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:00.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46e759e700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 0x7f46e8141070 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f46e0002f70 tx=0x7f46e0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:00.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e0006b40 con 0x7f46e813ec70 2022-07-12T08:50:00.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e00068b0 con 0x7f46e813ec70 2022-07-12T08:50:00.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e000b1e0 con 0x7f46e813ec70 2022-07-12T08:50:00.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e0011b40 con 0x7f46e813ec70 2022-07-12T08:50:00.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e8151730 con 0x7f46e813ec70 2022-07-12T08:50:00.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.392+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e8151450 con 0x7f46e813ec70 2022-07-12T08:50:00.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.396+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f46e0025030 con 0x7f46e813ec70 2022-07-12T08:50:00.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.396+0000 7f46d7fff700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46c4053920 0x7f46c4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:00.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.396+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f46e006ced0 con 0x7f46e813ec70 2022-07-12T08:50:00.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.396+0000 7f46e6d9d700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46c4053920 0x7f46c4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:00.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.396+0000 7f46e6d9d700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46c4053920 0x7f46c4055de0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f46d800d000 tx=0x7f46d8007e10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:00.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.404+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f46e0043ef0 con 0x7f46e813ec70 2022-07-12T08:50:00.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:00.524+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46e8057460 con 0x7f46e813ec70 2022-07-12T08:50:01.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.244+0000 7f46d7fff700 1 -- 172.21.15.61:0/2698489637 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v123) v1 ==== 173+0+0 (secure 0 0 0) 0x7f46e00230d0 con 0x7f46e813ec70 2022-07-12T08:50:01.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46c4053920 msgr2=0x7f46c4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46c4053920 0x7f46c4055de0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f46d800d000 tx=0x7f46d8007e10).stop 2022-07-12T08:50:01.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 msgr2=0x7f46e8141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 0x7f46e8141070 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f46e0002f70 tx=0x7f46e0015000).stop 2022-07-12T08:50:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 shutdown_connections 2022-07-12T08:50:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46e813ec70 0x7f46e8141070 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f46c4053920 0x7f46c4055de0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46e80b1320 0x7f46e8140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 --2- 172.21.15.61:0/2698489637 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46e80b0120 0x7f46e8064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 >> 172.21.15.61:0/2698489637 conn(0x7f46e80b5780 msgr2=0x7f46e8144270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 shutdown_connections 2022-07-12T08:50:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.248+0000 7f46ec974700 1 -- 172.21.15.61:0/2698489637 wait complete. 2022-07-12T08:50:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:50:01.272 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:50:01.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.384+0000 7f445c5b8700 1 Processor -- start 2022-07-12T08:50:01.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.384+0000 7f445c5b8700 1 -- start start 2022-07-12T08:50:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b52e0 0x7f44540b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4454141e50 0x7f44541422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44540b56c0 con 0x7f44540b4050 2022-07-12T08:50:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4454142f00 con 0x7f4454141e50 2022-07-12T08:50:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44540b1320 con 0x7f44540b52e0 2022-07-12T08:50:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459b53700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f44540b4050 0x7f44540b26a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48838/0 (socket says 172.21.15.61:48838) 2022-07-12T08:50:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459b53700 1 -- 172.21.15.61:0/2662001456 learned_addr learned my addr 172.21.15.61:0/2662001456 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445ab55700 1 --2- 172.21.15.61:0/2662001456 >> v2:172.21.15.111:3300/0 conn(0x7f4454141e50 0x7f44541422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445a354700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b52e0 0x7f44540b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459352700 1 -- 172.21.15.61:0/2662001456 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213439398 0 0) 0x7f44540b56c0 con 0x7f44540b4050 2022-07-12T08:50:01.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459352700 1 -- 172.21.15.61:0/2662001456 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f443c003260 con 0x7f44540b4050 2022-07-12T08:50:01.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445ab55700 1 -- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b52e0 msgr2=0x7f44540b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:01.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445ab55700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b52e0 0x7f44540b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445ab55700 1 -- 172.21.15.61:0/2662001456 >> v1:172.21.15.61:6789/0 conn(0x7f44540b4050 legacy=0x7f44540b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445ab55700 1 -- 172.21.15.61:0/2662001456 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540b4bd0 con 0x7f4454141e50 2022-07-12T08:50:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445a354700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b52e0 0x7f44540b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445ab55700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f44541422c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f44540b1460 tx=0x7f444c0091d0).ready entity=mon.1 client_cookie=39a7ad23f81236d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:01.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459352700 1 -- 172.21.15.61:0/2662001456 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f444c00b600 con 0x7f4454141e50 2022-07-12T08:50:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459352700 1 -- 172.21.15.61:0/2662001456 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c00b760 con 0x7f4454141e50 2022-07-12T08:50:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f4459352700 1 -- 172.21.15.61:0/2662001456 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f444c00b8d0 con 0x7f4454141e50 2022-07-12T08:50:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 msgr2=0x7f44541422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f44541422c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f44540b1460 tx=0x7f444c0091d0).stop 2022-07-12T08:50:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- 172.21.15.61:0/2662001456 shutdown_connections 2022-07-12T08:50:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f44541422c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 --2- 172.21.15.61:0/2662001456 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b52e0 0x7f44540b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- 172.21.15.61:0/2662001456 >> 172.21.15.61:0/2662001456 conn(0x7f4454067020 msgr2=0x7f4454067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- 172.21.15.61:0/2662001456 shutdown_connections 2022-07-12T08:50:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- 172.21.15.61:0/2662001456 wait complete. 2022-07-12T08:50:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 Processor -- start 2022-07-12T08:50:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.388+0000 7f445c5b8700 1 -- start start 2022-07-12T08:50:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b4050 0x7f445413e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 0x7f4454148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f4454148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4454144400 con 0x7f44540b52e0 2022-07-12T08:50:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44540afad0 con 0x7f4454141e50 2022-07-12T08:50:01.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44540af870 con 0x7f44540b4050 2022-07-12T08:50:01.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 0x7f4454148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445a354700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b4050 0x7f445413e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445ab55700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f4454148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 0x7f4454148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60628/0 (socket says 172.21.15.61:60628) 2022-07-12T08:50:01.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 -- 172.21.15.61:0/4232521653 learned_addr learned my addr 172.21.15.61:0/4232521653 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:01.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 -- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b4050 msgr2=0x7f445413e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b4050 0x7f445413e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 -- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 msgr2=0x7f4454148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f4454148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 -- 172.21.15.61:0/4232521653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444800b180 con 0x7f44540b52e0 2022-07-12T08:50:01.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f4459b53700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 0x7f4454148700 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f4450001110 tx=0x7f4450007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:01.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f445000ce80 con 0x7f44540b52e0 2022-07-12T08:50:01.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44500070d0 con 0x7f44540b52e0 2022-07-12T08:50:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c009470 con 0x7f44540b52e0 2022-07-12T08:50:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4450007430 con 0x7f44540b52e0 2022-07-12T08:50:01.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44540afd90 con 0x7f44540b52e0 2022-07-12T08:50:01.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454057460 con 0x7f44540b52e0 2022-07-12T08:50:01.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4450010070 con 0x7f44540b52e0 2022-07-12T08:50:01.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f44477fe700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4434053a40 0x7f4434055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f445006b210 con 0x7f44540b52e0 2022-07-12T08:50:01.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445a354700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4434053a40 0x7f4434055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.396+0000 7f445a354700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4434053a40 0x7f4434055f00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f444800b000 tx=0x7f4448006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:01.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.408+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4450042340 con 0x7f44540b52e0 2022-07-12T08:50:01.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.532+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f44540b36a0 con 0x7f44540b52e0 2022-07-12T08:50:01.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.536+0000 7f44477fe700 1 -- 172.21.15.61:0/4232521653 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4450018020 con 0x7f44540b52e0 2022-07-12T08:50:01.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.536+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4434053a40 msgr2=0x7f4434055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.536+0000 7f445c5b8700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4434053a40 0x7f4434055f00 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f444800b000 tx=0x7f4448006df0).stop 2022-07-12T08:50:01.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.536+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 msgr2=0x7f4454148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.536+0000 7f445c5b8700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 0x7f4454148700 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f4450001110 tx=0x7f4450007aa0).stop 2022-07-12T08:50:01.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 shutdown_connections 2022-07-12T08:50:01.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4434053a40 0x7f4434055f00 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4454141e50 0x7f4454148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44540b52e0 0x7f4454148700 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 --2- 172.21.15.61:0/4232521653 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44540b4050 0x7f445413e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 >> 172.21.15.61:0/4232521653 conn(0x7f4454067020 msgr2=0x7f4454067400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:01.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 shutdown_connections 2022-07-12T08:50:01.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.540+0000 7f445c5b8700 1 -- 172.21.15.61:0/4232521653 wait complete. 2022-07-12T08:50:01.557 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-12T08:50:01.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6155311700 1 Processor -- start 2022-07-12T08:50:01.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6155311700 1 -- start start 2022-07-12T08:50:01.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6155311700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f6150140300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6155311700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61500b02e0 con 0x7f61500af0b0 2022-07-12T08:50:01.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6155311700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6150067040 con 0x7f6150061030 2022-07-12T08:50:01.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6155311700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61500af490 con 0x7f615013fe70 2022-07-12T08:50:01.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f614ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f61500af0b0 0x7f6150060920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48852/0 (socket says 172.21.15.61:48852) 2022-07-12T08:50:01.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f614ffff700 1 -- 172.21.15.61:0/834964920 learned_addr learned my addr 172.21.15.61:0/834964920 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:01.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.672+0000 7f6154b10700 1 --2- 172.21.15.61:0/834964920 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f6150140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740858990 0 0) 0x7f61500b02e0 con 0x7f61500af0b0 2022-07-12T08:50:01.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f613c003260 con 0x7f61500af0b0 2022-07-12T08:50:01.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671951426 0 0) 0x7f6150067040 con 0x7f6150061030 2022-07-12T08:50:01.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61500b02e0 con 0x7f6150061030 2022-07-12T08:50:01.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6154b10700 1 -- 172.21.15.61:0/834964920 >> v1:172.21.15.111:6789/0 conn(0x7f6150061030 legacy=0x7f615013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6154b10700 1 -- 172.21.15.61:0/834964920 >> v1:172.21.15.61:6789/0 conn(0x7f61500af0b0 legacy=0x7f6150060920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6154b10700 1 -- 172.21.15.61:0/834964920 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6150061490 con 0x7f615013fe70 2022-07-12T08:50:01.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6154b10700 1 --2- 172.21.15.61:0/834964920 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f6150140300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f61480032f0 tx=0x7f61480091d0).ready entity=mon.2 client_cookie=297cfc9bf36586f3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:01.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f614800b640 con 0x7f615013fe70 2022-07-12T08:50:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f614800b7a0 con 0x7f615013fe70 2022-07-12T08:50:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f614effd700 1 -- 172.21.15.61:0/834964920 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f614800b910 con 0x7f615013fe70 2022-07-12T08:50:01.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 -- 172.21.15.61:0/834964920 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 msgr2=0x7f6150140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 --2- 172.21.15.61:0/834964920 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f6150140300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f61480032f0 tx=0x7f61480091d0).stop 2022-07-12T08:50:01.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 -- 172.21.15.61:0/834964920 shutdown_connections 2022-07-12T08:50:01.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 --2- 172.21.15.61:0/834964920 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f6150140300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 -- 172.21.15.61:0/834964920 >> 172.21.15.61:0/834964920 conn(0x7f61500b5780 msgr2=0x7f61500b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 -- 172.21.15.61:0/834964920 shutdown_connections 2022-07-12T08:50:01.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 -- 172.21.15.61:0/834964920 wait complete. 2022-07-12T08:50:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 Processor -- start 2022-07-12T08:50:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.676+0000 7f6155311700 1 -- start start 2022-07-12T08:50:01.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 0x7f6150064680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61500af0b0 0x7f615013d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f615013d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150142b00 con 0x7f6150061030 2022-07-12T08:50:01.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150142830 con 0x7f61500af0b0 2022-07-12T08:50:01.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6150064b70 con 0x7f615013fe70 2022-07-12T08:50:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 0x7f6150064680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 0x7f6150064680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60640/0 (socket says 172.21.15.61:60640) 2022-07-12T08:50:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 -- 172.21.15.61:0/1760240030 learned_addr learned my addr 172.21.15.61:0/1760240030 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6154b10700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f615013d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614f7fe700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61500af0b0 0x7f615013d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 -- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 msgr2=0x7f615013d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f615013d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 -- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61500af0b0 msgr2=0x7f615013d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61500af0b0 0x7f615013d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 -- 172.21.15.61:0/1760240030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6148009470 con 0x7f6150061030 2022-07-12T08:50:01.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6154b10700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f615013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:01.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614ffff700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 0x7f6150064680 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f61440010d0 tx=0x7f6144007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:01.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f614400ce90 con 0x7f6150061030 2022-07-12T08:50:01.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61440072e0 con 0x7f6150061030 2022-07-12T08:50:01.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6150066160 con 0x7f6150061030 2022-07-12T08:50:01.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6144005800 con 0x7f6150061030 2022-07-12T08:50:01.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615013ee70 con 0x7f6150061030 2022-07-12T08:50:01.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6150057460 con 0x7f6150061030 2022-07-12T08:50:01.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6144010070 con 0x7f6150061030 2022-07-12T08:50:01.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614d7fa700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6130053ad0 0x7f6130055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:01.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.680+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f614406b310 con 0x7f6150061030 2022-07-12T08:50:01.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.684+0000 7f614f7fe700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6130053ad0 0x7f6130055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:01.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.684+0000 7f614f7fe700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6130053ad0 0x7f6130055f90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f614000b000 tx=0x7f6140009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:01.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.684+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6144042550 con 0x7f6150061030 2022-07-12T08:50:01.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:01.852+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f615013eaf0 con 0x7f6150061030 2022-07-12T08:50:02.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.260+0000 7f614d7fa700 1 -- 172.21.15.61:0/1760240030 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v409) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6144045b70 con 0x7f6150061030 2022-07-12T08:50:02.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6130053ad0 msgr2=0x7f6130055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6130053ad0 0x7f6130055f90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f614000b000 tx=0x7f6140009040).stop 2022-07-12T08:50:02.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 msgr2=0x7f6150064680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 0x7f6150064680 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f61440010d0 tx=0x7f6144007b00).stop 2022-07-12T08:50:02.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 shutdown_connections 2022-07-12T08:50:02.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f615013fe70 0x7f615013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6130053ad0 0x7f6130055f90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61500af0b0 0x7f615013d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 --2- 172.21.15.61:0/1760240030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6150061030 0x7f6150064680 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 >> 172.21.15.61:0/1760240030 conn(0x7f61500b5780 msgr2=0x7f61500b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:02.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 shutdown_connections 2022-07-12T08:50:02.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.264+0000 7f6155311700 1 -- 172.21.15.61:0/1760240030 wait complete. 2022-07-12T08:50:02.284 INFO:tasks.cephfs_test_runner:test_fs_new (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:50:02.286 INFO:tasks.cephfs_test_runner:Starting test: test_getattr_caps (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:50:02.287 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-12T08:50:02.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd190533700 1 Processor -- start 2022-07-12T08:50:02.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd190533700 1 -- start start 2022-07-12T08:50:02.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd190533700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd1880b2db0 0x7fd1880aee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd190533700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd188140020 con 0x7fd18813fb70 2022-07-12T08:50:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd190533700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1880b2810 con 0x7fd1880b2300 2022-07-12T08:50:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd190533700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1881432c0 con 0x7fd1880b2db0 2022-07-12T08:50:02.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18e2cf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd1880b2db0 0x7fd1880aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:02.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18ead0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd18813fb70 0x7fd188143760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48870/0 (socket says 172.21.15.61:48870) 2022-07-12T08:50:02.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18ead0700 1 -- 172.21.15.61:0/3316537602 learned_addr learned my addr 172.21.15.61:0/3316537602 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:02.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18e2cf700 1 -- 172.21.15.61:0/3316537602 >> v1:172.21.15.111:6789/0 conn(0x7fd1880b2300 legacy=0x7fd188139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18e2cf700 1 -- 172.21.15.61:0/3316537602 >> v1:172.21.15.61:6789/0 conn(0x7fd18813fb70 legacy=0x7fd188143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18d2cd700 1 -- 172.21.15.61:0/3316537602 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606903904 0 0) 0x7fd188140020 con 0x7fd18813fb70 2022-07-12T08:50:02.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18e2cf700 1 -- 172.21.15.61:0/3316537602 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188139f20 con 0x7fd1880b2db0 2022-07-12T08:50:02.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18e2cf700 1 --2- 172.21.15.61:0/3316537602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880aee80 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7fd184002f70 tx=0x7fd184006df0).ready entity=mon.0 client_cookie=5c3a13a860da4d68 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:02.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18d2cd700 1 -- 172.21.15.61:0/3316537602 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400d610 con 0x7fd1880b2db0 2022-07-12T08:50:02.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18d2cd700 1 -- 172.21.15.61:0/3316537602 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd184000e90 con 0x7fd1880b2db0 2022-07-12T08:50:02.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.408+0000 7fd18d2cd700 1 -- 172.21.15.61:0/3316537602 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18400d960 con 0x7fd1880b2db0 2022-07-12T08:50:02.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 -- 172.21.15.61:0/3316537602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 msgr2=0x7fd1880aee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 --2- 172.21.15.61:0/3316537602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880aee80 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 rx=0x7fd184002f70 tx=0x7fd184006df0).stop 2022-07-12T08:50:02.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 -- 172.21.15.61:0/3316537602 shutdown_connections 2022-07-12T08:50:02.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 --2- 172.21.15.61:0/3316537602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880aee80 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 -- 172.21.15.61:0/3316537602 >> 172.21.15.61:0/3316537602 conn(0x7fd188063dc0 msgr2=0x7fd1880641a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:02.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 -- 172.21.15.61:0/3316537602 shutdown_connections 2022-07-12T08:50:02.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 -- 172.21.15.61:0/3316537602 wait complete. 2022-07-12T08:50:02.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 Processor -- start 2022-07-12T08:50:02.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.412+0000 7fd190533700 1 -- start start 2022-07-12T08:50:02.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1880b2300 0x7fd18813aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:02.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880676e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:02.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd18813fb70 0x7fd1880b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:02.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880b4970 con 0x7fd1880b2db0 2022-07-12T08:50:02.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880b5110 con 0x7fd1880b2300 2022-07-12T08:50:02.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1880b53b0 con 0x7fd18813fb70 2022-07-12T08:50:02.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880676e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:02.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880676e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60656/0 (socket says 172.21.15.61:60656) 2022-07-12T08:50:02.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 -- 172.21.15.61:0/2237496767 learned_addr learned my addr 172.21.15.61:0/2237496767 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:02.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18e2cf700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1880b2300 0x7fd18813aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:02.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18ead0700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd18813fb70 0x7fd1880b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:02.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 -- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd18813fb70 msgr2=0x7fd1880b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd18813fb70 0x7fd1880b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 -- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1880b2300 msgr2=0x7fd18813aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:02.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1880b2300 0x7fd18813aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:02.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 -- 172.21.15.61:0/2237496767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18400b180 con 0x7fd1880b2db0 2022-07-12T08:50:02.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd18dace700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880676e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7fd18813d780 tx=0x7fd17c001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:02.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd17c00f470 con 0x7fd1880b2db0 2022-07-12T08:50:02.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c00f5d0 con 0x7fd1880b2db0 2022-07-12T08:50:02.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1880b45f0 con 0x7fd1880b2db0 2022-07-12T08:50:02.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd17c00f930 con 0x7fd1880b2db0 2022-07-12T08:50:02.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.416+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1880b4b90 con 0x7fd1880b2db0 2022-07-12T08:50:02.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.420+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188057460 con 0x7fd1880b2db0 2022-07-12T08:50:02.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.420+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd17c019030 con 0x7fd1880b2db0 2022-07-12T08:50:02.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.420+0000 7fd17b7fe700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd168053af0 0x7fd168055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:02.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.420+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd17c06b3b0 con 0x7fd1880b2db0 2022-07-12T08:50:02.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.420+0000 7fd18e2cf700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd168053af0 0x7fd168055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:02.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.420+0000 7fd18e2cf700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd168053af0 0x7fd168055fb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fd18401c000 tx=0x7fd184015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:02.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.428+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd17c042550 con 0x7fd1880b2db0 2022-07-12T08:50:02.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:02.596+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7fd1880b47e0 con 0x7fd1880b2db0 2022-07-12T08:50:03.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.272+0000 7fd17b7fe700 1 -- 172.21.15.61:0/2237496767 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v410) v1 ==== 131+0+0 (secure 0 0 0) 0x7fd17c069e60 con 0x7fd1880b2db0 2022-07-12T08:50:03.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd168053af0 msgr2=0x7fd168055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd168053af0 0x7fd168055fb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fd18401c000 tx=0x7fd184015000).stop 2022-07-12T08:50:03.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 msgr2=0x7fd1880676e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880676e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7fd18813d780 tx=0x7fd17c001400).stop 2022-07-12T08:50:03.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 shutdown_connections 2022-07-12T08:50:03.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd18813fb70 0x7fd1880b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1880b2db0 0x7fd1880676e0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd168053af0 0x7fd168055fb0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 --2- 172.21.15.61:0/2237496767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd1880b2300 0x7fd18813aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 >> 172.21.15.61:0/2237496767 conn(0x7fd188063dc0 msgr2=0x7fd18805fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:03.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 shutdown_connections 2022-07-12T08:50:03.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.276+0000 7fd190533700 1 -- 172.21.15.61:0/2237496767 wait complete. 2022-07-12T08:50:03.299 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:50:03.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.412+0000 7f053a37f700 1 Processor -- start 2022-07-12T08:50:03.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053a37f700 1 -- start start 2022-07-12T08:50:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053a37f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05340afc00 0x7f0534145c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053a37f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340b5e80 con 0x7f05340b01b0 2022-07-12T08:50:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053a37f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340b0910 con 0x7f05340b5aa0 2022-07-12T08:50:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053a37f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340b0720 con 0x7f05340afc00 2022-07-12T08:50:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f05337fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f05340b01b0 0x7f05340addc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48882/0 (socket says 172.21.15.61:48882) 2022-07-12T08:50:03.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f05337fe700 1 -- 172.21.15.61:0/4240592235 learned_addr learned my addr 172.21.15.61:0/4240592235 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:03.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053891c700 1 --2- 172.21.15.61:0/4240592235 >> v2:172.21.15.111:3300/0 conn(0x7f05340afc00 0x7f0534145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:03.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f0532ffd700 1 -- 172.21.15.61:0/4240592235 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184460578 0 0) 0x7f05340b5e80 con 0x7f05340b01b0 2022-07-12T08:50:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f0532ffd700 1 -- 172.21.15.61:0/4240592235 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0520003260 con 0x7f05340b01b0 2022-07-12T08:50:03.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053891c700 1 -- 172.21.15.61:0/4240592235 >> v1:172.21.15.111:6789/0 conn(0x7f05340b5aa0 legacy=0x7f0534145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053891c700 1 -- 172.21.15.61:0/4240592235 >> v1:172.21.15.61:6789/0 conn(0x7f05340b01b0 legacy=0x7f05340addc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f0532ffd700 1 -- 172.21.15.61:0/4240592235 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3098064073 0 0) 0x7f0520003260 con 0x7f05340b01b0 2022-07-12T08:50:03.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053891c700 1 -- 172.21.15.61:0/4240592235 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340b0bb0 con 0x7f05340afc00 2022-07-12T08:50:03.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f053891c700 1 --2- 172.21.15.61:0/4240592235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340afc00 0x7f0534145c50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f0524001800 tx=0x7f052c0091d0).ready entity=mon.1 client_cookie=eda0c263f3f522ca server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:03.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f0532ffd700 1 -- 172.21.15.61:0/4240592235 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0524001660 con 0x7f05340afc00 2022-07-12T08:50:03.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f0532ffd700 1 -- 172.21.15.61:0/4240592235 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c00b580 con 0x7f05340afc00 2022-07-12T08:50:03.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.416+0000 7f0532ffd700 1 -- 172.21.15.61:0/4240592235 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f052c00b6f0 con 0x7f05340afc00 2022-07-12T08:50:03.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- 172.21.15.61:0/4240592235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340afc00 msgr2=0x7f0534145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 --2- 172.21.15.61:0/4240592235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340afc00 0x7f0534145c50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f0524001800 tx=0x7f052c0091d0).stop 2022-07-12T08:50:03.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- 172.21.15.61:0/4240592235 shutdown_connections 2022-07-12T08:50:03.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 --2- 172.21.15.61:0/4240592235 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340afc00 0x7f0534145c50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- 172.21.15.61:0/4240592235 >> 172.21.15.61:0/4240592235 conn(0x7f05340634d0 msgr2=0x7f05340b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:03.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- 172.21.15.61:0/4240592235 shutdown_connections 2022-07-12T08:50:03.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- 172.21.15.61:0/4240592235 wait complete. 2022-07-12T08:50:03.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 Processor -- start 2022-07-12T08:50:03.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- start start 2022-07-12T08:50:03.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 0x7f0534146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:03.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05340b01b0 0x7f0534146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:03.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340b5aa0 0x7f0534145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:03.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0534151df0 con 0x7f05340afc00 2022-07-12T08:50:03.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0534147160 con 0x7f05340b5aa0 2022-07-12T08:50:03.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f053a37f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05341463c0 con 0x7f05340b01b0 2022-07-12T08:50:03.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f0533fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 0x7f0534146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:03.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f0533fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 0x7f0534146710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60668/0 (socket says 172.21.15.61:60668) 2022-07-12T08:50:03.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f0533fff700 1 -- 172.21.15.61:0/1483591724 learned_addr learned my addr 172.21.15.61:0/1483591724 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:03.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.420+0000 7f05337fe700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05340b01b0 0x7f0534146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:03.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f053891c700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340b5aa0 0x7f0534145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:03.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f0533fff700 1 -- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05340b01b0 msgr2=0x7f0534146c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f0533fff700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05340b01b0 0x7f0534146c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f0533fff700 1 -- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340b5aa0 msgr2=0x7f0534145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f0533fff700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340b5aa0 0x7f0534145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f0533fff700 1 -- 172.21.15.61:0/1483591724 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009470 con 0x7f05340afc00 2022-07-12T08:50:03.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f0533fff700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 0x7f0534146710 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f053405d230 tx=0x7f0528007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:03.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0528012400 con 0x7f05340afc00 2022-07-12T08:50:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528012560 con 0x7f05340afc00 2022-07-12T08:50:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0534151fe0 con 0x7f05340afc00 2022-07-12T08:50:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05280128c0 con 0x7f05340afc00 2022-07-12T08:50:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0534145f30 con 0x7f05340afc00 2022-07-12T08:50:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0534057460 con 0x7f05340afc00 2022-07-12T08:50:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.424+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0528019030 con 0x7f05340afc00 2022-07-12T08:50:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.428+0000 7f05317fa700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0514053ad0 0x7f0514055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.428+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f052806af20 con 0x7f05340afc00 2022-07-12T08:50:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.428+0000 7f05337fe700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0514053ad0 0x7f0514055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:03.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.428+0000 7f05337fe700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0514053ad0 0x7f0514055f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f052400d000 tx=0x7f0524001790).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:03.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.436+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0528042050 con 0x7f05340afc00 2022-07-12T08:50:03.558 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:03.562 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:03.567 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:03.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.564+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f053405d010 con 0x7f05340afc00 2022-07-12T08:50:03.578 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:03.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f05317fa700 1 -- 172.21.15.61:0/1483591724 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0528045670 con 0x7f05340afc00 2022-07-12T08:50:03.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0514053ad0 msgr2=0x7f0514055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0514053ad0 0x7f0514055f90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f052400d000 tx=0x7f0524001790).stop 2022-07-12T08:50:03.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 msgr2=0x7f0534146710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:03.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 0x7f0534146710 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f053405d230 tx=0x7f0528007ca0).stop 2022-07-12T08:50:03.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 shutdown_connections 2022-07-12T08:50:03.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0514053ad0 0x7f0514055f90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05340b5aa0 0x7f0534145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05340b01b0 0x7f0534146c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 --2- 172.21.15.61:0/1483591724 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05340afc00 0x7f0534146710 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:03.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 >> 172.21.15.61:0/1483591724 conn(0x7f05340634d0 msgr2=0x7f053405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:03.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 shutdown_connections 2022-07-12T08:50:03.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:03.576+0000 7f053a37f700 1 -- 172.21.15.61:0/1483591724 wait complete. 2022-07-12T08:50:03.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:03.593 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:50:03.594 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:03.672 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:03.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:03.738 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:03.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:03.791 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:50:03.792 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:03.799 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:03.800 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:03.833 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:50:03.834 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:03.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:03.836 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:03.836 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:03.861 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:50:03.862 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:03.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:03.880 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:03.881 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:50:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:03.893 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:50:03.894 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:50:03.895 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:50:04.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.012+0000 7f94affff700 1 Processor -- start 2022-07-12T08:50:04.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.012+0000 7f94affff700 1 -- start start 2022-07-12T08:50:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94affff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f94b00abdf0 0x7f94b0060200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94affff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00ab3f0 con 0x7f94b00b5780 2022-07-12T08:50:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94affff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b00ab950 con 0x7f94b0146c50 2022-07-12T08:50:04.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94affff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b00ab690 con 0x7f94b00abdf0 2022-07-12T08:50:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f94b00b5780 0x7f94b0146540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48894/0 (socket says 172.21.15.61:48894) 2022-07-12T08:50:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 -- 172.21.15.61:0/2649172044 learned_addr learned my addr 172.21.15.61:0/2649172044 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94af7fe700 1 --2- 172.21.15.61:0/2649172044 >> v2:172.21.15.111:3300/0 conn(0x7f94b00abdf0 0x7f94b0060200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516477413 0 0) 0x7f94b00ab3f0 con 0x7f94b00b5780 2022-07-12T08:50:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490003260 con 0x7f94b00b5780 2022-07-12T08:50:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308188019 0 0) 0x7f9490003260 con 0x7f94b00b5780 2022-07-12T08:50:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 >> v2:172.21.15.111:3300/0 conn(0x7f94b00abdf0 msgr2=0x7f94b0060200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 --2- 172.21.15.61:0/2649172044 >> v2:172.21.15.111:3300/0 conn(0x7f94b00abdf0 0x7f94b0060200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 >> v1:172.21.15.111:6789/0 conn(0x7f94b0146c50 legacy=0x7f94b005faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00b5b60 con 0x7f94b00b5780 2022-07-12T08:50:04.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f94a80019e0 con 0x7f94b00b5780 2022-07-12T08:50:04.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 >> v1:172.21.15.61:6789/0 conn(0x7f94b00b5780 legacy=0x7f94b0146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f9490003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9490004500 0x7f9490004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9490004e70 0x7f949000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b00ab690 con 0x7f9490003780 2022-07-12T08:50:04.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949000bd20 con 0x7f9490004e70 2022-07-12T08:50:04.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949000bee0 con 0x7f9490004500 2022-07-12T08:50:04.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f9490003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94af7fe700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9490004e70 0x7f949000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94ae7fc700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9490004500 0x7f9490004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 -- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9490004500 msgr2=0x7f9490004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:04.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9490004500 0x7f9490004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 -- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9490004e70 msgr2=0x7f949000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:04.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9490004e70 0x7f949000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 -- 172.21.15.61:0/2649172044 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9490003260 con 0x7f9490003780 2022-07-12T08:50:04.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94af7fe700 1 -- 172.21.15.61:0/2649172044 reap_dead start 2022-07-12T08:50:04.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94ae7fc700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9490004500 0x7f9490004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:04.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94aeffd700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f9490003ba0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f94a80016f0 tx=0x7f94a8002190).ready entity=mon.0 client_cookie=eed791a559de220 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:04.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a800cc90 con 0x7f9490003780 2022-07-12T08:50:04.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a800cdf0 con 0x7f9490003780 2022-07-12T08:50:04.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.016+0000 7f94adffb700 1 -- 172.21.15.61:0/2649172044 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a8005b80 con 0x7f9490003780 2022-07-12T08:50:04.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 -- 172.21.15.61:0/2649172044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 msgr2=0x7f9490003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f9490003ba0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f94a80016f0 tx=0x7f94a8002190).stop 2022-07-12T08:50:04.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 -- 172.21.15.61:0/2649172044 shutdown_connections 2022-07-12T08:50:04.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 --2- 172.21.15.61:0/2649172044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f9490003ba0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 -- 172.21.15.61:0/2649172044 >> 172.21.15.61:0/2649172044 conn(0x7f94b0063380 msgr2=0x7f94b00b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:04.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 -- 172.21.15.61:0/2649172044 shutdown_connections 2022-07-12T08:50:04.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 -- 172.21.15.61:0/2649172044 wait complete. 2022-07-12T08:50:04.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 Processor -- start 2022-07-12T08:50:04.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 -- start start 2022-07-12T08:50:04.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f94b00ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.020+0000 7f94affff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0097920 0x7f94b0148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94b0065220 0x7f94b0149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0065c10 con 0x7f9490003780 2022-07-12T08:50:04.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0065eb0 con 0x7f94b0097920 2022-07-12T08:50:04.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b0066150 con 0x7f94b0065220 2022-07-12T08:50:04.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f94b00ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f94b00ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60688/0 (socket says 172.21.15.61:60688) 2022-07-12T08:50:04.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 -- 172.21.15.61:0/3366089328 learned_addr learned my addr 172.21.15.61:0/3366089328 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:04.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94af7fe700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94b0065220 0x7f94b0149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94ae7fc700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0097920 0x7f94b0148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 -- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94b0065220 msgr2=0x7f94b0149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94b0065220 0x7f94b0149320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 -- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0097920 msgr2=0x7f94b0148e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0097920 0x7f94b0148e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 -- 172.21.15.61:0/3366089328 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b00665f0 con 0x7f9490003780 2022-07-12T08:50:04.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94aeffd700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f94b00ae1e0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f94a80016f0 tx=0x7f94a8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:04.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a80074d0 con 0x7f9490003780 2022-07-12T08:50:04.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8007630 con 0x7f9490003780 2022-07-12T08:50:04.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b0064ed0 con 0x7f9490003780 2022-07-12T08:50:04.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a8005e20 con 0x7f9490003780 2022-07-12T08:50:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b0149a30 con 0x7f9490003780 2022-07-12T08:50:04.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b0066310 con 0x7f9490003780 2022-07-12T08:50:04.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.024+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f94a8010070 con 0x7f9490003780 2022-07-12T08:50:04.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.028+0000 7f94a7fff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9494053af0 0x7f9494055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.028+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f94a806cb30 con 0x7f9490003780 2022-07-12T08:50:04.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.028+0000 7f94ae7fc700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9494053af0 0x7f9494055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.028+0000 7f94ae7fc700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9494053af0 0x7f9494055fb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f949800d000 tx=0x7f9498009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:04.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.032+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94a8043d70 con 0x7f9490003780 2022-07-12T08:50:04.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.160+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f94b0057460 con 0x7f9490003780 2022-07-12T08:50:04.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94a7fff700 1 -- 172.21.15.61:0/3366089328 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v123) v1 ==== 81+0+28885 (secure 0 0 0) 0x7f94a8025030 con 0x7f9490003780 2022-07-12T08:50:04.167 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:04.167 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:50:04.168 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 123, 2022-07-12T08:50:04.173 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:50:04.174 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:50:04.174 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:50:01.243147+0000", 2022-07-12T08:50:04.174 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:04.175 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:04.175 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:50:04.176 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:50:04.176 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:50:04.177 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:50:04.177 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:50:04.178 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:50:04.178 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:50:04.182 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:04.183 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:50:04.183 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:50:04.183 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:50:04.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:50:04.184 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:50:04.185 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 24, 2022-07-12T08:50:04.186 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:50:04.186 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:50:04.186 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:50:04.187 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:50:04.187 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:50:04.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.188 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:50:04.189 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:50:04.189 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:50:04.189 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:04.191 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:04.192 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:04.192 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:04.193 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:04.193 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:04.194 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:50:04.194 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:50:04.194 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:50:04.195 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:50:04.195 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:50:04.196 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:04.196 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:04.196 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:04.197 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:04.197 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:04.198 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:04.198 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:04.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.199 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:50:04.200 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:04.200 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:04.200 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:04.201 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:04.203 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:04.203 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:04.204 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:04.204 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:04.205 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:04.205 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:04.205 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:04.206 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:04.206 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:04.207 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:04.207 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:04.207 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:04.208 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:04.208 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:04.209 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:04.209 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:04.210 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:04.210 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:04.210 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:04.211 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:04.211 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:04.212 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:04.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.213 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:04.213 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:04.213 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:04.214 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:04.214 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:04.215 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:04.216 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:04.216 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:04.217 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:04.217 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:04.217 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:04.218 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:50:04.218 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:50:04.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.219 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:04.219 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:50:04.220 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.221 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.221 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.222 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:50:04.222 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:50:04.222 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:50:04.223 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:04.223 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:04.224 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:04.224 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:04.224 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:04.225 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:04.226 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:04.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:04.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:04.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:04.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:04.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:04.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:04.229 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:04.229 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:04.230 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:04.230 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:04.230 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:04.231 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:04.231 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:04.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.232 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:50:04.232 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:04.233 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:04.233 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:04.233 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:04.234 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:04.234 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:04.235 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:04.235 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:04.235 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:04.236 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:04.237 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:04.237 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:04.237 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:04.237 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:04.238 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:04.238 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:04.238 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:04.238 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:04.239 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:04.239 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:04.239 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:04.247 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:04.247 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:04.247 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:04.248 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:04.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:04.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.250 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:04.250 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:04.251 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:04.251 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:04.251 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:04.252 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:04.252 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:04.253 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:04.253 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:04.254 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:04.254 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:04.255 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:50:04.255 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:04.256 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:50:04.256 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.256 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.257 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:04.257 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:50:04.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.258 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:04.259 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:50:04.259 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.260 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.260 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.260 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.262 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.262 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:04.263 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.263 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.263 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.265 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:50:04.265 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.268 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:50:04.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.269 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.270 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.270 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.271 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.271 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.272 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:50:04.272 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.274 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.274 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:50:04.275 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.275 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.276 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.278 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:50:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:50:04.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.281 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.284 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:50:04.285 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.285 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.286 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.286 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:50:04.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:04.287 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:50:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:50:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:50:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:50:04.290 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.290 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.291 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.291 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.292 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:04.293 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:50:04.293 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.293 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.294 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.294 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.296 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.296 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:04.296 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.297 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.297 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.299 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:50:04.300 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.300 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:50:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:50:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.306 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.306 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:50:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.308 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.309 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.309 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.319 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.320 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.320 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:50:04.320 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:50:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.322 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.322 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.322 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.323 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.323 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.323 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.323 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.323 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:50:04.324 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.324 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.325 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:50:04.326 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:04.327 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.328 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.328 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:50:04.329 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:50:04.329 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:50:04.330 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:50:04.330 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.331 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.331 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.332 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.333 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.333 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:04.334 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:50:04.334 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.335 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.335 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.336 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.336 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.337 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.337 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.337 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 114, 2022-07-12T08:50:04.338 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.338 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.339 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.339 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.340 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.340 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:50:04.341 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.341 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.342 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.342 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:50:04.343 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.343 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.343 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.344 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.344 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.345 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.345 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:50:04.346 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.346 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.346 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.347 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.347 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:50:04.347 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.347 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.348 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.349 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.349 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.350 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.351 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:50:04.351 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.352 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.352 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.353 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.353 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:50:04.353 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.354 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.354 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.354 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.355 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.355 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.356 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.356 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:50:04.357 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.358 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:50:04.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:04.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.360 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.360 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:50:04.361 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:50:04.361 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:50:04.361 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:50:04.362 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.362 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.363 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.363 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.366 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:04.367 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:50:04.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.368 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.368 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.369 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.369 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.370 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.370 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.370 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:04.371 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.371 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.372 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.372 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.372 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.373 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.373 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:50:04.373 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.374 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:50:04.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.375 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.376 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.376 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.376 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.378 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.378 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:50:04.378 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.379 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.379 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:50:04.380 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.381 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.382 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.382 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.382 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.383 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.383 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.384 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.384 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:50:04.384 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.385 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.385 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.385 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.386 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:50:04.386 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.387 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.387 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.387 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.388 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.389 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:50:04.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.399 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:50:04.400 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:04.401 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.402 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:50:04.402 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:50:04.402 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:50:04.403 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:50:04.403 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.404 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.404 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.404 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.405 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.405 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.405 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:04.406 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:50:04.406 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.408 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.408 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.408 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 114, 2022-07-12T08:50:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.410 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.410 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.411 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:50:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.413 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:50:04.414 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.415 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.415 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:50:04.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.419 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.419 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.420 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:50:04.420 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.420 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:50:04.427 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.427 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.427 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.428 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:50:04.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.429 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.431 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:50:04.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.437 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:50:04.438 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:04.438 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.439 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.439 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.439 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:50:04.440 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:50:04.440 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:50:04.440 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:50:04.441 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.441 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.441 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.442 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.442 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.442 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.443 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:04.443 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:50:04.443 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.444 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.444 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.444 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.445 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.445 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.445 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.446 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:04.452 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.453 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.453 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.454 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.454 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:50:04.454 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.455 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.455 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:50:04.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.456 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.456 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.456 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.457 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.457 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.458 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:50:04.458 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.458 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.458 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.459 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.459 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:50:04.459 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.459 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.460 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.460 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.460 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.460 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.460 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.461 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.461 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:50:04.461 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.462 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.462 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.463 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:50:04.463 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.463 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.463 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.464 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.466 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.466 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.467 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.467 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.467 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:50:04.467 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.468 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.468 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.468 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.468 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:50:04.468 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:04.469 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.469 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.469 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.469 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:50:04.470 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:50:04.470 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:50:04.470 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:50:04.470 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.471 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.471 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.471 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.472 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.474 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:04.475 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:50:04.475 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.475 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.475 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.476 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.476 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.476 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.477 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.477 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:04.478 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.478 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.479 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.479 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.479 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.480 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.480 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:50:04.481 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.481 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.482 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.482 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.482 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:50:04.483 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.484 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.484 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.485 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.485 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.488 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.489 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.489 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:50:04.490 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.505 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.506 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.506 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.508 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:50:04.508 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.509 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.509 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.509 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.510 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.510 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.513 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.513 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.542 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:50:04.543 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.543 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.544 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.544 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.544 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:50:04.545 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.545 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.546 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.546 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.547 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.547 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.548 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.549 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:50:04.549 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.550 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.550 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.550 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:50:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:04.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.552 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:50:04.553 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:50:04.569 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:50:04.570 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:50:04.570 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.576 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.577 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.577 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.578 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.578 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.579 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:04.579 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:50:04.579 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:04.580 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:04.580 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:04.580 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:04.581 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:04.582 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:04.582 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:04.582 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:04.583 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:04.583 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:04.601 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:04.602 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.602 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.602 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.602 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:50:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:50:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.606 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:50:04.615 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.617 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.619 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.619 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.620 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:50:04.620 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.620 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.620 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.621 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.621 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:04.621 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.621 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.622 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.622 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:50:04.622 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.622 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.623 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.623 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.623 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:50:04.623 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.625 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.626 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.626 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.626 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:04.627 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:04.627 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.636 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:04.636 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:50:04.637 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.637 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.637 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.637 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:04.641 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:50:04.642 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:04.643 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.643 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.644 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.644 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:50:04.645 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:50:04.645 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:50:04.645 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:50:04.646 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:04.647 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:04.647 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:04.647 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:04.648 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.648 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:04.649 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:50:04.649 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.650 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:04.650 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.651 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.651 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.651 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.652 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.652 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:50:04.653 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.653 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.654 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.654 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:04.655 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.655 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.656 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.656 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.657 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.657 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:50:04.658 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.658 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.659 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.659 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:04.660 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.660 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.661 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.661 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.661 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.662 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:50:04.662 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.663 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.663 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.664 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:04.664 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.664 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.665 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.665 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.665 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.667 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:50:04.668 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.668 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.668 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.669 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:04.669 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.670 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.670 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.671 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.671 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.672 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:50:04.672 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.672 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.673 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.673 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:04.674 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.674 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.675 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.675 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.676 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.676 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:50:04.677 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.677 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.677 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.678 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:04.678 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.679 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.679 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.679 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.680 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.680 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:50:04.681 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.681 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.682 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:04.682 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:04.683 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:04.683 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:04.683 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:04.684 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:04.684 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:04.685 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:50:04.685 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:04.685 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.686 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:04.687 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:50:04.687 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:50:04.687 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:50:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:50:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:50:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2867307616": "2022-07-13T08:49:42.084294+0000", 2022-07-12T08:50:04.689 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2867307616": "2022-07-13T08:49:42.084294+0000" 2022-07-12T08:50:04.689 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.690 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:50:04.690 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:50:04.691 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:50:04.691 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:50:04.692 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:50:04.692 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:50:04.692 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:50:04.694 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:50:04.694 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:04.694 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:04.695 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:50:04.695 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:50:04.696 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:50:04.696 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:50:04.697 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:50:04.697 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:50:04.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9494053af0 msgr2=0x7f9494055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9494053af0 0x7f9494055fb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f949800d000 tx=0x7f9498009040).stop 2022-07-12T08:50:04.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 msgr2=0x7f94b00ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f94b00ae1e0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 rx=0x7f94a80016f0 tx=0x7f94a8015000).stop 2022-07-12T08:50:04.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 shutdown_connections 2022-07-12T08:50:04.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94b0065220 0x7f94b0149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9494053af0 0x7f9494055fb0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94b0097920 0x7f94b0148e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 --2- 172.21.15.61:0/3366089328 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9490003780 0x7f94b00ae1e0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 >> 172.21.15.61:0/3366089328 conn(0x7f94b0063380 msgr2=0x7f94b0147a90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 shutdown_connections 2022-07-12T08:50:04.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.164+0000 7f94affff700 1 -- 172.21.15.61:0/3366089328 wait complete. 2022-07-12T08:50:04.707 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:04.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 Processor -- start 2022-07-12T08:50:04.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 -- start start 2022-07-12T08:50:04.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47900ab290 0x7f47900ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f479005f140 0x7f479005ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47900abc70 con 0x7f47900afa60 2022-07-12T08:50:04.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479005f580 con 0x7f47900ab290 2022-07-12T08:50:04.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47950be700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479005c720 con 0x7f479005f140 2022-07-12T08:50:04.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f478ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47900ab290 0x7f47900ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f478f7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f479005f140 0x7f479005ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47948bd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f47900afa60 0x7f479014a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34288/0 (socket says 172.21.15.61:34288) 2022-07-12T08:50:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.816+0000 7f47948bd700 1 -- 172.21.15.61:0/4271529831 learned_addr learned my addr 172.21.15.61:0/4271529831 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:04.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478f7fe700 1 -- 172.21.15.61:0/4271529831 >> v2:172.21.15.61:3300/0 conn(0x7f47900ab290 msgr2=0x7f47900ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:04.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478f7fe700 1 --2- 172.21.15.61:0/4271529831 >> v2:172.21.15.61:3300/0 conn(0x7f47900ab290 0x7f47900ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478f7fe700 1 -- 172.21.15.61:0/4271529831 >> v1:172.21.15.111:6789/0 conn(0x7f47900afa60 legacy=0x7f479014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478f7fe700 1 -- 172.21.15.61:0/4271529831 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479005c9c0 con 0x7f479005f140 2022-07-12T08:50:04.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478f7fe700 1 --2- 172.21.15.61:0/4271529831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f479005f140 0x7f479005ce60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4788001490 tx=0x7f4780006df0).ready entity=mon.2 client_cookie=977003f018675a41 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:04.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478effd700 1 -- 172.21.15.61:0/4271529831 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f478000d4a0 con 0x7f479005f140 2022-07-12T08:50:04.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478effd700 1 -- 172.21.15.61:0/4271529831 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4780000e90 con 0x7f479005f140 2022-07-12T08:50:04.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f478effd700 1 -- 172.21.15.61:0/4271529831 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f478000d7f0 con 0x7f479005f140 2022-07-12T08:50:04.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 -- 172.21.15.61:0/4271529831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f479005f140 msgr2=0x7f479005ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 --2- 172.21.15.61:0/4271529831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f479005f140 0x7f479005ce60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4788001490 tx=0x7f4780006df0).stop 2022-07-12T08:50:04.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 -- 172.21.15.61:0/4271529831 shutdown_connections 2022-07-12T08:50:04.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 --2- 172.21.15.61:0/4271529831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f479005f140 0x7f479005ce60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 --2- 172.21.15.61:0/4271529831 >> v2:172.21.15.61:3300/0 conn(0x7f47900ab290 0x7f47900ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 -- 172.21.15.61:0/4271529831 >> 172.21.15.61:0/4271529831 conn(0x7f47900b1950 msgr2=0x7f47900b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:04.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 -- 172.21.15.61:0/4271529831 shutdown_connections 2022-07-12T08:50:04.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.820+0000 7f47950be700 1 -- 172.21.15.61:0/4271529831 wait complete. 2022-07-12T08:50:04.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 Processor -- start 2022-07-12T08:50:04.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 -- start start 2022-07-12T08:50:04.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f479005f140 0x7f4790156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47900ab290 0x7f4790066740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 0x7f47900ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47900639b0 con 0x7f47900afa60 2022-07-12T08:50:04.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4790157000 con 0x7f479005f140 2022-07-12T08:50:04.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47900671b0 con 0x7f47900ab290 2022-07-12T08:50:04.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 0x7f47900ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 0x7f47900ad1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60706/0 (socket says 172.21.15.61:60706) 2022-07-12T08:50:04.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 -- 172.21.15.61:0/874004238 learned_addr learned my addr 172.21.15.61:0/874004238 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:04.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f478ffff700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f479005f140 0x7f4790156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f478f7fe700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47900ab290 0x7f4790066740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 -- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47900ab290 msgr2=0x7f4790066740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47900ab290 0x7f4790066740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 -- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f479005f140 msgr2=0x7f4790156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:04.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f479005f140 0x7f4790156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:04.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 -- 172.21.15.61:0/874004238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f478000b180 con 0x7f47900afa60 2022-07-12T08:50:04.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47948bd700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 0x7f47900ad1e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f4788001360 tx=0x7f4788007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:04.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4788005ba0 con 0x7f47900afa60 2022-07-12T08:50:04.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788005d00 con 0x7f47900afa60 2022-07-12T08:50:04.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4790067450 con 0x7f47900afa60 2022-07-12T08:50:04.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.824+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4788007300 con 0x7f47900afa60 2022-07-12T08:50:04.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47900aea80 con 0x7f47900afa60 2022-07-12T08:50:04.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4790156910 con 0x7f47900afa60 2022-07-12T08:50:04.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4788010030 con 0x7f47900afa60 2022-07-12T08:50:04.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f478d7fa700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4770053a40 0x7f4770055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:04.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f478806bc50 con 0x7f47900afa60 2022-07-12T08:50:04.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f478ffff700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4770053a40 0x7f4770055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:04.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.828+0000 7f478ffff700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4770053a40 0x7f4770055f00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f478400b000 tx=0x7f4784006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:04.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.836+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4788042df0 con 0x7f47900afa60 2022-07-12T08:50:04.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.996+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47900aed60 con 0x7f47900afa60 2022-07-12T08:50:05.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:04.996+0000 7f478d7fa700 1 -- 172.21.15.61:0/874004238 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+504 (secure 0 0 0) 0x7f4788046410 con 0x7f47900afa60 2022-07-12T08:50:05.001 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:05.012 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-12T08:50:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4770053a40 msgr2=0x7f4770055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4770053a40 0x7f4770055f00 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7f478400b000 tx=0x7f4784006df0).stop 2022-07-12T08:50:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 msgr2=0x7f47900ad1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 0x7f47900ad1e0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7f4788001360 tx=0x7f4788007f40).stop 2022-07-12T08:50:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 shutdown_connections 2022-07-12T08:50:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47900afa60 0x7f47900ad1e0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f47900ab290 0x7f4790066740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4770053a40 0x7f4770055f00 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 --2- 172.21.15.61:0/874004238 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f479005f140 0x7f4790156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 >> 172.21.15.61:0/874004238 conn(0x7f47900b1950 msgr2=0x7f479005bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:05.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 shutdown_connections 2022-07-12T08:50:05.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.000+0000 7f47950be700 1 -- 172.21.15.61:0/874004238 wait complete. 2022-07-12T08:50:05.025 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 69 2022-07-12T08:50:05.027 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:05.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 Processor -- start 2022-07-12T08:50:05.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 -- start start 2022-07-12T08:50:05.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55840ab290 0x7f55840ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f55840afa60 0x7f55840aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55840abc70 con 0x7f558405f140 2022-07-12T08:50:05.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558405f580 con 0x7f55840afa60 2022-07-12T08:50:05.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5589257700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558405c720 con 0x7f55840ab290 2022-07-12T08:50:05.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5588a56700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f55840afa60 0x7f55840aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f55837fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f558405f140 0x7f558405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34298/0 (socket says 172.21.15.61:34298) 2022-07-12T08:50:05.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f55837fe700 1 -- 172.21.15.61:0/1591661493 learned_addr learned my addr 172.21.15.61:0/1591661493 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:05.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5583fff700 1 --2- 172.21.15.61:0/1591661493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55840ab290 0x7f55840ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5582ffd700 1 -- 172.21.15.61:0/1591661493 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284268949 0 0) 0x7f55840abc70 con 0x7f558405f140 2022-07-12T08:50:05.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5588a56700 1 -- 172.21.15.61:0/1591661493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55840ab290 msgr2=0x7f55840ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5588a56700 1 --2- 172.21.15.61:0/1591661493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55840ab290 0x7f55840ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5588a56700 1 -- 172.21.15.61:0/1591661493 >> v1:172.21.15.111:6789/0 conn(0x7f558405f140 legacy=0x7f558405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5588a56700 1 -- 172.21.15.61:0/1591661493 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558405c9c0 con 0x7f55840afa60 2022-07-12T08:50:05.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5588a56700 1 --2- 172.21.15.61:0/1591661493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f55840aff10 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f557c0032f0 tx=0x7f557c0091d0).ready entity=mon.0 client_cookie=c42eba379ecf9444 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:05.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5582ffd700 1 -- 172.21.15.61:0/1591661493 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f557c00b6c0 con 0x7f55840afa60 2022-07-12T08:50:05.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5582ffd700 1 -- 172.21.15.61:0/1591661493 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c00b820 con 0x7f55840afa60 2022-07-12T08:50:05.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.156+0000 7f5582ffd700 1 -- 172.21.15.61:0/1591661493 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f557c00b990 con 0x7f55840afa60 2022-07-12T08:50:05.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- 172.21.15.61:0/1591661493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 msgr2=0x7f55840aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 --2- 172.21.15.61:0/1591661493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f55840aff10 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 rx=0x7f557c0032f0 tx=0x7f557c0091d0).stop 2022-07-12T08:50:05.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- 172.21.15.61:0/1591661493 shutdown_connections 2022-07-12T08:50:05.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 --2- 172.21.15.61:0/1591661493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f55840aff10 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 --2- 172.21.15.61:0/1591661493 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55840ab290 0x7f55840ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- 172.21.15.61:0/1591661493 >> 172.21.15.61:0/1591661493 conn(0x7f55840b1950 msgr2=0x7f55840b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:05.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- 172.21.15.61:0/1591661493 shutdown_connections 2022-07-12T08:50:05.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- 172.21.15.61:0/1591661493 wait complete. 2022-07-12T08:50:05.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 Processor -- start 2022-07-12T08:50:05.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- start start 2022-07-12T08:50:05.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f558405f140 0x7f55840669d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55840ab290 0x7f5584097030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f5584097520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5584097a10 con 0x7f55840afa60 2022-07-12T08:50:05.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5584156aa0 con 0x7f55840ab290 2022-07-12T08:50:05.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5589257700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55841560c0 con 0x7f558405f140 2022-07-12T08:50:05.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f5584097520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f5584097520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60720/0 (socket says 172.21.15.61:60720) 2022-07-12T08:50:05.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 -- 172.21.15.61:0/3147493810 learned_addr learned my addr 172.21.15.61:0/3147493810 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:05.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5583fff700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f558405f140 0x7f55840669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f55837fe700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55840ab290 0x7f5584097030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 -- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f558405f140 msgr2=0x7f55840669d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f558405f140 0x7f55840669d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 -- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55840ab290 msgr2=0x7f5584097030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55840ab290 0x7f5584097030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 -- 172.21.15.61:0/3147493810 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c009470 con 0x7f55840afa60 2022-07-12T08:50:05.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f55837fe700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55840ab290 0x7f5584097030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:05.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.160+0000 7f5588a56700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f5584097520 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f557c0032f0 tx=0x7f557c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:05.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f557c00b990 con 0x7f55840afa60 2022-07-12T08:50:05.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c0107a0 con 0x7f55840afa60 2022-07-12T08:50:05.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5584156360 con 0x7f55840afa60 2022-07-12T08:50:05.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f557c00b6c0 con 0x7f55840afa60 2022-07-12T08:50:05.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55840678f0 con 0x7f55840afa60 2022-07-12T08:50:05.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5584057460 con 0x7f55840afa60 2022-07-12T08:50:05.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f557c027020 con 0x7f55840afa60 2022-07-12T08:50:05.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f55817fa700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55640539d0 0x7f5564055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f557c06d100 con 0x7f55840afa60 2022-07-12T08:50:05.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.164+0000 7f5583fff700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55640539d0 0x7f5564055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.168+0000 7f5583fff700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55640539d0 0x7f5564055e90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f557800b000 tx=0x7f5578006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:05.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.168+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f557c044230 con 0x7f55840afa60 2022-07-12T08:50:05.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55841567d0 con 0x7f55840afa60 2022-07-12T08:50:05.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f55817fa700 1 -- 172.21.15.61:0/3147493810 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+504 (secure 0 0 0) 0x7f557c015090 con 0x7f55840afa60 2022-07-12T08:50:05.339 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:05.339 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-12T08:50:05.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55640539d0 msgr2=0x7f5564055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55640539d0 0x7f5564055e90 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f557800b000 tx=0x7f5578006df0).stop 2022-07-12T08:50:05.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 msgr2=0x7f5584097520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f5584097520 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7f557c0032f0 tx=0x7f557c016000).stop 2022-07-12T08:50:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 shutdown_connections 2022-07-12T08:50:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55840afa60 0x7f5584097520 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55640539d0 0x7f5564055e90 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55840ab290 0x7f5584097030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 --2- 172.21.15.61:0/3147493810 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f558405f140 0x7f55840669d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 >> 172.21.15.61:0/3147493810 conn(0x7f55840b1950 msgr2=0x7f55840b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 shutdown_connections 2022-07-12T08:50:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.332+0000 7f5589257700 1 -- 172.21.15.61:0/3147493810 wait complete. 2022-07-12T08:50:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 69 2022-07-12T08:50:05.349 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:50:05.349 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:50:05.350 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:50:05.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:50:05.362 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:50:05.363 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:50:05.363 DEBUG:teuthology.parallel:result is None 2022-07-12T08:50:05.364 DEBUG:teuthology.parallel:result is None 2022-07-12T08:50:05.364 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:50:05.463 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:50:05.464 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:50:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.484+0000 7f105cc2b700 1 Processor -- start 2022-07-12T08:50:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.484+0000 7f105cc2b700 1 -- start start 2022-07-12T08:50:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f1058146220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10581420b0 0x7f1058142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580b6ea0 con 0x7f105805f550 2022-07-12T08:50:05.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1058146710 con 0x7f10581420b0 2022-07-12T08:50:05.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1058060b40 con 0x7f10580b6ac0 2022-07-12T08:50:05.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10581420b0 0x7f1058142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10581420b0 0x7f1058142540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60740/0 (socket says 172.21.15.61:60740) 2022-07-12T08:50:05.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 -- 172.21.15.61:0/2398809340 learned_addr learned my addr 172.21.15.61:0/2398809340 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:05.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f10577fe700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f1058146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 -- 172.21.15.61:0/2398809340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 msgr2=0x7f1058146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f1058146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 -- 172.21.15.61:0/2398809340 >> v1:172.21.15.61:6789/0 conn(0x7f105805f550 legacy=0x7f1058060430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 -- 172.21.15.61:0/2398809340 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1058060200 con 0x7f10581420b0 2022-07-12T08:50:05.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1056ffd700 1 -- 172.21.15.61:0/2398809340 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415222450 0 0) 0x7f10580b6ea0 con 0x7f105805f550 2022-07-12T08:50:05.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f10577fe700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f1058146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:05.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1057fff700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f1058142540 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f10500032f0 tx=0x7f1050009290).ready entity=mon.0 client_cookie=195c25e0fc0571b4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:05.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1056ffd700 1 -- 172.21.15.61:0/2398809340 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f105000b8a0 con 0x7f10581420b0 2022-07-12T08:50:05.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1056ffd700 1 -- 172.21.15.61:0/2398809340 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105000ba00 con 0x7f10581420b0 2022-07-12T08:50:05.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f1056ffd700 1 -- 172.21.15.61:0/2398809340 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f105000bb70 con 0x7f10581420b0 2022-07-12T08:50:05.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- 172.21.15.61:0/2398809340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 msgr2=0x7f1058142540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f1058142540 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f10500032f0 tx=0x7f1050009290).stop 2022-07-12T08:50:05.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- 172.21.15.61:0/2398809340 shutdown_connections 2022-07-12T08:50:05.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f1058142540 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 --2- 172.21.15.61:0/2398809340 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f1058146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- 172.21.15.61:0/2398809340 >> 172.21.15.61:0/2398809340 conn(0x7f1058064570 msgr2=0x7f10580b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- 172.21.15.61:0/2398809340 shutdown_connections 2022-07-12T08:50:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- 172.21.15.61:0/2398809340 wait complete. 2022-07-12T08:50:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 Processor -- start 2022-07-12T08:50:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.488+0000 7f105cc2b700 1 -- start start 2022-07-12T08:50:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f105805f550 0x7f10580af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f10580657d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f105813de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580ad830 con 0x7f10581420b0 2022-07-12T08:50:05.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580afd90 con 0x7f105805f550 2022-07-12T08:50:05.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580ad590 con 0x7f10580b6ac0 2022-07-12T08:50:05.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f105813de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f105813de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60746/0 (socket says 172.21.15.61:60746) 2022-07-12T08:50:05.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 -- 172.21.15.61:0/4130677210 learned_addr learned my addr 172.21.15.61:0/4130677210 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:05.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f104effd700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f10580657d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f10577fe700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f105805f550 0x7f10580af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 -- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 msgr2=0x7f10580657d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f10580657d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 -- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f105805f550 msgr2=0x7f10580af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:05.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f105805f550 0x7f10580af1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:05.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 -- 172.21.15.61:0/4130677210 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1050009530 con 0x7f10581420b0 2022-07-12T08:50:05.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f10577fe700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f105805f550 0x7f10580af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:05.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f1057fff700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f105813de70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f10500032f0 tx=0x7f1050016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:05.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1050021670 con 0x7f10581420b0 2022-07-12T08:50:05.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10500217d0 con 0x7f10581420b0 2022-07-12T08:50:05.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10580b5790 con 0x7f10581420b0 2022-07-12T08:50:05.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1050021b30 con 0x7f10581420b0 2022-07-12T08:50:05.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580ada00 con 0x7f10581420b0 2022-07-12T08:50:05.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058057460 con 0x7f10581420b0 2022-07-12T08:50:05.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.492+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f105000f070 con 0x7f10581420b0 2022-07-12T08:50:05.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.496+0000 7f10557fa700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f103c0539b0 0x7f103c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:05.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.496+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f105006cd70 con 0x7f10581420b0 2022-07-12T08:50:05.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.496+0000 7f10577fe700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f103c0539b0 0x7f103c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:05.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.496+0000 7f10577fe700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f103c0539b0 0x7f103c055e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f104800c000 tx=0x7f1048008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:05.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.500+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1050044060 con 0x7f10581420b0 2022-07-12T08:50:05.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:05.640+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f1058068350 con 0x7f10581420b0 2022-07-12T08:50:06.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.440+0000 7f10557fa700 1 -- 172.21.15.61:0/4130677210 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v124) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1050015070 con 0x7f10581420b0 2022-07-12T08:50:06.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f103c0539b0 msgr2=0x7f103c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f103c0539b0 0x7f103c055e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f104800c000 tx=0x7f1048008040).stop 2022-07-12T08:50:06.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 msgr2=0x7f105813de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f105813de70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f10500032f0 tx=0x7f1050016000).stop 2022-07-12T08:50:06.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 shutdown_connections 2022-07-12T08:50:06.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581420b0 0x7f105813de70 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10580b6ac0 0x7f10580657d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f103c0539b0 0x7f103c055e70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 --2- 172.21.15.61:0/4130677210 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f105805f550 0x7f10580af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 >> 172.21.15.61:0/4130677210 conn(0x7f1058064570 msgr2=0x7f1058064aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:06.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 shutdown_connections 2022-07-12T08:50:06.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.444+0000 7f105cc2b700 1 -- 172.21.15.61:0/4130677210 wait complete. 2022-07-12T08:50:06.473 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:50:06.477 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:50:06.478 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:50:06.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.632+0000 7f7d2cc54700 1 Processor -- start 2022-07-12T08:50:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.632+0000 7f7d2cc54700 1 -- start start 2022-07-12T08:50:06.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d280b0df0 0x7f7d28146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:06.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d2813fcb0 0x7f7d28140140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:06.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2805f1e0 con 0x7f7d28065490 2022-07-12T08:50:06.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d280b11d0 con 0x7f7d2813fcb0 2022-07-12T08:50:06.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28060170 con 0x7f7d280b0df0 2022-07-12T08:50:06.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d27fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7d2813fcb0 0x7f7d28140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:06.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d26ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7d28065490 0x7f7d280b6060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34340/0 (socket says 172.21.15.61:34340) 2022-07-12T08:50:06.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d26ffd700 1 -- 172.21.15.61:0/2026758555 learned_addr learned my addr 172.21.15.61:0/2026758555 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:06.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d277fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d280b0df0 0x7f7d28146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:06.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d27fff700 1 -- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d280b0df0 msgr2=0x7f7d28146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:06.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d27fff700 1 --2- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d280b0df0 0x7f7d28146140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d27fff700 1 -- 172.21.15.61:0/2026758555 >> v1:172.21.15.111:6789/0 conn(0x7f7d28065490 legacy=0x7f7d280b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d27fff700 1 -- 172.21.15.61:0/2026758555 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2805f410 con 0x7f7d2813fcb0 2022-07-12T08:50:06.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d27fff700 1 --2- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d2813fcb0 0x7f7d28140140 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f7d200032f0 tx=0x7f7d200091d0).ready entity=mon.1 client_cookie=3ef888db61e2717 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:06.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d267fc700 1 -- 172.21.15.61:0/2026758555 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d2000b6c0 con 0x7f7d2813fcb0 2022-07-12T08:50:06.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d267fc700 1 -- 172.21.15.61:0/2026758555 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000b820 con 0x7f7d2813fcb0 2022-07-12T08:50:06.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d267fc700 1 -- 172.21.15.61:0/2026758555 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d2000b990 con 0x7f7d2813fcb0 2022-07-12T08:50:06.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d2813fcb0 msgr2=0x7f7d28140140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d2813fcb0 0x7f7d28140140 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f7d200032f0 tx=0x7f7d200091d0).stop 2022-07-12T08:50:06.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- 172.21.15.61:0/2026758555 shutdown_connections 2022-07-12T08:50:06.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d2813fcb0 0x7f7d28140140 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/2026758555 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d280b0df0 0x7f7d28146140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- 172.21.15.61:0/2026758555 >> 172.21.15.61:0/2026758555 conn(0x7f7d28063ad0 msgr2=0x7f7d28063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:06.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- 172.21.15.61:0/2026758555 shutdown_connections 2022-07-12T08:50:06.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.636+0000 7f7d2cc54700 1 -- 172.21.15.61:0/2026758555 wait complete. 2022-07-12T08:50:06.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 Processor -- start 2022-07-12T08:50:06.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 -- start start 2022-07-12T08:50:06.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d280b0df0 0x7f7d280b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:06.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d2813fcb0 0x7f7d28146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:06.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 0x7f7d280b5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:06.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28146190 con 0x7f7d281552d0 2022-07-12T08:50:06.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281450e0 con 0x7f7d280b0df0 2022-07-12T08:50:06.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d281463c0 con 0x7f7d2813fcb0 2022-07-12T08:50:06.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 0x7f7d280b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:06.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 0x7f7d280b5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60762/0 (socket says 172.21.15.61:60762) 2022-07-12T08:50:06.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 -- 172.21.15.61:0/968739168 learned_addr learned my addr 172.21.15.61:0/968739168 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:06.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d26ffd700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d2813fcb0 0x7f7d28146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:06.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d277fe700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d280b0df0 0x7f7d280b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:06.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 -- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d2813fcb0 msgr2=0x7f7d28146860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d2813fcb0 0x7f7d28146860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 -- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d280b0df0 msgr2=0x7f7d280b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d280b0df0 0x7f7d280b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 -- 172.21.15.61:0/968739168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20009470 con 0x7f7d281552d0 2022-07-12T08:50:06.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d27fff700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 0x7f7d280b5470 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7f7d200032f0 tx=0x7f7d20016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:06.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d20011e20 con 0x7f7d281552d0 2022-07-12T08:50:06.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d20021440 con 0x7f7d281552d0 2022-07-12T08:50:06.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d280b03a0 con 0x7f7d281552d0 2022-07-12T08:50:06.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d200217a0 con 0x7f7d281552d0 2022-07-12T08:50:06.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.640+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d280adff0 con 0x7f7d281552d0 2022-07-12T08:50:06.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.644+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d281465c0 con 0x7f7d281552d0 2022-07-12T08:50:06.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.644+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7d20027020 con 0x7f7d281552d0 2022-07-12T08:50:06.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.644+0000 7f7d24ff9700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d080539b0 0x7f7d08055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:06.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.644+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7d2006cf20 con 0x7f7d281552d0 2022-07-12T08:50:06.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.644+0000 7f7d277fe700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d080539b0 0x7f7d08055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:06.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.644+0000 7f7d277fe700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d080539b0 0x7f7d08055e70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f7d1800b000 tx=0x7f7d18006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:06.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.648+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d281465c0 con 0x7f7d281552d0 2022-07-12T08:50:06.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.820+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7d281465c0 con 0x7f7d281552d0 2022-07-12T08:50:06.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.820+0000 7f7d24ff9700 1 -- 172.21.15.61:0/968739168 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v38) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f7d281465c0 con 0x7f7d281552d0 2022-07-12T08:50:06.826 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:06.826 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:50:06.827 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:50:06.827 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.827 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:50:06.828 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:50:06.828 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.829 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:06.829 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.830 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.830 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.830 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.831 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.831 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.832 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:50:06.833 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:50:06.834 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.834 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:06.835 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.835 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.838 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.838 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.839 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.839 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.842 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:50:06.843 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:50:06.843 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.843 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.844 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.845 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.848 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.849 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.850 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:50:06.850 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:50:06.851 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.851 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.851 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.852 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.855 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.855 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.856 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.856 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:50:06.856 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:50:06.856 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.857 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.857 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.857 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.857 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.858 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.858 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:50:06.858 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:50:06.858 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.859 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.859 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.859 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.859 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.860 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.860 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.860 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:50:06.860 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:50:06.860 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.861 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.861 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.861 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.861 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.862 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.862 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:50:06.862 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:50:06.862 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.863 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.863 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.863 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.864 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.864 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.865 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.865 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:50:06.865 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:50:06.866 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.866 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.866 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.867 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.867 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.868 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.868 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.868 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:50:06.875 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:50:06.875 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.876 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.876 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.876 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.876 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.877 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.877 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:50:06.877 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:50:06.877 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.888 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:06.889 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:50:06.890 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:50:06.891 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.891 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.892 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:50:06.892 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:50:06.893 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.893 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:06.893 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:50:06.894 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:50:06.894 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.895 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.895 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.895 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:50:06.896 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:50:06.896 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.896 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:06.896 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:50:06.896 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:50:06.897 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.897 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.897 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.897 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:50:06.898 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:50:06.898 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.899 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:50:06.899 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:06.899 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:06.900 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.900 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.900 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.901 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.901 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:50:06.901 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:50:06.902 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.902 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:50:06.903 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:50:06.903 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.903 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.904 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:06.904 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:06.904 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:50:06.905 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:50:06.905 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:06.906 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:50:06.906 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:50:06.907 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:06.907 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.908 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:06.909 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:06.909 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:50:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d080539b0 msgr2=0x7f7d08055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d080539b0 0x7f7d08055e70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 rx=0x7f7d1800b000 tx=0x7f7d18006df0).stop 2022-07-12T08:50:06.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 msgr2=0x7f7d280b5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:06.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 0x7f7d280b5470 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 rx=0x7f7d200032f0 tx=0x7f7d20016000).stop 2022-07-12T08:50:06.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 shutdown_connections 2022-07-12T08:50:06.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d281552d0 0x7f7d280b5470 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d2813fcb0 0x7f7d28146860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d080539b0 0x7f7d08055e70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 --2- 172.21.15.61:0/968739168 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d280b0df0 0x7f7d280b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:06.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 >> 172.21.15.61:0/968739168 conn(0x7f7d28063ad0 msgr2=0x7f7d28063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:06.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 shutdown_connections 2022-07-12T08:50:06.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:06.824+0000 7f7d2cc54700 1 -- 172.21.15.61:0/968739168 wait complete. 2022-07-12T08:50:06.920 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:07.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.024+0000 7fcd7ec3b700 1 Processor -- start 2022-07-12T08:50:07.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.024+0000 7fcd7ec3b700 1 -- start start 2022-07-12T08:50:07.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7ec3b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 0x7fcd78098910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7ec3b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7805f650 con 0x7fcd780b1450 2022-07-12T08:50:07.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7ec3b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd780aabd0 con 0x7fcd780aa7f0 2022-07-12T08:50:07.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7ec3b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78098e00 con 0x7fcd7805f140 2022-07-12T08:50:07.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7d438700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcd780b1450 0x7fcd78141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48984/0 (socket says 172.21.15.61:48984) 2022-07-12T08:50:07.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7d438700 1 -- 172.21.15.61:0/1145368485 learned_addr learned my addr 172.21.15.61:0/1145368485 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:07.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7e43a700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 0x7fcd78098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676756215 0 0) 0x7fcd7805f650 con 0x7fcd780b1450 2022-07-12T08:50:07.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd60003260 con 0x7fcd780b1450 2022-07-12T08:50:07.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127639115 0 0) 0x7fcd60003260 con 0x7fcd780b1450 2022-07-12T08:50:07.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 msgr2=0x7fcd78098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 0x7fcd78098910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 >> v1:172.21.15.111:6789/0 conn(0x7fcd780aa7f0 legacy=0x7fcd78098200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd78067c80 con 0x7fcd780b1450 2022-07-12T08:50:07.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fcd68001680 con 0x7fcd780b1450 2022-07-12T08:50:07.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 >> v1:172.21.15.61:6789/0 conn(0x7fcd780b1450 legacy=0x7fcd78141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd60003780 0x7fcd60003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd60004500 0x7fcd60004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 0x7fcd60004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78098e00 con 0x7fcd60004500 2022-07-12T08:50:07.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6000b750 con 0x7fcd60003780 2022-07-12T08:50:07.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6000b8c0 con 0x7fcd7805f140 2022-07-12T08:50:07.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd60003780 0x7fcd60003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7d438700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd60004500 0x7fcd60004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7e43a700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 0x7fcd60004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 -- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 msgr2=0x7fcd60004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd7805f140 0x7fcd60004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 -- 172.21.15.61:0/1145368485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd60004500 msgr2=0x7fcd60004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd60004500 0x7fcd60004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 -- 172.21.15.61:0/1145368485 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd60003260 con 0x7fcd60003780 2022-07-12T08:50:07.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7d438700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd60004500 0x7fcd60004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:07.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd60003780 0x7fcd60003b60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fcd74001020 tx=0x7fcd740079e0).ready entity=mon.1 client_cookie=58b449c4e5de8388 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd74007cf0 con 0x7fcd60003780 2022-07-12T08:50:07.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd74007030 con 0x7fcd60003780 2022-07-12T08:50:07.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.028+0000 7fcd7cc37700 1 -- 172.21.15.61:0/1145368485 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd74007390 con 0x7fcd60003780 2022-07-12T08:50:07.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd60003780 msgr2=0x7fcd60003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/1145368485 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd60003780 0x7fcd60003b60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fcd74001020 tx=0x7fcd740079e0).stop 2022-07-12T08:50:07.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 -- 172.21.15.61:0/1145368485 reap_dead start 2022-07-12T08:50:07.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/1145368485 shutdown_connections 2022-07-12T08:50:07.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/1145368485 >> 172.21.15.61:0/1145368485 conn(0x7fcd780b1950 msgr2=0x7fcd780b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/1145368485 shutdown_connections 2022-07-12T08:50:07.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/1145368485 wait complete. 2022-07-12T08:50:07.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 Processor -- start 2022-07-12T08:50:07.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- start start 2022-07-12T08:50:07.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd60003780 0x7fcd7813cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd780b4a50 0x7fcd78136420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 0x7fcd78136910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780643c0 con 0x7fcd780b4440 2022-07-12T08:50:07.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78136e00 con 0x7fcd780b4a50 2022-07-12T08:50:07.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7813c2f0 con 0x7fcd60003780 2022-07-12T08:50:07.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 0x7fcd78136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 0x7fcd78136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60782/0 (socket says 172.21.15.61:60782) 2022-07-12T08:50:07.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 -- 172.21.15.61:0/4023624400 learned_addr learned my addr 172.21.15.61:0/4023624400 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:07.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7d438700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd780b4a50 0x7fcd78136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd60003780 0x7fcd7813cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 -- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd60003780 msgr2=0x7fcd7813cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd60003780 0x7fcd7813cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 -- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd780b4a50 msgr2=0x7fcd78136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd780b4a50 0x7fcd78136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 -- 172.21.15.61:0/4023624400 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7813c590 con 0x7fcd780b4440 2022-07-12T08:50:07.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7e43a700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 0x7fcd78136910 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7fcd7000b000 tx=0x7fcd700091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7000f830 con 0x7fcd780b4440 2022-07-12T08:50:07.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7000f990 con 0x7fcd780b4440 2022-07-12T08:50:07.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd780ac690 con 0x7fcd780b4440 2022-07-12T08:50:07.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.032+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7000fcf0 con 0x7fcd780b4440 2022-07-12T08:50:07.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd780acd50 con 0x7fcd780b4440 2022-07-12T08:50:07.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd780ac4f0 con 0x7fcd780b4440 2022-07-12T08:50:07.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcd70014440 con 0x7fcd780b4440 2022-07-12T08:50:07.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd66ffd700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd58053920 0x7fcd58055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcd7006c2e0 con 0x7fcd780b4440 2022-07-12T08:50:07.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd58053920 0x7fcd58055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.036+0000 7fcd7dc39700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd58053920 0x7fcd58055de0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fcd7401b000 tx=0x7fcd74015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.048+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd780ac4f0 con 0x7fcd780b4440 2022-07-12T08:50:07.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.172+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd780ac4f0 con 0x7fcd780b4440 2022-07-12T08:50:07.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd66ffd700 1 -- 172.21.15.61:0/4023624400 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+13239 (secure 0 0 0) 0x7fcd780ac4f0 con 0x7fcd780b4440 2022-07-12T08:50:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:07.192 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":124,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:06.434889+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:07.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd58053920 msgr2=0x7fcd58055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd58053920 0x7fcd58055de0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7fcd7401b000 tx=0x7fcd74015000).stop 2022-07-12T08:50:07.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 msgr2=0x7fcd78136910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 0x7fcd78136910 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 rx=0x7fcd7000b000 tx=0x7fcd700091d0).stop 2022-07-12T08:50:07.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 shutdown_connections 2022-07-12T08:50:07.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd780b4440 0x7fcd78136910 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcd58053920 0x7fcd58055de0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcd780b4a50 0x7fcd78136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 --2- 172.21.15.61:0/4023624400 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fcd60003780 0x7fcd7813cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 >> 172.21.15.61:0/4023624400 conn(0x7fcd780b1950 msgr2=0x7fcd780afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 shutdown_connections 2022-07-12T08:50:07.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.176+0000 7fcd7ec3b700 1 -- 172.21.15.61:0/4023624400 wait complete. 2022-07-12T08:50:07.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:50:07.203 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:07.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88bb16b700 1 Processor -- start 2022-07-12T08:50:07.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88bb16b700 1 -- start start 2022-07-12T08:50:07.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88bb16b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f88b414b860 0x7f88b414dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88bb16b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b4142bc0 con 0x7f88b40ad970 2022-07-12T08:50:07.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88bb16b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40add50 con 0x7f88b40b0fd0 2022-07-12T08:50:07.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88bb16b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b4060b20 con 0x7f88b414b860 2022-07-12T08:50:07.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b8f07700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f88b40ad970 0x7f88b40b2de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49004/0 (socket says 172.21.15.61:49004) 2022-07-12T08:50:07.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b8f07700 1 -- 172.21.15.61:0/1637267200 learned_addr learned my addr 172.21.15.61:0/1637267200 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:07.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 --2- 172.21.15.61:0/1637267200 >> v2:172.21.15.111:3300/0 conn(0x7f88b414b860 0x7f88b414dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867600180 0 0) 0x7f889c001490 con 0x7f88b40ad970 2022-07-12T08:50:07.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8894003260 con 0x7f88b40ad970 2022-07-12T08:50:07.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222667599 0 0) 0x7f8894003260 con 0x7f88b40ad970 2022-07-12T08:50:07.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 >> v2:172.21.15.111:3300/0 conn(0x7f88b414b860 msgr2=0x7f88b414dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 --2- 172.21.15.61:0/1637267200 >> v2:172.21.15.111:3300/0 conn(0x7f88b414b860 0x7f88b414dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 >> v1:172.21.15.111:6789/0 conn(0x7f88b40b0fd0 legacy=0x7f88b41462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40b26a0 con 0x7f88b40ad970 2022-07-12T08:50:07.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f889c001b70 con 0x7f88b40ad970 2022-07-12T08:50:07.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 >> v1:172.21.15.61:6789/0 conn(0x7f88b40ad970 legacy=0x7f88b40b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894003750 0x7f8894003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8894004500 0x7f8894004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8894004e70 0x7f889400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4060b20 con 0x7f8894004e70 2022-07-12T08:50:07.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889400bd20 con 0x7f8894004500 2022-07-12T08:50:07.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889400bee0 con 0x7f8894003750 2022-07-12T08:50:07.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8894004e70 0x7f889400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b8f07700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894003750 0x7f8894003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b3fff700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8894004500 0x7f8894004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 -- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894003750 msgr2=0x7f8894003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894003750 0x7f8894003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 -- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8894004500 msgr2=0x7f8894004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8894004500 0x7f8894004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b9708700 1 -- 172.21.15.61:0/1637267200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8894003260 con 0x7f8894004e70 2022-07-12T08:50:07.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b3fff700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8894004500 0x7f8894004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:07.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.312+0000 7f88b8f07700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894003750 0x7f8894003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:07.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88b9708700 1 -- 172.21.15.61:0/1637267200 reap_dead start 2022-07-12T08:50:07.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88b9708700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8894004e70 0x7f889400b750 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7f88ac00c000 tx=0x7f88ac006df0).ready entity=mon.0 client_cookie=dc7df97fb31a11a9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88ac00e830 con 0x7f8894004e70 2022-07-12T08:50:07.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac00e990 con 0x7f8894004e70 2022-07-12T08:50:07.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88b37fe700 1 -- 172.21.15.61:0/1637267200 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88ac00ecf0 con 0x7f8894004e70 2022-07-12T08:50:07.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 -- 172.21.15.61:0/1637267200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8894004e70 msgr2=0x7f889400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8894004e70 0x7f889400b750 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7f88ac00c000 tx=0x7f88ac006df0).stop 2022-07-12T08:50:07.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 -- 172.21.15.61:0/1637267200 shutdown_connections 2022-07-12T08:50:07.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 --2- 172.21.15.61:0/1637267200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8894004e70 0x7f889400b750 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 -- 172.21.15.61:0/1637267200 >> 172.21.15.61:0/1637267200 conn(0x7f88b40621d0 msgr2=0x7f88b4063bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 -- 172.21.15.61:0/1637267200 shutdown_connections 2022-07-12T08:50:07.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.316+0000 7f88bb16b700 1 -- 172.21.15.61:0/1637267200 wait complete. 2022-07-12T08:50:07.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 Processor -- start 2022-07-12T08:50:07.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 -- start start 2022-07-12T08:50:07.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894004e70 0x7f88b40b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 0x7f88b4156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b4066010 0x7f88b414aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4145880 con 0x7f88b4064e60 2022-07-12T08:50:07.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4155a30 con 0x7f88b4066010 2022-07-12T08:50:07.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4155cd0 con 0x7f8894004e70 2022-07-12T08:50:07.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 0x7f88b4156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 0x7f88b4156170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60798/0 (socket says 172.21.15.61:60798) 2022-07-12T08:50:07.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 -- 172.21.15.61:0/4098566902 learned_addr learned my addr 172.21.15.61:0/4098566902 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:07.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b8f07700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894004e70 0x7f88b40b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b9708700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b4066010 0x7f88b414aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 -- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894004e70 msgr2=0x7f88b40b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894004e70 0x7f88b40b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 -- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b4066010 msgr2=0x7f88b414aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b4066010 0x7f88b414aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 -- 172.21.15.61:0/4098566902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4065530 con 0x7f88b4064e60 2022-07-12T08:50:07.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b9708700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b4066010 0x7f88b414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:07.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b3fff700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 0x7f88b4156170 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f88a4008000 tx=0x7f88a4009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a4007e20 con 0x7f88b4064e60 2022-07-12T08:50:07.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a40104e0 con 0x7f88b4064e60 2022-07-12T08:50:07.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b4156660 con 0x7f88b4064e60 2022-07-12T08:50:07.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88a4010840 con 0x7f88b4064e60 2022-07-12T08:50:07.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.320+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b4145640 con 0x7f88b4064e60 2022-07-12T08:50:07.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.324+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b4057460 con 0x7f88b4064e60 2022-07-12T08:50:07.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.324+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f88a40109a0 con 0x7f88b4064e60 2022-07-12T08:50:07.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.324+0000 7f88b1ffb700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8898053a40 0x7f8898055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.324+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f88a406b770 con 0x7f88b4064e60 2022-07-12T08:50:07.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.324+0000 7f88b8f07700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8898053a40 0x7f8898055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.324+0000 7f88b8f07700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8898053a40 0x7f8898055f00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f889c00d000 tx=0x7f889c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.328+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88a40428f0 con 0x7f88b4064e60 2022-07-12T08:50:07.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.452+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f88b4155e90 con 0x7f88b4064e60 2022-07-12T08:50:07.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.452+0000 7f88b1ffb700 1 -- 172.21.15.61:0/4098566902 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v124) v1 ==== 110+0+10 (secure 0 0 0) 0x7f88a4045f10 con 0x7f88b4064e60 2022-07-12T08:50:07.461 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8898053a40 msgr2=0x7f8898055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8898053a40 0x7f8898055f00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f889c00d000 tx=0x7f889c009040).stop 2022-07-12T08:50:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 msgr2=0x7f88b4156170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 0x7f88b4156170 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f88a4008000 tx=0x7f88a4009040).stop 2022-07-12T08:50:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 shutdown_connections 2022-07-12T08:50:07.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8898053a40 0x7f8898055f00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88b4066010 0x7f88b414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88b4064e60 0x7f88b4156170 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 --2- 172.21.15.61:0/4098566902 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8894004e70 0x7f88b40b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 >> 172.21.15.61:0/4098566902 conn(0x7f88b40621d0 msgr2=0x7f88b405f650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 shutdown_connections 2022-07-12T08:50:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.456+0000 7f88bb16b700 1 -- 172.21.15.61:0/4098566902 wait complete. 2022-07-12T08:50:07.489 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:07.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a7403700 1 Processor -- start 2022-07-12T08:50:07.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a7403700 1 -- start start 2022-07-12T08:50:07.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a7403700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 0x7f09a0143680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a7403700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a00abfa0 con 0x7f09a00b5920 2022-07-12T08:50:07.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a7403700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a005cd00 con 0x7f09a00b0240 2022-07-12T08:50:07.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a7403700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a005c870 con 0x7f09a00b06b0 2022-07-12T08:50:07.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a6401700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f09a00b5920 0x7f09a00aa500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49022/0 (socket says 172.21.15.61:49022) 2022-07-12T08:50:07.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a6401700 1 -- 172.21.15.61:0/4203565594 learned_addr learned my addr 172.21.15.61:0/4203565594 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:07.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a6c02700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 0x7f09a0143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548297239 0 0) 0x7f09a00abfa0 con 0x7f09a00b5920 2022-07-12T08:50:07.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0988003260 con 0x7f09a00b5920 2022-07-12T08:50:07.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356469762 0 0) 0x7f09a005cd00 con 0x7f09a00b0240 2022-07-12T08:50:07.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a00abfa0 con 0x7f09a00b0240 2022-07-12T08:50:07.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4105015942 0 0) 0x7f0988003260 con 0x7f09a00b5920 2022-07-12T08:50:07.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 msgr2=0x7f09a0143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 0x7f09a0143680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 >> v1:172.21.15.111:6789/0 conn(0x7f09a00b0240 legacy=0x7f09a0142f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a005cad0 con 0x7f09a00b5920 2022-07-12T08:50:07.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f099c001660 con 0x7f09a00b5920 2022-07-12T08:50:07.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 >> v1:172.21.15.61:6789/0 conn(0x7f09a00b5920 legacy=0x7f09a00aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0988003910 0x7f0988003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0988004500 0x7f0988004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 0x7f0988004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a005c870 con 0x7f0988004500 2022-07-12T08:50:07.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098800b780 con 0x7f0988003910 2022-07-12T08:50:07.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098800b940 con 0x7f09a00b06b0 2022-07-12T08:50:07.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a5c00700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0988004500 0x7f0988004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a6401700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0988003910 0x7f0988003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.616+0000 7f09a6c02700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 0x7f0988004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a5c00700 1 -- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 msgr2=0x7f0988004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a5c00700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09a00b06b0 0x7f0988004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a5c00700 1 -- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0988003910 msgr2=0x7f0988003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a5c00700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0988003910 0x7f0988003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a5c00700 1 -- 172.21.15.61:0/4203565594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0988003260 con 0x7f0988004500 2022-07-12T08:50:07.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a5c00700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0988004500 0x7f0988004980 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f0990001110 tx=0x7f0990007a00).ready entity=mon.0 client_cookie=27986f5d730db571 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0990007d10 con 0x7f0988004500 2022-07-12T08:50:07.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0990007030 con 0x7f0988004500 2022-07-12T08:50:07.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a53ff700 1 -- 172.21.15.61:0/4203565594 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0990007390 con 0x7f0988004500 2022-07-12T08:50:07.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 -- 172.21.15.61:0/4203565594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0988004500 msgr2=0x7f0988004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 --2- 172.21.15.61:0/4203565594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0988004500 0x7f0988004980 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7f0990001110 tx=0x7f0990007a00).stop 2022-07-12T08:50:07.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a6c02700 1 -- 172.21.15.61:0/4203565594 reap_dead start 2022-07-12T08:50:07.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 -- 172.21.15.61:0/4203565594 shutdown_connections 2022-07-12T08:50:07.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 -- 172.21.15.61:0/4203565594 >> 172.21.15.61:0/4203565594 conn(0x7f09a0063340 msgr2=0x7f09a00b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 -- 172.21.15.61:0/4203565594 shutdown_connections 2022-07-12T08:50:07.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 -- 172.21.15.61:0/4203565594 wait complete. 2022-07-12T08:50:07.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 Processor -- start 2022-07-12T08:50:07.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.620+0000 7f09a7403700 1 -- start start 2022-07-12T08:50:07.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0988004500 0x7f09a0140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a0148790 0x7f09a00ac390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 0x7f09a0155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0147c80 con 0x7f09a00ae9c0 2022-07-12T08:50:07.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a01481e0 con 0x7f09a0148790 2022-07-12T08:50:07.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0148480 con 0x7f0988004500 2022-07-12T08:50:07.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 0x7f09a0155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 0x7f09a0155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60820/0 (socket says 172.21.15.61:60820) 2022-07-12T08:50:07.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6401700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0988004500 0x7f09a0140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 -- 172.21.15.61:0/2147279515 learned_addr learned my addr 172.21.15.61:0/2147279515 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:07.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a5c00700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a0148790 0x7f09a00ac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 -- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0988004500 msgr2=0x7f09a0140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0988004500 0x7f09a0140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 -- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a0148790 msgr2=0x7f09a00ac390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a0148790 0x7f09a00ac390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 -- 172.21.15.61:0/2147279515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00af870 con 0x7f09a00ae9c0 2022-07-12T08:50:07.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a6c02700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 0x7f09a0155490 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f099800b000 tx=0x7f09980091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0998013750 con 0x7f09a00ae9c0 2022-07-12T08:50:07.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09980138b0 con 0x7f09a00ae9c0 2022-07-12T08:50:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a0144300 con 0x7f09a00ae9c0 2022-07-12T08:50:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0998013c10 con 0x7f09a00ae9c0 2022-07-12T08:50:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.624+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a00aee70 con 0x7f09a00ae9c0 2022-07-12T08:50:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.628+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a0057460 con 0x7f09a00ae9c0 2022-07-12T08:50:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.628+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0998013d70 con 0x7f09a00ae9c0 2022-07-12T08:50:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.628+0000 7f098f7fe700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0980053a40 0x7f0980055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.628+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f099806c310 con 0x7f09a00ae9c0 2022-07-12T08:50:07.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.628+0000 7f09a6401700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0980053a40 0x7f0980055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.628+0000 7f09a6401700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0980053a40 0x7f0980055f00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f099c005000 tx=0x7f099c001db0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.636+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0998043490 con 0x7f09a00ae9c0 2022-07-12T08:50:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.756+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f09a01444f0 con 0x7f09a00ae9c0 2022-07-12T08:50:07.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f098f7fe700 1 -- 172.21.15.61:0/2147279515 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v124) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0998018020 con 0x7f09a00ae9c0 2022-07-12T08:50:07.770 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0980053a40 msgr2=0x7f0980055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0980053a40 0x7f0980055f00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f099c005000 tx=0x7f099c001db0).stop 2022-07-12T08:50:07.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 msgr2=0x7f09a0155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 0x7f09a0155490 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 rx=0x7f099800b000 tx=0x7f09980091d0).stop 2022-07-12T08:50:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 shutdown_connections 2022-07-12T08:50:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09a00ae9c0 0x7f09a0155490 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0980053a40 0x7f0980055f00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09a0148790 0x7f09a00ac390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 --2- 172.21.15.61:0/2147279515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0988004500 0x7f09a0140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 >> 172.21.15.61:0/2147279515 conn(0x7f09a0063340 msgr2=0x7f09a00b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 shutdown_connections 2022-07-12T08:50:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.760+0000 7f09a7403700 1 -- 172.21.15.61:0/2147279515 wait complete. 2022-07-12T08:50:07.793 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:50:07.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 Processor -- start 2022-07-12T08:50:07.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 -- start start 2022-07-12T08:50:07.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 0x7f0500066430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05001434a0 0x7f0500143930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0500066920 con 0x7f05000b29a0 2022-07-12T08:50:07.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0500066b70 con 0x7f05001434a0 2022-07-12T08:50:07.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f05054cd700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05000b2d80 con 0x7f05000b3270 2022-07-12T08:50:07.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04f7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 0x7f0500066430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f0504ccc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f05001434a0 0x7f0500143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04f7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 0x7f0500066430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39400/0 (socket says 172.21.15.61:39400) 2022-07-12T08:50:07.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04f7fff700 1 -- 172.21.15.61:0/894233254 learned_addr learned my addr 172.21.15.61:0/894233254 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:07.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f0504ccc700 1 -- 172.21.15.61:0/894233254 >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 msgr2=0x7f0500066430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:07.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f0504ccc700 1 --2- 172.21.15.61:0/894233254 >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 0x7f0500066430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f0504ccc700 1 -- 172.21.15.61:0/894233254 >> v1:172.21.15.111:6789/0 conn(0x7f05000b29a0 legacy=0x7f050005bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f0504ccc700 1 -- 172.21.15.61:0/894233254 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0500142ac0 con 0x7f05001434a0 2022-07-12T08:50:07.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04f7fff700 1 --2- 172.21.15.61:0/894233254 >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 0x7f0500066430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:07.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f0504ccc700 1 --2- 172.21.15.61:0/894233254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05001434a0 0x7f0500143930 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7f04f80032f0 tx=0x7f04f80091d0).ready entity=mon.0 client_cookie=751f4d2adf8b0fd0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04ff7fe700 1 -- 172.21.15.61:0/894233254 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f800b680 con 0x7f05001434a0 2022-07-12T08:50:07.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04ff7fe700 1 -- 172.21.15.61:0/894233254 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f800b7e0 con 0x7f05001434a0 2022-07-12T08:50:07.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.900+0000 7f04ff7fe700 1 -- 172.21.15.61:0/894233254 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04f800b950 con 0x7f05001434a0 2022-07-12T08:50:07.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- 172.21.15.61:0/894233254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05001434a0 msgr2=0x7f0500143930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 --2- 172.21.15.61:0/894233254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05001434a0 0x7f0500143930 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7f04f80032f0 tx=0x7f04f80091d0).stop 2022-07-12T08:50:07.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- 172.21.15.61:0/894233254 shutdown_connections 2022-07-12T08:50:07.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 --2- 172.21.15.61:0/894233254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05001434a0 0x7f0500143930 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 --2- 172.21.15.61:0/894233254 >> v2:172.21.15.111:3300/0 conn(0x7f05000b3270 0x7f0500066430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- 172.21.15.61:0/894233254 >> 172.21.15.61:0/894233254 conn(0x7f05000b70b0 msgr2=0x7f05000b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:07.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- 172.21.15.61:0/894233254 shutdown_connections 2022-07-12T08:50:07.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- 172.21.15.61:0/894233254 wait complete. 2022-07-12T08:50:07.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 Processor -- start 2022-07-12T08:50:07.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- start start 2022-07-12T08:50:07.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05000b29a0 0x7f050013eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 0x7f05000ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05001434a0 0x7f05000b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000aec60 con 0x7f05000b3270 2022-07-12T08:50:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000ac700 con 0x7f05000b29a0 2022-07-12T08:50:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f05054cd700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05000ad710 con 0x7f05001434a0 2022-07-12T08:50:07.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 0x7f05000ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04fffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05000b29a0 0x7f050013eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 0x7f05000ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60832/0 (socket says 172.21.15.61:60832) 2022-07-12T08:50:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 -- 172.21.15.61:0/966646307 learned_addr learned my addr 172.21.15.61:0/966646307 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:07.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f0504ccc700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05001434a0 0x7f05000b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 -- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05001434a0 msgr2=0x7f05000b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05001434a0 0x7f05000b0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 -- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05000b29a0 msgr2=0x7f050013eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:07.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05000b29a0 0x7f050013eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:07.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.904+0000 7f04f7fff700 1 -- 172.21.15.61:0/966646307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e800b1e0 con 0x7f05000b3270 2022-07-12T08:50:07.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f04f7fff700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 0x7f05000ade50 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f04e800c000 tx=0x7f04e8006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04e8012600 con 0x7f05000b3270 2022-07-12T08:50:07.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e8012760 con 0x7f05000b3270 2022-07-12T08:50:07.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f8009470 con 0x7f05000b3270 2022-07-12T08:50:07.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04e8012ac0 con 0x7f05000b3270 2022-07-12T08:50:07.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05000ac260 con 0x7f05000b3270 2022-07-12T08:50:07.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050013abc0 con 0x7f05000b3270 2022-07-12T08:50:07.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f04e8012c20 con 0x7f05000b3270 2022-07-12T08:50:07.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.908+0000 7f04fdffb700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04e40539b0 0x7f04e4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:07.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.912+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f04e806bde0 con 0x7f05000b3270 2022-07-12T08:50:07.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.912+0000 7f04fffff700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04e40539b0 0x7f04e4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:07.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.912+0000 7f04fffff700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04e40539b0 0x7f04e4055e70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f04f000d000 tx=0x7f04f0000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:07.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:07.916+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f050013abc0 con 0x7f05000b3270 2022-07-12T08:50:08.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f050013abc0 con 0x7f05000b3270 2022-07-12T08:50:08.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f04fdffb700 1 -- 172.21.15.61:0/966646307 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v124) v1 ==== 84+0+143 (secure 0 0 0) 0x7f050013abc0 con 0x7f05000b3270 2022-07-12T08:50:08.056 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:08.056 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:50:08.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:08.057 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:50:08.057 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:50:08.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:08.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:08.059 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:50:08.060 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:50:08.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:08.060 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:50:08.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04e40539b0 msgr2=0x7f04e4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04e40539b0 0x7f04e4055e70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7f04f000d000 tx=0x7f04f0000bc0).stop 2022-07-12T08:50:08.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 msgr2=0x7f05000ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 0x7f05000ade50 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7f04e800c000 tx=0x7f04e8006eb0).stop 2022-07-12T08:50:08.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 shutdown_connections 2022-07-12T08:50:08.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f05001434a0 0x7f05000b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f05000b3270 0x7f05000ade50 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f04e40539b0 0x7f04e4055e70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 --2- 172.21.15.61:0/966646307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f05000b29a0 0x7f050013eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 >> 172.21.15.61:0/966646307 conn(0x7f05000b70b0 msgr2=0x7f050005c450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:08.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 shutdown_connections 2022-07-12T08:50:08.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.044+0000 7f05054cd700 1 -- 172.21.15.61:0/966646307 wait complete. 2022-07-12T08:50:08.092 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:50:08.093 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:50:08.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 Processor -- start 2022-07-12T08:50:08.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 -- start start 2022-07-12T08:50:08.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 0x7f990c0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b05f0 0x7f990c142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990c0b3560 con 0x7f990c0b3730 2022-07-12T08:50:08.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990c0b5b80 con 0x7f990c0b5480 2022-07-12T08:50:08.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f99134b5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c0b5e20 con 0x7f990c0b05f0 2022-07-12T08:50:08.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911251700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 0x7f990c0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911a52700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b05f0 0x7f990c142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911251700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 0x7f990c0b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60838/0 (socket says 172.21.15.61:60838) 2022-07-12T08:50:08.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911251700 1 -- 172.21.15.61:0/347367180 learned_addr learned my addr 172.21.15.61:0/347367180 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:08.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9903fff700 1 -- 172.21.15.61:0/347367180 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243245990 0 0) 0x7f990c0b3560 con 0x7f990c0b3730 2022-07-12T08:50:08.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9903fff700 1 -- 172.21.15.61:0/347367180 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f0003260 con 0x7f990c0b3730 2022-07-12T08:50:08.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911a52700 1 -- 172.21.15.61:0/347367180 >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 msgr2=0x7f990c0b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:08.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911a52700 1 --2- 172.21.15.61:0/347367180 >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 0x7f990c0b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911a52700 1 -- 172.21.15.61:0/347367180 >> v1:172.21.15.111:6789/0 conn(0x7f990c0b3730 legacy=0x7f990c0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911a52700 1 -- 172.21.15.61:0/347367180 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c0b5860 con 0x7f990c0b05f0 2022-07-12T08:50:08.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911251700 1 --2- 172.21.15.61:0/347367180 >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 0x7f990c0b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:08.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9911a52700 1 --2- 172.21.15.61:0/347367180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b05f0 0x7f990c142f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f99040032f0 tx=0x7f99040091d0).ready entity=mon.2 client_cookie=c1348b5c3c83b343 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:08.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.208+0000 7f9903fff700 1 -- 172.21.15.61:0/347367180 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990400b600 con 0x7f990c0b05f0 2022-07-12T08:50:08.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f9903fff700 1 -- 172.21.15.61:0/347367180 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990400b760 con 0x7f990c0b05f0 2022-07-12T08:50:08.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f9903fff700 1 -- 172.21.15.61:0/347367180 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f990400b8d0 con 0x7f990c0b05f0 2022-07-12T08:50:08.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- 172.21.15.61:0/347367180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b05f0 msgr2=0x7f990c142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 --2- 172.21.15.61:0/347367180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b05f0 0x7f990c142f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f99040032f0 tx=0x7f99040091d0).stop 2022-07-12T08:50:08.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- 172.21.15.61:0/347367180 shutdown_connections 2022-07-12T08:50:08.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 --2- 172.21.15.61:0/347367180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b05f0 0x7f990c142f90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 --2- 172.21.15.61:0/347367180 >> v2:172.21.15.61:3300/0 conn(0x7f990c0b5480 0x7f990c0b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- 172.21.15.61:0/347367180 >> 172.21.15.61:0/347367180 conn(0x7f990c0b96b0 msgr2=0x7f990c0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:08.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- 172.21.15.61:0/347367180 shutdown_connections 2022-07-12T08:50:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- 172.21.15.61:0/347367180 wait complete. 2022-07-12T08:50:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 Processor -- start 2022-07-12T08:50:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- start start 2022-07-12T08:50:08.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990c0b05f0 0x7f990c144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b3730 0x7f990c1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 0x7f990c147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c146480 con 0x7f990c0b5480 2022-07-12T08:50:08.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c1457a0 con 0x7f990c0b05f0 2022-07-12T08:50:08.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.212+0000 7f99134b5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c146f40 con 0x7f990c0b3730 2022-07-12T08:50:08.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9911a52700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 0x7f990c147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b3730 0x7f990c1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9911a52700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 0x7f990c147370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60848/0 (socket says 172.21.15.61:60848) 2022-07-12T08:50:08.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9911a52700 1 -- 172.21.15.61:0/350967984 learned_addr learned my addr 172.21.15.61:0/350967984 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:08.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9911251700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990c0b05f0 0x7f990c144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 -- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990c0b05f0 msgr2=0x7f990c144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990c0b05f0 0x7f990c144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 -- 172.21.15.61:0/350967984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 msgr2=0x7f990c147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 0x7f990c147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 -- 172.21.15.61:0/350967984 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990800b1e0 con 0x7f990c0b3730 2022-07-12T08:50:08.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9911a52700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 0x7f990c147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:08.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f9910a50700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b3730 0x7f990c1557c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f98fc001110 tx=0x7f98fc007a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:08.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc012400 con 0x7f990c0b3730 2022-07-12T08:50:08.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc012560 con 0x7f990c0b3730 2022-07-12T08:50:08.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9904009470 con 0x7f990c0b3730 2022-07-12T08:50:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98fc0128c0 con 0x7f990c0b3730 2022-07-12T08:50:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c140210 con 0x7f990c0b3730 2022-07-12T08:50:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c147080 con 0x7f990c0b3730 2022-07-12T08:50:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f98fc019030 con 0x7f990c0b3730 2022-07-12T08:50:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.216+0000 7f99027fc700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98f4053a60 0x7f98f4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.220+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f98fc06afb0 con 0x7f990c0b3730 2022-07-12T08:50:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.220+0000 7f9911251700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98f4053a60 0x7f98f4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.220+0000 7f9911251700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98f4053a60 0x7f98f4055f20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f990800b000 tx=0x7f9908006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:08.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.228+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f98fc042050 con 0x7f990c0b3730 2022-07-12T08:50:08.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.352+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f990c057460 con 0x7f990c0b3730 2022-07-12T08:50:08.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.456+0000 7f99027fc700 1 -- 172.21.15.61:0/350967984 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v125) v1 ==== 133+0+0 (secure 0 0 0) 0x7f98fc045670 con 0x7f990c0b3730 2022-07-12T08:50:08.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98f4053a60 msgr2=0x7f98f4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98f4053a60 0x7f98f4055f20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f990800b000 tx=0x7f9908006eb0).stop 2022-07-12T08:50:08.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b3730 msgr2=0x7f990c1557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b3730 0x7f990c1557c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f98fc001110 tx=0x7f98fc007a20).stop 2022-07-12T08:50:08.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 shutdown_connections 2022-07-12T08:50:08.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f990c0b5480 0x7f990c147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f990c0b3730 0x7f990c1557c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f98f4053a60 0x7f98f4055f20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 --2- 172.21.15.61:0/350967984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f990c0b05f0 0x7f990c144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 >> 172.21.15.61:0/350967984 conn(0x7f990c0b96b0 msgr2=0x7f990c0b7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:08.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 shutdown_connections 2022-07-12T08:50:08.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.460+0000 7f99134b5700 1 -- 172.21.15.61:0/350967984 wait complete. 2022-07-12T08:50:08.474 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:50:08.477 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:50:08.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84bb698700 1 Processor -- start 2022-07-12T08:50:08.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84bb698700 1 -- start start 2022-07-12T08:50:08.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84bb698700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84bb698700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b40b3560 con 0x7f84b40b5480 2022-07-12T08:50:08.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84bb698700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b40b5b80 con 0x7f84b40b3730 2022-07-12T08:50:08.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84bb698700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b40b5e20 con 0x7f84b40b05f0 2022-07-12T08:50:08.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b9434700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f84b40b5480 0x7f84b40afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49070/0 (socket says 172.21.15.61:49070) 2022-07-12T08:50:08.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b9434700 1 -- 172.21.15.61:0/2633297000 learned_addr learned my addr 172.21.15.61:0/2633297000 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:08.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b8c33700 1 --2- 172.21.15.61:0/2633297000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118109539 0 0) 0x7f84b40b3560 con 0x7f84b40b5480 2022-07-12T08:50:08.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f849c003260 con 0x7f84b40b5480 2022-07-12T08:50:08.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98877903 0 0) 0x7f84b40b5b80 con 0x7f84b40b3730 2022-07-12T08:50:08.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84b40b3560 con 0x7f84b40b3730 2022-07-12T08:50:08.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b8c33700 1 -- 172.21.15.61:0/2633297000 >> v1:172.21.15.111:6789/0 conn(0x7f84b40b3730 legacy=0x7f84b4142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b8c33700 1 -- 172.21.15.61:0/2633297000 >> v1:172.21.15.61:6789/0 conn(0x7f84b40b5480 legacy=0x7f84b40afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 913051168 0 0) 0x7f849c003260 con 0x7f84b40b5480 2022-07-12T08:50:08.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b8c33700 1 -- 172.21.15.61:0/2633297000 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b40b5860 con 0x7f84b40b05f0 2022-07-12T08:50:08.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84b8c33700 1 --2- 172.21.15.61:0/2633297000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b40b45d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f84a4001110 tx=0x7f84b0006df0).ready entity=mon.2 client_cookie=323e3aa45dffd7e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:08.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a4001660 con 0x7f84b40b05f0 2022-07-12T08:50:08.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b0000e90 con 0x7f84b40b05f0 2022-07-12T08:50:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.616+0000 7f84a3fff700 1 -- 172.21.15.61:0/2633297000 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84b0012460 con 0x7f84b40b05f0 2022-07-12T08:50:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- 172.21.15.61:0/2633297000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 msgr2=0x7f84b40b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 --2- 172.21.15.61:0/2633297000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b40b45d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f84a4001110 tx=0x7f84b0006df0).stop 2022-07-12T08:50:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- 172.21.15.61:0/2633297000 shutdown_connections 2022-07-12T08:50:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 --2- 172.21.15.61:0/2633297000 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b40b45d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- 172.21.15.61:0/2633297000 >> 172.21.15.61:0/2633297000 conn(0x7f84b40b96b0 msgr2=0x7f84b40b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- 172.21.15.61:0/2633297000 shutdown_connections 2022-07-12T08:50:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- 172.21.15.61:0/2633297000 wait complete. 2022-07-12T08:50:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 Processor -- start 2022-07-12T08:50:08.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- start start 2022-07-12T08:50:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b41453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b40b3730 0x7f84b41445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84b40b5480 0x7f84b4152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b4147fe0 con 0x7f84b40b5480 2022-07-12T08:50:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41494e0 con 0x7f84b40b3730 2022-07-12T08:50:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84bb698700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41521f0 con 0x7f84b40b05f0 2022-07-12T08:50:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84b9c35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84b40b5480 0x7f84b4152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84b9434700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b41453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84b9434700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b41453a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:43054/0 (socket says 172.21.15.61:43054) 2022-07-12T08:50:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 -- 172.21.15.61:0/3980142865 learned_addr learned my addr 172.21.15.61:0/3980142865 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.620+0000 7f84b9c35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84b40b5480 0x7f84b4152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60862/0 (socket says 172.21.15.61:60862) 2022-07-12T08:50:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b8c33700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b40b3730 0x7f84b41445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 -- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b40b3730 msgr2=0x7f84b41445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b40b3730 0x7f84b41445d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 -- 172.21.15.61:0/3980142865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84b40b5480 msgr2=0x7f84b4152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:08.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84b40b5480 0x7f84b4152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:08.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 -- 172.21.15.61:0/3980142865 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b000b180 con 0x7f84b40b05f0 2022-07-12T08:50:08.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84b9434700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b41453a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f84a4001840 tx=0x7f84a4001720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:08.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a4007170 con 0x7f84b40b05f0 2022-07-12T08:50:08.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a40072d0 con 0x7f84b40b05f0 2022-07-12T08:50:08.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b4152490 con 0x7f84b40b05f0 2022-07-12T08:50:08.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a4004710 con 0x7f84b40b05f0 2022-07-12T08:50:08.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b4144ac0 con 0x7f84b40b05f0 2022-07-12T08:50:08.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b4057460 con 0x7f84b40b05f0 2022-07-12T08:50:08.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.624+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f84a401b070 con 0x7f84b40b05f0 2022-07-12T08:50:08.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.628+0000 7f84a27fc700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8494053af0 0x7f8494055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:08.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.628+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f84a406b340 con 0x7f84b40b05f0 2022-07-12T08:50:08.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.628+0000 7f84b8c33700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8494053af0 0x7f8494055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:08.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.628+0000 7f84b8c33700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8494053af0 0x7f8494055fb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f84b001a000 tx=0x7f84b0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:08.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.636+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84a4042360 con 0x7f84b40b05f0 2022-07-12T08:50:08.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:08.764+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f84b40ba5d0 con 0x7f84b40b05f0 2022-07-12T08:50:09.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.464+0000 7f84a27fc700 1 -- 172.21.15.61:0/3980142865 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v126) v1 ==== 125+0+0 (secure 0 0 0) 0x7f84a4045980 con 0x7f84b40b05f0 2022-07-12T08:50:09.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8494053af0 msgr2=0x7f8494055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8494053af0 0x7f8494055fb0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f84b001a000 tx=0x7f84b0014000).stop 2022-07-12T08:50:09.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 msgr2=0x7f84b41453a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b41453a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f84a4001840 tx=0x7f84a4001720).stop 2022-07-12T08:50:09.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 shutdown_connections 2022-07-12T08:50:09.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84b40b5480 0x7f84b4152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8494053af0 0x7f8494055fb0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84b40b3730 0x7f84b41445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 --2- 172.21.15.61:0/3980142865 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f84b40b05f0 0x7f84b41453a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 >> 172.21.15.61:0/3980142865 conn(0x7f84b40b96b0 msgr2=0x7f84b40b8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:09.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 shutdown_connections 2022-07-12T08:50:09.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.468+0000 7f84bb698700 1 -- 172.21.15.61:0/3980142865 wait complete. 2022-07-12T08:50:09.488 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:50:09.502 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:50:09.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 Processor -- start 2022-07-12T08:50:09.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 -- start start 2022-07-12T08:50:09.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f24fc066fc0 0x7f24fc0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24fc0b45d0 0x7f24fc0b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24fc0b2f20 con 0x7f24fc0b2940 2022-07-12T08:50:09.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0b3170 con 0x7f24fc0b45d0 2022-07-12T08:50:09.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f2500816700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24fc0b4170 con 0x7f24fc066fc0 2022-07-12T08:50:09.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f24f9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24fc0b45d0 0x7f24fc0b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f24fad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f24fc0b2940 0x7f24fc05c090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49088/0 (socket says 172.21.15.61:49088) 2022-07-12T08:50:09.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.620+0000 7f24fad9d700 1 -- 172.21.15.61:0/245702121 learned_addr learned my addr 172.21.15.61:0/245702121 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:09.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24fa59c700 1 --2- 172.21.15.61:0/245702121 >> v2:172.21.15.111:3300/0 conn(0x7f24fc066fc0 0x7f24fc0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f9d9b700 1 -- 172.21.15.61:0/245702121 >> v2:172.21.15.111:3300/0 conn(0x7f24fc066fc0 msgr2=0x7f24fc0b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:09.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f9d9b700 1 --2- 172.21.15.61:0/245702121 >> v2:172.21.15.111:3300/0 conn(0x7f24fc066fc0 0x7f24fc0b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f9d9b700 1 -- 172.21.15.61:0/245702121 >> v1:172.21.15.61:6789/0 conn(0x7f24fc0b2940 legacy=0x7f24fc05c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f959a700 1 -- 172.21.15.61:0/245702121 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282762302 0 0) 0x7f24fc0b2f20 con 0x7f24fc0b2940 2022-07-12T08:50:09.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f9d9b700 1 -- 172.21.15.61:0/245702121 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc0b56e0 con 0x7f24fc0b45d0 2022-07-12T08:50:09.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f9d9b700 1 --2- 172.21.15.61:0/245702121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc0b45d0 0x7f24fc0b51f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f24ec002f70 tx=0x7f24ec006df0).ready entity=mon.0 client_cookie=60a4f16dd73a7d87 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:09.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f959a700 1 -- 172.21.15.61:0/245702121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24ec00d570 con 0x7f24fc0b45d0 2022-07-12T08:50:09.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f959a700 1 -- 172.21.15.61:0/245702121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec000e90 con 0x7f24fc0b45d0 2022-07-12T08:50:09.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f24f959a700 1 -- 172.21.15.61:0/245702121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24ec00d8c0 con 0x7f24fc0b45d0 2022-07-12T08:50:09.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- 172.21.15.61:0/245702121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc0b45d0 msgr2=0x7f24fc0b51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 --2- 172.21.15.61:0/245702121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc0b45d0 0x7f24fc0b51f0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7f24ec002f70 tx=0x7f24ec006df0).stop 2022-07-12T08:50:09.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- 172.21.15.61:0/245702121 shutdown_connections 2022-07-12T08:50:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 --2- 172.21.15.61:0/245702121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc0b45d0 0x7f24fc0b51f0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 --2- 172.21.15.61:0/245702121 >> v2:172.21.15.111:3300/0 conn(0x7f24fc066fc0 0x7f24fc0b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- 172.21.15.61:0/245702121 >> 172.21.15.61:0/245702121 conn(0x7f24fc0b94d0 msgr2=0x7f24fc0b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:09.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- 172.21.15.61:0/245702121 shutdown_connections 2022-07-12T08:50:09.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- 172.21.15.61:0/245702121 wait complete. 2022-07-12T08:50:09.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 Processor -- start 2022-07-12T08:50:09.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- start start 2022-07-12T08:50:09.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 0x7f24fc1486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f24fc0b2940 0x7f24fc148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24fc0b45d0 0x7f24fc13e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc144160 con 0x7f24fc066fc0 2022-07-12T08:50:09.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc0af9e0 con 0x7f24fc0b45d0 2022-07-12T08:50:09.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.624+0000 7f2500816700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24fc144390 con 0x7f24fc0b2940 2022-07-12T08:50:09.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 0x7f24fc1486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 0x7f24fc1486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60872/0 (socket says 172.21.15.61:60872) 2022-07-12T08:50:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 -- 172.21.15.61:0/3701689569 learned_addr learned my addr 172.21.15.61:0/3701689569 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:09.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24f9d9b700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f24fc0b2940 0x7f24fc148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fad9d700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24fc0b45d0 0x7f24fc13e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 -- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f24fc0b2940 msgr2=0x7f24fc148bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f24fc0b2940 0x7f24fc148bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 -- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24fc0b45d0 msgr2=0x7f24fc13e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24fc0b45d0 0x7f24fc13e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 -- 172.21.15.61:0/3701689569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec00b180 con 0x7f24fc066fc0 2022-07-12T08:50:09.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24fa59c700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 0x7f24fc1486d0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f24f000c000 tx=0x7f24f0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:09.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24f0010780 con 0x7f24fc066fc0 2022-07-12T08:50:09.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f00108e0 con 0x7f24fc066fc0 2022-07-12T08:50:09.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24fc144630 con 0x7f24fc066fc0 2022-07-12T08:50:09.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24f0010c40 con 0x7f24fc066fc0 2022-07-12T08:50:09.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24fc147f30 con 0x7f24fc066fc0 2022-07-12T08:50:09.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.628+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24fc057460 con 0x7f24fc066fc0 2022-07-12T08:50:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.632+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f24f0026430 con 0x7f24fc066fc0 2022-07-12T08:50:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.632+0000 7f24eb7fe700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f24d8053ad0 0x7f24d8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.632+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f24f006cce0 con 0x7f24fc066fc0 2022-07-12T08:50:09.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.632+0000 7f24f9d9b700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f24d8053ad0 0x7f24d8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.632+0000 7f24f9d9b700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f24d8053ad0 0x7f24d8055f90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f24ec01a000 tx=0x7f24ec014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.636+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24f0043c50 con 0x7f24fc066fc0 2022-07-12T08:50:09.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.800+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f24fc068470 con 0x7f24fc066fc0 2022-07-12T08:50:09.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.808+0000 7f24eb7fe700 1 -- 172.21.15.61:0/3701689569 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 25 and data pool 26 v71) v1 ==== 189+0+0 (secure 0 0 0) 0x7f24f0047270 con 0x7f24fc066fc0 2022-07-12T08:50:09.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f24d8053ad0 msgr2=0x7f24d8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f24d8053ad0 0x7f24d8055f90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f24ec01a000 tx=0x7f24ec014000).stop 2022-07-12T08:50:09.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 msgr2=0x7f24fc1486d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 0x7f24fc1486d0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7f24f000c000 tx=0x7f24f0006df0).stop 2022-07-12T08:50:09.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 shutdown_connections 2022-07-12T08:50:09.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f24d8053ad0 0x7f24d8055f90 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24fc0b45d0 0x7f24fc13e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f24fc0b2940 0x7f24fc148bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 --2- 172.21.15.61:0/3701689569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24fc066fc0 0x7f24fc1486d0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 >> 172.21.15.61:0/3701689569 conn(0x7f24fc0b94d0 msgr2=0x7f24fc0b7530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:09.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 shutdown_connections 2022-07-12T08:50:09.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.816+0000 7f2500816700 1 -- 172.21.15.61:0/3701689569 wait complete. 2022-07-12T08:50:09.848 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 25 and data pool 26 2022-07-12T08:50:09.849 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:09.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 Processor -- start 2022-07-12T08:50:09.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- start start 2022-07-12T08:50:09.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f65ac05c050 0x7f65ac0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac0ad2f0 con 0x7f65ac14a8f0 2022-07-12T08:50:09.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac05c430 con 0x7f65ac0b01f0 2022-07-12T08:50:09.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ac0b0a40 con 0x7f65ac05c050 2022-07-12T08:50:09.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b1230700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f65ac14a8f0 0x7f65ac1443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49112/0 (socket says 172.21.15.61:49112) 2022-07-12T08:50:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b1230700 1 -- 172.21.15.61:0/2694019775 learned_addr learned my addr 172.21.15.61:0/2694019775 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b0a2f700 1 --2- 172.21.15.61:0/2694019775 >> v2:172.21.15.111:3300/0 conn(0x7f65ac05c050 0x7f65ac0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65ab7fe700 1 -- 172.21.15.61:0/2694019775 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696074740 0 0) 0x7f65ac0ad2f0 con 0x7f65ac14a8f0 2022-07-12T08:50:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65ab7fe700 1 -- 172.21.15.61:0/2694019775 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c003260 con 0x7f65ac14a8f0 2022-07-12T08:50:09.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b0a2f700 1 -- 172.21.15.61:0/2694019775 >> v1:172.21.15.111:6789/0 conn(0x7f65ac0b01f0 legacy=0x7f65ac05cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b0a2f700 1 -- 172.21.15.61:0/2694019775 >> v1:172.21.15.61:6789/0 conn(0x7f65ac14a8f0 legacy=0x7f65ac1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b0a2f700 1 -- 172.21.15.61:0/2694019775 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac0b2420 con 0x7f65ac05c050 2022-07-12T08:50:09.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b0a2f700 1 --2- 172.21.15.61:0/2694019775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac05c050 0x7f65ac0ac710 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f659c0032f0 tx=0x7f659c0091d0).ready entity=mon.1 client_cookie=c6448d20640b871c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:09.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65ab7fe700 1 -- 172.21.15.61:0/2694019775 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659c00b6c0 con 0x7f65ac05c050 2022-07-12T08:50:09.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65ab7fe700 1 -- 172.21.15.61:0/2694019775 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c00b820 con 0x7f65ac05c050 2022-07-12T08:50:09.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65ab7fe700 1 -- 172.21.15.61:0/2694019775 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f659c00b990 con 0x7f65ac05c050 2022-07-12T08:50:09.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- 172.21.15.61:0/2694019775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac05c050 msgr2=0x7f65ac0ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 --2- 172.21.15.61:0/2694019775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac05c050 0x7f65ac0ac710 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f659c0032f0 tx=0x7f659c0091d0).stop 2022-07-12T08:50:09.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- 172.21.15.61:0/2694019775 shutdown_connections 2022-07-12T08:50:09.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 --2- 172.21.15.61:0/2694019775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac05c050 0x7f65ac0ac710 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- 172.21.15.61:0/2694019775 >> 172.21.15.61:0/2694019775 conn(0x7f65ac062c00 msgr2=0x7f65ac062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:09.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- 172.21.15.61:0/2694019775 shutdown_connections 2022-07-12T08:50:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.964+0000 7f65b2c93700 1 -- 172.21.15.61:0/2694019775 wait complete. 2022-07-12T08:50:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 Processor -- start 2022-07-12T08:50:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 -- start start 2022-07-12T08:50:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65ac05c050 0x7f65ac0ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac0b01f0 0x7f65ac156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 0x7f65ac157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac157680 con 0x7f65ac14a8f0 2022-07-12T08:50:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac1492c0 con 0x7f65ac0b01f0 2022-07-12T08:50:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac144650 con 0x7f65ac05c050 2022-07-12T08:50:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 0x7f65ac157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 0x7f65ac157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60900/0 (socket says 172.21.15.61:60900) 2022-07-12T08:50:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 -- 172.21.15.61:0/3721226955 learned_addr learned my addr 172.21.15.61:0/3721226955 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b0a2f700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65ac05c050 0x7f65ac0ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65abfff700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac0b01f0 0x7f65ac156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 -- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65ac05c050 msgr2=0x7f65ac0ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65ac05c050 0x7f65ac0ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 -- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac0b01f0 msgr2=0x7f65ac156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac0b01f0 0x7f65ac156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 -- 172.21.15.61:0/3721226955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c009470 con 0x7f65ac14a8f0 2022-07-12T08:50:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b0a2f700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65ac05c050 0x7f65ac0ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:09.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b1230700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 0x7f65ac157190 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f65a4001110 tx=0x7f65a4007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:09.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65a400ce00 con 0x7f65ac14a8f0 2022-07-12T08:50:09.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a40072e0 con 0x7f65ac14a8f0 2022-07-12T08:50:09.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65a4005800 con 0x7f65ac14a8f0 2022-07-12T08:50:09.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac148d40 con 0x7f65ac14a8f0 2022-07-12T08:50:09.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.968+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac1568c0 con 0x7f65ac14a8f0 2022-07-12T08:50:09.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.972+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac1494c0 con 0x7f65ac14a8f0 2022-07-12T08:50:09.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.972+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f65a4010070 con 0x7f65ac14a8f0 2022-07-12T08:50:09.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.972+0000 7f65a9ffb700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6590053a40 0x7f6590055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:09.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.972+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f65a406b690 con 0x7f65ac14a8f0 2022-07-12T08:50:09.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.972+0000 7f65b0a2f700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6590053a40 0x7f6590055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:09.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.972+0000 7f65b0a2f700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6590053a40 0x7f6590055f00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f659c019000 tx=0x7f659c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:09.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:09.980+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65a4042550 con 0x7f65ac14a8f0 2022-07-12T08:50:10.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.108+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65ac057460 con 0x7f65ac14a8f0 2022-07-12T08:50:10.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65a9ffb700 1 -- 172.21.15.61:0/3721226955 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f65a4069e60 con 0x7f65ac14a8f0 2022-07-12T08:50:10.120 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:10.121 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":127,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:09.802964+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:08.408495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:08.818782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:10.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6590053a40 msgr2=0x7f6590055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6590053a40 0x7f6590055f00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f659c019000 tx=0x7f659c016000).stop 2022-07-12T08:50:10.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 msgr2=0x7f65ac157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 0x7f65ac157190 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f65a4001110 tx=0x7f65a4007af0).stop 2022-07-12T08:50:10.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 shutdown_connections 2022-07-12T08:50:10.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65ac14a8f0 0x7f65ac157190 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6590053a40 0x7f6590055f00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65ac0b01f0 0x7f65ac156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 --2- 172.21.15.61:0/3721226955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f65ac05c050 0x7f65ac0ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 >> 172.21.15.61:0/3721226955 conn(0x7f65ac062c00 msgr2=0x7f65ac062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:10.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 shutdown_connections 2022-07-12T08:50:10.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.112+0000 7f65b2c93700 1 -- 172.21.15.61:0/3721226955 wait complete. 2022-07-12T08:50:10.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:10.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d7fff700 1 Processor -- start 2022-07-12T08:50:10.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d7fff700 1 -- start start 2022-07-12T08:50:10.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe4d805f140 0x7fe4d8151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80abc70 con 0x7fe4d80ab290 2022-07-12T08:50:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d7fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d805c720 con 0x7fe4d80ab670 2022-07-12T08:50:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d7fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d80aba90 con 0x7fe4d805f140 2022-07-12T08:50:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe4d80ab290 0x7fe4d805ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49122/0 (socket says 172.21.15.61:49122) 2022-07-12T08:50:10.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d6ffd700 1 -- 172.21.15.61:0/971944844 learned_addr learned my addr 172.21.15.61:0/971944844 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d77fe700 1 --2- 172.21.15.61:0/971944844 >> v2:172.21.15.111:3300/0 conn(0x7fe4d805f140 0x7fe4d8151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584634725 0 0) 0x7fe4d80abc70 con 0x7fe4d80ab290 2022-07-12T08:50:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4bc003260 con 0x7fe4d80ab290 2022-07-12T08:50:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711733188 0 0) 0x7fe4bc003260 con 0x7fe4d80ab290 2022-07-12T08:50:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 >> v2:172.21.15.111:3300/0 conn(0x7fe4d805f140 msgr2=0x7fe4d8151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:10.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 --2- 172.21.15.61:0/971944844 >> v2:172.21.15.111:3300/0 conn(0x7fe4d805f140 0x7fe4d8151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 >> v1:172.21.15.111:6789/0 conn(0x7fe4d80ab670 legacy=0x7fe4d80afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d805c9c0 con 0x7fe4d80ab290 2022-07-12T08:50:10.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fe4d00019e0 con 0x7fe4d80ab290 2022-07-12T08:50:10.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 >> v1:172.21.15.61:6789/0 conn(0x7fe4d80ab290 legacy=0x7fe4d805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4bc003750 0x7fe4bc003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4bc004500 0x7fe4bc004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc004e70 0x7fe4bc00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d80aba90 con 0x7fe4bc004500 2022-07-12T08:50:10.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc00bdb0 con 0x7fe4bc004e70 2022-07-12T08:50:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc00bf70 con 0x7fe4bc003750 2022-07-12T08:50:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4bc004500 0x7fe4bc004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d77fe700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc004e70 0x7fe4bc00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.252+0000 7fe4d6ffd700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4bc003750 0x7fe4bc003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d67fc700 1 -- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4bc003750 msgr2=0x7fe4bc003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:10.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4bc003750 0x7fe4bc003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d67fc700 1 -- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc004e70 msgr2=0x7fe4bc00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc004e70 0x7fe4bc00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d67fc700 1 -- 172.21.15.61:0/971944844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc003260 con 0x7fe4bc004500 2022-07-12T08:50:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d77fe700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4bc004e70 0x7fe4bc00b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d77fe700 1 -- 172.21.15.61:0/971944844 reap_dead start 2022-07-12T08:50:10.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4bc004500 0x7fe4bc004980 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7fe4d80af510 tx=0x7fe4cc007a00).ready entity=mon.0 client_cookie=6e9ff20e531b1d69 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4cc007d10 con 0x7fe4bc004500 2022-07-12T08:50:10.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc007030 con 0x7fe4bc004500 2022-07-12T08:50:10.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d5ffb700 1 -- 172.21.15.61:0/971944844 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4cc007390 con 0x7fe4bc004500 2022-07-12T08:50:10.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- 172.21.15.61:0/971944844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4bc004500 msgr2=0x7fe4bc004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4bc004500 0x7fe4bc004980 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7fe4d80af510 tx=0x7fe4cc007a00).stop 2022-07-12T08:50:10.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- 172.21.15.61:0/971944844 shutdown_connections 2022-07-12T08:50:10.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/971944844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4bc004500 0x7fe4bc004980 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- 172.21.15.61:0/971944844 >> 172.21.15.61:0/971944844 conn(0x7fe4d80b1950 msgr2=0x7fe4d80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- 172.21.15.61:0/971944844 shutdown_connections 2022-07-12T08:50:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- 172.21.15.61:0/971944844 wait complete. 2022-07-12T08:50:10.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 Processor -- start 2022-07-12T08:50:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- start start 2022-07-12T08:50:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4d8145400 0x7fe4d8066270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 0x7fe4d80ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d80aeda0 0x7fe4d8143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d81457e0 con 0x7fe4d8063860 2022-07-12T08:50:10.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8063d00 con 0x7fe4d80aeda0 2022-07-12T08:50:10.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.256+0000 7fe4d7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8063fa0 con 0x7fe4d8145400 2022-07-12T08:50:10.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 0x7fe4d80ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d80aeda0 0x7fe4d8143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 0x7fe4d80ae8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60918/0 (socket says 172.21.15.61:60918) 2022-07-12T08:50:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 -- 172.21.15.61:0/967158109 learned_addr learned my addr 172.21.15.61:0/967158109 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:10.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d6ffd700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4d8145400 0x7fe4d8066270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 -- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4d8145400 msgr2=0x7fe4d8066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4d8145400 0x7fe4d8066270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 -- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d80aeda0 msgr2=0x7fe4d8143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d80aeda0 0x7fe4d8143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 -- 172.21.15.61:0/967158109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d80ae410 con 0x7fe4d8063860 2022-07-12T08:50:10.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d67fc700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 0x7fe4d80ae8b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7fe4d80af510 tx=0x7fe4cc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4cc007d10 con 0x7fe4d8063860 2022-07-12T08:50:10.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc020640 con 0x7fe4d8063860 2022-07-12T08:50:10.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d80672b0 con 0x7fe4d8063860 2022-07-12T08:50:10.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4cc0209a0 con 0x7fe4d8063860 2022-07-12T08:50:10.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d8064c80 con 0x7fe4d8063860 2022-07-12T08:50:10.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d80ae5a0 con 0x7fe4d8063860 2022-07-12T08:50:10.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe4cc028030 con 0x7fe4d8063860 2022-07-12T08:50:10.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4c7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4b4053a40 0x7fe4b4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe4cc001e40 con 0x7fe4d8063860 2022-07-12T08:50:10.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.260+0000 7fe4d6ffd700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4b4053a40 0x7fe4b4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.264+0000 7fe4d6ffd700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4b4053a40 0x7fe4b4055f00 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fe4d0005000 tx=0x7fe4d80aadc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.268+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4cc044100 con 0x7fe4d8063860 2022-07-12T08:50:10.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4d8057460 con 0x7fe4d8063860 2022-07-12T08:50:10.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4c7fff700 1 -- 172.21.15.61:0/967158109 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fe4cc047720 con 0x7fe4d8063860 2022-07-12T08:50:10.395 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:10.395 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":127,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:09.802964+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:08.408495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:08.818782+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:10.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4b4053a40 msgr2=0x7fe4b4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4b4053a40 0x7fe4b4055f00 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fe4d0005000 tx=0x7fe4d80aadc0).stop 2022-07-12T08:50:10.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 msgr2=0x7fe4d80ae8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 0x7fe4d80ae8b0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 rx=0x7fe4d80af510 tx=0x7fe4cc015000).stop 2022-07-12T08:50:10.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 shutdown_connections 2022-07-12T08:50:10.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4b4053a40 0x7fe4b4055f00 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4d80aeda0 0x7fe4d8143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4d8063860 0x7fe4d80ae8b0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 --2- 172.21.15.61:0/967158109 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4d8145400 0x7fe4d8066270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 >> 172.21.15.61:0/967158109 conn(0x7fe4d80b1950 msgr2=0x7fe4d80b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:10.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 shutdown_connections 2022-07-12T08:50:10.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.392+0000 7fe4d7fff700 1 -- 172.21.15.61:0/967158109 wait complete. 2022-07-12T08:50:10.409 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:50:10.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 Processor -- start 2022-07-12T08:50:10.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- start start 2022-07-12T08:50:10.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc099b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9abc142480 0x7f9abc142910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9abc0993f0 con 0x7f9abc0b1570 2022-07-12T08:50:10.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc099690 con 0x7f9abc142480 2022-07-12T08:50:10.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc060ff0 con 0x7f9abc060c10 2022-07-12T08:50:10.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9abc142480 0x7f9abc142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9abc142480 0x7f9abc142910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60928/0 (socket says 172.21.15.61:60928) 2022-07-12T08:50:10.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 -- 172.21.15.61:0/2372765779 learned_addr learned my addr 172.21.15.61:0/2372765779 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:10.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9abbfff700 1 --2- 172.21.15.61:0/2372765779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9abaffd700 1 -- 172.21.15.61:0/2372765779 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615065075 0 0) 0x7f9abc0993f0 con 0x7f9abc0b1570 2022-07-12T08:50:10.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9abaffd700 1 -- 172.21.15.61:0/2372765779 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8003260 con 0x7f9abc0b1570 2022-07-12T08:50:10.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 -- 172.21.15.61:0/2372765779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 msgr2=0x7f9abc099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:10.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 --2- 172.21.15.61:0/2372765779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc099b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 -- 172.21.15.61:0/2372765779 >> v1:172.21.15.111:6789/0 conn(0x7f9abc0b1570 legacy=0x7f9abc0ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 -- 172.21.15.61:0/2372765779 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc142e00 con 0x7f9abc142480 2022-07-12T08:50:10.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac0d5b700 1 --2- 172.21.15.61:0/2372765779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc142480 0x7f9abc142910 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f9ab40032f0 tx=0x7f9ab4009290).ready entity=mon.0 client_cookie=88597df3355418a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9abaffd700 1 -- 172.21.15.61:0/2372765779 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ab400b6c0 con 0x7f9abc142480 2022-07-12T08:50:10.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9abaffd700 1 -- 172.21.15.61:0/2372765779 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab400b820 con 0x7f9abc142480 2022-07-12T08:50:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9abaffd700 1 -- 172.21.15.61:0/2372765779 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ab400b990 con 0x7f9abc142480 2022-07-12T08:50:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- 172.21.15.61:0/2372765779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc142480 msgr2=0x7f9abc142910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 --2- 172.21.15.61:0/2372765779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc142480 0x7f9abc142910 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f9ab40032f0 tx=0x7f9ab4009290).stop 2022-07-12T08:50:10.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- 172.21.15.61:0/2372765779 shutdown_connections 2022-07-12T08:50:10.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 --2- 172.21.15.61:0/2372765779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc142480 0x7f9abc142910 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 --2- 172.21.15.61:0/2372765779 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc099b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- 172.21.15.61:0/2372765779 >> 172.21.15.61:0/2372765779 conn(0x7f9abc05bd10 msgr2=0x7f9abc05c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:10.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- 172.21.15.61:0/2372765779 shutdown_connections 2022-07-12T08:50:10.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.524+0000 7f9ac155c700 1 -- 172.21.15.61:0/2372765779 wait complete. 2022-07-12T08:50:10.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 Processor -- start 2022-07-12T08:50:10.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- start start 2022-07-12T08:50:10.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc0968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 0x7f9abc1404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9abc142480 0x7f9abc0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0ae090 con 0x7f9abc0b1570 2022-07-12T08:50:10.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0ae330 con 0x7f9abc142480 2022-07-12T08:50:10.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc0add80 con 0x7f9abc060c10 2022-07-12T08:50:10.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 0x7f9abc1404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 0x7f9abc1404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60932/0 (socket says 172.21.15.61:60932) 2022-07-12T08:50:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 -- 172.21.15.61:0/1434149794 learned_addr learned my addr 172.21.15.61:0/1434149794 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abbfff700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc0968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac0d5b700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9abc142480 0x7f9abc0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 -- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 msgr2=0x7f9abc0968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc0968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 -- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9abc142480 msgr2=0x7f9abc0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9abc142480 0x7f9abc0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 -- 172.21.15.61:0/1434149794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4009530 con 0x7f9abc0b1570 2022-07-12T08:50:10.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac0d5b700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9abc142480 0x7f9abc0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:10.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9abb7fe700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 0x7f9abc1404f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7f9ab0001110 tx=0x7f9ab0007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ab000f470 con 0x7f9abc0b1570 2022-07-12T08:50:10.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab000f5d0 con 0x7f9abc0b1570 2022-07-12T08:50:10.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc0973c0 con 0x7f9abc0b1570 2022-07-12T08:50:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ab000f930 con 0x7f9abc0b1570 2022-07-12T08:50:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc1521d0 con 0x7f9abc0b1570 2022-07-12T08:50:10.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.528+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc0ae4f0 con 0x7f9abc0b1570 2022-07-12T08:50:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.532+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9ab0010070 con 0x7f9abc0b1570 2022-07-12T08:50:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.532+0000 7f9ab97fa700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a9c053a40 0x7f9a9c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.532+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ab006b150 con 0x7f9abc0b1570 2022-07-12T08:50:10.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.532+0000 7f9abbfff700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a9c053a40 0x7f9a9c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.532+0000 7f9abbfff700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a9c053a40 0x7f9a9c055f00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f9aac00b000 tx=0x7f9aac008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.536+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ab00420b0 con 0x7f9abc0b1570 2022-07-12T08:50:10.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.704+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9abc0982e0 con 0x7f9abc0b1570 2022-07-12T08:50:10.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.820+0000 7f9ab97fa700 1 -- 172.21.15.61:0/1434149794 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9ab00456d0 con 0x7f9abc0b1570 2022-07-12T08:50:10.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a9c053a40 msgr2=0x7f9a9c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a9c053a40 0x7f9a9c055f00 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f9aac00b000 tx=0x7f9aac008040).stop 2022-07-12T08:50:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 msgr2=0x7f9abc1404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 0x7f9abc1404f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7f9ab0001110 tx=0x7f9ab0007b20).stop 2022-07-12T08:50:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 shutdown_connections 2022-07-12T08:50:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9a9c053a40 0x7f9a9c055f00 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9abc142480 0x7f9abc0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9abc0b1570 0x7f9abc1404f0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 --2- 172.21.15.61:0/1434149794 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9abc060c10 0x7f9abc0968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 >> 172.21.15.61:0/1434149794 conn(0x7f9abc05bd10 msgr2=0x7f9abc143810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:10.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 shutdown_connections 2022-07-12T08:50:10.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.824+0000 7f9ac155c700 1 -- 172.21.15.61:0/1434149794 wait complete. 2022-07-12T08:50:10.846 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:10.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.952+0000 7f36057d7700 1 Processor -- start 2022-07-12T08:50:10.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.952+0000 7f36057d7700 1 -- start start 2022-07-12T08:50:10.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3600144ad0 0x7f36000ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36000ad2f0 con 0x7f36000b01f0 2022-07-12T08:50:10.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f360005c430 con 0x7f360005c050 2022-07-12T08:50:10.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36000b0a40 con 0x7f3600144ad0 2022-07-12T08:50:10.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fe7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f36000b01f0 0x7f36001443c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49156/0 (socket says 172.21.15.61:49156) 2022-07-12T08:50:10.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fe7fc700 1 -- 172.21.15.61:0/150061153 learned_addr learned my addr 172.21.15.61:0/150061153 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:10.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35ff7fe700 1 --2- 172.21.15.61:0/150061153 >> v2:172.21.15.111:3300/0 conn(0x7f3600144ad0 0x7f36000ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234410578 0 0) 0x7f36000ad2f0 con 0x7f36000b01f0 2022-07-12T08:50:10.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e0003260 con 0x7f36000b01f0 2022-07-12T08:50:10.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372279668 0 0) 0x7f360005c430 con 0x7f360005c050 2022-07-12T08:50:10.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36000ad2f0 con 0x7f360005c050 2022-07-12T08:50:10.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35ff7fe700 1 -- 172.21.15.61:0/150061153 >> v1:172.21.15.111:6789/0 conn(0x7f360005c050 legacy=0x7f360005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459446982 0 0) 0x7f35e0003260 con 0x7f36000b01f0 2022-07-12T08:50:10.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35ff7fe700 1 -- 172.21.15.61:0/150061153 >> v1:172.21.15.61:6789/0 conn(0x7f36000b01f0 legacy=0x7f36001443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35ff7fe700 1 -- 172.21.15.61:0/150061153 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36000b2420 con 0x7f3600144ad0 2022-07-12T08:50:10.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35ff7fe700 1 --2- 172.21.15.61:0/150061153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f36000ac710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f35f0001110 tx=0x7f35f80091d0).ready entity=mon.1 client_cookie=4dc60cb7d3c32b90 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35f0001660 con 0x7f3600144ad0 2022-07-12T08:50:10.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f800f510 con 0x7f3600144ad0 2022-07-12T08:50:10.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f35fdffb700 1 -- 172.21.15.61:0/150061153 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35f800f680 con 0x7f3600144ad0 2022-07-12T08:50:10.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- 172.21.15.61:0/150061153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 msgr2=0x7f36000ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 --2- 172.21.15.61:0/150061153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f36000ac710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f35f0001110 tx=0x7f35f80091d0).stop 2022-07-12T08:50:10.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- 172.21.15.61:0/150061153 shutdown_connections 2022-07-12T08:50:10.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 --2- 172.21.15.61:0/150061153 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f36000ac710 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- 172.21.15.61:0/150061153 >> 172.21.15.61:0/150061153 conn(0x7f3600062c00 msgr2=0x7f3600062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:10.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- 172.21.15.61:0/150061153 shutdown_connections 2022-07-12T08:50:10.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.956+0000 7f36057d7700 1 -- 172.21.15.61:0/150061153 wait complete. 2022-07-12T08:50:10.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 Processor -- start 2022-07-12T08:50:10.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 -- start start 2022-07-12T08:50:10.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f360005c050 0x7f3600148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 0x7f36001570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f360014a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36000af170 con 0x7f36000b01f0 2022-07-12T08:50:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36001564e0 con 0x7f3600144ad0 2022-07-12T08:50:10.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36001566d0 con 0x7f360005c050 2022-07-12T08:50:10.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 0x7f36001570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 0x7f36001570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60944/0 (socket says 172.21.15.61:60944) 2022-07-12T08:50:10.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 -- 172.21.15.61:0/131158160 learned_addr learned my addr 172.21.15.61:0/131158160 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:10.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35ff7fe700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f360014a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35feffd700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f360005c050 0x7f3600148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 -- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f360005c050 msgr2=0x7f3600148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f360005c050 0x7f3600148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 -- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 msgr2=0x7f360014a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:10.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f360014a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:10.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 -- 172.21.15.61:0/131158160 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f8009470 con 0x7f36000b01f0 2022-07-12T08:50:10.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35feffd700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f360005c050 0x7f3600148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35fe7fc700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 0x7f36001570b0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f35f0000fe0 tx=0x7f35f0001b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35f0007170 con 0x7f36000b01f0 2022-07-12T08:50:10.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f00072d0 con 0x7f36000b01f0 2022-07-12T08:50:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3600156970 con 0x7f36000b01f0 2022-07-12T08:50:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35f0004920 con 0x7f36000b01f0 2022-07-12T08:50:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.960+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36001575a0 con 0x7f36000b01f0 2022-07-12T08:50:10.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.964+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3600057460 con 0x7f36000b01f0 2022-07-12T08:50:10.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.964+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f35f001b070 con 0x7f36000b01f0 2022-07-12T08:50:10.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.964+0000 7f35f7fff700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35e4053af0 0x7f35e4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:10.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.964+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f35f006b7c0 con 0x7f36000b01f0 2022-07-12T08:50:10.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.964+0000 7f35feffd700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35e4053af0 0x7f35e4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:10.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.968+0000 7f35feffd700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35e4053af0 0x7f35e4055fb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f35e8008000 tx=0x7f35e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:10.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:10.972+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35f0042720 con 0x7f36000b01f0 2022-07-12T08:50:11.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.136+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36000447a0 con 0x7f36000b01f0 2022-07-12T08:50:11.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.136+0000 7f35f7fff700 1 -- 172.21.15.61:0/131158160 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f35f0045d40 con 0x7f36000b01f0 2022-07-12T08:50:11.147 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:11.147 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:10.807048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8082":{"gid":8082,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2724832679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2724832679},{"type":"v1","addr":"172.21.15.61:6833","nonce":2724832679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:11.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.136+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35e4053af0 msgr2=0x7f35e4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.136+0000 7f36057d7700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35e4053af0 0x7f35e4055fb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f35e8008000 tx=0x7f35e8009040).stop 2022-07-12T08:50:11.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.136+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 msgr2=0x7f36001570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.136+0000 7f36057d7700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 0x7f36001570b0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f35f0000fe0 tx=0x7f35f0001b60).stop 2022-07-12T08:50:11.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 shutdown_connections 2022-07-12T08:50:11.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f35e4053af0 0x7f35e4055fb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3600144ad0 0x7f360014a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36000b01f0 0x7f36001570b0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 --2- 172.21.15.61:0/131158160 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f360005c050 0x7f3600148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 >> 172.21.15.61:0/131158160 conn(0x7f3600062c00 msgr2=0x7f3600142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:11.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 shutdown_connections 2022-07-12T08:50:11.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.140+0000 7f36057d7700 1 -- 172.21.15.61:0/131158160 wait complete. 2022-07-12T08:50:11.155 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-12T08:50:11.157 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:11.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 Processor -- start 2022-07-12T08:50:11.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- start start 2022-07-12T08:50:11.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa3c00ab290 0x7fa3c00ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa3c005f140 0x7fa3c005ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c00abc70 con 0x7fa3c00afa60 2022-07-12T08:50:11.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c005f580 con 0x7fa3c005f140 2022-07-12T08:50:11.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c005c720 con 0x7fa3c00ab290 2022-07-12T08:50:11.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa3c005f140 0x7fa3c005ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa3c005f140 0x7fa3c005ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60952/0 (socket says 172.21.15.61:60952) 2022-07-12T08:50:11.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 -- 172.21.15.61:0/1079183359 learned_addr learned my addr 172.21.15.61:0/1079183359 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:11.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c4a2c700 1 --2- 172.21.15.61:0/1079183359 >> v2:172.21.15.111:3300/0 conn(0x7fa3c00ab290 0x7fa3c00ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 -- 172.21.15.61:0/1079183359 >> v2:172.21.15.111:3300/0 conn(0x7fa3c00ab290 msgr2=0x7fa3c00ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:11.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 --2- 172.21.15.61:0/1079183359 >> v2:172.21.15.111:3300/0 conn(0x7fa3c00ab290 0x7fa3c00ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 -- 172.21.15.61:0/1079183359 >> v1:172.21.15.111:6789/0 conn(0x7fa3c00afa60 legacy=0x7fa3c014a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:11.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 -- 172.21.15.61:0/1079183359 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c005c9c0 con 0x7fa3c005f140 2022-07-12T08:50:11.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bffff700 1 --2- 172.21.15.61:0/1079183359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c005f140 0x7fa3c005ce60 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7fa3c00615d0 tx=0x7fa3b8009290).ready entity=mon.0 client_cookie=4d890e7b8c0a00a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bf7fe700 1 -- 172.21.15.61:0/1079183359 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b800b830 con 0x7fa3c005f140 2022-07-12T08:50:11.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bf7fe700 1 -- 172.21.15.61:0/1079183359 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b800b990 con 0x7fa3c005f140 2022-07-12T08:50:11.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3bf7fe700 1 -- 172.21.15.61:0/1079183359 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b800bb40 con 0x7fa3c005f140 2022-07-12T08:50:11.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/1079183359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c005f140 msgr2=0x7fa3c005ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/1079183359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c005f140 0x7fa3c005ce60 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7fa3c00615d0 tx=0x7fa3b8009290).stop 2022-07-12T08:50:11.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/1079183359 shutdown_connections 2022-07-12T08:50:11.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/1079183359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c005f140 0x7fa3c005ce60 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/1079183359 >> v2:172.21.15.111:3300/0 conn(0x7fa3c00ab290 0x7fa3c00ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/1079183359 >> 172.21.15.61:0/1079183359 conn(0x7fa3c00b1950 msgr2=0x7fa3c00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:11.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/1079183359 shutdown_connections 2022-07-12T08:50:11.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.256+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/1079183359 wait complete. 2022-07-12T08:50:11.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 Processor -- start 2022-07-12T08:50:11.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 -- start start 2022-07-12T08:50:11.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c005f140 0x7fa3c00aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 0x7fa3c00ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3c00afa60 0x7fa3c00ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c0099cd0 con 0x7fa3c00ab290 2022-07-12T08:50:11.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c0099f70 con 0x7fa3c005f140 2022-07-12T08:50:11.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c00aebc0 con 0x7fa3c00afa60 2022-07-12T08:50:11.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 0x7fa3c00ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 0x7fa3c00ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60960/0 (socket says 172.21.15.61:60960) 2022-07-12T08:50:11.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 -- 172.21.15.61:0/4100923176 learned_addr learned my addr 172.21.15.61:0/4100923176 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:11.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c4a2c700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c005f140 0x7fa3c00aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c522d700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3c00afa60 0x7fa3c00ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 -- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3c00afa60 msgr2=0x7fa3c00ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3c00afa60 0x7fa3c00ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 -- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c005f140 msgr2=0x7fa3c00aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c005f140 0x7fa3c00aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 -- 172.21.15.61:0/4100923176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8009530 con 0x7fa3c00ab290 2022-07-12T08:50:11.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c522d700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3c00afa60 0x7fa3c00ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:11.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bffff700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 0x7fa3c00ae300 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7fa3c00615d0 tx=0x7fa3b8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b8021440 con 0x7fa3c00ab290 2022-07-12T08:50:11.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b80215a0 con 0x7fa3c00ab290 2022-07-12T08:50:11.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c0065ed0 con 0x7fa3c00ab290 2022-07-12T08:50:11.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b8021900 con 0x7fa3c00ab290 2022-07-12T08:50:11.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.260+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c0099a50 con 0x7fa3c00ab290 2022-07-12T08:50:11.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.264+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c0057460 con 0x7fa3c00ab290 2022-07-12T08:50:11.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.264+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa3b8027020 con 0x7fa3c00ab290 2022-07-12T08:50:11.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.264+0000 7fa3bdffb700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa398053a60 0x7fa398055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.264+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa3b806d380 con 0x7fa3c00ab290 2022-07-12T08:50:11.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.264+0000 7fa3c4a2c700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa398053a60 0x7fa398055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.264+0000 7fa3c4a2c700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa398053a60 0x7fa398055f20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fa3a800c000 tx=0x7fa3a8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.272+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3b80442e0 con 0x7fa3c00ab290 2022-07-12T08:50:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.400+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c0066060 con 0x7fa3c00ab290 2022-07-12T08:50:11.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.404+0000 7fa3bdffb700 1 -- 172.21.15.61:0/4100923176 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fa3b8015070 con 0x7fa3c00ab290 2022-07-12T08:50:11.409 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:11.410 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":128,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:10.806045+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:08.408495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:08.818782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:11.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa398053a60 msgr2=0x7fa398055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa398053a60 0x7fa398055f20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fa3a800c000 tx=0x7fa3a8008040).stop 2022-07-12T08:50:11.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 msgr2=0x7fa3c00ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 0x7fa3c00ae300 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7fa3c00615d0 tx=0x7fa3b8016000).stop 2022-07-12T08:50:11.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 shutdown_connections 2022-07-12T08:50:11.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa3c00afa60 0x7fa3c00ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c00ab290 0x7fa3c00ae300 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa398053a60 0x7fa398055f20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 --2- 172.21.15.61:0/4100923176 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa3c005f140 0x7fa3c00aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 >> 172.21.15.61:0/4100923176 conn(0x7fa3c00b1950 msgr2=0x7fa3c014d870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:11.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 shutdown_connections 2022-07-12T08:50:11.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.408+0000 7fa3c5a2e700 1 -- 172.21.15.61:0/4100923176 wait complete. 2022-07-12T08:50:11.426 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:50:11.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 Processor -- start 2022-07-12T08:50:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 -- start start 2022-07-12T08:50:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02bc0b52e0 0x7f02bc0b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc0b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02bc0b56c0 con 0x7f02bc0b2b90 2022-07-12T08:50:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02bc0b3000 con 0x7f02bc0b52e0 2022-07-12T08:50:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c2b56700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc142f00 con 0x7f02bc0b4050 2022-07-12T08:50:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02bc0b52e0 0x7f02bc0b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02bc0b52e0 0x7f02bc0b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60966/0 (socket says 172.21.15.61:60966) 2022-07-12T08:50:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 -- 172.21.15.61:0/1225280509 learned_addr learned my addr 172.21.15.61:0/1225280509 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02bbfff700 1 --2- 172.21.15.61:0/1225280509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc0b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02bb7fe700 1 -- 172.21.15.61:0/1225280509 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828000785 0 0) 0x7f02bc0b56c0 con 0x7f02bc0b2b90 2022-07-12T08:50:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02bb7fe700 1 -- 172.21.15.61:0/1225280509 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02a0003260 con 0x7f02bc0b2b90 2022-07-12T08:50:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 -- 172.21.15.61:0/1225280509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 msgr2=0x7f02bc0b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 --2- 172.21.15.61:0/1225280509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc0b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 -- 172.21.15.61:0/1225280509 >> v1:172.21.15.111:6789/0 conn(0x7f02bc0b2b90 legacy=0x7f02bc05bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 -- 172.21.15.61:0/1225280509 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0b4bd0 con 0x7f02bc0b52e0 2022-07-12T08:50:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02c08f2700 1 --2- 172.21.15.61:0/1225280509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0b21b0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f02b0002f70 tx=0x7f02b0006eb0).ready entity=mon.0 client_cookie=4320aa379e8ce5ff server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02bb7fe700 1 -- 172.21.15.61:0/1225280509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02b000d5c0 con 0x7f02bc0b52e0 2022-07-12T08:50:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02bb7fe700 1 -- 172.21.15.61:0/1225280509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b0000e90 con 0x7f02bc0b52e0 2022-07-12T08:50:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.548+0000 7f02bb7fe700 1 -- 172.21.15.61:0/1225280509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02b000d950 con 0x7f02bc0b52e0 2022-07-12T08:50:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- 172.21.15.61:0/1225280509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b52e0 msgr2=0x7f02bc0b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 --2- 172.21.15.61:0/1225280509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0b21b0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f02b0002f70 tx=0x7f02b0006eb0).stop 2022-07-12T08:50:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- 172.21.15.61:0/1225280509 shutdown_connections 2022-07-12T08:50:11.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 --2- 172.21.15.61:0/1225280509 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc0b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 --2- 172.21.15.61:0/1225280509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0b21b0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- 172.21.15.61:0/1225280509 >> 172.21.15.61:0/1225280509 conn(0x7f02bc067020 msgr2=0x7f02bc067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- 172.21.15.61:0/1225280509 shutdown_connections 2022-07-12T08:50:11.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- 172.21.15.61:0/1225280509 wait complete. 2022-07-12T08:50:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 Processor -- start 2022-07-12T08:50:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- start start 2022-07-12T08:50:11.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 0x7f02bc148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc112c50 con 0x7f02bc0b2b90 2022-07-12T08:50:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc1530d0 con 0x7f02bc0b52e0 2022-07-12T08:50:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c2b56700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc1473d0 con 0x7f02bc0b4050 2022-07-12T08:50:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 0x7f02bc148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 0x7f02bc148b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:60972/0 (socket says 172.21.15.61:60972) 2022-07-12T08:50:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 -- 172.21.15.61:0/2156530355 learned_addr learned my addr 172.21.15.61:0/2156530355 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02bbfff700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c10f3700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 -- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 msgr2=0x7f02bc151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 -- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0b52e0 msgr2=0x7f02bc0adaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0adaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.552+0000 7f02c08f2700 1 -- 172.21.15.61:0/2156530355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b000b1e0 con 0x7f02bc0b2b90 2022-07-12T08:50:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02c08f2700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 0x7f02bc148b90 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7f02b0002f70 tx=0x7f02b0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02b0011860 con 0x7f02bc0b2b90 2022-07-12T08:50:11.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b00119c0 con 0x7f02bc0b2b90 2022-07-12T08:50:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02b0011d20 con 0x7f02bc0b2b90 2022-07-12T08:50:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02bc1129b0 con 0x7f02bc0b2b90 2022-07-12T08:50:11.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc0af880 con 0x7f02bc0b2b90 2022-07-12T08:50:11.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02bc147020 con 0x7f02bc0b2b90 2022-07-12T08:50:11.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f02b0025030 con 0x7f02bc0b2b90 2022-07-12T08:50:11.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02b9ffb700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f029c053ad0 0x7f029c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f02b006cf40 con 0x7f02bc0b2b90 2022-07-12T08:50:11.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.556+0000 7f02bbfff700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f029c053ad0 0x7f029c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.560+0000 7f02bbfff700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f029c053ad0 0x7f029c055f90 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f02a800b000 tx=0x7f02a8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.564+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02b0043ea0 con 0x7f02bc0b2b90 2022-07-12T08:50:11.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.732+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f02bc0af880 con 0x7f02bc0b2b90 2022-07-12T08:50:11.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02b9ffb700 1 -- 172.21.15.61:0/2156530355 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7f02bc0af880 con 0x7f02bc0b2b90 2022-07-12T08:50:11.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f029c053ad0 msgr2=0x7f029c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f029c053ad0 0x7f029c055f90 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f02a800b000 tx=0x7f02a8008040).stop 2022-07-12T08:50:11.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 msgr2=0x7f02bc148b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 0x7f02bc148b90 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 rx=0x7f02b0002f70 tx=0x7f02b0015000).stop 2022-07-12T08:50:11.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 shutdown_connections 2022-07-12T08:50:11.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f029c053ad0 0x7f029c055f90 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0b52e0 0x7f02bc0adaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f02bc0b4050 0x7f02bc151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 --2- 172.21.15.61:0/2156530355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02bc0b2b90 0x7f02bc148b90 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 >> 172.21.15.61:0/2156530355 conn(0x7f02bc067020 msgr2=0x7f02bc0b8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:11.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 shutdown_connections 2022-07-12T08:50:11.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.740+0000 7f02c2b56700 1 -- 172.21.15.61:0/2156530355 wait complete. 2022-07-12T08:50:11.760 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:50:11.762 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:50:11.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 Processor -- start 2022-07-12T08:50:11.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 -- start start 2022-07-12T08:50:11.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 0x7f0a4c0618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c138f40 0x7f0a4c140370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4c142b90 con 0x7f0a4c0b02f0 2022-07-12T08:50:11.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4c060ef0 con 0x7f0a4c1427b0 2022-07-12T08:50:11.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a51485700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c061160 con 0x7f0a4c138f40 2022-07-12T08:50:11.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a4bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 0x7f0a4c0618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a50c84700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c138f40 0x7f0a4c140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a4bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 0x7f0a4c0618d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39554/0 (socket says 172.21.15.61:39554) 2022-07-12T08:50:11.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a4bfff700 1 -- 172.21.15.61:0/2548974644 learned_addr learned my addr 172.21.15.61:0/2548974644 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:11.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a4b7fe700 1 --1- 172.21.15.61:0/2548974644 >> v1:172.21.15.111:6789/0 conn(0x7f0a4c0b02f0 0x7f0a4c138830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34572/0 (socket says 172.21.15.61:34572) 2022-07-12T08:50:11.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a50c84700 1 -- 172.21.15.61:0/2548974644 >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 msgr2=0x7f0a4c0618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:11.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a50c84700 1 --2- 172.21.15.61:0/2548974644 >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 0x7f0a4c0618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a50c84700 1 -- 172.21.15.61:0/2548974644 >> v1:172.21.15.111:6789/0 conn(0x7f0a4c0b02f0 legacy=0x7f0a4c138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a50c84700 1 -- 172.21.15.61:0/2548974644 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4c061400 con 0x7f0a4c138f40 2022-07-12T08:50:11.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a4bfff700 1 --2- 172.21.15.61:0/2548974644 >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 0x7f0a4c0618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:11.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a50c84700 1 --2- 172.21.15.61:0/2548974644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c138f40 0x7f0a4c140370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0a440032f0 tx=0x7f0a440091d0).ready entity=mon.2 client_cookie=e0e3329fc497f96b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.884+0000 7f0a4affd700 1 -- 172.21.15.61:0/2548974644 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a4400b6c0 con 0x7f0a4c138f40 2022-07-12T08:50:11.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a4affd700 1 -- 172.21.15.61:0/2548974644 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4400b820 con 0x7f0a4c138f40 2022-07-12T08:50:11.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a4affd700 1 -- 172.21.15.61:0/2548974644 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a4400b990 con 0x7f0a4c138f40 2022-07-12T08:50:11.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 -- 172.21.15.61:0/2548974644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c138f40 msgr2=0x7f0a4c140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 --2- 172.21.15.61:0/2548974644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c138f40 0x7f0a4c140370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0a440032f0 tx=0x7f0a440091d0).stop 2022-07-12T08:50:11.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 -- 172.21.15.61:0/2548974644 shutdown_connections 2022-07-12T08:50:11.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 --2- 172.21.15.61:0/2548974644 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c138f40 0x7f0a4c140370 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 --2- 172.21.15.61:0/2548974644 >> v2:172.21.15.111:3300/0 conn(0x7f0a4c1427b0 0x7f0a4c0618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 -- 172.21.15.61:0/2548974644 >> 172.21.15.61:0/2548974644 conn(0x7f0a4c0b5860 msgr2=0x7f0a4c0b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:11.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 -- 172.21.15.61:0/2548974644 shutdown_connections 2022-07-12T08:50:11.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 -- 172.21.15.61:0/2548974644 wait complete. 2022-07-12T08:50:11.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 Processor -- start 2022-07-12T08:50:11.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.888+0000 7f0a51485700 1 -- start start 2022-07-12T08:50:11.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c0b02f0 0x7f0a4c0ac570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a4c138f40 0x7f0a4c132970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 0x7f0a4c132e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c133860 con 0x7f0a4c138f40 2022-07-12T08:50:11.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c133ac0 con 0x7f0a4c1427b0 2022-07-12T08:50:11.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4c133350 con 0x7f0a4c0b02f0 2022-07-12T08:50:11.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 0x7f0a4c132e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a4bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c0b02f0 0x7f0a4c0ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a4b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a4c138f40 0x7f0a4c132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 0x7f0a4c132e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39564/0 (socket says 172.21.15.61:39564) 2022-07-12T08:50:11.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 -- 172.21.15.61:0/1364467046 learned_addr learned my addr 172.21.15.61:0/1364467046 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:11.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 -- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c0b02f0 msgr2=0x7f0a4c0ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c0b02f0 0x7f0a4c0ac570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 -- 172.21.15.61:0/1364467046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a4c138f40 msgr2=0x7f0a4c132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:11.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a4c138f40 0x7f0a4c132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:11.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 -- 172.21.15.61:0/1364467046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c00b1e0 con 0x7f0a4c1427b0 2022-07-12T08:50:11.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a50c84700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 0x7f0a4c132e60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f0a440032f0 tx=0x7f0a44016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a44011bd0 con 0x7f0a4c1427b0 2022-07-12T08:50:11.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a44011d30 con 0x7f0a4c1427b0 2022-07-12T08:50:11.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a44009470 con 0x7f0a4c1427b0 2022-07-12T08:50:11.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a44011ea0 con 0x7f0a4c1427b0 2022-07-12T08:50:11.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4c135000 con 0x7f0a4c1427b0 2022-07-12T08:50:11.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.892+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4c134e50 con 0x7f0a4c1427b0 2022-07-12T08:50:11.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.896+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0a4400f070 con 0x7f0a4c1427b0 2022-07-12T08:50:11.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.896+0000 7f0a497fa700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0a30053920 0x7f0a30055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:11.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.896+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0a4406eda0 con 0x7f0a4c1427b0 2022-07-12T08:50:11.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.896+0000 7f0a4bfff700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0a30053920 0x7f0a30055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:11.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.896+0000 7f0a4bfff700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0a30053920 0x7f0a30055de0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f0a3c00b000 tx=0x7f0a3c006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:11.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:11.904+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0a44045bf0 con 0x7f0a4c1427b0 2022-07-12T08:50:12.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.068+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f0a4c142e00 con 0x7f0a4c1427b0 2022-07-12T08:50:12.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.080+0000 7f0a497fa700 1 -- 172.21.15.61:0/1364467046 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v40) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0a44022020 con 0x7f0a4c1427b0 2022-07-12T08:50:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0a30053920 msgr2=0x7f0a30055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0a30053920 0x7f0a30055de0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f0a3c00b000 tx=0x7f0a3c006eb0).stop 2022-07-12T08:50:12.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 msgr2=0x7f0a4c132e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 0x7f0a4c132e60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f0a440032f0 tx=0x7f0a44016000).stop 2022-07-12T08:50:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 shutdown_connections 2022-07-12T08:50:12.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0a30053920 0x7f0a30055de0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0a4c1427b0 0x7f0a4c132e60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a4c138f40 0x7f0a4c132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 --2- 172.21.15.61:0/1364467046 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0a4c0b02f0 0x7f0a4c0ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 >> 172.21.15.61:0/1364467046 conn(0x7f0a4c0b5860 msgr2=0x7f0a4c140f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:12.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 shutdown_connections 2022-07-12T08:50:12.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.084+0000 7f0a51485700 1 -- 172.21.15.61:0/1364467046 wait complete. 2022-07-12T08:50:12.096 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:50:12.108 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:50:12.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf8f71700 1 Processor -- start 2022-07-12T08:50:12.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf8f71700 1 -- start start 2022-07-12T08:50:12.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf8f71700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40677d0 0x7f1cf40b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf8f71700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf40b6fc0 con 0x7f1cf40b6be0 2022-07-12T08:50:12.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf8f71700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cf40a9460 con 0x7f1cf41431e0 2022-07-12T08:50:12.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf8f71700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40b2370 con 0x7f1cf40677d0 2022-07-12T08:50:12.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1cf40b6be0 0x7f1cf40b1770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49212/0 (socket says 172.21.15.61:49212) 2022-07-12T08:50:12.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf259c700 1 -- 172.21.15.61:0/1568943900 learned_addr learned my addr 172.21.15.61:0/1568943900 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:12.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf2d9d700 1 --2- 172.21.15.61:0/1568943900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40677d0 0x7f1cf40b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723863619 0 0) 0x7f1cf40b6fc0 con 0x7f1cf40b6be0 2022-07-12T08:50:12.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd8003260 con 0x7f1cf40b6be0 2022-07-12T08:50:12.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641413209 0 0) 0x7f1cf40a9460 con 0x7f1cf41431e0 2022-07-12T08:50:12.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf40b6fc0 con 0x7f1cf41431e0 2022-07-12T08:50:12.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/1568943900 >> v1:172.21.15.111:6789/0 conn(0x7f1cf41431e0 legacy=0x7f1cf4140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/1568943900 >> v1:172.21.15.61:6789/0 conn(0x7f1cf40b6be0 legacy=0x7f1cf40b1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/1568943900 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40a9700 con 0x7f1cf40677d0 2022-07-12T08:50:12.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941836841 0 0) 0x7f1cd8003260 con 0x7f1cf40b6be0 2022-07-12T08:50:12.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.228+0000 7f1cf2d9d700 1 --2- 172.21.15.61:0/1568943900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40677d0 0x7f1cf40b1e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1cec0032f0 tx=0x7f1cec0091d0).ready entity=mon.2 client_cookie=472811e692bf6fb7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:12.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec00b640 con 0x7f1cf40677d0 2022-07-12T08:50:12.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec00b7a0 con 0x7f1cf40677d0 2022-07-12T08:50:12.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf159a700 1 -- 172.21.15.61:0/1568943900 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec00b910 con 0x7f1cf40677d0 2022-07-12T08:50:12.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 -- 172.21.15.61:0/1568943900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40677d0 msgr2=0x7f1cf40b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/1568943900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40677d0 0x7f1cf40b1e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1cec0032f0 tx=0x7f1cec0091d0).stop 2022-07-12T08:50:12.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 -- 172.21.15.61:0/1568943900 shutdown_connections 2022-07-12T08:50:12.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/1568943900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40677d0 0x7f1cf40b1e80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 -- 172.21.15.61:0/1568943900 >> 172.21.15.61:0/1568943900 conn(0x7f1cf4064560 msgr2=0x7f1cf40b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:12.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 -- 172.21.15.61:0/1568943900 shutdown_connections 2022-07-12T08:50:12.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 -- 172.21.15.61:0/1568943900 wait complete. 2022-07-12T08:50:12.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 Processor -- start 2022-07-12T08:50:12.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.232+0000 7f1cf8f71700 1 -- start start 2022-07-12T08:50:12.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf40677d0 0x7f1cf41402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40b6be0 0x7f1cf4152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 0x7f1cf4146af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40ac530 con 0x7f1cf41431e0 2022-07-12T08:50:12.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4152520 con 0x7f1cf40677d0 2022-07-12T08:50:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf40adab0 con 0x7f1cf40b6be0 2022-07-12T08:50:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 0x7f1cf4146af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 0x7f1cf4146af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32772/0 (socket says 172.21.15.61:32772) 2022-07-12T08:50:12.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/3241366233 learned_addr learned my addr 172.21.15.61:0/3241366233 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:12.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf1d9b700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40b6be0 0x7f1cf4152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf259c700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf40677d0 0x7f1cf41402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40b6be0 msgr2=0x7f1cf4152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40b6be0 0x7f1cf4152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf40677d0 msgr2=0x7f1cf41402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf40677d0 0x7f1cf41402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 -- 172.21.15.61:0/3241366233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec009470 con 0x7f1cf41431e0 2022-07-12T08:50:12.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf2d9d700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 0x7f1cf4146af0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f1cec0032f0 tx=0x7f1cec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:12.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec00b910 con 0x7f1cf41431e0 2022-07-12T08:50:12.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec011e10 con 0x7f1cf41431e0 2022-07-12T08:50:12.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cec00b640 con 0x7f1cf41431e0 2022-07-12T08:50:12.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf40ac290 con 0x7f1cf41431e0 2022-07-12T08:50:12.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf41453a0 con 0x7f1cf41431e0 2022-07-12T08:50:12.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf4057460 con 0x7f1cf41431e0 2022-07-12T08:50:12.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.236+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1cec026030 con 0x7f1cf41431e0 2022-07-12T08:50:12.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.240+0000 7f1ce37fe700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1cd0053920 0x7f1cd0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.240+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1cec06ca30 con 0x7f1cf41431e0 2022-07-12T08:50:12.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.240+0000 7f1cf259c700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1cd0053920 0x7f1cd0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.240+0000 7f1cf259c700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1cd0053920 0x7f1cd0055de0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f1ce400d000 tx=0x7f1ce4001760).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:12.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.244+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1cec043880 con 0x7f1cf41431e0 2022-07-12T08:50:12.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.412+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1cf4065380 con 0x7f1cf41431e0 2022-07-12T08:50:12.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.420+0000 7f1ce37fe700 1 -- 172.21.15.61:0/3241366233 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v41) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1cec0240e0 con 0x7f1cf41431e0 2022-07-12T08:50:12.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1cd0053920 msgr2=0x7f1cd0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1cd0053920 0x7f1cd0055de0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7f1ce400d000 tx=0x7f1ce4001760).stop 2022-07-12T08:50:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 msgr2=0x7f1cf4146af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 0x7f1cf4146af0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7f1cec0032f0 tx=0x7f1cec016000).stop 2022-07-12T08:50:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 shutdown_connections 2022-07-12T08:50:12.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cf41431e0 0x7f1cf4146af0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1cf40b6be0 0x7f1cf4152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1cd0053920 0x7f1cd0055de0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 --2- 172.21.15.61:0/3241366233 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1cf40677d0 0x7f1cf41402a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 >> 172.21.15.61:0/3241366233 conn(0x7f1cf4064560 msgr2=0x7f1cf40b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:12.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 shutdown_connections 2022-07-12T08:50:12.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.424+0000 7f1cf8f71700 1 -- 172.21.15.61:0/3241366233 wait complete. 2022-07-12T08:50:12.436 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:50:12.437 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:12.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 Processor -- start 2022-07-12T08:50:12.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- start start 2022-07-12T08:50:12.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f768005c050 0x7f76800ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f768014a8f0 0x7f76801443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800ad2f0 con 0x7f76800b01f0 2022-07-12T08:50:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76800b0a40 con 0x7f768014a8f0 2022-07-12T08:50:12.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768005c430 con 0x7f768005c050 2022-07-12T08:50:12.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f768014a8f0 0x7f76801443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76861f5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f768005c050 0x7f76800ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f768014a8f0 0x7f76801443c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32780/0 (socket says 172.21.15.61:32780) 2022-07-12T08:50:12.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 -- 172.21.15.61:0/2907346230 learned_addr learned my addr 172.21.15.61:0/2907346230 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:12.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76851f3700 1 -- 172.21.15.61:0/2907346230 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382574290 0 0) 0x7f76800ad2f0 con 0x7f76800b01f0 2022-07-12T08:50:12.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 -- 172.21.15.61:0/2907346230 >> v2:172.21.15.111:3300/0 conn(0x7f768005c050 msgr2=0x7f76800ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:12.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 --2- 172.21.15.61:0/2907346230 >> v2:172.21.15.111:3300/0 conn(0x7f768005c050 0x7f76800ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 -- 172.21.15.61:0/2907346230 >> v1:172.21.15.111:6789/0 conn(0x7f76800b01f0 legacy=0x7f768005cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 -- 172.21.15.61:0/2907346230 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800b2420 con 0x7f768014a8f0 2022-07-12T08:50:12.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76869f6700 1 --2- 172.21.15.61:0/2907346230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768014a8f0 0x7f76801443c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f76780032f0 tx=0x7f7678009290).ready entity=mon.0 client_cookie=694db586345bbefe server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:12.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76851f3700 1 -- 172.21.15.61:0/2907346230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f767800b870 con 0x7f768014a8f0 2022-07-12T08:50:12.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76851f3700 1 -- 172.21.15.61:0/2907346230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767800b9d0 con 0x7f768014a8f0 2022-07-12T08:50:12.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f76851f3700 1 -- 172.21.15.61:0/2907346230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f767800bb40 con 0x7f768014a8f0 2022-07-12T08:50:12.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- 172.21.15.61:0/2907346230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768014a8f0 msgr2=0x7f76801443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 --2- 172.21.15.61:0/2907346230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768014a8f0 0x7f76801443c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 rx=0x7f76780032f0 tx=0x7f7678009290).stop 2022-07-12T08:50:12.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- 172.21.15.61:0/2907346230 shutdown_connections 2022-07-12T08:50:12.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 --2- 172.21.15.61:0/2907346230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f768014a8f0 0x7f76801443c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 --2- 172.21.15.61:0/2907346230 >> v2:172.21.15.111:3300/0 conn(0x7f768005c050 0x7f76800ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.540+0000 7f7688459700 1 -- 172.21.15.61:0/2907346230 >> 172.21.15.61:0/2907346230 conn(0x7f7680062c00 msgr2=0x7f7680062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 -- 172.21.15.61:0/2907346230 shutdown_connections 2022-07-12T08:50:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 -- 172.21.15.61:0/2907346230 wait complete. 2022-07-12T08:50:12.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 Processor -- start 2022-07-12T08:50:12.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 -- start start 2022-07-12T08:50:12.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768005c050 0x7f7680146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 0x7f7680148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768014a8f0 0x7f7680149480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7680143660 con 0x7f76800b01f0 2022-07-12T08:50:12.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76801428f0 con 0x7f768014a8f0 2022-07-12T08:50:12.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f7688459700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7680159ac0 con 0x7f768005c050 2022-07-12T08:50:12.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 0x7f7680148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 0x7f7680148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32784/0 (socket says 172.21.15.61:32784) 2022-07-12T08:50:12.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 -- 172.21.15.61:0/1255954124 learned_addr learned my addr 172.21.15.61:0/1255954124 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:12.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76861f5700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768005c050 0x7f7680146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76869f6700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768014a8f0 0x7f7680149480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 -- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768005c050 msgr2=0x7f7680146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768005c050 0x7f7680146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 -- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768014a8f0 msgr2=0x7f7680149480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768014a8f0 0x7f7680149480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.544+0000 7f76859f4700 1 -- 172.21.15.61:0/1255954124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7678009530 con 0x7f76800b01f0 2022-07-12T08:50:12.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76859f4700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 0x7f7680148b50 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f767c000fe0 tx=0x7f767c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:12.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f767c00ce50 con 0x7f76800b01f0 2022-07-12T08:50:12.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c00f470 con 0x7f76800b01f0 2022-07-12T08:50:12.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f767c00f7d0 con 0x7f76800b01f0 2022-07-12T08:50:12.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7680149040 con 0x7f76800b01f0 2022-07-12T08:50:12.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76801487f0 con 0x7f76800b01f0 2022-07-12T08:50:12.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7680057460 con 0x7f76800b01f0 2022-07-12T08:50:12.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f767c019030 con 0x7f76800b01f0 2022-07-12T08:50:12.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76737fe700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7660053af0 0x7f7660055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:12.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f767c06b1d0 con 0x7f76800b01f0 2022-07-12T08:50:12.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.548+0000 7f76861f5700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7660053af0 0x7f7660055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:12.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.552+0000 7f76861f5700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7660053af0 0x7f7660055fb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f767400b000 tx=0x7f7674006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:12.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.556+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f767c0421e0 con 0x7f76800b01f0 2022-07-12T08:50:12.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.716+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7680143810 con 0x7f76800b01f0 2022-07-12T08:50:12.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.716+0000 7f76737fe700 1 -- 172.21.15.61:0/1255954124 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f767c045800 con 0x7f76800b01f0 2022-07-12T08:50:12.724 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:12.725 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:10.807048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8082":{"gid":8082,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2724832679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2724832679},{"type":"v1","addr":"172.21.15.61:6833","nonce":2724832679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:12.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7660053af0 msgr2=0x7f7660055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7660053af0 0x7f7660055fb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7f767400b000 tx=0x7f7674006df0).stop 2022-07-12T08:50:12.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 msgr2=0x7f7680148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:12.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 0x7f7680148b50 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7f767c000fe0 tx=0x7f767c007b50).stop 2022-07-12T08:50:12.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 shutdown_connections 2022-07-12T08:50:12.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7660053af0 0x7f7660055fb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f768014a8f0 0x7f7680149480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76800b01f0 0x7f7680148b50 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 --2- 172.21.15.61:0/1255954124 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f768005c050 0x7f7680146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 >> 172.21.15.61:0/1255954124 conn(0x7f7680062c00 msgr2=0x7f76800b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 shutdown_connections 2022-07-12T08:50:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:12.720+0000 7f7688459700 1 -- 172.21.15.61:0/1255954124 wait complete. 2022-07-12T08:50:12.753 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-12T08:50:12.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:09.803335+0000', 'modified': '2022-07-12T08:50:10.807048+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8082}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8082': {'gid': 8082, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2724832679', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2724832679}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2724832679}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:12.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:12.971 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:13.090 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8082,"want_state":"up:active","state":"up:active","rank_uptime":3.2728405669999998,"mdsmap_epoch":72,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":7.6490187650000001} 2022-07-12T08:50:13.091 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8082, "mdsmap_epoch": 72, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 3.272840567, "state": "up:active", "uptime": 7.649018765, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:13.092 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:13.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ffb962700 1 Processor -- start 2022-07-12T08:50:13.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ffb962700 1 -- start start 2022-07-12T08:50:13.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ffb962700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ff4144ad0 0x7f2ff40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ffb962700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff40ad2f0 con 0x7f2ff405c050 2022-07-12T08:50:13.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ffb962700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff405c430 con 0x7f2ff40b01f0 2022-07-12T08:50:13.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ffb962700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff40b0a40 con 0x7f2ff4144ad0 2022-07-12T08:50:13.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff96fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2ff405c050 0x7f2ff405cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49242/0 (socket says 172.21.15.61:49242) 2022-07-12T08:50:13.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff96fe700 1 -- 172.21.15.61:0/385235000 learned_addr learned my addr 172.21.15.61:0/385235000 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:13.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff9eff700 1 --2- 172.21.15.61:0/385235000 >> v2:172.21.15.111:3300/0 conn(0x7f2ff4144ad0 0x7f2ff40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff9eff700 1 -- 172.21.15.61:0/385235000 >> v1:172.21.15.111:6789/0 conn(0x7f2ff40b01f0 legacy=0x7f2ff41443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff9eff700 1 -- 172.21.15.61:0/385235000 >> v1:172.21.15.61:6789/0 conn(0x7f2ff405c050 legacy=0x7f2ff405cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2febfff700 1 -- 172.21.15.61:0/385235000 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282202100 0 0) 0x7f2ff40ad2f0 con 0x7f2ff405c050 2022-07-12T08:50:13.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff9eff700 1 -- 172.21.15.61:0/385235000 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff40b2420 con 0x7f2ff4144ad0 2022-07-12T08:50:13.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.196+0000 7f2ff9eff700 1 --2- 172.21.15.61:0/385235000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff4144ad0 0x7f2ff40ac710 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f2fec0032f0 tx=0x7f2fec0091d0).ready entity=mon.1 client_cookie=22fcd4ca7df9c939 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2febfff700 1 -- 172.21.15.61:0/385235000 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fec009ea0 con 0x7f2ff4144ad0 2022-07-12T08:50:13.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2febfff700 1 -- 172.21.15.61:0/385235000 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec00b8a0 con 0x7f2ff4144ad0 2022-07-12T08:50:13.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2febfff700 1 -- 172.21.15.61:0/385235000 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fec00ba10 con 0x7f2ff4144ad0 2022-07-12T08:50:13.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 -- 172.21.15.61:0/385235000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff4144ad0 msgr2=0x7f2ff40ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 --2- 172.21.15.61:0/385235000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff4144ad0 0x7f2ff40ac710 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f2fec0032f0 tx=0x7f2fec0091d0).stop 2022-07-12T08:50:13.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 -- 172.21.15.61:0/385235000 shutdown_connections 2022-07-12T08:50:13.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 --2- 172.21.15.61:0/385235000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff4144ad0 0x7f2ff40ac710 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 -- 172.21.15.61:0/385235000 >> 172.21.15.61:0/385235000 conn(0x7f2ff4062c00 msgr2=0x7f2ff4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:13.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 -- 172.21.15.61:0/385235000 shutdown_connections 2022-07-12T08:50:13.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 -- 172.21.15.61:0/385235000 wait complete. 2022-07-12T08:50:13.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 Processor -- start 2022-07-12T08:50:13.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.200+0000 7f2ffb962700 1 -- start start 2022-07-12T08:50:13.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 0x7f2ff4063d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff40b01f0 0x7f2ff41457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff4144ad0 0x7f2ff4156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff414b770 con 0x7f2ff405c050 2022-07-12T08:50:13.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4145080 con 0x7f2ff40b01f0 2022-07-12T08:50:13.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4145320 con 0x7f2ff4144ad0 2022-07-12T08:50:13.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 0x7f2ff4063d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 0x7f2ff4063d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32796/0 (socket says 172.21.15.61:32796) 2022-07-12T08:50:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 -- 172.21.15.61:0/1812556525 learned_addr learned my addr 172.21.15.61:0/1812556525 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff8efd700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff40b01f0 0x7f2ff41457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff9eff700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff4144ad0 0x7f2ff4156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 -- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff4144ad0 msgr2=0x7f2ff4156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff4144ad0 0x7f2ff4156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 -- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff40b01f0 msgr2=0x7f2ff41457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff40b01f0 0x7f2ff41457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 -- 172.21.15.61:0/1812556525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec009470 con 0x7f2ff405c050 2022-07-12T08:50:13.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ff96fe700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 0x7f2ff4063d10 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f2fe4000bc0 tx=0x7f2fe4007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe40070d0 con 0x7f2ff405c050 2022-07-12T08:50:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe4007230 con 0x7f2ff405c050 2022-07-12T08:50:13.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fe4005610 con 0x7f2ff405c050 2022-07-12T08:50:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff4064200 con 0x7f2ff405c050 2022-07-12T08:50:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff4146e30 con 0x7f2ff405c050 2022-07-12T08:50:13.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.204+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff41454e0 con 0x7f2ff405c050 2022-07-12T08:50:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.208+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2fe4010070 con 0x7f2ff405c050 2022-07-12T08:50:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.208+0000 7f2fea7fc700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fdc053ad0 0x7f2fdc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.208+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2fe406b6f0 con 0x7f2ff405c050 2022-07-12T08:50:13.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.208+0000 7f2ff8efd700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fdc053ad0 0x7f2fdc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.208+0000 7f2ff8efd700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fdc053ad0 0x7f2fdc055f90 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f2ff000d000 tx=0x7f2ff0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.212+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2fe40424c0 con 0x7f2ff405c050 2022-07-12T08:50:13.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.340+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ff4057460 con 0x7f2ff405c050 2022-07-12T08:50:13.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.340+0000 7f2fea7fc700 1 -- 172.21.15.61:0/1812556525 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f2fe4045ae0 con 0x7f2ff405c050 2022-07-12T08:50:13.344 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:13.345 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":128,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:10.806045+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:08.408495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:08.818782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:13.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fdc053ad0 msgr2=0x7f2fdc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fdc053ad0 0x7f2fdc055f90 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f2ff000d000 tx=0x7f2ff0007b90).stop 2022-07-12T08:50:13.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 msgr2=0x7f2ff4063d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 0x7f2ff4063d10 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 rx=0x7f2fe4000bc0 tx=0x7f2fe4007af0).stop 2022-07-12T08:50:13.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 shutdown_connections 2022-07-12T08:50:13.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff4144ad0 0x7f2ff4156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fdc053ad0 0x7f2fdc055f90 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff40b01f0 0x7f2ff41457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 --2- 172.21.15.61:0/1812556525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405c050 0x7f2ff4063d10 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 >> 172.21.15.61:0/1812556525 conn(0x7f2ff4062c00 msgr2=0x7f2ff40b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 shutdown_connections 2022-07-12T08:50:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.344+0000 7f2ffb962700 1 -- 172.21.15.61:0/1812556525 wait complete. 2022-07-12T08:50:13.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:50:08.408495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:50:08.818782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:50:13.358 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:13.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.472+0000 7f3325c41700 1 Processor -- start 2022-07-12T08:50:13.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.472+0000 7f3325c41700 1 -- start start 2022-07-12T08:50:13.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f33200b1a00 0x7f33200b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3320060520 0x7f332013f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f332013fd50 0x7f3320142620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332013f160 con 0x7f332013fd50 2022-07-12T08:50:13.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332013f3c0 con 0x7f3320060520 2022-07-12T08:50:13.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33200b0410 con 0x7f33200b1a00 2022-07-12T08:50:13.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f332013fd50 0x7f3320142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f332013fd50 0x7f3320142620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32808/0 (socket says 172.21.15.61:32808) 2022-07-12T08:50:13.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 -- 172.21.15.61:0/1457064413 learned_addr learned my addr 172.21.15.61:0/1457064413 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:13.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331f7fe700 1 --2- 172.21.15.61:0/1457064413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f33200b1a00 0x7f33200b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331effd700 1 --2- 172.21.15.61:0/1457064413 >> v2:172.21.15.111:3300/0 conn(0x7f3320060520 0x7f332013f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 -- 172.21.15.61:0/1457064413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f33200b1a00 msgr2=0x7f33200b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 --2- 172.21.15.61:0/1457064413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f33200b1a00 0x7f33200b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 -- 172.21.15.61:0/1457064413 >> v2:172.21.15.111:3300/0 conn(0x7f3320060520 msgr2=0x7f332013f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:13.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 --2- 172.21.15.61:0/1457064413 >> v2:172.21.15.111:3300/0 conn(0x7f3320060520 0x7f332013f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 -- 172.21.15.61:0/1457064413 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200b0a50 con 0x7f332013fd50 2022-07-12T08:50:13.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331ffff700 1 --2- 172.21.15.61:0/1457064413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332013fd50 0x7f3320142620 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7f33180032f0 tx=0x7f3318009290).ready entity=mon.0 client_cookie=f0de62ac6cca9492 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331e7fc700 1 -- 172.21.15.61:0/1457064413 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f331800da80 con 0x7f332013fd50 2022-07-12T08:50:13.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331e7fc700 1 -- 172.21.15.61:0/1457064413 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331800dbe0 con 0x7f332013fd50 2022-07-12T08:50:13.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f331e7fc700 1 -- 172.21.15.61:0/1457064413 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3318010470 con 0x7f332013fd50 2022-07-12T08:50:13.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.476+0000 7f3325c41700 1 -- 172.21.15.61:0/1457064413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332013fd50 msgr2=0x7f3320142620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.480+0000 7f3325c41700 1 --2- 172.21.15.61:0/1457064413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332013fd50 0x7f3320142620 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7f33180032f0 tx=0x7f3318009290).stop 2022-07-12T08:50:13.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.480+0000 7f3325c41700 1 -- 172.21.15.61:0/1457064413 shutdown_connections 2022-07-12T08:50:13.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.480+0000 7f3325c41700 1 --2- 172.21.15.61:0/1457064413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f332013fd50 0x7f3320142620 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.480+0000 7f3325c41700 1 --2- 172.21.15.61:0/1457064413 >> v2:172.21.15.111:3300/0 conn(0x7f3320060520 0x7f332013f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.480+0000 7f3325c41700 1 --2- 172.21.15.61:0/1457064413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f33200b1a00 0x7f33200b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.480+0000 7f3325c41700 1 -- 172.21.15.61:0/1457064413 >> 172.21.15.61:0/1457064413 conn(0x7f33200b23f0 msgr2=0x7f33200b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:13.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 -- 172.21.15.61:0/1457064413 shutdown_connections 2022-07-12T08:50:13.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 -- 172.21.15.61:0/1457064413 wait complete. 2022-07-12T08:50:13.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 Processor -- start 2022-07-12T08:50:13.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 -- start start 2022-07-12T08:50:13.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3320060520 0x7f33200af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 0x7f33201414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332013fd50 0x7f3320093110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332008fe40 con 0x7f33200b1a00 2022-07-12T08:50:13.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3320090aa0 con 0x7f332013fd50 2022-07-12T08:50:13.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f3325c41700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3320092c70 con 0x7f3320060520 2022-07-12T08:50:13.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 0x7f33201414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 0x7f33201414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32812/0 (socket says 172.21.15.61:32812) 2022-07-12T08:50:13.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 -- 172.21.15.61:0/4252145421 learned_addr learned my addr 172.21.15.61:0/4252145421 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:13.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331f7fe700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3320060520 0x7f33200af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331ffff700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332013fd50 0x7f3320093110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 -- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3320060520 msgr2=0x7f33200af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3320060520 0x7f33200af2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 -- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332013fd50 msgr2=0x7f3320093110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332013fd50 0x7f3320093110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 -- 172.21.15.61:0/4252145421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318009530 con 0x7f33200b1a00 2022-07-12T08:50:13.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.484+0000 7f331effd700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 0x7f33201414b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f330800c000 tx=0x7f3308006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3308010780 con 0x7f33200b1a00 2022-07-12T08:50:13.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33080108e0 con 0x7f33200b1a00 2022-07-12T08:50:13.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33201419a0 con 0x7f33200b1a00 2022-07-12T08:50:13.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3308010c40 con 0x7f33200b1a00 2022-07-12T08:50:13.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332008f830 con 0x7f33200b1a00 2022-07-12T08:50:13.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3320093940 con 0x7f33200b1a00 2022-07-12T08:50:13.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f330801c460 con 0x7f33200b1a00 2022-07-12T08:50:13.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f331cff9700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3304053ab0 0x7f3304055f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.488+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f330806c340 con 0x7f33200b1a00 2022-07-12T08:50:13.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.492+0000 7f331f7fe700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3304053ab0 0x7f3304055f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.492+0000 7f331f7fe700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3304053ab0 0x7f3304055f70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f3310009000 tx=0x7f3310006dc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.496+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3320093940 con 0x7f33200b1a00 2022-07-12T08:50:13.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.620+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3320093940 con 0x7f33200b1a00 2022-07-12T08:50:13.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f331cff9700 1 -- 172.21.15.61:0/4252145421 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v128) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3320093940 con 0x7f33200b1a00 2022-07-12T08:50:13.628 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:13.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3304053ab0 msgr2=0x7f3304055f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3304053ab0 0x7f3304055f70 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f3310009000 tx=0x7f3310006dc0).stop 2022-07-12T08:50:13.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 msgr2=0x7f33201414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 0x7f33201414b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7f330800c000 tx=0x7f3308006df0).stop 2022-07-12T08:50:13.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 shutdown_connections 2022-07-12T08:50:13.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3304053ab0 0x7f3304055f70 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f332013fd50 0x7f3320093110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33200b1a00 0x7f33201414b0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 --2- 172.21.15.61:0/4252145421 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3320060520 0x7f33200af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 >> 172.21.15.61:0/4252145421 conn(0x7f33200b23f0 msgr2=0x7f3320066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:13.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 shutdown_connections 2022-07-12T08:50:13.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.624+0000 7f3325c41700 1 -- 172.21.15.61:0/4252145421 wait complete. 2022-07-12T08:50:13.645 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:13.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 Processor -- start 2022-07-12T08:50:13.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- start start 2022-07-12T08:50:13.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fab4c0b0280 0x7fab4c14b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fab4c0aa830 0x7fab4c0aac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c05c3e0 con 0x7fab4c0b5920 2022-07-12T08:50:13.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4c0b5d00 con 0x7fab4c0b0280 2022-07-12T08:50:13.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4c05c610 con 0x7fab4c0aa830 2022-07-12T08:50:13.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51620700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fab4c0b0280 0x7fab4c14b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51e21700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fab4c0b5920 0x7fab4c14a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49268/0 (socket says 172.21.15.61:49268) 2022-07-12T08:50:13.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51e21700 1 -- 172.21.15.61:0/2796647256 learned_addr learned my addr 172.21.15.61:0/2796647256 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:13.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52622700 1 --2- 172.21.15.61:0/2796647256 >> v2:172.21.15.111:3300/0 conn(0x7fab4c0aa830 0x7fab4c0aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51620700 1 -- 172.21.15.61:0/2796647256 >> v2:172.21.15.111:3300/0 conn(0x7fab4c0aa830 msgr2=0x7fab4c0aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:13.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51620700 1 --2- 172.21.15.61:0/2796647256 >> v2:172.21.15.111:3300/0 conn(0x7fab4c0aa830 0x7fab4c0aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51620700 1 -- 172.21.15.61:0/2796647256 >> v1:172.21.15.61:6789/0 conn(0x7fab4c0b5920 legacy=0x7fab4c14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51620700 1 -- 172.21.15.61:0/2796647256 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c05c8b0 con 0x7fab4c0b0280 2022-07-12T08:50:13.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab51620700 1 --2- 172.21.15.61:0/2796647256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0b0280 0x7fab4c14b0f0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7fab48002f70 tx=0x7fab48006df0).ready entity=mon.0 client_cookie=1089fca7e42d532e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab50e1f700 1 -- 172.21.15.61:0/2796647256 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab4800d5c0 con 0x7fab4c0b0280 2022-07-12T08:50:13.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab50e1f700 1 -- 172.21.15.61:0/2796647256 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48000e90 con 0x7fab4c0b0280 2022-07-12T08:50:13.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab50e1f700 1 -- 172.21.15.61:0/2796647256 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab4800d910 con 0x7fab4c0b0280 2022-07-12T08:50:13.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- 172.21.15.61:0/2796647256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0b0280 msgr2=0x7fab4c14b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 --2- 172.21.15.61:0/2796647256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0b0280 0x7fab4c14b0f0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 rx=0x7fab48002f70 tx=0x7fab48006df0).stop 2022-07-12T08:50:13.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- 172.21.15.61:0/2796647256 shutdown_connections 2022-07-12T08:50:13.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 --2- 172.21.15.61:0/2796647256 >> v2:172.21.15.111:3300/0 conn(0x7fab4c0aa830 0x7fab4c0aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 --2- 172.21.15.61:0/2796647256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0b0280 0x7fab4c14b0f0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- 172.21.15.61:0/2796647256 >> 172.21.15.61:0/2796647256 conn(0x7fab4c063340 msgr2=0x7fab4c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:13.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- 172.21.15.61:0/2796647256 shutdown_connections 2022-07-12T08:50:13.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.784+0000 7fab52e23700 1 -- 172.21.15.61:0/2796647256 wait complete. 2022-07-12T08:50:13.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 Processor -- start 2022-07-12T08:50:13.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 -- start start 2022-07-12T08:50:13.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 0x7fab4c064180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fab4c0b0280 0x7fab4c098b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab4c0b5920 0x7fab4c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c0ae990 con 0x7fab4c0aa830 2022-07-12T08:50:13.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c1596c0 con 0x7fab4c0b5920 2022-07-12T08:50:13.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c1598f0 con 0x7fab4c0b0280 2022-07-12T08:50:13.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 0x7fab4c064180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 0x7fab4c064180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32824/0 (socket says 172.21.15.61:32824) 2022-07-12T08:50:13.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 -- 172.21.15.61:0/3487678870 learned_addr learned my addr 172.21.15.61:0/3487678870 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:13.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52622700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab4c0b5920 0x7fab4c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51620700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fab4c0b0280 0x7fab4c098b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 -- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fab4c0b0280 msgr2=0x7fab4c098b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fab4c0b0280 0x7fab4c098b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 -- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab4c0b5920 msgr2=0x7fab4c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab4c0b5920 0x7fab4c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 -- 172.21.15.61:0/3487678870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4800b180 con 0x7fab4c0aa830 2022-07-12T08:50:13.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab51e21700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 0x7fab4c064180 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7fab4c0b5f30 tx=0x7fab3c007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3c0070d0 con 0x7fab4c0aa830 2022-07-12T08:50:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c007230 con 0x7fab4c0aa830 2022-07-12T08:50:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c159b90 con 0x7fab4c0aa830 2022-07-12T08:50:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3c005610 con 0x7fab4c0aa830 2022-07-12T08:50:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.788+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c14a1a0 con 0x7fab4c0aa830 2022-07-12T08:50:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.792+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c057460 con 0x7fab4c0aa830 2022-07-12T08:50:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.792+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fab3c010070 con 0x7fab4c0aa830 2022-07-12T08:50:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.792+0000 7fab3affd700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fab2c053a40 0x7fab2c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.792+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fab3c06b430 con 0x7fab4c0aa830 2022-07-12T08:50:13.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.792+0000 7fab51620700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fab2c053a40 0x7fab2c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:13.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.792+0000 7fab51620700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fab2c053a40 0x7fab2c055f00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fab4801a000 tx=0x7fab48014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:13.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.796+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab3c0422f0 con 0x7fab4c0aa830 2022-07-12T08:50:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.920+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fab4c06b2b0 con 0x7fab4c0aa830 2022-07-12T08:50:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab3affd700 1 -- 172.21.15.61:0/3487678870 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7fab3c018030 con 0x7fab4c0aa830 2022-07-12T08:50:13.929 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:13.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fab2c053a40 msgr2=0x7fab2c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fab2c053a40 0x7fab2c055f00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7fab4801a000 tx=0x7fab48014000).stop 2022-07-12T08:50:13.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 msgr2=0x7fab4c064180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:13.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 0x7fab4c064180 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7fab4c0b5f30 tx=0x7fab3c007ca0).stop 2022-07-12T08:50:13.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 shutdown_connections 2022-07-12T08:50:13.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fab2c053a40 0x7fab2c055f00 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fab4c0b5920 0x7fab4c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fab4c0b0280 0x7fab4c098b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 --2- 172.21.15.61:0/3487678870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab4c0aa830 0x7fab4c064180 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 >> 172.21.15.61:0/3487678870 conn(0x7fab4c063340 msgr2=0x7fab4c14d850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 shutdown_connections 2022-07-12T08:50:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:13.924+0000 7fab52e23700 1 -- 172.21.15.61:0/3487678870 wait complete. 2022-07-12T08:50:13.947 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:50:14.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 Processor -- start 2022-07-12T08:50:14.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 -- start start 2022-07-12T08:50:14.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b4142eb0 0x7f98b405c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98b405d130 0x7f98b40b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f98b40b5f10 0x7f98b4146090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b4143500 con 0x7f98b40b5f10 2022-07-12T08:50:14.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b4143290 con 0x7f98b405d130 2022-07-12T08:50:14.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98ba763700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4143760 con 0x7f98b4142eb0 2022-07-12T08:50:14.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.060+0000 7f98b9f62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f98b40b5f10 0x7f98b4146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f98b40b5f10 0x7f98b4146090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32836/0 (socket says 172.21.15.61:32836) 2022-07-12T08:50:14.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 -- 172.21.15.61:0/2416264279 learned_addr learned my addr 172.21.15.61:0/2416264279 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:14.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9761700 1 --2- 172.21.15.61:0/2416264279 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b4142eb0 0x7f98b405c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b8f60700 1 --2- 172.21.15.61:0/2416264279 >> v2:172.21.15.111:3300/0 conn(0x7f98b405d130 0x7f98b40b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 -- 172.21.15.61:0/2416264279 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b4142eb0 msgr2=0x7f98b405c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:14.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 --2- 172.21.15.61:0/2416264279 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b4142eb0 0x7f98b405c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 -- 172.21.15.61:0/2416264279 >> v2:172.21.15.111:3300/0 conn(0x7f98b405d130 msgr2=0x7f98b40b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:14.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 --2- 172.21.15.61:0/2416264279 >> v2:172.21.15.111:3300/0 conn(0x7f98b405d130 0x7f98b40b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 -- 172.21.15.61:0/2416264279 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40b4030 con 0x7f98b40b5f10 2022-07-12T08:50:14.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98b9f62700 1 --2- 172.21.15.61:0/2416264279 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40b5f10 0x7f98b4146090 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f98ac0032f0 tx=0x7f98ac009290).ready entity=mon.0 client_cookie=5bf8750939b7162f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98a7fff700 1 -- 172.21.15.61:0/2416264279 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98ac00b680 con 0x7f98b40b5f10 2022-07-12T08:50:14.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98a7fff700 1 -- 172.21.15.61:0/2416264279 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac00b7e0 con 0x7f98b40b5f10 2022-07-12T08:50:14.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98a7fff700 1 -- 172.21.15.61:0/2416264279 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98ac00b950 con 0x7f98b40b5f10 2022-07-12T08:50:14.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 -- 172.21.15.61:0/2416264279 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40b5f10 msgr2=0x7f98b4146090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 --2- 172.21.15.61:0/2416264279 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40b5f10 0x7f98b4146090 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7f98ac0032f0 tx=0x7f98ac009290).stop 2022-07-12T08:50:14.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 -- 172.21.15.61:0/2416264279 shutdown_connections 2022-07-12T08:50:14.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 --2- 172.21.15.61:0/2416264279 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b40b5f10 0x7f98b4146090 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 --2- 172.21.15.61:0/2416264279 >> v2:172.21.15.111:3300/0 conn(0x7f98b405d130 0x7f98b40b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 --2- 172.21.15.61:0/2416264279 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b4142eb0 0x7f98b405c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 -- 172.21.15.61:0/2416264279 >> 172.21.15.61:0/2416264279 conn(0x7f98b4066d70 msgr2=0x7f98b4067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:14.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 -- 172.21.15.61:0/2416264279 shutdown_connections 2022-07-12T08:50:14.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 -- 172.21.15.61:0/2416264279 wait complete. 2022-07-12T08:50:14.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.064+0000 7f98ba763700 1 Processor -- start 2022-07-12T08:50:14.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- start start 2022-07-12T08:50:14.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b405d130 0x7f98b413e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b5f10 0x7f98b413d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 0x7f98b413d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b413da00 con 0x7f98b4142eb0 2022-07-12T08:50:14.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4144030 con 0x7f98b405d130 2022-07-12T08:50:14.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4144290 con 0x7f98b40b5f10 2022-07-12T08:50:14.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 0x7f98b413d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 0x7f98b413d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32842/0 (socket says 172.21.15.61:32842) 2022-07-12T08:50:14.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 -- 172.21.15.61:0/2838534360 learned_addr learned my addr 172.21.15.61:0/2838534360 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:14.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9761700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b405d130 0x7f98b413e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b8f60700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b5f10 0x7f98b413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 -- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b5f10 msgr2=0x7f98b413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b5f10 0x7f98b413d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 -- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b405d130 msgr2=0x7f98b413e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b405d130 0x7f98b413e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 -- 172.21.15.61:0/2838534360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009530 con 0x7f98b4142eb0 2022-07-12T08:50:14.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98b9f62700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 0x7f98b413d510 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7f98ac0032f0 tx=0x7f98ac017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98ac020070 con 0x7f98b4142eb0 2022-07-12T08:50:14.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac011e80 con 0x7f98b4142eb0 2022-07-12T08:50:14.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b40ada10 con 0x7f98b4142eb0 2022-07-12T08:50:14.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98ac00b680 con 0x7f98b4142eb0 2022-07-12T08:50:14.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b40ad520 con 0x7f98b4142eb0 2022-07-12T08:50:14.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b4057460 con 0x7f98b4142eb0 2022-07-12T08:50:14.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.068+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f98ac00f070 con 0x7f98b4142eb0 2022-07-12T08:50:14.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.072+0000 7f98a67fc700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f989c0539b0 0x7f989c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.072+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f98ac06d480 con 0x7f98b4142eb0 2022-07-12T08:50:14.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.072+0000 7f98b9761700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f989c0539b0 0x7f989c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.072+0000 7f98b9761700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f989c0539b0 0x7f989c055e70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f98b000b000 tx=0x7f98b0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.080+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f98ac0443e0 con 0x7f98b4142eb0 2022-07-12T08:50:14.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.204+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98b4144460 con 0x7f98b4142eb0 2022-07-12T08:50:14.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.204+0000 7f98a67fc700 1 -- 172.21.15.61:0/2838534360 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7f98ac047a00 con 0x7f98b4142eb0 2022-07-12T08:50:14.209 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:14.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f989c0539b0 msgr2=0x7f989c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f989c0539b0 0x7f989c055e70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f98b000b000 tx=0x7f98b0008040).stop 2022-07-12T08:50:14.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 msgr2=0x7f98b413d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 0x7f98b413d510 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7f98ac0032f0 tx=0x7f98ac017000).stop 2022-07-12T08:50:14.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 shutdown_connections 2022-07-12T08:50:14.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98b4142eb0 0x7f98b413d510 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98b40b5f10 0x7f98b413d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f989c0539b0 0x7f989c055e70 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 --2- 172.21.15.61:0/2838534360 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98b405d130 0x7f98b413e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 >> 172.21.15.61:0/2838534360 conn(0x7f98b4066d70 msgr2=0x7f98b405c460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:14.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 shutdown_connections 2022-07-12T08:50:14.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.208+0000 7f98ba763700 1 -- 172.21.15.61:0/2838534360 wait complete. 2022-07-12T08:50:14.233 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:50:14.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.368+0000 7ff47ceba700 1 Processor -- start 2022-07-12T08:50:14.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.368+0000 7ff47ceba700 1 -- start start 2022-07-12T08:50:14.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff4780b5480 0x7ff4780b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4780b05f0 0x7ff478142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4780b3560 con 0x7ff4780b3730 2022-07-12T08:50:14.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4780b5b80 con 0x7ff4780b05f0 2022-07-12T08:50:14.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4780b5e20 con 0x7ff4780b5480 2022-07-12T08:50:14.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4780b05f0 0x7ff478142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff4780b05f0 0x7ff478142f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32850/0 (socket says 172.21.15.61:32850) 2022-07-12T08:50:14.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 -- 172.21.15.61:0/2976611235 learned_addr learned my addr 172.21.15.61:0/2976611235 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:14.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47659c700 1 --2- 172.21.15.61:0/2976611235 >> v2:172.21.15.111:3300/0 conn(0x7ff4780b5480 0x7ff4780b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 -- 172.21.15.61:0/2976611235 >> v2:172.21.15.111:3300/0 conn(0x7ff4780b5480 msgr2=0x7ff4780b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:14.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 --2- 172.21.15.61:0/2976611235 >> v2:172.21.15.111:3300/0 conn(0x7ff4780b5480 0x7ff4780b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 -- 172.21.15.61:0/2976611235 >> v1:172.21.15.111:6789/0 conn(0x7ff4780b3730 legacy=0x7ff4780afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:14.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 -- 172.21.15.61:0/2976611235 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4780b5860 con 0x7ff4780b05f0 2022-07-12T08:50:14.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff476d9d700 1 --2- 172.21.15.61:0/2976611235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b05f0 0x7ff478142f90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7ff478066150 tx=0x7ff470009290).ready entity=mon.0 client_cookie=89f6a5cf182733e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47559a700 1 -- 172.21.15.61:0/2976611235 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff47000b830 con 0x7ff4780b05f0 2022-07-12T08:50:14.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47559a700 1 -- 172.21.15.61:0/2976611235 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47000b990 con 0x7ff4780b05f0 2022-07-12T08:50:14.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47559a700 1 -- 172.21.15.61:0/2976611235 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff47000bb40 con 0x7ff4780b05f0 2022-07-12T08:50:14.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- 172.21.15.61:0/2976611235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b05f0 msgr2=0x7ff478142f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 --2- 172.21.15.61:0/2976611235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b05f0 0x7ff478142f90 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 rx=0x7ff478066150 tx=0x7ff470009290).stop 2022-07-12T08:50:14.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- 172.21.15.61:0/2976611235 shutdown_connections 2022-07-12T08:50:14.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 --2- 172.21.15.61:0/2976611235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b05f0 0x7ff478142f90 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 --2- 172.21.15.61:0/2976611235 >> v2:172.21.15.111:3300/0 conn(0x7ff4780b5480 0x7ff4780b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- 172.21.15.61:0/2976611235 >> 172.21.15.61:0/2976611235 conn(0x7ff4780b96b0 msgr2=0x7ff4780b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:14.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- 172.21.15.61:0/2976611235 shutdown_connections 2022-07-12T08:50:14.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.372+0000 7ff47ceba700 1 -- 172.21.15.61:0/2976611235 wait complete. 2022-07-12T08:50:14.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 Processor -- start 2022-07-12T08:50:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 -- start start 2022-07-12T08:50:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4780b05f0 0x7ff47814a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 0x7ff4781149e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4780b5480 0x7ff478114ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780aee50 con 0x7ff4780b3730 2022-07-12T08:50:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780ae910 con 0x7ff4780b05f0 2022-07-12T08:50:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780aebb0 con 0x7ff4780b5480 2022-07-12T08:50:14.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 0x7ff4781149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 0x7ff4781149e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32854/0 (socket says 172.21.15.61:32854) 2022-07-12T08:50:14.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 -- 172.21.15.61:0/1632775037 learned_addr learned my addr 172.21.15.61:0/1632775037 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:14.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47659c700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4780b05f0 0x7ff47814a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff476d9d700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4780b5480 0x7ff478114ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 -- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4780b5480 msgr2=0x7ff478114ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4780b5480 0x7ff478114ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 -- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4780b05f0 msgr2=0x7ff47814a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4780b05f0 0x7ff47814a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 -- 172.21.15.61:0/1632775037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470009530 con 0x7ff4780b3730 2022-07-12T08:50:14.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff475d9b700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 0x7ff4781149e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7ff468001fa0 tx=0x7ff468007c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff468007030 con 0x7ff4780b3730 2022-07-12T08:50:14.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff468007190 con 0x7ff4780b3730 2022-07-12T08:50:14.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4780b0190 con 0x7ff4780b3730 2022-07-12T08:50:14.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff468006510 con 0x7ff4780b3730 2022-07-12T08:50:14.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.376+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4780af0a0 con 0x7ff4780b3730 2022-07-12T08:50:14.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.380+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff468010030 con 0x7ff4780b3730 2022-07-12T08:50:14.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.380+0000 7ff4677fe700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff454053ad0 0x7ff454055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.380+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff46806ba60 con 0x7ff4780b3730 2022-07-12T08:50:14.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.380+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff478114780 con 0x7ff4780b3730 2022-07-12T08:50:14.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.380+0000 7ff47659c700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff454053ad0 0x7ff454055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.380+0000 7ff47659c700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff454053ad0 0x7ff454055f90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7ff46c00c000 tx=0x7ff46c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.384+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4680429c0 con 0x7ff4780b3730 2022-07-12T08:50:14.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff478057460 con 0x7ff4780b3730 2022-07-12T08:50:14.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff4677fe700 1 -- 172.21.15.61:0/1632775037 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff468045fe0 con 0x7ff4780b3730 2022-07-12T08:50:14.522 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:14.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff454053ad0 msgr2=0x7ff454055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff454053ad0 0x7ff454055f90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7ff46c00c000 tx=0x7ff46c008040).stop 2022-07-12T08:50:14.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 msgr2=0x7ff4781149e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 0x7ff4781149e0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7ff468001fa0 tx=0x7ff468007c30).stop 2022-07-12T08:50:14.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 shutdown_connections 2022-07-12T08:50:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4780b5480 0x7ff478114ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4780b3730 0x7ff4781149e0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff454053ad0 0x7ff454055f90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 --2- 172.21.15.61:0/1632775037 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4780b05f0 0x7ff47814a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 >> 172.21.15.61:0/1632775037 conn(0x7ff4780b96b0 msgr2=0x7ff47805c950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 shutdown_connections 2022-07-12T08:50:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.516+0000 7ff47ceba700 1 -- 172.21.15.61:0/1632775037 wait complete. 2022-07-12T08:50:14.543 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:50:14.545 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:14.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 Processor -- start 2022-07-12T08:50:14.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 -- start start 2022-07-12T08:50:14.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc4240ab670 0x7fc42405f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4240afa60 0x7fc4240afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4240abc70 con 0x7fc4240ab290 2022-07-12T08:50:14.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4240aba90 con 0x7fc4240afa60 2022-07-12T08:50:14.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc429284700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42405c720 con 0x7fc4240ab670 2022-07-12T08:50:14.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc423fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc4240ab290 0x7fc42405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49310/0 (socket says 172.21.15.61:49310) 2022-07-12T08:50:14.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.648+0000 7fc428a83700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4240afa60 0x7fc4240afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc423fff700 1 -- 172.21.15.61:0/2535971204 learned_addr learned my addr 172.21.15.61:0/2535971204 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:14.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc4237fe700 1 --2- 172.21.15.61:0/2535971204 >> v2:172.21.15.111:3300/0 conn(0x7fc4240ab670 0x7fc42405f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc422ffd700 1 -- 172.21.15.61:0/2535971204 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121384854 0 0) 0x7fc4240abc70 con 0x7fc4240ab290 2022-07-12T08:50:14.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc422ffd700 1 -- 172.21.15.61:0/2535971204 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc410003260 con 0x7fc4240ab290 2022-07-12T08:50:14.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc4237fe700 1 -- 172.21.15.61:0/2535971204 >> v2:172.21.15.61:3300/0 conn(0x7fc4240afa60 msgr2=0x7fc4240afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:14.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc4237fe700 1 --2- 172.21.15.61:0/2535971204 >> v2:172.21.15.61:3300/0 conn(0x7fc4240afa60 0x7fc4240afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc4237fe700 1 -- 172.21.15.61:0/2535971204 >> v1:172.21.15.61:6789/0 conn(0x7fc4240ab290 legacy=0x7fc42405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc4237fe700 1 -- 172.21.15.61:0/2535971204 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42405c9c0 con 0x7fc4240ab670 2022-07-12T08:50:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc4237fe700 1 --2- 172.21.15.61:0/2535971204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc42405f140 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fc418002f70 tx=0x7fc418006df0).ready entity=mon.1 client_cookie=b7fe2590bcb2d8d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc422ffd700 1 -- 172.21.15.61:0/2535971204 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc41800d4f0 con 0x7fc4240ab670 2022-07-12T08:50:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc422ffd700 1 -- 172.21.15.61:0/2535971204 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc418000e90 con 0x7fc4240ab670 2022-07-12T08:50:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc422ffd700 1 -- 172.21.15.61:0/2535971204 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc41800d840 con 0x7fc4240ab670 2022-07-12T08:50:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 -- 172.21.15.61:0/2535971204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 msgr2=0x7fc42405f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 --2- 172.21.15.61:0/2535971204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc42405f140 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fc418002f70 tx=0x7fc418006df0).stop 2022-07-12T08:50:14.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 -- 172.21.15.61:0/2535971204 shutdown_connections 2022-07-12T08:50:14.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 --2- 172.21.15.61:0/2535971204 >> v2:172.21.15.61:3300/0 conn(0x7fc4240afa60 0x7fc4240afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 --2- 172.21.15.61:0/2535971204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc42405f140 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 -- 172.21.15.61:0/2535971204 >> 172.21.15.61:0/2535971204 conn(0x7fc4240b1950 msgr2=0x7fc4240b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:14.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 -- 172.21.15.61:0/2535971204 shutdown_connections 2022-07-12T08:50:14.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 -- 172.21.15.61:0/2535971204 wait complete. 2022-07-12T08:50:14.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 Processor -- start 2022-07-12T08:50:14.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.652+0000 7fc429284700 1 -- start start 2022-07-12T08:50:14.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4240ab290 0x7fc4240672b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc424148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 0x7fc424156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42414b8f0 con 0x7fc4240afa60 2022-07-12T08:50:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc424155ec0 con 0x7fc4240ab670 2022-07-12T08:50:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc424156160 con 0x7fc4240ab290 2022-07-12T08:50:14.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 0x7fc424156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 0x7fc424156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32870/0 (socket says 172.21.15.61:32870) 2022-07-12T08:50:14.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc423fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4240ab290 0x7fc4240672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 -- 172.21.15.61:0/3827936769 learned_addr learned my addr 172.21.15.61:0/3827936769 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:14.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc4237fe700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc424148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 -- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4240ab290 msgr2=0x7fc4240672b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4240ab290 0x7fc4240672b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 -- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 msgr2=0x7fc424148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc424148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 -- 172.21.15.61:0/3827936769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41800b180 con 0x7fc4240afa60 2022-07-12T08:50:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc428a83700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 0x7fc424156600 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fc41400c000 tx=0x7fc414006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc414011750 con 0x7fc4240afa60 2022-07-12T08:50:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4140118b0 con 0x7fc4240afa60 2022-07-12T08:50:14.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4240677a0 con 0x7fc4240afa60 2022-07-12T08:50:14.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc414011c10 con 0x7fc4240afa60 2022-07-12T08:50:14.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4240665f0 con 0x7fc4240afa60 2022-07-12T08:50:14.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.656+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc42414b740 con 0x7fc4240afa60 2022-07-12T08:50:14.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.660+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc414011d70 con 0x7fc4240afa60 2022-07-12T08:50:14.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.660+0000 7fc4217fa700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc404053a40 0x7fc404055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:14.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.660+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc41406c2a0 con 0x7fc4240afa60 2022-07-12T08:50:14.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.660+0000 7fc423fff700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc404053a40 0x7fc404055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:14.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.660+0000 7fc423fff700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc404053a40 0x7fc404055f00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fc41c00d000 tx=0x7fc41c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:14.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.664+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4140430f0 con 0x7fc4240afa60 2022-07-12T08:50:14.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.828+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc42406b2b0 con 0x7fc4240afa60 2022-07-12T08:50:14.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc4217fa700 1 -- 172.21.15.61:0/3827936769 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2145 (secure 0 0 0) 0x7fc414046710 con 0x7fc4240afa60 2022-07-12T08:50:14.836 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:14.836 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:10.807048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8082":{"gid":8082,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2724832679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2724832679},{"type":"v1","addr":"172.21.15.61:6833","nonce":2724832679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc404053a40 msgr2=0x7fc404055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc404053a40 0x7fc404055f00 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fc41c00d000 tx=0x7fc41c009040).stop 2022-07-12T08:50:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 msgr2=0x7fc424156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 0x7fc424156600 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7fc41400c000 tx=0x7fc414006df0).stop 2022-07-12T08:50:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 shutdown_connections 2022-07-12T08:50:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4240afa60 0x7fc424156600 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc404053a40 0x7fc404055f00 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc4240ab670 0x7fc424148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 --2- 172.21.15.61:0/3827936769 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc4240ab290 0x7fc4240672b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:14.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 >> 172.21.15.61:0/3827936769 conn(0x7fc4240b1950 msgr2=0x7fc4240b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:14.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 shutdown_connections 2022-07-12T08:50:14.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:14.832+0000 7fc429284700 1 -- 172.21.15.61:0/3827936769 wait complete. 2022-07-12T08:50:14.856 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-12T08:50:15.065 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:09.803335+0000', 'modified': '2022-07-12T08:50:10.807048+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8082}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8082': {'gid': 8082, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2724832679', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2724832679}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2724832679}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:15.066 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:15.067 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:15.191 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8082,"want_state":"up:active","state":"up:active","rank_uptime":5.3678209240000001,"mdsmap_epoch":72,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":9.7440077410000008} 2022-07-12T08:50:15.192 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8082, "mdsmap_epoch": 72, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 5.367820924, "state": "up:active", "uptime": 9.744007741, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:15.192 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:50:15.193 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:50:15.194 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:50:15.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.196 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:15.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:50:15.284 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:15.285 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:15.354 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:15.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.356 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:50:15.368 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:50:15.373 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:15.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:50:15.435 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:15.436 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:15.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:50:15.482+0000 7fc518453080 -1 init, newargv = 0x560d8fc6f830 newargc=9 2022-07-12T08:50:15.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22144]: starting ceph client 2022-07-12T08:50:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:15.510 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:15.510 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.511 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:50:15.512 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22144]: starting fuse 2022-07-12T08:50:15.548 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:50:15.549 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:50:15.550 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:50:15.552 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:50:15.553 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:50:15.554 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:50:15.554 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:50:15.555 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:15.563 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:50:15.564 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:50:15.565 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:50:15.566 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:15.572 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:50:15.574 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:50:15.575 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:50:15.611 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:50:15.611 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:15.612 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:50:15.612 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:50:15.613 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:50:15.613 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:50:15.613 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:50:15.616 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:15.617 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:50:15.617 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:50:15.691 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.22144.asok 2022-07-12T08:50:15.703 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22144.asok 2022-07-12T08:50:15.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.705 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22144.asok status 2022-07-12T08:50:15.859 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:50:15.860 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:50:15.861 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:50:15.861 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:50:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:50:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:50:15.863 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:50:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "22144", 2022-07-12T08:50:15.864 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:50:15.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:15.865 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:50:15.865 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:50:15.866 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 7097, 2022-07-12T08:50:15.866 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:50:15.867 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:50:15.867 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:50:15.867 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 7097 2022-07-12T08:50:15.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:15.868 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:50:15.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:50:15.869 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:50:15.869 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 461828418 2022-07-12T08:50:15.871 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:50:15.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:15.872 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:50:15.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:50:15.873 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:50:15.873 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 461828418 2022-07-12T08:50:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:15.874 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.7097 172.21.15.121:0/461828418", 2022-07-12T08:50:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/461828418", 2022-07-12T08:50:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:50:15.875 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 72, 2022-07-12T08:50:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 128, 2022-07-12T08:50:15.876 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:50:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:50:15.877 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:50:15.877 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:50:15.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.880 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:15.881 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:15.904 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:50:15.904 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:15.905 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:15.906 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:15.976 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:16.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 Processor -- start 2022-07-12T08:50:16.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- start start 2022-07-12T08:50:16.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa12c0af280 0x7fa12c060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c0af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa12c0b3f60 0x7fa12c0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12c143b00 con 0x7fa12c0b3f60 2022-07-12T08:50:16.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12c0aecb0 con 0x7fa12c0af280 2022-07-12T08:50:16.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c1435c0 con 0x7fa12c09bce0 2022-07-12T08:50:16.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa12c0b3f60 0x7fa12c0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa12c0b3f60 0x7fa12c0b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32878/0 (socket says 172.21.15.61:32878) 2022-07-12T08:50:16.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 -- 172.21.15.61:0/2549384435 learned_addr learned my addr 172.21.15.61:0/2549384435 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:16.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa129d9b700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12a59c700 1 --2- 172.21.15.61:0/2549384435 >> v2:172.21.15.111:3300/0 conn(0x7fa12c0af280 0x7fa12c060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 -- 172.21.15.61:0/2549384435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 msgr2=0x7fa12c0af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c0af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 -- 172.21.15.61:0/2549384435 >> v2:172.21.15.111:3300/0 conn(0x7fa12c0af280 msgr2=0x7fa12c060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 --2- 172.21.15.61:0/2549384435 >> v2:172.21.15.111:3300/0 conn(0x7fa12c0af280 0x7fa12c060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 -- 172.21.15.61:0/2549384435 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12c143860 con 0x7fa12c0b3f60 2022-07-12T08:50:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa129d9b700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c0af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12ad9d700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0b3f60 0x7fa12c0b4810 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7fa1240032f0 tx=0x7fa124009290).ready entity=mon.0 client_cookie=44f72c7ceb37ded3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12959a700 1 -- 172.21.15.61:0/2549384435 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa12400b680 con 0x7fa12c0b3f60 2022-07-12T08:50:16.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12959a700 1 -- 172.21.15.61:0/2549384435 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12400b7e0 con 0x7fa12c0b3f60 2022-07-12T08:50:16.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12959a700 1 -- 172.21.15.61:0/2549384435 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa12400b950 con 0x7fa12c0b3f60 2022-07-12T08:50:16.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- 172.21.15.61:0/2549384435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0b3f60 msgr2=0x7fa12c0b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0b3f60 0x7fa12c0b4810 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7fa1240032f0 tx=0x7fa124009290).stop 2022-07-12T08:50:16.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- 172.21.15.61:0/2549384435 shutdown_connections 2022-07-12T08:50:16.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c0af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- 172.21.15.61:0/2549384435 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0b3f60 0x7fa12c0b4810 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 --2- 172.21.15.61:0/2549384435 >> v2:172.21.15.111:3300/0 conn(0x7fa12c0af280 0x7fa12c060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- 172.21.15.61:0/2549384435 >> 172.21.15.61:0/2549384435 conn(0x7fa12c0b92c0 msgr2=0x7fa12c0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- 172.21.15.61:0/2549384435 shutdown_connections 2022-07-12T08:50:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.088+0000 7fa12b59e700 1 -- 172.21.15.61:0/2549384435 wait complete. 2022-07-12T08:50:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 Processor -- start 2022-07-12T08:50:16.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 -- start start 2022-07-12T08:50:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c09d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 0x7fa12c1406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa12c0b3f60 0x7fa12c13c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c09c7f0 con 0x7fa12c0af280 2022-07-12T08:50:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c09ca50 con 0x7fa12c0b3f60 2022-07-12T08:50:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c09da50 con 0x7fa12c09bce0 2022-07-12T08:50:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 0x7fa12c1406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 0x7fa12c1406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32884/0 (socket says 172.21.15.61:32884) 2022-07-12T08:50:16.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 -- 172.21.15.61:0/826258204 learned_addr learned my addr 172.21.15.61:0/826258204 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:16.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12a59c700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c09d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12ad9d700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa12c0b3f60 0x7fa12c13c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 -- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 msgr2=0x7fa12c09d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c09d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 -- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa12c0b3f60 msgr2=0x7fa12c13c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa12c0b3f60 0x7fa12c13c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 -- 172.21.15.61:0/826258204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11400b180 con 0x7fa12c0af280 2022-07-12T08:50:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa129d9b700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 0x7fa12c1406f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7fa11400c000 tx=0x7fa114006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa11400c7e0 con 0x7fa12c0af280 2022-07-12T08:50:16.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11400c940 con 0x7fa12c0af280 2022-07-12T08:50:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa124009530 con 0x7fa12c0af280 2022-07-12T08:50:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa11400cca0 con 0x7fa12c0af280 2022-07-12T08:50:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.092+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12c151c10 con 0x7fa12c0af280 2022-07-12T08:50:16.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.096+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12c057460 con 0x7fa12c0af280 2022-07-12T08:50:16.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.096+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa11401c460 con 0x7fa12c0af280 2022-07-12T08:50:16.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.096+0000 7fa1237fe700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa110053ad0 0x7fa110055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.096+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa11406c4c0 con 0x7fa12c0af280 2022-07-12T08:50:16.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.096+0000 7fa12a59c700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa110053ad0 0x7fa110055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.096+0000 7fa12a59c700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa110053ad0 0x7fa110055f90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7fa11c006000 tx=0x7fa11c006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.104+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1140432b0 con 0x7fa12c0af280 2022-07-12T08:50:16.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.232+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa12c0afeb0 con 0x7fa12c0af280 2022-07-12T08:50:16.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.232+0000 7fa1237fe700 1 -- 172.21.15.61:0/826258204 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fa11400ce00 con 0x7fa12c0af280 2022-07-12T08:50:16.245 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:16.246 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":128,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:10.806045+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:08.408495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:08.818782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:16.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa110053ad0 msgr2=0x7fa110055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa110053ad0 0x7fa110055f90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7fa11c006000 tx=0x7fa11c006d90).stop 2022-07-12T08:50:16.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 msgr2=0x7fa12c1406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 0x7fa12c1406f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7fa11400c000 tx=0x7fa114006df0).stop 2022-07-12T08:50:16.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 shutdown_connections 2022-07-12T08:50:16.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa110053ad0 0x7fa110055f90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa12c0b3f60 0x7fa12c13c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa12c0af280 0x7fa12c1406f0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 --2- 172.21.15.61:0/826258204 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa12c09bce0 0x7fa12c09d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 >> 172.21.15.61:0/826258204 conn(0x7fa12c0b92c0 msgr2=0x7fa12c1488c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 shutdown_connections 2022-07-12T08:50:16.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.236+0000 7fa12b59e700 1 -- 172.21.15.61:0/826258204 wait complete. 2022-07-12T08:50:16.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:50:08.408495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:50:08.818782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:50:16.263 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:16.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.372+0000 7fc6bef0a700 1 Processor -- start 2022-07-12T08:50:16.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.372+0000 7fc6bef0a700 1 -- start start 2022-07-12T08:50:16.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b81433e0 0x7fc6b814e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b80ab010 con 0x7fc6b80abe90 2022-07-12T08:50:16.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b8097aa0 con 0x7fc6b8060a70 2022-07-12T08:50:16.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8143ea0 con 0x7fc6b81433e0 2022-07-12T08:50:16.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bd707700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc6b80abe90 0x7fc6b8142cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49340/0 (socket says 172.21.15.61:49340) 2022-07-12T08:50:16.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bd707700 1 -- 172.21.15.61:0/118122187 learned_addr learned my addr 172.21.15.61:0/118122187 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:16.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6be709700 1 --2- 172.21.15.61:0/118122187 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b81433e0 0x7fc6b814e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034629347 0 0) 0x7fc6b80ab010 con 0x7fc6b80abe90 2022-07-12T08:50:16.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a4003260 con 0x7fc6b80abe90 2022-07-12T08:50:16.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147148125 0 0) 0x7fc6b8097aa0 con 0x7fc6b8060a70 2022-07-12T08:50:16.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b80ab010 con 0x7fc6b8060a70 2022-07-12T08:50:16.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6be709700 1 -- 172.21.15.61:0/118122187 >> v1:172.21.15.111:6789/0 conn(0x7fc6b8060a70 legacy=0x7fc6b809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6be709700 1 -- 172.21.15.61:0/118122187 >> v1:172.21.15.61:6789/0 conn(0x7fc6b80abe90 legacy=0x7fc6b8142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6be709700 1 -- 172.21.15.61:0/118122187 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b809b040 con 0x7fc6b81433e0 2022-07-12T08:50:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6be709700 1 --2- 172.21.15.61:0/118122187 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b81433e0 0x7fc6b814e650 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc6b00032f0 tx=0x7fc6b00091d0).ready entity=mon.2 client_cookie=8cccdf210968be82 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b000b640 con 0x7fc6b81433e0 2022-07-12T08:50:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b000b7a0 con 0x7fc6b81433e0 2022-07-12T08:50:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bcf06700 1 -- 172.21.15.61:0/118122187 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b000b910 con 0x7fc6b81433e0 2022-07-12T08:50:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- 172.21.15.61:0/118122187 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b81433e0 msgr2=0x7fc6b814e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/118122187 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b81433e0 0x7fc6b814e650 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc6b00032f0 tx=0x7fc6b00091d0).stop 2022-07-12T08:50:16.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- 172.21.15.61:0/118122187 shutdown_connections 2022-07-12T08:50:16.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/118122187 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b81433e0 0x7fc6b814e650 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- 172.21.15.61:0/118122187 >> 172.21.15.61:0/118122187 conn(0x7fc6b805bd10 msgr2=0x7fc6b805c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- 172.21.15.61:0/118122187 shutdown_connections 2022-07-12T08:50:16.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.376+0000 7fc6bef0a700 1 -- 172.21.15.61:0/118122187 wait complete. 2022-07-12T08:50:16.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 Processor -- start 2022-07-12T08:50:16.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- start start 2022-07-12T08:50:16.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 0x7fc6b8147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b80abe90 0x7fc6b8148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6b81433e0 0x7fc6b8149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b8152cc0 con 0x7fc6b8060a70 2022-07-12T08:50:16.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b813d370 con 0x7fc6b81433e0 2022-07-12T08:50:16.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b813d610 con 0x7fc6b80abe90 2022-07-12T08:50:16.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 0x7fc6b8147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 0x7fc6b8147c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32894/0 (socket says 172.21.15.61:32894) 2022-07-12T08:50:16.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 -- 172.21.15.61:0/3458444567 learned_addr learned my addr 172.21.15.61:0/3458444567 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:16.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6be709700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6b81433e0 0x7fc6b8149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bd707700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b80abe90 0x7fc6b8148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 -- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b80abe90 msgr2=0x7fc6b8148170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b80abe90 0x7fc6b8148170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 -- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6b81433e0 msgr2=0x7fc6b8149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6b81433e0 0x7fc6b8149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 -- 172.21.15.61:0/3458444567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b0009470 con 0x7fc6b8060a70 2022-07-12T08:50:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6be709700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6b81433e0 0x7fc6b8149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:16.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bd707700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b80abe90 0x7fc6b8148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:16.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bdf08700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 0x7fc6b8147c80 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fc6a8001110 tx=0x7fc6a8007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6a800f470 con 0x7fc6b8060a70 2022-07-12T08:50:16.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a800f5d0 con 0x7fc6b8060a70 2022-07-12T08:50:16.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b8146ab0 con 0x7fc6b8060a70 2022-07-12T08:50:16.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6a800f930 con 0x7fc6b8060a70 2022-07-12T08:50:16.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b8152a70 con 0x7fc6b8060a70 2022-07-12T08:50:16.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.380+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b813d7d0 con 0x7fc6b8060a70 2022-07-12T08:50:16.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.384+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc6a8010070 con 0x7fc6b8060a70 2022-07-12T08:50:16.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.384+0000 7fc6b6ffd700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6980539b0 0x7fc698055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.384+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc6a806b140 con 0x7fc6b8060a70 2022-07-12T08:50:16.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.384+0000 7fc6bd707700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6980539b0 0x7fc698055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.384+0000 7fc6bd707700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6980539b0 0x7fc698055e70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fc6ac008000 tx=0x7fc6ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.388+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6a80420a0 con 0x7fc6b8060a70 2022-07-12T08:50:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.516+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc6b8057460 con 0x7fc6b8060a70 2022-07-12T08:50:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.516+0000 7fc6b6ffd700 1 -- 172.21.15.61:0/3458444567 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v128) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc6a80456c0 con 0x7fc6b8060a70 2022-07-12T08:50:16.522 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.516+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6980539b0 msgr2=0x7fc698055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.516+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6980539b0 0x7fc698055e70 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7fc6ac008000 tx=0x7fc6ac009040).stop 2022-07-12T08:50:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.516+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 msgr2=0x7fc6b8147c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.516+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 0x7fc6b8147c80 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fc6a8001110 tx=0x7fc6a8007b20).stop 2022-07-12T08:50:16.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 shutdown_connections 2022-07-12T08:50:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc6980539b0 0x7fc698055e70 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc6b81433e0 0x7fc6b8149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc6b80abe90 0x7fc6b8148170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 --2- 172.21.15.61:0/3458444567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6b8060a70 0x7fc6b8147c80 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 >> 172.21.15.61:0/3458444567 conn(0x7fc6b805bd10 msgr2=0x7fc6b8142930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 shutdown_connections 2022-07-12T08:50:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.520+0000 7fc6bef0a700 1 -- 172.21.15.61:0/3458444567 wait complete. 2022-07-12T08:50:16.533 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:16.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.640+0000 7f1477fff700 1 Processor -- start 2022-07-12T08:50:16.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.640+0000 7f1477fff700 1 -- start start 2022-07-12T08:50:16.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f14780b2e40 0x7f147805f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f14780b1950 0x7f1478140410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14780b3220 con 0x7f14780b0a70 2022-07-12T08:50:16.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147813fa30 con 0x7f14780b2e40 2022-07-12T08:50:16.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147813fcd0 con 0x7f14780b1950 2022-07-12T08:50:16.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f14780b2e40 0x7f147805f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f14780b2e40 0x7f147805f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32902/0 (socket says 172.21.15.61:32902) 2022-07-12T08:50:16.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 -- 172.21.15.61:0/3896527074 learned_addr learned my addr 172.21.15.61:0/3896527074 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:16.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f146ffff700 1 --2- 172.21.15.61:0/3896527074 >> v2:172.21.15.111:3300/0 conn(0x7f14780b1950 0x7f1478140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 -- 172.21.15.61:0/3896527074 >> v2:172.21.15.111:3300/0 conn(0x7f14780b1950 msgr2=0x7f1478140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:16.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 --2- 172.21.15.61:0/3896527074 >> v2:172.21.15.111:3300/0 conn(0x7f14780b1950 0x7f1478140410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 -- 172.21.15.61:0/3896527074 >> v1:172.21.15.111:6789/0 conn(0x7f14780b0a70 legacy=0x7f1478128910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 -- 172.21.15.61:0/3896527074 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147813ff70 con 0x7f14780b2e40 2022-07-12T08:50:16.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1476ffd700 1 --2- 172.21.15.61:0/3896527074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14780b2e40 0x7f147805f140 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f1468002f70 tx=0x7f1468006eb0).ready entity=mon.0 client_cookie=acad639424164318 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f14767fc700 1 -- 172.21.15.61:0/3896527074 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f146800d610 con 0x7f14780b2e40 2022-07-12T08:50:16.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f14767fc700 1 -- 172.21.15.61:0/3896527074 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1468000e90 con 0x7f14780b2e40 2022-07-12T08:50:16.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f14767fc700 1 -- 172.21.15.61:0/3896527074 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f146800d9a0 con 0x7f14780b2e40 2022-07-12T08:50:16.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- 172.21.15.61:0/3896527074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14780b2e40 msgr2=0x7f147805f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 --2- 172.21.15.61:0/3896527074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14780b2e40 0x7f147805f140 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f1468002f70 tx=0x7f1468006eb0).stop 2022-07-12T08:50:16.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- 172.21.15.61:0/3896527074 shutdown_connections 2022-07-12T08:50:16.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 --2- 172.21.15.61:0/3896527074 >> v2:172.21.15.111:3300/0 conn(0x7f14780b1950 0x7f1478140410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 --2- 172.21.15.61:0/3896527074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14780b2e40 0x7f147805f140 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- 172.21.15.61:0/3896527074 >> 172.21.15.61:0/3896527074 conn(0x7f1478063ab0 msgr2=0x7f1478063e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- 172.21.15.61:0/3896527074 shutdown_connections 2022-07-12T08:50:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.644+0000 7f1477fff700 1 -- 172.21.15.61:0/3896527074 wait complete. 2022-07-12T08:50:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 Processor -- start 2022-07-12T08:50:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 -- start start 2022-07-12T08:50:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14780b0a70 0x7f14780b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14780b1950 0x7f1478141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1478143940 0x7f1478065460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478066c50 con 0x7f1478143940 2022-07-12T08:50:16.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1478066ef0 con 0x7f14780b1950 2022-07-12T08:50:16.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1478067190 con 0x7f14780b0a70 2022-07-12T08:50:16.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f14777fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1478143940 0x7f1478065460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f14777fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1478143940 0x7f1478065460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32912/0 (socket says 172.21.15.61:32912) 2022-07-12T08:50:16.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14780b1950 0x7f1478141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f14777fe700 1 -- 172.21.15.61:0/1985615162 learned_addr learned my addr 172.21.15.61:0/1985615162 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:16.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1476ffd700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14780b0a70 0x7f14780b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 -- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14780b0a70 msgr2=0x7f14780b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14780b0a70 0x7f14780b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 -- 172.21.15.61:0/1985615162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1478143940 msgr2=0x7f1478065460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1478143940 0x7f1478065460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 -- 172.21.15.61:0/1985615162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146800b1e0 con 0x7f14780b1950 2022-07-12T08:50:16.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f146ffff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14780b1950 0x7f1478141f50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f146000a000 tx=0x7f1460008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1460011590 con 0x7f14780b1950 2022-07-12T08:50:16.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1478127310 con 0x7f14780b1950 2022-07-12T08:50:16.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14600116f0 con 0x7f14780b1950 2022-07-12T08:50:16.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147813f0a0 con 0x7f14780b1950 2022-07-12T08:50:16.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.648+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1460011a50 con 0x7f14780b1950 2022-07-12T08:50:16.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.652+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1478057460 con 0x7f14780b1950 2022-07-12T08:50:16.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.652+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1460011bb0 con 0x7f14780b1950 2022-07-12T08:50:16.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.652+0000 7f1474ff9700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f145c053a40 0x7f145c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.652+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f146006bf60 con 0x7f14780b1950 2022-07-12T08:50:16.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.652+0000 7f1476ffd700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f145c053a40 0x7f145c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.652+0000 7f1476ffd700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f145c053a40 0x7f145c055f00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f146801c000 tx=0x7f1468015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.660+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1460042dc0 con 0x7f14780b1950 2022-07-12T08:50:16.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.788+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1478068350 con 0x7f14780b1950 2022-07-12T08:50:16.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.788+0000 7f1474ff9700 1 -- 172.21.15.61:0/1985615162 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7f14600463e0 con 0x7f14780b1950 2022-07-12T08:50:16.799 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:16.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f145c053a40 msgr2=0x7f145c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f145c053a40 0x7f145c055f00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f146801c000 tx=0x7f1468015000).stop 2022-07-12T08:50:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14780b1950 msgr2=0x7f1478141f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14780b1950 0x7f1478141f50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f146000a000 tx=0x7f1460008040).stop 2022-07-12T08:50:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 shutdown_connections 2022-07-12T08:50:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1478143940 0x7f1478065460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f145c053a40 0x7f145c055f00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14780b1950 0x7f1478141f50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 --2- 172.21.15.61:0/1985615162 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f14780b0a70 0x7f14780b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 >> 172.21.15.61:0/1985615162 conn(0x7f1478063ab0 msgr2=0x7f14781287b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 shutdown_connections 2022-07-12T08:50:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.792+0000 7f1477fff700 1 -- 172.21.15.61:0/1985615162 wait complete. 2022-07-12T08:50:16.813 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:50:16.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 Processor -- start 2022-07-12T08:50:16.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 -- start start 2022-07-12T08:50:16.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2e640987f0 0x7f2e64098c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64143a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e64097c80 con 0x7f2e640b5920 2022-07-12T08:50:16.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e640b5d00 con 0x7f2e640987f0 2022-07-12T08:50:16.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e69082700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6405c7f0 con 0x7f2e64143580 2022-07-12T08:50:16.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e637fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2e640987f0 0x7f2e64098c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e63fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2e640b5920 0x7f2e640b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49366/0 (socket says 172.21.15.61:49366) 2022-07-12T08:50:16.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e637fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2e640987f0 0x7f2e64098c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32916/0 (socket says 172.21.15.61:32916) 2022-07-12T08:50:16.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e63fff700 1 -- 172.21.15.61:0/3245573807 learned_addr learned my addr 172.21.15.61:0/3245573807 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:16.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.936+0000 7f2e68881700 1 --2- 172.21.15.61:0/3245573807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e637fe700 1 -- 172.21.15.61:0/3245573807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 msgr2=0x7f2e64143a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:16.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e637fe700 1 --2- 172.21.15.61:0/3245573807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64143a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e637fe700 1 -- 172.21.15.61:0/3245573807 >> v1:172.21.15.61:6789/0 conn(0x7f2e640b5920 legacy=0x7f2e640b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e62ffd700 1 -- 172.21.15.61:0/3245573807 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810961023 0 0) 0x7f2e64097c80 con 0x7f2e640b5920 2022-07-12T08:50:16.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e637fe700 1 -- 172.21.15.61:0/3245573807 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6405cbf0 con 0x7f2e640987f0 2022-07-12T08:50:16.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e637fe700 1 --2- 172.21.15.61:0/3245573807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e64098c60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7f2e4c002f70 tx=0x7f2e4c006df0).ready entity=mon.0 client_cookie=e973456f77dde254 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e62ffd700 1 -- 172.21.15.61:0/3245573807 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4c00d570 con 0x7f2e640987f0 2022-07-12T08:50:16.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e62ffd700 1 -- 172.21.15.61:0/3245573807 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c000e90 con 0x7f2e640987f0 2022-07-12T08:50:16.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e62ffd700 1 -- 172.21.15.61:0/3245573807 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e4c00d8c0 con 0x7f2e640987f0 2022-07-12T08:50:16.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 -- 172.21.15.61:0/3245573807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 msgr2=0x7f2e64098c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 --2- 172.21.15.61:0/3245573807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e64098c60 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 rx=0x7f2e4c002f70 tx=0x7f2e4c006df0).stop 2022-07-12T08:50:16.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 -- 172.21.15.61:0/3245573807 shutdown_connections 2022-07-12T08:50:16.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 --2- 172.21.15.61:0/3245573807 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64143a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 --2- 172.21.15.61:0/3245573807 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e64098c60 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 -- 172.21.15.61:0/3245573807 >> 172.21.15.61:0/3245573807 conn(0x7f2e64063340 msgr2=0x7f2e640b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:16.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 -- 172.21.15.61:0/3245573807 shutdown_connections 2022-07-12T08:50:16.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 -- 172.21.15.61:0/3245573807 wait complete. 2022-07-12T08:50:16.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 Processor -- start 2022-07-12T08:50:16.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.940+0000 7f2e69082700 1 -- start start 2022-07-12T08:50:16.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e640678a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e640b5920 0x7f2e64064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64064b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e640aed10 con 0x7f2e640987f0 2022-07-12T08:50:16.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64065640 con 0x7f2e640b5920 2022-07-12T08:50:16.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e64063b20 con 0x7f2e64143580 2022-07-12T08:50:16.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e640678a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e640678a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32922/0 (socket says 172.21.15.61:32922) 2022-07-12T08:50:16.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 -- 172.21.15.61:0/1807360765 learned_addr learned my addr 172.21.15.61:0/1807360765 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:16.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e68881700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64064b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e637fe700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e640b5920 0x7f2e64064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 -- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 msgr2=0x7f2e64064b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64064b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 -- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e640b5920 msgr2=0x7f2e64064210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:16.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e640b5920 0x7f2e64064210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:16.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 -- 172.21.15.61:0/1807360765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c00b180 con 0x7f2e640987f0 2022-07-12T08:50:16.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e63fff700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e640678a0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f2e54000fe0 tx=0x7f2e54007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e540070d0 con 0x7f2e640987f0 2022-07-12T08:50:16.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54007230 con 0x7f2e640987f0 2022-07-12T08:50:16.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e54005610 con 0x7f2e640987f0 2022-07-12T08:50:16.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e640638e0 con 0x7f2e640987f0 2022-07-12T08:50:16.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.944+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e64150a30 con 0x7f2e640987f0 2022-07-12T08:50:16.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.948+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e64067640 con 0x7f2e640987f0 2022-07-12T08:50:16.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.948+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2e54010070 con 0x7f2e640987f0 2022-07-12T08:50:16.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.948+0000 7f2e617fa700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e480539d0 0x7f2e48055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:16.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.948+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2e5406b450 con 0x7f2e640987f0 2022-07-12T08:50:16.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.948+0000 7f2e637fe700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e480539d0 0x7f2e48055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:16.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.948+0000 7f2e637fe700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e480539d0 0x7f2e48055e90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f2e4c01a000 tx=0x7f2e4c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:16.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:16.952+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2e540422a0 con 0x7f2e640987f0 2022-07-12T08:50:17.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.076+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e640afc50 con 0x7f2e640987f0 2022-07-12T08:50:17.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e617fa700 1 -- 172.21.15.61:0/1807360765 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2e540458c0 con 0x7f2e640987f0 2022-07-12T08:50:17.085 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:17.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e480539d0 msgr2=0x7f2e48055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e480539d0 0x7f2e48055e90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f2e4c01a000 tx=0x7f2e4c014000).stop 2022-07-12T08:50:17.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 msgr2=0x7f2e640678a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e640678a0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f2e54000fe0 tx=0x7f2e54007b50).stop 2022-07-12T08:50:17.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 shutdown_connections 2022-07-12T08:50:17.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2e64143580 0x7f2e64064b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2e480539d0 0x7f2e48055e90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2e640b5920 0x7f2e64064210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 --2- 172.21.15.61:0/1807360765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2e640987f0 0x7f2e640678a0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 >> 172.21.15.61:0/1807360765 conn(0x7f2e64063340 msgr2=0x7f2e640b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:17.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 shutdown_connections 2022-07-12T08:50:17.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.080+0000 7f2e69082700 1 -- 172.21.15.61:0/1807360765 wait complete. 2022-07-12T08:50:17.100 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:50:17.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 Processor -- start 2022-07-12T08:50:17.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 -- start start 2022-07-12T08:50:17.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f800413ec40 0x7f80040aaf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80040ab470 0x7f80040ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8004143170 con 0x7f800405fe30 2022-07-12T08:50:17.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80040b1790 con 0x7f800413ec40 2022-07-12T08:50:17.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f800359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80040b14e0 con 0x7f80040ab470 2022-07-12T08:50:17.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f800413ec40 0x7f80040aaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f800413ec40 0x7f80040aaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32934/0 (socket says 172.21.15.61:32934) 2022-07-12T08:50:17.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 -- 172.21.15.61:0/2484990828 learned_addr learned my addr 172.21.15.61:0/2484990828 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:17.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8002d9d700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80040ab470 0x7f80040ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 -- 172.21.15.61:0/2484990828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80040ab470 msgr2=0x7f80040ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:17.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80040ab470 0x7f80040ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 -- 172.21.15.61:0/2484990828 >> v1:172.21.15.111:6789/0 conn(0x7f800405fe30 legacy=0x7f800413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 -- 172.21.15.61:0/2484990828 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8004060210 con 0x7f800413ec40 2022-07-12T08:50:17.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8002d9d700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80040ab470 0x7f80040ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:17.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.212+0000 7f8001d9b700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800413ec40 0x7f80040aaf80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f7ff4002f70 tx=0x7f7ff4006f40).ready entity=mon.0 client_cookie=b1e5e98e92fc325c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:17.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800159a700 1 -- 172.21.15.61:0/2484990828 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff400dc70 con 0x7f800413ec40 2022-07-12T08:50:17.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800159a700 1 -- 172.21.15.61:0/2484990828 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4000e90 con 0x7f800413ec40 2022-07-12T08:50:17.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800159a700 1 -- 172.21.15.61:0/2484990828 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff400be90 con 0x7f800413ec40 2022-07-12T08:50:17.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 -- 172.21.15.61:0/2484990828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800413ec40 msgr2=0x7f80040aaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800413ec40 0x7f80040aaf80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f7ff4002f70 tx=0x7f7ff4006f40).stop 2022-07-12T08:50:17.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 -- 172.21.15.61:0/2484990828 shutdown_connections 2022-07-12T08:50:17.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80040ab470 0x7f80040ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 --2- 172.21.15.61:0/2484990828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800413ec40 0x7f80040aaf80 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 -- 172.21.15.61:0/2484990828 >> 172.21.15.61:0/2484990828 conn(0x7f8004063340 msgr2=0x7f80040b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:17.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 -- 172.21.15.61:0/2484990828 shutdown_connections 2022-07-12T08:50:17.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 -- 172.21.15.61:0/2484990828 wait complete. 2022-07-12T08:50:17.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 Processor -- start 2022-07-12T08:50:17.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.216+0000 7f800359e700 1 -- start start 2022-07-12T08:50:17.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 0x7f80040b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80040ab470 0x7f8004142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f800413ec40 0x7f80040aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8004142a50 con 0x7f800405fe30 2022-07-12T08:50:17.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80040ae4d0 con 0x7f80040ab470 2022-07-12T08:50:17.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80040ae770 con 0x7f800413ec40 2022-07-12T08:50:17.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 0x7f80040b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 0x7f80040b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32936/0 (socket says 172.21.15.61:32936) 2022-07-12T08:50:17.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 -- 172.21.15.61:0/4098606910 learned_addr learned my addr 172.21.15.61:0/4098606910 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:17.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f8002d9d700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f800413ec40 0x7f80040aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f8001d9b700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80040ab470 0x7f8004142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 -- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f800413ec40 msgr2=0x7f80040aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f800413ec40 0x7f80040aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 -- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80040ab470 msgr2=0x7f8004142560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80040ab470 0x7f8004142560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 -- 172.21.15.61:0/4098606910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff400b460 con 0x7f800405fe30 2022-07-12T08:50:17.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800259c700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 0x7f80040b30f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f800405f350 tx=0x7f7fec007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:17.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fec00ce30 con 0x7f800405fe30 2022-07-12T08:50:17.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec00f470 con 0x7f800405fe30 2022-07-12T08:50:17.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80040af100 con 0x7f800405fe30 2022-07-12T08:50:17.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fec00f7d0 con 0x7f800405fe30 2022-07-12T08:50:17.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80040acfc0 con 0x7f800405fe30 2022-07-12T08:50:17.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80040ae930 con 0x7f800405fe30 2022-07-12T08:50:17.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.220+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7fec019030 con 0x7f800405fe30 2022-07-12T08:50:17.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.224+0000 7f7ffb7fe700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7fe8053a60 0x7f7fe8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.224+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7fec06b2a0 con 0x7f800405fe30 2022-07-12T08:50:17.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.224+0000 7f8001d9b700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7fe8053a60 0x7f7fe8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.224+0000 7f8001d9b700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7fe8053a60 0x7f7fe8055f20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f7ff401c000 tx=0x7f7ff4006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:17.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.232+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7fec0421a0 con 0x7f800405fe30 2022-07-12T08:50:17.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8004057460 con 0x7f800405fe30 2022-07-12T08:50:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f7ffb7fe700 1 -- 172.21.15.61:0/4098606910 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7fec0457c0 con 0x7f800405fe30 2022-07-12T08:50:17.368 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:17.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7fe8053a60 msgr2=0x7f7fe8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7fe8053a60 0x7f7fe8055f20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f7ff401c000 tx=0x7f7ff4006f40).stop 2022-07-12T08:50:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 msgr2=0x7f80040b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 0x7f80040b30f0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f800405f350 tx=0x7f7fec007c50).stop 2022-07-12T08:50:17.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 shutdown_connections 2022-07-12T08:50:17.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f800413ec40 0x7f80040aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7fe8053a60 0x7f7fe8055f20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80040ab470 0x7f8004142560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 --2- 172.21.15.61:0/4098606910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f800405fe30 0x7f80040b30f0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 >> 172.21.15.61:0/4098606910 conn(0x7f8004063340 msgr2=0x7f800405bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:17.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 shutdown_connections 2022-07-12T08:50:17.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.360+0000 7f800359e700 1 -- 172.21.15.61:0/4098606910 wait complete. 2022-07-12T08:50:17.384 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:50:17.385 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:17.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.500+0000 7f89f5cc9700 1 Processor -- start 2022-07-12T08:50:17.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.500+0000 7f89f5cc9700 1 -- start start 2022-07-12T08:50:17.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89f5cc9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f00b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89f5cc9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f0140010 con 0x7f89f01436b0 2022-07-12T08:50:17.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89f5cc9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f00668f0 con 0x7f89f00b6780 2022-07-12T08:50:17.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89f5cc9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f0142f70 con 0x7f89f00b25c0 2022-07-12T08:50:17.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89f54c8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f89f01436b0 0x7f89f0065a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49398/0 (socket says 172.21.15.61:49398) 2022-07-12T08:50:17.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89f54c8700 1 -- 172.21.15.61:0/2579979900 learned_addr learned my addr 172.21.15.61:0/2579979900 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:17.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89effff700 1 --2- 172.21.15.61:0/2579979900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f00b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89effff700 1 -- 172.21.15.61:0/2579979900 >> v1:172.21.15.111:6789/0 conn(0x7f89f00b6780 legacy=0x7f89f013f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89effff700 1 -- 172.21.15.61:0/2579979900 >> v1:172.21.15.61:6789/0 conn(0x7f89f01436b0 legacy=0x7f89f0065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89effff700 1 -- 172.21.15.61:0/2579979900 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f0143210 con 0x7f89f00b25c0 2022-07-12T08:50:17.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.504+0000 7f89effff700 1 --2- 172.21.15.61:0/2579979900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f00b6b60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f89dc002f70 tx=0x7f89dc006df0).ready entity=mon.2 client_cookie=a65f2191c5afe067 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:17.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.508+0000 7f89ef7fe700 1 -- 172.21.15.61:0/2579979900 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89dc00d5c0 con 0x7f89f00b25c0 2022-07-12T08:50:17.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.508+0000 7f89ef7fe700 1 -- 172.21.15.61:0/2579979900 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc000e90 con 0x7f89f00b25c0 2022-07-12T08:50:17.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89ef7fe700 1 -- 172.21.15.61:0/2579979900 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89dc00d910 con 0x7f89f00b25c0 2022-07-12T08:50:17.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2579979900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 msgr2=0x7f89f00b6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2579979900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f00b6b60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f89dc002f70 tx=0x7f89dc006df0).stop 2022-07-12T08:50:17.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2579979900 shutdown_connections 2022-07-12T08:50:17.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2579979900 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f00b6b60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2579979900 >> 172.21.15.61:0/2579979900 conn(0x7f89f0064570 msgr2=0x7f89f00b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:17.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2579979900 shutdown_connections 2022-07-12T08:50:17.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2579979900 wait complete. 2022-07-12T08:50:17.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 Processor -- start 2022-07-12T08:50:17.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.512+0000 7f89f5cc9700 1 -- start start 2022-07-12T08:50:17.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f0127450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89f00b6780 0x7f89f00af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 0x7f89f0140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00aea90 con 0x7f89f01436b0 2022-07-12T08:50:17.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f01425b0 con 0x7f89f00b6780 2022-07-12T08:50:17.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f00aecf0 con 0x7f89f00b25c0 2022-07-12T08:50:17.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 0x7f89f0140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 0x7f89f0140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32954/0 (socket says 172.21.15.61:32954) 2022-07-12T08:50:17.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 -- 172.21.15.61:0/2854851116 learned_addr learned my addr 172.21.15.61:0/2854851116 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:17.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f4cc7700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f0127450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89effff700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89f00b6780 0x7f89f00af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 -- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 msgr2=0x7f89f0127450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f0127450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 -- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89f00b6780 msgr2=0x7f89f00af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89f00b6780 0x7f89f00af190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 -- 172.21.15.61:0/2854851116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89dc00b180 con 0x7f89f01436b0 2022-07-12T08:50:17.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f54c8700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 0x7f89f0140dd0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f89f0139fb0 tx=0x7f89e80013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:17.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89e8005ba0 con 0x7f89f01436b0 2022-07-12T08:50:17.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e8005d00 con 0x7f89f01436b0 2022-07-12T08:50:17.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f00b03a0 con 0x7f89f01436b0 2022-07-12T08:50:17.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89e80072c0 con 0x7f89f01436b0 2022-07-12T08:50:17.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.516+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f013e530 con 0x7f89f01436b0 2022-07-12T08:50:17.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.520+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f0057460 con 0x7f89f01436b0 2022-07-12T08:50:17.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.520+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f89e8010070 con 0x7f89f01436b0 2022-07-12T08:50:17.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.520+0000 7f89edffb700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89d00539d0 0x7f89d0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:17.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.520+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f89e806b660 con 0x7f89f01436b0 2022-07-12T08:50:17.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.520+0000 7f89f4cc7700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89d00539d0 0x7f89d0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:17.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.520+0000 7f89f4cc7700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89d00539d0 0x7f89d0055e90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f89e400d000 tx=0x7f89e4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:17.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.528+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f89e80425c0 con 0x7f89f01436b0 2022-07-12T08:50:17.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.688+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f0067b80 con 0x7f89f01436b0 2022-07-12T08:50:17.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.688+0000 7f89edffb700 1 -- 172.21.15.61:0/2854851116 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f89e8045be0 con 0x7f89f01436b0 2022-07-12T08:50:17.698 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:17.698 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:10.807048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8082":{"gid":8082,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2724832679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2724832679},{"type":"v1","addr":"172.21.15.61:6833","nonce":2724832679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:17.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.688+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89d00539d0 msgr2=0x7f89d0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.688+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89d00539d0 0x7f89d0055e90 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f89e400d000 tx=0x7f89e4007b90).stop 2022-07-12T08:50:17.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.688+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 msgr2=0x7f89f0140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:17.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.688+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 0x7f89f0140dd0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 rx=0x7f89f0139fb0 tx=0x7f89e80013a0).stop 2022-07-12T08:50:17.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 shutdown_connections 2022-07-12T08:50:17.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89f01436b0 0x7f89f0140dd0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f89d00539d0 0x7f89d0055e90 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f89f00b6780 0x7f89f00af190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 --2- 172.21.15.61:0/2854851116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f89f00b25c0 0x7f89f0127450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 >> 172.21.15.61:0/2854851116 conn(0x7f89f0064570 msgr2=0x7f89f0065eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 shutdown_connections 2022-07-12T08:50:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:17.692+0000 7f89f5cc9700 1 -- 172.21.15.61:0/2854851116 wait complete. 2022-07-12T08:50:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-12T08:50:17.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:09.803335+0000', 'modified': '2022-07-12T08:50:10.807048+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8082}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8082': {'gid': 8082, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2724832679', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2724832679}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2724832679}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:17.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:17.911 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:18.036 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8082,"want_state":"up:active","state":"up:active","rank_uptime":8.2195763320000008,"mdsmap_epoch":72,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":12.595749797} 2022-07-12T08:50:18.037 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8082, "mdsmap_epoch": 72, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 8.219576332, "state": "up:active", "uptime": 12.595749797, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:18.038 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:50:18.040 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:50:18.041 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:50:18.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.046 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.058 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:18.088 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:18.089 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:18.150 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:18.151 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.151 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:18.166 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:50:18.167 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.172 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:18.206 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:18.210 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:18.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:50:18.241+0000 7efc0fe27080 -1 init, newargv = 0x557c2c8b0540 newargc=9ceph-fuse[21574]: starting ceph client 2022-07-12T08:50:18.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr: 2022-07-12T08:50:18.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21574]: starting fuse 2022-07-12T08:50:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:18.264 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:18.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.265 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:18.277 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:50:18.278 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:50:18.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.280 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:50:18.281 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:50:18.282 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:50:18.283 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:50:18.283 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:50:18.284 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:18.285 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:50:18.286 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:50:18.286 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:50:18.287 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:18.288 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:50:18.288 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:50:18.289 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:50:18.290 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:50:18.291 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:18.291 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:50:18.292 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:50:18.293 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:50:18.293 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:50:18.294 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:50:18.295 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:18.295 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:50:18.297 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:50:18.377 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21574.asok 2022-07-12T08:50:18.390 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21574.asok 2022-07-12T08:50:18.390 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.391 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21574.asok status 2022-07-12T08:50:18.485 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:50:18.485 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:50:18.485 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:50:18.486 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:50:18.486 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:50:18.486 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:50:18.486 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:50:18.486 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21574", 2022-07-12T08:50:18.487 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:50:18.488 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:18.488 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:50:18.488 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:50:18.488 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 8286, 2022-07-12T08:50:18.488 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:50:18.489 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:50:18.489 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:50:18.489 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 8286 2022-07-12T08:50:18.490 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:18.490 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:50:18.490 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:50:18.490 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:50:18.491 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 1130946127 2022-07-12T08:50:18.491 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:50:18.491 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:18.491 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:50:18.491 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:50:18.492 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:50:18.492 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 1130946127 2022-07-12T08:50:18.492 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:18.493 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.8286 172.21.15.130:0/1130946127", 2022-07-12T08:50:18.493 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/1130946127", 2022-07-12T08:50:18.493 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:50:18.493 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 72, 2022-07-12T08:50:18.494 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 128, 2022-07-12T08:50:18.494 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:50:18.494 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:50:18.494 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:50:18.494 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:50:18.502 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.503 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:18.503 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.524 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:18.525 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:18.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.526 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.589 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:18.590 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.599 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:18.602 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:18.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.603 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:18.604 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.653 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:18.655 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:18.656 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:50:18.657 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:50:18.658 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21574]: fuse finished with error 0 and tester_r 0 2022-07-12T08:50:18.683 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:18.683 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:18.684 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:18.697 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:50:18.698 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:50:18.699 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:50:24.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:24.807 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:24.827 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:24.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:24.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:24.852 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:24.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:24.871 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2022-07-12T08:50:24.872 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2022-07-12T08:50:24.872 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:24.873 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2022-07-12T08:50:24.873 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.874 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2022-07-12T08:50:24.874 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.875 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:24.875 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.875 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:24.876 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2022-07-12T08:50:24.876 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.877 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2022-07-12T08:50:24.877 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2022-07-12T08:50:24.877 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.879 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:24.879 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:24.879 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.880 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:24.880 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:24.881 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.881 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2022-07-12T08:50:24.882 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.882 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:24.883 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2022-07-12T08:50:24.883 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:24.883 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:24.884 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:24.884 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:24.885 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2022-07-12T08:50:24.885 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:24.885 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:24.886 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.886 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:24.887 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.887 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2022-07-12T08:50:24.888 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.888 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2022-07-12T08:50:24.889 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.890 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2022-07-12T08:50:24.891 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2022-07-12T08:50:24.891 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2022-07-12T08:50:24.891 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:24.892 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:24.892 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.893 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2022-07-12T08:50:24.893 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2022-07-12T08:50:24.895 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.895 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:24.896 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2022-07-12T08:50:24.896 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.896 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2022-07-12T08:50:24.897 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:24.897 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.898 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2022-07-12T08:50:24.898 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2022-07-12T08:50:24.899 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:24.899 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.900 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2022-07-12T08:50:24.901 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2022-07-12T08:50:24.901 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.902 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:24.902 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.903 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2022-07-12T08:50:24.903 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.904 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2022-07-12T08:50:24.904 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.904 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:24.905 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:24.905 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.906 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2022-07-12T08:50:24.906 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2022-07-12T08:50:24.907 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:24.907 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 20 2022-07-12T08:50:24.907 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = filestore 2022-07-12T08:50:24.908 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-12T08:50:24.908 INFO:teuthology.orchestra.run.smithi061.stdout: osd sloppy crc = True 2022-07-12T08:50:24.909 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.909 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2022-07-12T08:50:24.909 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:24.910 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2022-07-12T08:50:24.911 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-12T08:50:24.912 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-12T08:50:24.912 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:24.912 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:24.912 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:24.913 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.913 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2022-07-12T08:50:24.913 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:24.914 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-12T08:50:24.914 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2022-07-12T08:50:24.915 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-12T08:50:24.915 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2022-07-12T08:50:24.915 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:24.916 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:24.916 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:24.917 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2022-07-12T08:50:24.917 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:24.918 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.918 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:24.919 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:24.935 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:24.935 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.936 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:24.936 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:24.937 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:24.937 INFO:teuthology.orchestra.run.smithi061.stdout: mon op complaint time = 120 2022-07-12T08:50:24.938 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:24.938 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2022-07-12T08:50:24.938 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2022-07-12T08:50:24.939 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2022-07-12T08:50:24.939 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2022-07-12T08:50:24.940 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:24.940 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:24.941 INFO:teuthology.orchestra.run.smithi061.stdout: client mount timeout = 600 2022-07-12T08:50:24.941 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 20 2022-07-12T08:50:24.941 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:24.942 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-12T08:50:24.943 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-12T08:50:24.943 INFO:teuthology.orchestra.run.smithi061.stdout: client debug getattr caps = true 2022-07-12T08:50:24.944 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2022-07-12T08:50:24.944 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.b] 2022-07-12T08:50:24.946 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.c] 2022-07-12T08:50:24.946 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-12T08:50:24.946 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-12T08:50:24.947 INFO:teuthology.orchestra.run.smithi061.stdout:[mds] 2022-07-12T08:50:24.947 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 20 2022-07-12T08:50:24.948 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:24.948 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal frag = True 2022-07-12T08:50:24.949 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:24.949 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal merge size = 5 2022-07-12T08:50:24.949 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split bits = 3 2022-07-12T08:50:24.950 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split size = 100 2022-07-12T08:50:24.950 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug frag = True 2022-07-12T08:50:24.951 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug scatterstat = True 2022-07-12T08:50:24.951 INFO:teuthology.orchestra.run.smithi061.stdout: mds op complaint time = 180 2022-07-12T08:50:24.952 INFO:teuthology.orchestra.run.smithi061.stdout: mds verify scatter = True 2022-07-12T08:50:24.952 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-12T08:50:24.953 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-12T08:50:24.954 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-12T08:50:24.963 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2022-07-12T08:50:24.964 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2022-07-12T08:50:24.965 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:24.965 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2022-07-12T08:50:24.966 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.966 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2022-07-12T08:50:24.967 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.968 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:24.968 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.969 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:24.970 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2022-07-12T08:50:24.971 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.971 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2022-07-12T08:50:24.972 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2022-07-12T08:50:24.972 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.973 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:24.973 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:24.974 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.974 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:24.975 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:24.976 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.976 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2022-07-12T08:50:24.977 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.977 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:24.978 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2022-07-12T08:50:24.978 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:24.979 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:24.979 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:24.980 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:24.980 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2022-07-12T08:50:24.981 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:24.981 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:24.982 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.982 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:24.983 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.983 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2022-07-12T08:50:24.984 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.984 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2022-07-12T08:50:24.984 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.985 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2022-07-12T08:50:24.985 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2022-07-12T08:50:24.985 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2022-07-12T08:50:24.986 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:24.987 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:24.987 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.987 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2022-07-12T08:50:24.988 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2022-07-12T08:50:24.988 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.989 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:24.989 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2022-07-12T08:50:24.990 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.990 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2022-07-12T08:50:24.991 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:24.991 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.992 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2022-07-12T08:50:24.992 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2022-07-12T08:50:24.993 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:24.994 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.994 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2022-07-12T08:50:24.994 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2022-07-12T08:50:24.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.995 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:24.996 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.996 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2022-07-12T08:50:24.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.997 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2022-07-12T08:50:24.997 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.998 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:24.998 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:24.998 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:24.999 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2022-07-12T08:50:24.999 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2022-07-12T08:50:24.999 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:24.999 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2022-07-12T08:50:25.000 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = filestore 2022-07-12T08:50:25.000 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2022-07-12T08:50:25.000 INFO:teuthology.orchestra.run.smithi137.stdout: osd sloppy crc = True 2022-07-12T08:50:25.000 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:25.001 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2022-07-12T08:50:25.001 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:25.001 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2022-07-12T08:50:25.002 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2022-07-12T08:50:25.002 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2022-07-12T08:50:25.002 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.002 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.003 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:25.003 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:25.003 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2022-07-12T08:50:25.003 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:25.004 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2022-07-12T08:50:25.004 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2022-07-12T08:50:25.004 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2022-07-12T08:50:25.004 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2022-07-12T08:50:25.005 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:25.005 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.005 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2022-07-12T08:50:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:25.006 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:25.007 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: mon op complaint time = 120 2022-07-12T08:50:25.008 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:25.009 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2022-07-12T08:50:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2022-07-12T08:50:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2022-07-12T08:50:25.009 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2022-07-12T08:50:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: client mount timeout = 600 2022-07-12T08:50:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2022-07-12T08:50:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.011 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.019 INFO:teuthology.orchestra.run.smithi137.stdout: client debug getattr caps = true 2022-07-12T08:50:25.020 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2022-07-12T08:50:25.020 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2022-07-12T08:50:25.021 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2022-07-12T08:50:25.021 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2022-07-12T08:50:25.022 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2022-07-12T08:50:25.022 INFO:teuthology.orchestra.run.smithi137.stdout:[mds] 2022-07-12T08:50:25.022 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds = 20 2022-07-12T08:50:25.023 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:25.023 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal frag = True 2022-07-12T08:50:25.024 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:25.025 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal merge size = 5 2022-07-12T08:50:25.025 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal split bits = 3 2022-07-12T08:50:25.026 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal split size = 100 2022-07-12T08:50:25.026 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug frag = True 2022-07-12T08:50:25.027 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug scatterstat = True 2022-07-12T08:50:25.027 INFO:teuthology.orchestra.run.smithi137.stdout: mds op complaint time = 180 2022-07-12T08:50:25.027 INFO:teuthology.orchestra.run.smithi137.stdout: mds verify scatter = True 2022-07-12T08:50:25.028 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2022-07-12T08:50:25.028 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.029 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.029 INFO:teuthology.orchestra.run.smithi130.stdout:[global] 2022-07-12T08:50:25.030 INFO:teuthology.orchestra.run.smithi130.stdout: chdir = "" 2022-07-12T08:50:25.030 INFO:teuthology.orchestra.run.smithi130.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:25.031 INFO:teuthology.orchestra.run.smithi130.stdout: auth supported = cephx 2022-07-12T08:50:25.031 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.032 INFO:teuthology.orchestra.run.smithi130.stdout: filestore xattr use omap = true 2022-07-12T08:50:25.032 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.032 INFO:teuthology.orchestra.run.smithi130.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:25.033 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.033 INFO:teuthology.orchestra.run.smithi130.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:25.034 INFO:teuthology.orchestra.run.smithi130.stdout: auth debug = true 2022-07-12T08:50:25.034 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.035 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on old message = true 2022-07-12T08:50:25.036 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on bug = true 2022-07-12T08:50:25.036 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.037 INFO:teuthology.orchestra.run.smithi130.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:25.037 INFO:teuthology.orchestra.run.smithi130.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:25.038 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.039 INFO:teuthology.orchestra.run.smithi130.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:25.039 INFO:teuthology.orchestra.run.smithi130.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:25.039 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.040 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default size = 2 2022-07-12T08:50:25.040 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.041 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:25.041 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow pg remap = true 2022-07-12T08:50:25.041 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:25.042 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:25.042 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:25.043 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:25.043 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on too few osds = false 2022-07-12T08:50:25.043 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:25.044 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:25.044 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.045 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:25.045 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.045 INFO:teuthology.orchestra.run.smithi130.stdout: osd default data pool replay window = 5 2022-07-12T08:50:25.046 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.046 INFO:teuthology.orchestra.run.smithi130.stdout: mon allow pool delete = true 2022-07-12T08:50:25.047 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.047 INFO:teuthology.orchestra.run.smithi130.stdout: mon cluster log file level = debug 2022-07-12T08:50:25.047 INFO:teuthology.orchestra.run.smithi130.stdout: debug asserts on shutdown = true 2022-07-12T08:50:25.048 INFO:teuthology.orchestra.run.smithi130.stdout: mon health detail to clog = false 2022-07-12T08:50:25.048 INFO:teuthology.orchestra.run.smithi130.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:25.048 INFO:teuthology.orchestra.run.smithi130.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:25.049 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.049 INFO:teuthology.orchestra.run.smithi130.stdout:[osd] 2022-07-12T08:50:25.050 INFO:teuthology.orchestra.run.smithi130.stdout: osd journal size = 100 2022-07-12T08:50:25.050 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.050 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:25.051 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub max interval = 600 2022-07-12T08:50:25.051 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.052 INFO:teuthology.orchestra.run.smithi130.stdout: osd recover clone overlap = true 2022-07-12T08:50:25.052 INFO:teuthology.orchestra.run.smithi130.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:25.052 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.053 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug shutdown = true 2022-07-12T08:50:25.053 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug op order = true 2022-07-12T08:50:25.054 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:25.054 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.055 INFO:teuthology.orchestra.run.smithi130.stdout: osd open classes on start = true 2022-07-12T08:50:25.055 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug pg log writeout = true 2022-07-12T08:50:25.055 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.056 INFO:teuthology.orchestra.run.smithi130.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:25.056 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.056 INFO:teuthology.orchestra.run.smithi130.stdout: osd map max advance = 10 2022-07-12T08:50:25.056 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.056 INFO:teuthology.orchestra.run.smithi130.stdout: journal zero on create = true 2022-07-12T08:50:25.057 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.057 INFO:teuthology.orchestra.run.smithi130.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:25.057 INFO:teuthology.orchestra.run.smithi130.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:25.057 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.058 INFO:teuthology.orchestra.run.smithi130.stdout: bdev debug aio = true 2022-07-12T08:50:25.058 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug misdirected ops = true 2022-07-12T08:50:25.058 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:25.058 INFO:teuthology.orchestra.run.smithi130.stdout: debug osd = 20 2022-07-12T08:50:25.058 INFO:teuthology.orchestra.run.smithi130.stdout: osd objectstore = filestore 2022-07-12T08:50:25.059 INFO:teuthology.orchestra.run.smithi130.stdout: osd op complaint time = 180 2022-07-12T08:50:25.059 INFO:teuthology.orchestra.run.smithi130.stdout: osd sloppy crc = True 2022-07-12T08:50:25.059 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.060 INFO:teuthology.orchestra.run.smithi130.stdout:[mgr] 2022-07-12T08:50:25.060 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:25.060 INFO:teuthology.orchestra.run.smithi130.stdout: debug mgr = 20 2022-07-12T08:50:25.060 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2022-07-12T08:50:25.060 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2022-07-12T08:50:25.061 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.061 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.061 INFO:teuthology.orchestra.run.smithi130.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:25.061 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.062 INFO:teuthology.orchestra.run.smithi130.stdout:[mon] 2022-07-12T08:50:25.062 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:25.062 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2022-07-12T08:50:25.062 INFO:teuthology.orchestra.run.smithi130.stdout: debug paxos = 20 2022-07-12T08:50:25.062 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2022-07-12T08:50:25.063 INFO:teuthology.orchestra.run.smithi130.stdout: mon data avail warn = 5 2022-07-12T08:50:25.063 INFO:teuthology.orchestra.run.smithi130.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:25.063 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.063 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:25.064 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd prime pg temp = true 2022-07-12T08:50:25.064 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.064 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.064 INFO:teuthology.orchestra.run.smithi130.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:25.064 INFO:teuthology.orchestra.run.smithi130.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:25.065 INFO:teuthology.orchestra.run.smithi130.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:25.065 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.065 INFO:teuthology.orchestra.run.smithi130.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:25.065 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:25.066 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:25.066 INFO:teuthology.orchestra.run.smithi130.stdout: mon op complaint time = 120 2022-07-12T08:50:25.066 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:25.066 INFO:teuthology.orchestra.run.smithi130.stdout:[client] 2022-07-12T08:50:25.067 INFO:teuthology.orchestra.run.smithi130.stdout: rgw cache enabled = true 2022-07-12T08:50:25.067 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable ops log = true 2022-07-12T08:50:25.067 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable usage log = true 2022-07-12T08:50:25.067 INFO:teuthology.orchestra.run.smithi130.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:25.068 INFO:teuthology.orchestra.run.smithi130.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:25.068 INFO:teuthology.orchestra.run.smithi130.stdout: client mount timeout = 600 2022-07-12T08:50:25.068 INFO:teuthology.orchestra.run.smithi130.stdout: debug client = 20 2022-07-12T08:50:25.068 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:25.068 INFO:teuthology.orchestra.run.smithi130.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.069 INFO:teuthology.orchestra.run.smithi130.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.069 INFO:teuthology.orchestra.run.smithi130.stdout: client debug getattr caps = true 2022-07-12T08:50:25.069 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.a] 2022-07-12T08:50:25.069 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.b] 2022-07-12T08:50:25.070 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.c] 2022-07-12T08:50:25.070 INFO:teuthology.orchestra.run.smithi130.stdout:[mds.a] 2022-07-12T08:50:25.070 INFO:teuthology.orchestra.run.smithi130.stdout:[mds.b] 2022-07-12T08:50:25.070 INFO:teuthology.orchestra.run.smithi130.stdout:[mds] 2022-07-12T08:50:25.071 INFO:teuthology.orchestra.run.smithi130.stdout: debug mds = 20 2022-07-12T08:50:25.071 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:25.071 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal frag = True 2022-07-12T08:50:25.071 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:25.072 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal merge size = 5 2022-07-12T08:50:25.072 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal split bits = 3 2022-07-12T08:50:25.072 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal split size = 100 2022-07-12T08:50:25.072 INFO:teuthology.orchestra.run.smithi130.stdout: mds debug frag = True 2022-07-12T08:50:25.072 INFO:teuthology.orchestra.run.smithi130.stdout: mds debug scatterstat = True 2022-07-12T08:50:25.073 INFO:teuthology.orchestra.run.smithi130.stdout: mds op complaint time = 180 2022-07-12T08:50:25.073 INFO:teuthology.orchestra.run.smithi130.stdout: mds verify scatter = True 2022-07-12T08:50:25.073 INFO:teuthology.orchestra.run.smithi130.stdout: osd op complaint time = 180 2022-07-12T08:50:25.073 INFO:teuthology.orchestra.run.smithi130.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.074 INFO:teuthology.orchestra.run.smithi130.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.074 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2022-07-12T08:50:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2022-07-12T08:50:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2022-07-12T08:50:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2022-07-12T08:50:25.075 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2022-07-12T08:50:25.076 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2022-07-12T08:50:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2022-07-12T08:50:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.077 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:25.078 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2022-07-12T08:50:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:25.079 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2022-07-12T08:50:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:25.080 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2022-07-12T08:50:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:25.081 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2022-07-12T08:50:25.082 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2022-07-12T08:50:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2022-07-12T08:50:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2022-07-12T08:50:25.083 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2022-07-12T08:50:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:25.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.084 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2022-07-12T08:50:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2022-07-12T08:50:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:25.085 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2022-07-12T08:50:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2022-07-12T08:50:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:25.086 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2022-07-12T08:50:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2022-07-12T08:50:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.087 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2022-07-12T08:50:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2022-07-12T08:50:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:25.088 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2022-07-12T08:50:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2022-07-12T08:50:25.089 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.090 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2022-07-12T08:50:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2022-07-12T08:50:25.091 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2022-07-12T08:50:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = filestore 2022-07-12T08:50:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-07-12T08:50:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: osd sloppy crc = True 2022-07-12T08:50:25.092 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.093 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2022-07-12T08:50:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2022-07-12T08:50:25.093 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-07-12T08:50:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-07-12T08:50:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.094 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.095 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2022-07-12T08:50:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:25.095 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-07-12T08:50:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2022-07-12T08:50:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-07-12T08:50:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2022-07-12T08:50:25.096 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2022-07-12T08:50:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.097 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:25.098 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2022-07-12T08:50:25.099 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:25.100 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2022-07-12T08:50:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2022-07-12T08:50:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2022-07-12T08:50:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2022-07-12T08:50:25.100 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2022-07-12T08:50:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2022-07-12T08:50:25.101 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: client debug getattr caps = true 2022-07-12T08:50:25.102 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2022-07-12T08:50:25.103 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2022-07-12T08:50:25.103 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2022-07-12T08:50:25.103 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2022-07-12T08:50:25.103 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2022-07-12T08:50:25.104 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2022-07-12T08:50:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2022-07-12T08:50:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:25.104 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal frag = True 2022-07-12T08:50:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal merge size = 5 2022-07-12T08:50:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split bits = 3 2022-07-12T08:50:25.105 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split size = 100 2022-07-12T08:50:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2022-07-12T08:50:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2022-07-12T08:50:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2022-07-12T08:50:25.106 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2022-07-12T08:50:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-07-12T08:50:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.107 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.107 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2022-07-12T08:50:25.108 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2022-07-12T08:50:25.108 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:25.108 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2022-07-12T08:50:25.108 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.109 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2022-07-12T08:50:25.109 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.109 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:25.109 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.110 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:25.110 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2022-07-12T08:50:25.110 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.110 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2022-07-12T08:50:25.111 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2022-07-12T08:50:25.111 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.111 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:25.111 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:25.111 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.112 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:25.112 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:25.112 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.113 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2022-07-12T08:50:25.113 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.113 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:25.113 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2022-07-12T08:50:25.114 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:25.114 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:25.114 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:25.114 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:25.115 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2022-07-12T08:50:25.115 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:25.115 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:25.115 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.116 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:25.116 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.116 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2022-07-12T08:50:25.116 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.117 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2022-07-12T08:50:25.117 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.117 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2022-07-12T08:50:25.117 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2022-07-12T08:50:25.118 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2022-07-12T08:50:25.118 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:25.118 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:25.118 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.119 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2022-07-12T08:50:25.119 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2022-07-12T08:50:25.119 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.119 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2022-07-12T08:50:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2022-07-12T08:50:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:25.121 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.121 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2022-07-12T08:50:25.121 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2022-07-12T08:50:25.122 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:25.122 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.122 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2022-07-12T08:50:25.122 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2022-07-12T08:50:25.123 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.123 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:25.123 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.123 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2022-07-12T08:50:25.124 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.124 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2022-07-12T08:50:25.124 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.124 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:25.125 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:25.125 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.125 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2022-07-12T08:50:25.125 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2022-07-12T08:50:25.125 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:25.126 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2022-07-12T08:50:25.126 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = filestore 2022-07-12T08:50:25.126 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2022-07-12T08:50:25.126 INFO:teuthology.orchestra.run.smithi111.stdout: osd sloppy crc = True 2022-07-12T08:50:25.127 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.127 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2022-07-12T08:50:25.127 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:25.127 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2022-07-12T08:50:25.127 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2022-07-12T08:50:25.128 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2022-07-12T08:50:25.128 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.128 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.128 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:25.128 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.129 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2022-07-12T08:50:25.129 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:25.129 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2022-07-12T08:50:25.129 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2022-07-12T08:50:25.130 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2022-07-12T08:50:25.130 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2022-07-12T08:50:25.130 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:25.130 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:25.130 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:25.131 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2022-07-12T08:50:25.131 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:25.131 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.131 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:25.132 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:25.132 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:25.132 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.132 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:25.133 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:25.133 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:25.133 INFO:teuthology.orchestra.run.smithi111.stdout: mon op complaint time = 120 2022-07-12T08:50:25.133 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:25.134 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2022-07-12T08:50:25.134 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2022-07-12T08:50:25.134 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2022-07-12T08:50:25.134 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2022-07-12T08:50:25.135 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:25.135 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:25.135 INFO:teuthology.orchestra.run.smithi111.stdout: client mount timeout = 600 2022-07-12T08:50:25.135 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2022-07-12T08:50:25.136 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:25.136 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.136 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.136 INFO:teuthology.orchestra.run.smithi111.stdout: client debug getattr caps = true 2022-07-12T08:50:25.137 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2022-07-12T08:50:25.137 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2022-07-12T08:50:25.137 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2022-07-12T08:50:25.137 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2022-07-12T08:50:25.137 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2022-07-12T08:50:25.138 INFO:teuthology.orchestra.run.smithi111.stdout:[mds] 2022-07-12T08:50:25.138 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds = 20 2022-07-12T08:50:25.139 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:25.139 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal frag = True 2022-07-12T08:50:25.139 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:25.140 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal merge size = 5 2022-07-12T08:50:25.140 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal split bits = 3 2022-07-12T08:50:25.140 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal split size = 100 2022-07-12T08:50:25.141 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug frag = True 2022-07-12T08:50:25.141 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug scatterstat = True 2022-07-12T08:50:25.141 INFO:teuthology.orchestra.run.smithi111.stdout: mds op complaint time = 180 2022-07-12T08:50:25.141 INFO:teuthology.orchestra.run.smithi111.stdout: mds verify scatter = True 2022-07-12T08:50:25.142 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2022-07-12T08:50:25.142 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 15m 2022-07-12T08:50:25.142 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 15m 2022-07-12T08:50:25.143 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:25.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 Processor -- start 2022-07-12T08:50:25.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 -- start start 2022-07-12T08:50:25.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f80b00b5be0 0x7f80b00af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f80b0144830 0x7f80b013a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b00674c0 con 0x7f80b005d0f0 2022-07-12T08:50:25.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b00af6a0 con 0x7f80b00b5be0 2022-07-12T08:50:25.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b7997700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b00b5fc0 con 0x7f80b0144830 2022-07-12T08:50:25.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f80b00b5be0 0x7f80b00af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f80b00b5be0 0x7f80b00af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32958/0 (socket says 172.21.15.61:32958) 2022-07-12T08:50:25.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 -- 172.21.15.61:0/1310516224 learned_addr learned my addr 172.21.15.61:0/1310516224 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:25.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5f34700 1 --2- 172.21.15.61:0/1310516224 >> v2:172.21.15.111:3300/0 conn(0x7f80b0144830 0x7f80b013a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 -- 172.21.15.61:0/1310516224 >> v2:172.21.15.111:3300/0 conn(0x7f80b0144830 msgr2=0x7f80b013a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:25.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 --2- 172.21.15.61:0/1310516224 >> v2:172.21.15.111:3300/0 conn(0x7f80b0144830 0x7f80b013a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 -- 172.21.15.61:0/1310516224 >> v1:172.21.15.111:6789/0 conn(0x7f80b005d0f0 legacy=0x7f80b0144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:25.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 -- 172.21.15.61:0/1310516224 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b005c580 con 0x7f80b00b5be0 2022-07-12T08:50:25.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5f34700 1 --2- 172.21.15.61:0/1310516224 >> v2:172.21.15.111:3300/0 conn(0x7f80b0144830 0x7f80b013a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:25.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f80b5733700 1 --2- 172.21.15.61:0/1310516224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b00af1b0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f80b0063930 tx=0x7f80ac006eb0).ready entity=mon.0 client_cookie=f54cd4953901c178 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.252+0000 7f809ffff700 1 -- 172.21.15.61:0/1310516224 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80ac00d5c0 con 0x7f80b00b5be0 2022-07-12T08:50:25.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f809ffff700 1 -- 172.21.15.61:0/1310516224 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac000e90 con 0x7f80b00b5be0 2022-07-12T08:50:25.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f809ffff700 1 -- 172.21.15.61:0/1310516224 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80ac00d910 con 0x7f80b00b5be0 2022-07-12T08:50:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 -- 172.21.15.61:0/1310516224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 msgr2=0x7f80b00af1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 --2- 172.21.15.61:0/1310516224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b00af1b0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f80b0063930 tx=0x7f80ac006eb0).stop 2022-07-12T08:50:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 -- 172.21.15.61:0/1310516224 shutdown_connections 2022-07-12T08:50:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 --2- 172.21.15.61:0/1310516224 >> v2:172.21.15.111:3300/0 conn(0x7f80b0144830 0x7f80b013a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 --2- 172.21.15.61:0/1310516224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b00af1b0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 -- 172.21.15.61:0/1310516224 >> 172.21.15.61:0/1310516224 conn(0x7f80b0063450 msgr2=0x7f80b00b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 -- 172.21.15.61:0/1310516224 shutdown_connections 2022-07-12T08:50:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 -- 172.21.15.61:0/1310516224 wait complete. 2022-07-12T08:50:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.256+0000 7f80b7997700 1 Processor -- start 2022-07-12T08:50:25.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 -- start start 2022-07-12T08:50:25.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80b005d0f0 0x7f80b01417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b013d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80b0144830 0x7f80b0152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b013bf40 con 0x7f80b00b5be0 2022-07-12T08:50:25.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b009e240 con 0x7f80b0144830 2022-07-12T08:50:25.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b00ac1a0 con 0x7f80b005d0f0 2022-07-12T08:50:25.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b013d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32966/0 (socket says 172.21.15.61:32966) 2022-07-12T08:50:25.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 -- 172.21.15.61:0/881590078 learned_addr learned my addr 172.21.15.61:0/881590078 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:25.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b5733700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80b005d0f0 0x7f80b01417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b5f34700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80b0144830 0x7f80b0152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 -- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80b005d0f0 msgr2=0x7f80b01417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80b005d0f0 0x7f80b01417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 -- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80b0144830 msgr2=0x7f80b0152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80b0144830 0x7f80b0152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 -- 172.21.15.61:0/881590078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac00b1e0 con 0x7f80b00b5be0 2022-07-12T08:50:25.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b4f32700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b013d250 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7f80a0001fa0 tx=0x7f80a0007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a0007030 con 0x7f80b00b5be0 2022-07-12T08:50:25.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a0007190 con 0x7f80b00b5be0 2022-07-12T08:50:25.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b009e470 con 0x7f80b00b5be0 2022-07-12T08:50:25.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80a0006510 con 0x7f80b00b5be0 2022-07-12T08:50:25.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.260+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b01510a0 con 0x7f80b00b5be0 2022-07-12T08:50:25.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.264+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b0057460 con 0x7f80b00b5be0 2022-07-12T08:50:25.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.264+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f80a0010030 con 0x7f80b00b5be0 2022-07-12T08:50:25.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.264+0000 7f809e7fc700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8090053920 0x7f8090055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.264+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f80a006bac0 con 0x7f80b00b5be0 2022-07-12T08:50:25.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.264+0000 7f80b5733700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8090053920 0x7f8090055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.264+0000 7f80b5733700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8090053920 0x7f8090055de0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f80ac01c000 tx=0x7f80ac015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.268+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80a0042a20 con 0x7f80b00b5be0 2022-07-12T08:50:25.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.392+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80b006a5f0 con 0x7f80b00b5be0 2022-07-12T08:50:25.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f809e7fc700 1 -- 172.21.15.61:0/881590078 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f80a0046040 con 0x7f80b00b5be0 2022-07-12T08:50:25.404 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:25.405 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":128,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:10.806045+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:08.408495+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:08.818782+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8090053920 msgr2=0x7f8090055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8090053920 0x7f8090055de0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f80ac01c000 tx=0x7f80ac015000).stop 2022-07-12T08:50:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 msgr2=0x7f80b013d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b013d250 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7f80a0001fa0 tx=0x7f80a0007c00).stop 2022-07-12T08:50:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 shutdown_connections 2022-07-12T08:50:25.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8090053920 0x7f8090055de0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80b0144830 0x7f80b0152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80b00b5be0 0x7f80b013d250 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 --2- 172.21.15.61:0/881590078 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f80b005d0f0 0x7f80b01417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.396+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 >> 172.21.15.61:0/881590078 conn(0x7f80b0063450 msgr2=0x7f80b00b4050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:25.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.400+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 shutdown_connections 2022-07-12T08:50:25.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.400+0000 7f80b7997700 1 -- 172.21.15.61:0/881590078 wait complete. 2022-07-12T08:50:25.416 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:50:08.408495+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:50:08.818782+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '128', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:50:25.417 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:25.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.536+0000 7f60b8cac700 1 Processor -- start 2022-07-12T08:50:25.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- start start 2022-07-12T08:50:25.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b56a0 0x7f60b40b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f60b40b3380 0x7f60b40b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f60b40b06d0 0x7f60b4143130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b40b0b40 con 0x7f60b40b3380 2022-07-12T08:50:25.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b40b5f70 con 0x7f60b40b06d0 2022-07-12T08:50:25.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40b1560 con 0x7f60b40b56a0 2022-07-12T08:50:25.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f60b40b3380 0x7f60b40b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f60b40b3380 0x7f60b40b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32974/0 (socket says 172.21.15.61:32974) 2022-07-12T08:50:25.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 -- 172.21.15.61:0/3233609054 learned_addr learned my addr 172.21.15.61:0/3233609054 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:25.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b2d9d700 1 --2- 172.21.15.61:0/3233609054 >> v2:172.21.15.111:3300/0 conn(0x7f60b40b06d0 0x7f60b4143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b259c700 1 --2- 172.21.15.61:0/3233609054 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b56a0 0x7f60b40b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 -- 172.21.15.61:0/3233609054 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b56a0 msgr2=0x7f60b40b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:25.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 --2- 172.21.15.61:0/3233609054 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b56a0 0x7f60b40b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 -- 172.21.15.61:0/3233609054 >> v2:172.21.15.111:3300/0 conn(0x7f60b40b06d0 msgr2=0x7f60b4143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:25.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 --2- 172.21.15.61:0/3233609054 >> v2:172.21.15.111:3300/0 conn(0x7f60b40b06d0 0x7f60b4143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 -- 172.21.15.61:0/3233609054 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40b4710 con 0x7f60b40b3380 2022-07-12T08:50:25.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60abfff700 1 --2- 172.21.15.61:0/3233609054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b3380 0x7f60b40b01e0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f609c002f70 tx=0x7f609c006eb0).ready entity=mon.0 client_cookie=c3e8c150bc376c36 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b1d9b700 1 -- 172.21.15.61:0/3233609054 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f609c00d570 con 0x7f60b40b3380 2022-07-12T08:50:25.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b1d9b700 1 -- 172.21.15.61:0/3233609054 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c000e90 con 0x7f60b40b3380 2022-07-12T08:50:25.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b1d9b700 1 -- 172.21.15.61:0/3233609054 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f609c00d900 con 0x7f60b40b3380 2022-07-12T08:50:25.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- 172.21.15.61:0/3233609054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b3380 msgr2=0x7f60b40b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3233609054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b3380 0x7f60b40b01e0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f609c002f70 tx=0x7f609c006eb0).stop 2022-07-12T08:50:25.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- 172.21.15.61:0/3233609054 shutdown_connections 2022-07-12T08:50:25.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3233609054 >> v2:172.21.15.111:3300/0 conn(0x7f60b40b06d0 0x7f60b4143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3233609054 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b3380 0x7f60b40b01e0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3233609054 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b56a0 0x7f60b40b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- 172.21.15.61:0/3233609054 >> 172.21.15.61:0/3233609054 conn(0x7f60b40b6350 msgr2=0x7f60b40b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:25.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- 172.21.15.61:0/3233609054 shutdown_connections 2022-07-12T08:50:25.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.540+0000 7f60b8cac700 1 -- 172.21.15.61:0/3233609054 wait complete. 2022-07-12T08:50:25.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 Processor -- start 2022-07-12T08:50:25.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 -- start start 2022-07-12T08:50:25.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 0x7f60b4155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40b3380 0x7f60b40ba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 0x7f60b4152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b41524e0 con 0x7f60b40b56a0 2022-07-12T08:50:25.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4112ad0 con 0x7f60b40b3380 2022-07-12T08:50:25.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b8cac700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4115b40 con 0x7f60b40b06d0 2022-07-12T08:50:25.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 0x7f60b4152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b259c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 0x7f60b4155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60abfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40b3380 0x7f60b40ba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 0x7f60b4152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32982/0 (socket says 172.21.15.61:32982) 2022-07-12T08:50:25.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b259c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 0x7f60b4155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:43406/0 (socket says 172.21.15.61:43406) 2022-07-12T08:50:25.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 -- 172.21.15.61:0/3953919542 learned_addr learned my addr 172.21.15.61:0/3953919542 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:25.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 -- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 msgr2=0x7f60b4155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 0x7f60b4155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 -- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40b3380 msgr2=0x7f60b40ba0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.544+0000 7f60b2d9d700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40b3380 0x7f60b40ba0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60b2d9d700 1 -- 172.21.15.61:0/3953919542 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c00b1e0 con 0x7f60b40b56a0 2022-07-12T08:50:25.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60b259c700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 0x7f60b4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:25.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60abfff700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40b3380 0x7f60b40ba0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:25.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60b2d9d700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 0x7f60b4152910 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7f60ac00a000 tx=0x7f60ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60ac0127e0 con 0x7f60b40b56a0 2022-07-12T08:50:25.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac012940 con 0x7f60b40b56a0 2022-07-12T08:50:25.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60ac012ca0 con 0x7f60b40b56a0 2022-07-12T08:50:25.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b4115de0 con 0x7f60b40b56a0 2022-07-12T08:50:25.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40afbe0 con 0x7f60b40b56a0 2022-07-12T08:50:25.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b4152670 con 0x7f60b40b56a0 2022-07-12T08:50:25.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.548+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f60ac013480 con 0x7f60b40b56a0 2022-07-12T08:50:25.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.552+0000 7f60ab7fe700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6098053ad0 0x7f6098055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.552+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f60ac06c170 con 0x7f60b40b56a0 2022-07-12T08:50:25.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.552+0000 7f60b259c700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6098053ad0 0x7f6098055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.552+0000 7f60b259c700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6098053ad0 0x7f6098055f90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f60a400c000 tx=0x7f60a4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.556+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60b4152670 con 0x7f60b40b56a0 2022-07-12T08:50:25.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f60b4152670 con 0x7f60b40b56a0 2022-07-12T08:50:25.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60ab7fe700 1 -- 172.21.15.61:0/3953919542 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v128) v1 ==== 110+0+10 (secure 0 0 0) 0x7f60b4152670 con 0x7f60b40b56a0 2022-07-12T08:50:25.687 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:25.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6098053ad0 msgr2=0x7f6098055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6098053ad0 0x7f6098055f90 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f60a400c000 tx=0x7f60a4008040).stop 2022-07-12T08:50:25.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 msgr2=0x7f60b4152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 0x7f60b4152910 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 rx=0x7f60ac00a000 tx=0x7f60ac008040).stop 2022-07-12T08:50:25.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 shutdown_connections 2022-07-12T08:50:25.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b40b56a0 0x7f60b4152910 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6098053ad0 0x7f6098055f90 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60b40b3380 0x7f60b40ba0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 --2- 172.21.15.61:0/3953919542 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f60b40b06d0 0x7f60b4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 >> 172.21.15.61:0/3953919542 conn(0x7f60b40b6350 msgr2=0x7f60b40b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:25.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 shutdown_connections 2022-07-12T08:50:25.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.684+0000 7f60b8cac700 1 -- 172.21.15.61:0/3953919542 wait complete. 2022-07-12T08:50:25.701 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:25.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0646413700 1 Processor -- start 2022-07-12T08:50:25.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0646413700 1 -- start start 2022-07-12T08:50:25.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0646413700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f06400b2850 0x7f06401428f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0646413700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06400b43a0 con 0x7f06400b6070 2022-07-12T08:50:25.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0646413700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0640146ef0 con 0x7f06400b3500 2022-07-12T08:50:25.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0646413700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0640147190 con 0x7f06400b2850 2022-07-12T08:50:25.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0645c12700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f06400b2850 0x7f06401428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0645411700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f06400b6070 0x7f06400b2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49436/0 (socket says 172.21.15.61:49436) 2022-07-12T08:50:25.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0645411700 1 -- 172.21.15.61:0/4246150185 learned_addr learned my addr 172.21.15.61:0/4246150185 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:25.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0645c12700 1 -- 172.21.15.61:0/4246150185 >> v1:172.21.15.111:6789/0 conn(0x7f06400b3500 legacy=0x7f06401421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0645c12700 1 -- 172.21.15.61:0/4246150185 >> v1:172.21.15.61:6789/0 conn(0x7f06400b6070 legacy=0x7f06400b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.808+0000 7f0645c12700 1 -- 172.21.15.61:0/4246150185 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06400b6450 con 0x7f06400b2850 2022-07-12T08:50:25.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0645c12700 1 --2- 172.21.15.61:0/4246150185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b2850 0x7f06401428f0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f06380032f0 tx=0x7f06380091d0).ready entity=mon.0 client_cookie=84b50f790343b7a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f062bfff700 1 -- 172.21.15.61:0/4246150185 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0638009ea0 con 0x7f06400b2850 2022-07-12T08:50:25.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f062bfff700 1 -- 172.21.15.61:0/4246150185 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f063800b8a0 con 0x7f06400b2850 2022-07-12T08:50:25.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f062bfff700 1 -- 172.21.15.61:0/4246150185 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f063800ba10 con 0x7f06400b2850 2022-07-12T08:50:25.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- 172.21.15.61:0/4246150185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b2850 msgr2=0x7f06401428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 --2- 172.21.15.61:0/4246150185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b2850 0x7f06401428f0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7f06380032f0 tx=0x7f06380091d0).stop 2022-07-12T08:50:25.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- 172.21.15.61:0/4246150185 shutdown_connections 2022-07-12T08:50:25.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 --2- 172.21.15.61:0/4246150185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b2850 0x7f06401428f0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- 172.21.15.61:0/4246150185 >> 172.21.15.61:0/4246150185 conn(0x7f0640066c10 msgr2=0x7f0640066ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:25.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- 172.21.15.61:0/4246150185 shutdown_connections 2022-07-12T08:50:25.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- 172.21.15.61:0/4246150185 wait complete. 2022-07-12T08:50:25.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 Processor -- start 2022-07-12T08:50:25.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- start start 2022-07-12T08:50:25.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400b2850 0x7f06400b7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 0x7f064013e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b6070 0x7f064013eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0640146960 con 0x7f06400b3500 2022-07-12T08:50:25.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06400ad600 con 0x7f06400b6070 2022-07-12T08:50:25.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0646413700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06400ad8a0 con 0x7f06400b2850 2022-07-12T08:50:25.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.812+0000 7f0644c10700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 0x7f064013e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 0x7f064013e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:32994/0 (socket says 172.21.15.61:32994) 2022-07-12T08:50:25.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 -- 172.21.15.61:0/1246791144 learned_addr learned my addr 172.21.15.61:0/1246791144 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:25.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0645411700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400b2850 0x7f06400b7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0645c12700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b6070 0x7f064013eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 -- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400b2850 msgr2=0x7f06400b7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400b2850 0x7f06400b7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 -- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b6070 msgr2=0x7f064013eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b6070 0x7f064013eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 -- 172.21.15.61:0/1246791144 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638009470 con 0x7f06400b3500 2022-07-12T08:50:25.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0644c10700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 0x7f064013e8b0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7f06400b6800 tx=0x7f0634007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0634012400 con 0x7f06400b3500 2022-07-12T08:50:25.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0634012560 con 0x7f06400b3500 2022-07-12T08:50:25.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06400afbf0 con 0x7f06400b3500 2022-07-12T08:50:25.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06340128c0 con 0x7f06400b3500 2022-07-12T08:50:25.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06400acca0 con 0x7f06400b3500 2022-07-12T08:50:25.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0640146760 con 0x7f06400b3500 2022-07-12T08:50:25.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.816+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0634019030 con 0x7f06400b3500 2022-07-12T08:50:25.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.820+0000 7f0632ffd700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c0539b0 0x7f061c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:25.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.820+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f063406b190 con 0x7f06400b3500 2022-07-12T08:50:25.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.820+0000 7f0645411700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c0539b0 0x7f061c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:25.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.820+0000 7f0645411700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c0539b0 0x7f061c055e70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f063c00d000 tx=0x7f063c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:25.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.824+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06340420f0 con 0x7f06400b3500 2022-07-12T08:50:25.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.944+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0640057460 con 0x7f06400b3500 2022-07-12T08:50:25.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0632ffd700 1 -- 172.21.15.61:0/1246791144 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v128) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0634045710 con 0x7f06400b3500 2022-07-12T08:50:25.950 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c0539b0 msgr2=0x7f061c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c0539b0 0x7f061c055e70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f063c00d000 tx=0x7f063c009040).stop 2022-07-12T08:50:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 msgr2=0x7f064013e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 0x7f064013e8b0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 rx=0x7f06400b6800 tx=0x7f0634007c50).stop 2022-07-12T08:50:25.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 shutdown_connections 2022-07-12T08:50:25.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f061c0539b0 0x7f061c055e70 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06400b6070 0x7f064013eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06400b3500 0x7f064013e8b0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 --2- 172.21.15.61:0/1246791144 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f06400b2850 0x7f06400b7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:25.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 >> 172.21.15.61:0/1246791144 conn(0x7f0640066c10 msgr2=0x7f06400b93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:25.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 shutdown_connections 2022-07-12T08:50:25.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:25.948+0000 7f0646413700 1 -- 172.21.15.61:0/1246791144 wait complete. 2022-07-12T08:50:25.964 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:50:26.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 Processor -- start 2022-07-12T08:50:26.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 -- start start 2022-07-12T08:50:26.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b3500 0x7fa9b40b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa9b40b2850 0x7fa9b41421e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b40b43a0 con 0x7fa9b40b6070 2022-07-12T08:50:26.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b4146ef0 con 0x7fa9b40b2850 2022-07-12T08:50:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b8ac3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4147190 con 0x7fa9b40b3500 2022-07-12T08:50:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa9b40b6070 0x7fa9b40b2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49450/0 (socket says 172.21.15.61:49450) 2022-07-12T08:50:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b37fe700 1 -- 172.21.15.61:0/2628959613 learned_addr learned my addr 172.21.15.61:0/2628959613 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.080+0000 7fa9b2ffd700 1 --2- 172.21.15.61:0/2628959613 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b3500 0x7fa9b40b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b3fff700 1 --2- 172.21.15.61:0/2628959613 >> v2:172.21.15.111:3300/0 conn(0x7fa9b40b2850 0x7fa9b41421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b27fc700 1 -- 172.21.15.61:0/2628959613 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46169441 0 0) 0x7fa9b40b43a0 con 0x7fa9b40b6070 2022-07-12T08:50:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b27fc700 1 -- 172.21.15.61:0/2628959613 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa994003260 con 0x7fa9b40b6070 2022-07-12T08:50:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b2ffd700 1 -- 172.21.15.61:0/2628959613 >> v2:172.21.15.111:3300/0 conn(0x7fa9b40b2850 msgr2=0x7fa9b41421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:26.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b2ffd700 1 --2- 172.21.15.61:0/2628959613 >> v2:172.21.15.111:3300/0 conn(0x7fa9b40b2850 0x7fa9b41421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b2ffd700 1 -- 172.21.15.61:0/2628959613 >> v1:172.21.15.61:6789/0 conn(0x7fa9b40b6070 legacy=0x7fa9b40b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b2ffd700 1 -- 172.21.15.61:0/2628959613 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b40b6450 con 0x7fa9b40b3500 2022-07-12T08:50:26.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b3fff700 1 --2- 172.21.15.61:0/2628959613 >> v2:172.21.15.111:3300/0 conn(0x7fa9b40b2850 0x7fa9b41421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:26.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b2ffd700 1 --2- 172.21.15.61:0/2628959613 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b3500 0x7fa9b40b01c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fa9a4002f70 tx=0x7fa9a4006df0).ready entity=mon.2 client_cookie=19b52b47a384b6d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b27fc700 1 -- 172.21.15.61:0/2628959613 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a400d4a0 con 0x7fa9b40b3500 2022-07-12T08:50:26.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b27fc700 1 -- 172.21.15.61:0/2628959613 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a4000e90 con 0x7fa9b40b3500 2022-07-12T08:50:26.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b27fc700 1 -- 172.21.15.61:0/2628959613 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9a400d7f0 con 0x7fa9b40b3500 2022-07-12T08:50:26.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2628959613 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b3500 msgr2=0x7fa9b40b01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2628959613 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b3500 0x7fa9b40b01c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fa9a4002f70 tx=0x7fa9a4006df0).stop 2022-07-12T08:50:26.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2628959613 shutdown_connections 2022-07-12T08:50:26.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2628959613 >> v2:172.21.15.111:3300/0 conn(0x7fa9b40b2850 0x7fa9b41421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2628959613 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b3500 0x7fa9b40b01c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2628959613 >> 172.21.15.61:0/2628959613 conn(0x7fa9b4066c10 msgr2=0x7fa9b4066ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:26.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2628959613 shutdown_connections 2022-07-12T08:50:26.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.084+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2628959613 wait complete. 2022-07-12T08:50:26.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 Processor -- start 2022-07-12T08:50:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 -- start start 2022-07-12T08:50:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9b40b2850 0x7fa9b4143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b40b3500 0x7fa9b41492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 0x7fa9b41474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4148a30 con 0x7fa9b40b2850 2022-07-12T08:50:26.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b41479c0 con 0x7fa9b40b3500 2022-07-12T08:50:26.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4147c50 con 0x7fa9b40b6070 2022-07-12T08:50:26.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 0x7fa9b41474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9b40b2850 0x7fa9b4143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 0x7fa9b41474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:43436/0 (socket says 172.21.15.61:43436) 2022-07-12T08:50:26.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 -- 172.21.15.61:0/2473426361 learned_addr learned my addr 172.21.15.61:0/2473426361 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:26.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b2ffd700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b40b3500 0x7fa9b41492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 -- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b40b3500 msgr2=0x7fa9b41492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b40b3500 0x7fa9b41492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 -- 172.21.15.61:0/2473426361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9b40b2850 msgr2=0x7fa9b4143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9b40b2850 0x7fa9b4143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 -- 172.21.15.61:0/2473426361 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac009470 con 0x7fa9b40b6070 2022-07-12T08:50:26.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b3fff700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 0x7fa9b41474d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa9b4145d40 tx=0x7fa9ac0091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9ac00c850 con 0x7fa9b40b6070 2022-07-12T08:50:26.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac00c9b0 con 0x7fa9b40b6070 2022-07-12T08:50:26.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a400b180 con 0x7fa9b40b6070 2022-07-12T08:50:26.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9ac00cd10 con 0x7fa9b40b6070 2022-07-12T08:50:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.088+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b4148e50 con 0x7fa9b40b6070 2022-07-12T08:50:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.092+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b4057460 con 0x7fa9b40b6070 2022-07-12T08:50:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.092+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa9ac022070 con 0x7fa9b40b6070 2022-07-12T08:50:26.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.092+0000 7fa9b0ff9700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa998053a40 0x7fa998055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.092+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa9ac06c5f0 con 0x7fa9b40b6070 2022-07-12T08:50:26.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.092+0000 7fa9b37fe700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa998053a40 0x7fa998055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.092+0000 7fa9b37fe700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa998053a40 0x7fa998055f00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fa99c00d000 tx=0x7fa99c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.100+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9ac043550 con 0x7fa9b40b6070 2022-07-12T08:50:26.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.228+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa9b405d950 con 0x7fa9b40b6070 2022-07-12T08:50:26.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.228+0000 7fa9b0ff9700 1 -- 172.21.15.61:0/2473426361 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa9ac016020 con 0x7fa9b40b6070 2022-07-12T08:50:26.232 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:26.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa998053a40 msgr2=0x7fa998055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa998053a40 0x7fa998055f00 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fa99c00d000 tx=0x7fa99c009040).stop 2022-07-12T08:50:26.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 msgr2=0x7fa9b41474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 0x7fa9b41474d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fa9b4145d40 tx=0x7fa9ac0091d0).stop 2022-07-12T08:50:26.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 shutdown_connections 2022-07-12T08:50:26.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9b40b6070 0x7fa9b41474d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa998053a40 0x7fa998055f00 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9b40b3500 0x7fa9b41492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 --2- 172.21.15.61:0/2473426361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9b40b2850 0x7fa9b4143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 >> 172.21.15.61:0/2473426361 conn(0x7fa9b4066c10 msgr2=0x7fa9b40b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:26.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 shutdown_connections 2022-07-12T08:50:26.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.232+0000 7fa9b8ac3700 1 -- 172.21.15.61:0/2473426361 wait complete. 2022-07-12T08:50:26.245 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:50:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 Processor -- start 2022-07-12T08:50:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 -- start start 2022-07-12T08:50:26.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feba80b07e0 0x7feba80b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feba80601f0 0x7feba80b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba81402a0 con 0x7feba80b2b90 2022-07-12T08:50:26.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba80b2670 con 0x7feba80601f0 2022-07-12T08:50:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febadcfa700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba8140510 con 0x7feba80b07e0 2022-07-12T08:50:26.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febad4f9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feba80601f0 0x7feba80b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febaccf8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feba80b2b90 0x7feba81395f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49464/0 (socket says 172.21.15.61:49464) 2022-07-12T08:50:26.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febaccf8700 1 -- 172.21.15.61:0/1235549051 learned_addr learned my addr 172.21.15.61:0/1235549051 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:26.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7feb9ffff700 1 --2- 172.21.15.61:0/1235549051 >> v2:172.21.15.111:3300/0 conn(0x7feba80b07e0 0x7feba80b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febad4f9700 1 -- 172.21.15.61:0/1235549051 >> v2:172.21.15.111:3300/0 conn(0x7feba80b07e0 msgr2=0x7feba80b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:26.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febad4f9700 1 --2- 172.21.15.61:0/1235549051 >> v2:172.21.15.111:3300/0 conn(0x7feba80b07e0 0x7feba80b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febad4f9700 1 -- 172.21.15.61:0/1235549051 >> v1:172.21.15.61:6789/0 conn(0x7feba80b2b90 legacy=0x7feba81395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.364+0000 7febad4f9700 1 -- 172.21.15.61:0/1235549051 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba81407b0 con 0x7feba80601f0 2022-07-12T08:50:26.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febad4f9700 1 --2- 172.21.15.61:0/1235549051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80601f0 0x7feba80b14f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7feba00032f0 tx=0x7feba00091d0).ready entity=mon.0 client_cookie=36677df12f5db76a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7feb9f7fe700 1 -- 172.21.15.61:0/1235549051 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feba0009ea0 con 0x7feba80601f0 2022-07-12T08:50:26.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7feb9f7fe700 1 -- 172.21.15.61:0/1235549051 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba000b870 con 0x7feba80601f0 2022-07-12T08:50:26.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7feb9f7fe700 1 -- 172.21.15.61:0/1235549051 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feba000b9e0 con 0x7feba80601f0 2022-07-12T08:50:26.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 -- 172.21.15.61:0/1235549051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80601f0 msgr2=0x7feba80b14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 --2- 172.21.15.61:0/1235549051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80601f0 0x7feba80b14f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7feba00032f0 tx=0x7feba00091d0).stop 2022-07-12T08:50:26.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 -- 172.21.15.61:0/1235549051 shutdown_connections 2022-07-12T08:50:26.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 --2- 172.21.15.61:0/1235549051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80601f0 0x7feba80b14f0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 --2- 172.21.15.61:0/1235549051 >> v2:172.21.15.111:3300/0 conn(0x7feba80b07e0 0x7feba80b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 -- 172.21.15.61:0/1235549051 >> 172.21.15.61:0/1235549051 conn(0x7feba8063970 msgr2=0x7feba8063d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 -- 172.21.15.61:0/1235549051 shutdown_connections 2022-07-12T08:50:26.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 -- 172.21.15.61:0/1235549051 wait complete. 2022-07-12T08:50:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 Processor -- start 2022-07-12T08:50:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.368+0000 7febadcfa700 1 -- start start 2022-07-12T08:50:26.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feba80601f0 0x7feba8139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feba80b07e0 0x7feba8140d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 0x7feba8141220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80af890 con 0x7feba80b2b90 2022-07-12T08:50:26.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba80aeeb0 con 0x7feba80601f0 2022-07-12T08:50:26.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba80af150 con 0x7feba80b07e0 2022-07-12T08:50:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 0x7feba8141220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 0x7feba8141220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33024/0 (socket says 172.21.15.61:33024) 2022-07-12T08:50:26.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 -- 172.21.15.61:0/3429507376 learned_addr learned my addr 172.21.15.61:0/3429507376 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:26.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7feb9ffff700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feba80b07e0 0x7feba8140d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febaccf8700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feba80601f0 0x7feba8139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 -- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feba80b07e0 msgr2=0x7feba8140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feba80b07e0 0x7feba8140d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 -- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feba80601f0 msgr2=0x7feba8139750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feba80601f0 0x7feba8139750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 -- 172.21.15.61:0/3429507376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0009470 con 0x7feba80b2b90 2022-07-12T08:50:26.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febaccf8700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feba80601f0 0x7feba8139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:26.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febad4f9700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 0x7feba8141220 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7feba00032f0 tx=0x7feba0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feba0011ca0 con 0x7feba80b2b90 2022-07-12T08:50:26.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba0011e00 con 0x7feba80b2b90 2022-07-12T08:50:26.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba80af3f0 con 0x7feba80b2b90 2022-07-12T08:50:26.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feba001f5e0 con 0x7feba80b2b90 2022-07-12T08:50:26.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba80aea10 con 0x7feba80b2b90 2022-07-12T08:50:26.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.372+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba80aed20 con 0x7feba80b2b90 2022-07-12T08:50:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.376+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feba0026030 con 0x7feba80b2b90 2022-07-12T08:50:26.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.376+0000 7feb9dffb700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feb880539b0 0x7feb88055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.376+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feba006cbc0 con 0x7feba80b2b90 2022-07-12T08:50:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.376+0000 7febaccf8700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feb880539b0 0x7feb88055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.376+0000 7febaccf8700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feb880539b0 0x7feb88055e70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7feba400b000 tx=0x7feba4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.384+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feba0043a10 con 0x7feba80b2b90 2022-07-12T08:50:26.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feba8131e40 con 0x7feba80b2b90 2022-07-12T08:50:26.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7feb9dffb700 1 -- 172.21.15.61:0/3429507376 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+10 (secure 0 0 0) 0x7feba8131e40 con 0x7feba80b2b90 2022-07-12T08:50:26.512 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:26.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feb880539b0 msgr2=0x7feb88055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feb880539b0 0x7feb88055e70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7feba400b000 tx=0x7feba4009040).stop 2022-07-12T08:50:26.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 msgr2=0x7feba8141220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 0x7feba8141220 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 rx=0x7feba00032f0 tx=0x7feba0016000).stop 2022-07-12T08:50:26.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 shutdown_connections 2022-07-12T08:50:26.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feba80b2b90 0x7feba8141220 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feba80b07e0 0x7feba8140d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feb880539b0 0x7feb88055e70 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 --2- 172.21.15.61:0/3429507376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feba80601f0 0x7feba8139750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 >> 172.21.15.61:0/3429507376 conn(0x7feba8063970 msgr2=0x7feba813f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:26.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 shutdown_connections 2022-07-12T08:50:26.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.508+0000 7febadcfa700 1 -- 172.21.15.61:0/3429507376 wait complete. 2022-07-12T08:50:26.528 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:50:26.529 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:26.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44ebe7f700 1 Processor -- start 2022-07-12T08:50:26.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44ebe7f700 1 -- start start 2022-07-12T08:50:26.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44ebe7f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f44e40afca0 0x7f44e414a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44ebe7f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40b0c70 con 0x7f44e40b0710 2022-07-12T08:50:26.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44ebe7f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40b2b90 con 0x7f44e40b27b0 2022-07-12T08:50:26.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44ebe7f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e40b2290 con 0x7f44e40afca0 2022-07-12T08:50:26.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44e941a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f44e40b0710 0x7f44e40b1460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49478/0 (socket says 172.21.15.61:49478) 2022-07-12T08:50:26.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.640+0000 7f44e941a700 1 -- 172.21.15.61:0/2234326122 learned_addr learned my addr 172.21.15.61:0/2234326122 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:26.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ea41c700 1 --2- 172.21.15.61:0/2234326122 >> v2:172.21.15.111:3300/0 conn(0x7f44e40afca0 0x7f44e414a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44e8c19700 1 -- 172.21.15.61:0/2234326122 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065924254 0 0) 0x7f44e40b0c70 con 0x7f44e40b0710 2022-07-12T08:50:26.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44e8c19700 1 -- 172.21.15.61:0/2234326122 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc003260 con 0x7f44e40b0710 2022-07-12T08:50:26.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ea41c700 1 -- 172.21.15.61:0/2234326122 >> v1:172.21.15.111:6789/0 conn(0x7f44e40b27b0 legacy=0x7f44e40ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ea41c700 1 -- 172.21.15.61:0/2234326122 >> v1:172.21.15.61:6789/0 conn(0x7f44e40b0710 legacy=0x7f44e40b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ea41c700 1 -- 172.21.15.61:0/2234326122 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e40b0ea0 con 0x7f44e40afca0 2022-07-12T08:50:26.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ea41c700 1 --2- 172.21.15.61:0/2234326122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40afca0 0x7f44e414a0b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f44dc0032f0 tx=0x7f44dc0091d0).ready entity=mon.1 client_cookie=96576aff3108121d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44e8c19700 1 -- 172.21.15.61:0/2234326122 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44dc00b6c0 con 0x7f44e40afca0 2022-07-12T08:50:26.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44e8c19700 1 -- 172.21.15.61:0/2234326122 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44dc00b820 con 0x7f44e40afca0 2022-07-12T08:50:26.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44e8c19700 1 -- 172.21.15.61:0/2234326122 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44dc00b990 con 0x7f44e40afca0 2022-07-12T08:50:26.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 -- 172.21.15.61:0/2234326122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40afca0 msgr2=0x7f44e414a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/2234326122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40afca0 0x7f44e414a0b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f44dc0032f0 tx=0x7f44dc0091d0).stop 2022-07-12T08:50:26.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 -- 172.21.15.61:0/2234326122 shutdown_connections 2022-07-12T08:50:26.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/2234326122 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40afca0 0x7f44e414a0b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 -- 172.21.15.61:0/2234326122 >> 172.21.15.61:0/2234326122 conn(0x7f44e4063170 msgr2=0x7f44e4063580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:26.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 -- 172.21.15.61:0/2234326122 shutdown_connections 2022-07-12T08:50:26.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 -- 172.21.15.61:0/2234326122 wait complete. 2022-07-12T08:50:26.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 Processor -- start 2022-07-12T08:50:26.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.644+0000 7f44ebe7f700 1 -- start start 2022-07-12T08:50:26.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e40afca0 0x7f44e41477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 0x7f44e4144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44e40b27b0 0x7f44e4065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4144220 con 0x7f44e40afca0 2022-07-12T08:50:26.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4143cc0 con 0x7f44e40b0710 2022-07-12T08:50:26.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4144590 con 0x7f44e40b27b0 2022-07-12T08:50:26.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e9c1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e40afca0 0x7f44e41477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 0x7f44e4144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e9c1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e40afca0 0x7f44e41477e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33034/0 (socket says 172.21.15.61:33034) 2022-07-12T08:50:26.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 0x7f44e4144ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:39844/0 (socket says 172.21.15.61:39844) 2022-07-12T08:50:26.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e9c1b700 1 -- 172.21.15.61:0/3702507936 learned_addr learned my addr 172.21.15.61:0/3702507936 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:26.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ea41c700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44e40b27b0 0x7f44e4065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 -- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44e40b27b0 msgr2=0x7f44e4065730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44e40b27b0 0x7f44e4065730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 -- 172.21.15.61:0/3702507936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e40afca0 msgr2=0x7f44e41477e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e40afca0 0x7f44e41477e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 -- 172.21.15.61:0/3702507936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc009470 con 0x7f44e40b0710 2022-07-12T08:50:26.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44e941a700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 0x7f44e4144ae0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f44e0001110 tx=0x7f44e0007ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e00070d0 con 0x7f44e40b0710 2022-07-12T08:50:26.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e4144fd0 con 0x7f44e40b0710 2022-07-12T08:50:26.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0007230 con 0x7f44e40b0710 2022-07-12T08:50:26.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e41414c0 con 0x7f44e40b0710 2022-07-12T08:50:26.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e00073a0 con 0x7f44e40b0710 2022-07-12T08:50:26.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e4144750 con 0x7f44e40b0710 2022-07-12T08:50:26.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.648+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f44e0010070 con 0x7f44e40b0710 2022-07-12T08:50:26.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.652+0000 7f44d2ffd700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f44c4053920 0x7f44c4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:26.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.652+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f44e006b440 con 0x7f44e40b0710 2022-07-12T08:50:26.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.652+0000 7f44e9c1b700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f44c4053920 0x7f44c4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:26.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.652+0000 7f44e9c1b700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f44c4053920 0x7f44c4055de0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f44d400d000 tx=0x7f44d4007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:26.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.660+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44e00422c0 con 0x7f44e40b0710 2022-07-12T08:50:26.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.828+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44e4141740 con 0x7f44e40b0710 2022-07-12T08:50:26.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.828+0000 7f44d2ffd700 1 -- 172.21.15.61:0/3702507936 <== mon.1 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f44e00458e0 con 0x7f44e40b0710 2022-07-12T08:50:26.834 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:26.835 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:10.807048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8082":{"gid":8082,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2724832679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2724832679},{"type":"v1","addr":"172.21.15.61:6833","nonce":2724832679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:26.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f44c4053920 msgr2=0x7f44c4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f44c4053920 0x7f44c4055de0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f44d400d000 tx=0x7f44d4007c00).stop 2022-07-12T08:50:26.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 msgr2=0x7f44e4144ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:26.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 0x7f44e4144ae0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f44e0001110 tx=0x7f44e0007ac0).stop 2022-07-12T08:50:26.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 shutdown_connections 2022-07-12T08:50:26.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f44e40b27b0 0x7f44e4065730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f44c4053920 0x7f44c4055de0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44e40b0710 0x7f44e4144ae0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 --2- 172.21.15.61:0/3702507936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e40afca0 0x7f44e41477e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:26.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 >> 172.21.15.61:0/3702507936 conn(0x7f44e4063170 msgr2=0x7f44e40b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:26.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 shutdown_connections 2022-07-12T08:50:26.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:26.832+0000 7f44ebe7f700 1 -- 172.21.15.61:0/3702507936 wait complete. 2022-07-12T08:50:26.844 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-12T08:50:27.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:09.803335+0000', 'modified': '2022-07-12T08:50:10.807048+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8082}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8082': {'gid': 8082, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2724832679', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2724832679}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2724832679}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:27.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:27.051 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:27.171 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8082,"want_state":"up:active","state":"up:active","rank_uptime":17.353968037000001,"mdsmap_epoch":72,"osdmap_epoch":128,"osdmap_epoch_barrier":128,"uptime":21.730139456} 2022-07-12T08:50:27.172 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8082, "mdsmap_epoch": 72, "osdmap_epoch": 128, "osdmap_epoch_barrier": 128, "rank_uptime": 17.353968037, "state": "up:active", "uptime": 21.730139456, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:27.173 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:50:27.174 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:50:27.174 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:50:27.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.176 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:27.193 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:27.217 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:27.218 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:27.277 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:27.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.282 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:27.296 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:50:27.297 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:27.341 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:27.359 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:27.360 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:27.374 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:27.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.378 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:27.415 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:50:27.409+0000 7f5090185080 -1 init, newargv = 0x56376a92f890 newargc=9ceph-fuse[21653 2022-07-12T08:50:27.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:]: starting ceph client 2022-07-12T08:50:27.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21653]: starting fuse 2022-07-12T08:50:27.428 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:50:27.429 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:50:27.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.432 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:50:27.432 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:50:27.433 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:50:27.433 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:50:27.434 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:50:27.435 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:27.436 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:50:27.437 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:50:27.439 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:50:27.440 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:27.440 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:50:27.442 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:50:27.442 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:50:27.443 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:50:27.444 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:27.445 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:50:27.446 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:50:27.447 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:50:27.448 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:50:27.450 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:50:27.451 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:27.452 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:50:27.452 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:50:27.526 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21653.asok 2022-07-12T08:50:27.532 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21653.asok 2022-07-12T08:50:27.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.539 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21653.asok status 2022-07-12T08:50:27.645 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:50:27.646 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:50:27.646 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:50:27.646 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:50:27.647 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:50:27.647 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:50:27.647 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:50:27.647 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21653", 2022-07-12T08:50:27.648 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:50:27.648 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:27.648 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:50:27.648 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:50:27.649 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 8140, 2022-07-12T08:50:27.649 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:50:27.649 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:50:27.650 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:50:27.650 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 8140 2022-07-12T08:50:27.651 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:27.651 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:50:27.651 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:50:27.651 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:50:27.652 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2156289645 2022-07-12T08:50:27.652 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:50:27.652 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:27.652 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:50:27.653 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:50:27.653 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:50:27.653 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2156289645 2022-07-12T08:50:27.653 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:27.654 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.8140 172.21.15.130:0/2156289645", 2022-07-12T08:50:27.654 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/2156289645", 2022-07-12T08:50:27.654 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:50:27.654 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 72, 2022-07-12T08:50:27.655 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 128, 2022-07-12T08:50:27.655 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:50:27.655 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:50:27.655 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:50:27.656 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:50:27.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.657 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:27.673 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:27.687 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:27.687 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:27.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.689 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:27.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:27.710 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:27.711 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:27.721 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:50:27.721 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:27.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-12T08:50:27.723 DEBUG:teuthology.orchestra.run.smithi121:> import time 2022-07-12T08:50:27.724 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:27.724 DEBUG:teuthology.orchestra.run.smithi121:> with open("/home/ubuntu/cephtest/mnt.0/testfile", '"'"'w'"'"') as f: 2022-07-12T08:50:27.725 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content'"'"') 2022-07-12T08:50:27.726 DEBUG:teuthology.orchestra.run.smithi121:> f.flush() 2022-07-12T08:50:27.726 DEBUG:teuthology.orchestra.run.smithi121:> f.write('"'"'content2'"'"') 2022-07-12T08:50:27.727 DEBUG:teuthology.orchestra.run.smithi121:> while True: 2022-07-12T08:50:27.728 DEBUG:teuthology.orchestra.run.smithi121:> time.sleep(1) 2022-07-12T08:50:27.728 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:50:27.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-12T08:50:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-12T08:50:27.796 INFO:teuthology.orchestra.run.smithi121.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/testfile': No such file or directory 2022-07-12T08:50:28.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-12T08:50:28.815 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.0/testfile 2022-07-12T08:50:28.817 DEBUG:tasks.cephfs.mount:File testfile became visible from 0 after 1s 2022-07-12T08:50:28.818 DEBUG:teuthology.orchestra.run.smithi130:> sudo ls /home/ubuntu/cephtest/mnt.1/testfile 2022-07-12T08:50:28.854 INFO:teuthology.orchestra.run.smithi130.stdout:/home/ubuntu/cephtest/mnt.1/testfile 2022-07-12T08:50:28.856 DEBUG:tasks.cephfs.mount:File testfile became visible from 1 after 0s 2022-07-12T08:50:28.858 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-12T08:50:28.858 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest/mnt.1 && sudo cat testfile 2022-07-12T08:50:28.937 INFO:teuthology.orchestra.run.smithi130.stdout:content 2022-07-12T08:50:28.953 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:50:28.953 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:28.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:29.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:29.024 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:29.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:29.079 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:50:29.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:29.082 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:29.083 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:29.095 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:50:29.096 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:29.096 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:50:29.097 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:50:29.098 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:29.154 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22144]: fuse finished with error 0 and tester_r 0 2022-07-12T08:50:29.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:29.157 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:29.157 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:29.163 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:50:29.164 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:50:29.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:29.284 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:29.297 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:29.298 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:29.298 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:29.312 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:29.313 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:29.314 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:50:29.315 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:50:29.316 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:29.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21653]: fuse finished with error 0 and tester_r 0 2022-07-12T08:50:29.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:29.374 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:29.374 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:29.389 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:50:29.391 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:50:29.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:29.411 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:29.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:29.463 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:29.463 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:50:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:29.474 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:50:29.476 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:50:29.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:29.477 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:50:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:29.537 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:50:29.537 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:50:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 Processor -- start 2022-07-12T08:50:29.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- start start 2022-07-12T08:50:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b9eb0 0x7f9958142140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99580b08e0 0x7f995805bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:29.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580b2ba0 con 0x7f99580b34a0 2022-07-12T08:50:29.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99580b5c30 con 0x7f99580b08e0 2022-07-12T08:50:29.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99580ba290 con 0x7f99580b9eb0 2022-07-12T08:50:29.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9955d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f99580b34a0 0x7f9958142630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49492/0 (socket says 172.21.15.61:49492) 2022-07-12T08:50:29.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9955d9b700 1 -- 172.21.15.61:0/591733121 learned_addr learned my addr 172.21.15.61:0/591733121 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:29.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9956d9d700 1 --2- 172.21.15.61:0/591733121 >> v2:172.21.15.61:3300/0 conn(0x7f99580b08e0 0x7f995805bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:29.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995659c700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b9eb0 0x7f9958142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:29.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995559a700 1 -- 172.21.15.61:0/591733121 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507741132 0 0) 0x7f99580b2ba0 con 0x7f99580b34a0 2022-07-12T08:50:29.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995559a700 1 -- 172.21.15.61:0/591733121 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f993c003260 con 0x7f99580b34a0 2022-07-12T08:50:29.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9956d9d700 1 -- 172.21.15.61:0/591733121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b9eb0 msgr2=0x7f9958142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9956d9d700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b9eb0 0x7f9958142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9956d9d700 1 -- 172.21.15.61:0/591733121 >> v1:172.21.15.61:6789/0 conn(0x7f99580b34a0 legacy=0x7f9958142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9956d9d700 1 -- 172.21.15.61:0/591733121 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580b3e00 con 0x7f99580b08e0 2022-07-12T08:50:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995659c700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b9eb0 0x7f9958142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:29.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f9956d9d700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b08e0 0x7f995805bdb0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7f99500032f0 tx=0x7f99500091d0).ready entity=mon.0 client_cookie=ec2110c8ad2937cd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:29.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995559a700 1 -- 172.21.15.61:0/591733121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995000b6c0 con 0x7f99580b08e0 2022-07-12T08:50:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995559a700 1 -- 172.21.15.61:0/591733121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995000b820 con 0x7f99580b08e0 2022-07-12T08:50:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995559a700 1 -- 172.21.15.61:0/591733121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995000b990 con 0x7f99580b08e0 2022-07-12T08:50:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- 172.21.15.61:0/591733121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b08e0 msgr2=0x7f995805bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b08e0 0x7f995805bdb0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7f99500032f0 tx=0x7f99500091d0).stop 2022-07-12T08:50:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- 172.21.15.61:0/591733121 shutdown_connections 2022-07-12T08:50:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b08e0 0x7f995805bdb0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 --2- 172.21.15.61:0/591733121 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b9eb0 0x7f9958142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- 172.21.15.61:0/591733121 >> 172.21.15.61:0/591733121 conn(0x7f99580676e0 msgr2=0x7f99580b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- 172.21.15.61:0/591733121 shutdown_connections 2022-07-12T08:50:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.656+0000 7f995759e700 1 -- 172.21.15.61:0/591733121 wait complete. 2022-07-12T08:50:29.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 Processor -- start 2022-07-12T08:50:29.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 -- start start 2022-07-12T08:50:29.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b08e0 0x7f9958146970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:29.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 0x7f995813f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:29.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99580b9eb0 0x7f995813fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:29.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99580afcf0 con 0x7f99580b34a0 2022-07-12T08:50:29.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99580ae520 con 0x7f99580b9eb0 2022-07-12T08:50:29.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9958112760 con 0x7f99580b08e0 2022-07-12T08:50:29.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 0x7f995813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:29.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 0x7f995813f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33050/0 (socket says 172.21.15.61:33050) 2022-07-12T08:50:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 -- 172.21.15.61:0/3846298401 learned_addr learned my addr 172.21.15.61:0/3846298401 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995659c700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b08e0 0x7f9958146970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9956d9d700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99580b9eb0 0x7f995813fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 -- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b08e0 msgr2=0x7f9958146970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b08e0 0x7f9958146970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 -- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99580b9eb0 msgr2=0x7f995813fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99580b9eb0 0x7f995813fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 -- 172.21.15.61:0/3846298401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9950009470 con 0x7f99580b34a0 2022-07-12T08:50:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f9955d9b700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 0x7f995813f670 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f9948001110 tx=0x7f9948007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99480072e0 con 0x7f99580b34a0 2022-07-12T08:50:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948007440 con 0x7f99580b34a0 2022-07-12T08:50:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9958112a00 con 0x7f99580b34a0 2022-07-12T08:50:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9948005800 con 0x7f99580b34a0 2022-07-12T08:50:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.660+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99580ad770 con 0x7f99580b34a0 2022-07-12T08:50:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.664+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f994801a070 con 0x7f99580b34a0 2022-07-12T08:50:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.664+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9958112b90 con 0x7f99580b34a0 2022-07-12T08:50:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.664+0000 7f99477fe700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9934053a40 0x7f9934055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:29.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.664+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f994806b340 con 0x7f99580b34a0 2022-07-12T08:50:29.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.664+0000 7f995659c700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9934053a40 0x7f9934055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:29.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.664+0000 7f995659c700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9934053a40 0x7f9934055f00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f994c00c000 tx=0x7f994c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:29.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.672+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99480421a0 con 0x7f99580b34a0 2022-07-12T08:50:29.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.796+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f9958057460 con 0x7f99580b34a0 2022-07-12T08:50:29.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.796+0000 7f99477fe700 1 -- 172.21.15.61:0/3846298401 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v128) v1 ==== 81+0+33707 (secure 0 0 0) 0x7f99480457c0 con 0x7f99580b34a0 2022-07-12T08:50:29.801 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:29.801 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:50:29.801 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 128, 2022-07-12T08:50:29.802 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:50:29.802 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:50:29.802 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:50:10.806045+0000", 2022-07-12T08:50:29.802 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:29.803 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:29.803 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:50:29.803 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:50:29.803 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:50:29.804 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:50:29.804 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:50:29.804 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:50:29.804 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:50:29.805 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:29.805 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:50:29.805 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:50:29.805 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:50:29.806 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:50:29.807 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:50:29.807 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 26, 2022-07-12T08:50:29.807 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:50:29.807 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:50:29.807 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:50:29.808 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:50:29.808 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:50:29.808 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.808 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:50:29.809 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:50:29.809 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:50:29.809 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:29.810 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:29.810 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:29.810 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:29.810 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:29.810 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:29.811 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:29.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:29.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:50:29.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:50:29.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:50:29.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:50:29.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:50:29.812 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:29.812 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:29.813 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:29.813 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:29.813 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:29.813 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:29.814 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:29.814 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:50:29.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:29.815 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:29.815 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:29.815 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:29.815 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:29.815 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:29.816 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:29.816 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:29.816 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:29.816 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:29.817 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:29.817 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:29.817 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:29.817 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:29.817 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:29.818 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:29.818 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:29.818 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:29.818 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:29.819 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:29.819 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:29.819 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:29.819 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:29.820 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:29.820 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:29.821 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:29.821 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.821 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:29.822 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:29.822 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:29.822 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:29.822 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:29.823 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:29.823 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:29.823 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:29.823 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:29.823 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:29.825 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:29.825 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:50:29.826 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:50:29.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.826 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:29.827 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:50:29.827 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.828 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.828 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:50:29.828 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:50:29.829 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:50:29.829 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:29.829 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:29.829 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:29.830 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:29.830 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:29.830 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:29.831 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:29.831 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:29.831 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:29.832 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:29.832 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:29.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:29.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:29.834 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:29.834 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:29.834 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:29.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:29.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:29.835 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:29.836 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:29.836 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.837 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:50:29.837 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:29.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:29.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:29.838 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:29.838 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:29.839 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:29.839 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:29.839 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:29.840 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:29.840 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:29.840 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:29.841 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:29.842 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:29.843 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:29.843 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:29.843 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:29.844 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:29.844 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:29.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:29.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:29.846 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:29.846 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:29.847 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:29.847 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:29.848 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:29.848 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:29.848 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.849 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:29.849 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:29.849 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:29.850 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:29.850 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:29.850 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:29.851 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:29.851 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:29.852 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:29.852 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:29.852 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:29.853 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:50:29.853 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:29.854 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:50:29.854 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.855 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.855 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.856 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 25, 2022-07-12T08:50:29.856 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:50:29.856 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:50:08.408495+0000", 2022-07-12T08:50:29.857 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:29.857 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:29.858 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:29.858 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:29.858 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:29.859 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:29.859 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:29.859 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:29.860 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:29.860 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:29.860 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:29.861 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:29.861 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:29.862 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:29.862 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:29.862 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:29.862 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:29.863 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:29.863 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:29.863 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:29.863 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.864 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "127", 2022-07-12T08:50:29.864 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:29.864 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:29.864 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:29.865 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:29.865 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:29.865 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:29.865 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:29.866 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:29.866 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:29.867 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:29.867 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:29.868 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:29.868 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:29.868 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:29.869 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:29.869 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:29.869 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:29.870 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:29.870 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:29.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:29.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:29.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:29.872 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:29.872 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:29.873 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:29.873 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:29.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.874 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:29.874 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:29.874 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:29.874 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:29.875 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:29.875 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:29.875 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:29.876 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:29.876 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:29.876 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:29.876 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:29.877 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:50:29.877 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:50:29.877 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:50:29.877 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:50:29.878 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.878 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:29.879 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:50:29.879 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:50:29.879 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.880 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.881 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.881 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.881 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 26, 2022-07-12T08:50:29.882 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:50:29.882 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:50:08.818782+0000", 2022-07-12T08:50:29.894 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:29.895 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:29.895 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:29.895 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:29.896 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:29.897 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:29.897 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:29.898 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:29.898 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:29.906 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:29.906 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "128", 2022-07-12T08:50:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:29.910 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:29.910 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:29.911 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:29.911 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:29.912 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:29.912 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:29.912 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:29.912 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:29.913 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:29.913 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:29.913 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:29.913 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:29.913 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:29.914 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:29.914 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:29.915 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:29.915 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:29.915 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:29.916 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:29.917 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:29.917 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:29.918 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:29.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:29.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.919 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:29.920 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:29.920 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:29.921 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:29.921 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:29.922 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:29.922 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:29.922 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:29.923 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:29.923 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:29.924 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:29.924 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:50:29.925 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:29.925 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:50:29.925 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:50:29.926 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.926 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.927 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.927 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:29.927 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:50:29.928 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.928 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:29.929 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:50:29.929 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:29.929 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:29.930 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:29.930 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:29.931 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:29.931 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:29.932 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:29.932 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 125, 2022-07-12T08:50:29.932 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:29.933 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:29.933 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:29.934 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.934 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.935 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.935 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:50:29.935 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.936 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.936 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.937 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.937 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:50:29.937 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.938 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.938 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.939 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.939 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:29.939 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.940 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.940 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.940 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:50:29.941 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.941 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.941 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.942 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.942 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:50:29.942 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.943 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.943 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.943 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.944 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:29.944 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.944 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.944 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.945 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:50:29.945 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.945 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.945 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.946 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.946 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:50:29.946 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.947 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.947 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.947 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.947 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:29.947 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.948 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.948 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:50:29.949 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.949 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.949 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.949 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.950 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:50:29.950 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:29.950 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.950 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.951 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.951 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:50:29.951 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:50:29.951 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:50:29.952 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:50:29.952 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:29.952 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:29.952 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:29.952 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.953 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.953 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.953 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:29.954 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:50:29.954 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:29.955 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:29.955 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:29.956 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:29.956 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:29.956 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:29.957 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:29.957 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:29.958 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:29.958 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:29.958 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:29.959 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.959 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.959 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.960 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:50:29.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.961 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.961 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.962 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.962 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:50:29.962 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.963 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.963 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.964 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.964 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:29.964 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.965 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.965 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.965 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:50:29.966 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.966 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.967 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.967 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.967 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:50:29.968 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.968 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.969 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.969 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.969 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:29.970 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.970 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.971 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.971 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:50:29.971 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.972 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.972 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.973 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.973 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:50:29.973 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.974 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.974 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.975 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.975 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:29.975 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.976 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.976 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.976 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:50:29.977 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.977 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.978 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.979 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:50:29.979 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:29.979 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.980 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.980 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.981 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:50:29.981 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:50:29.982 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:50:29.982 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:50:29.983 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:29.983 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:29.983 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:29.984 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.985 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.985 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:29.986 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:50:29.986 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:29.987 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:29.987 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:29.988 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:29.988 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:29.989 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:29.989 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:29.990 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:29.990 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:29.991 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:29.991 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:29.992 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.992 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.993 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:29.993 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:50:29.993 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:29.994 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.994 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.995 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:29.995 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:50:29.996 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:29.996 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:29.997 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:29.997 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:29.998 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:29.998 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:29.999 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:29.999 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.000 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:50:30.001 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:30.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.001 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.001 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.002 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:50:30.002 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:30.002 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.003 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.003 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.003 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:30.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.004 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:50:30.005 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:30.005 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.005 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.005 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:50:30.006 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:30.006 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.006 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.006 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.007 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:30.007 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.007 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.007 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:50:30.008 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:30.008 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.008 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.009 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:50:30.009 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:30.009 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.009 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.010 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.010 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:50:30.010 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:50:30.010 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:50:30.011 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:50:30.011 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:30.011 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:30.011 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:30.012 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.012 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.012 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:30.013 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:50:30.013 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:30.013 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:30.013 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:30.014 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:30.014 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:30.014 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:30.015 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:30.015 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 125, 2022-07-12T08:50:30.015 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:30.015 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:30.015 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:30.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:50:30.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.017 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.018 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:50:30.018 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.019 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.019 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.019 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:30.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.020 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.021 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.021 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:50:30.021 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.022 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.022 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.022 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:50:30.023 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.023 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.024 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.024 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.025 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:30.025 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.026 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.026 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.027 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:50:30.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.028 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.028 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.029 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:50:30.029 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.030 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.030 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.031 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:30.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.032 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.033 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:50:30.033 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.034 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.035 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:50:30.036 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:30.036 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.037 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.038 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:50:30.038 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:50:30.039 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:50:30.039 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:50:30.040 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:30.040 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:30.041 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:30.041 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.043 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:30.043 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:50:30.043 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:30.044 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:30.044 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:30.045 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:30.045 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:30.046 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:30.046 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:30.047 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:30.047 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:30.047 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:30.048 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:30.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:50:30.050 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.051 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:50:30.052 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.053 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.053 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.054 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.054 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:30.055 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.055 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.055 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:50:30.056 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.057 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:50:30.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.059 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.059 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.060 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:30.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.061 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:50:30.062 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:50:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:30.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:50:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.068 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:50:30.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:30.070 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.070 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:50:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:50:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:50:30.072 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:50:30.072 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:30.072 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.074 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:50:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:30.077 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:30.077 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:30.077 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:30.078 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:30.078 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:30.079 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:30.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:50:30.081 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.082 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:50:30.083 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.083 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.084 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.086 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:30.086 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:50:30.088 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.088 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.089 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.090 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:50:30.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.090 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.091 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.092 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:30.092 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.093 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.093 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:50:30.094 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.094 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.095 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.095 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.095 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:50:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:30.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:50:30.098 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.098 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.098 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.098 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:50:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.100 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:50:30.100 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:50:30.100 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:50:30.100 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:50:30.101 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:30.101 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:30.101 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:30.101 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:50:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:30.105 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:30.105 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.105 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:50:30.106 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.106 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.107 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:50:30.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.107 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.108 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.108 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:30.108 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:50:30.109 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.109 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.110 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.110 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.110 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:50:30.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.111 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.111 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:30.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.112 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:50:30.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.113 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.113 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:50:30.114 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.114 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.114 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.115 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.115 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:30.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.116 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.117 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.117 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:50:30.118 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.118 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.118 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.119 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:50:30.120 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:30.120 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.121 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.122 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:50:30.122 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:50:30.122 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:50:30.123 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:50:30.123 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:30.124 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:30.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:30.124 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.125 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.125 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:30.126 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:50:30.126 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:30.127 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:30.127 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:30.127 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:30.128 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:30.128 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:30.128 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:30.129 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:30.129 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:30.129 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:30.130 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:30.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:50:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:50:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.134 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.134 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:50:30.134 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:50:30.136 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.136 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.136 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.137 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:30.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:50:30.138 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.138 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:50:30.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.140 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:30.141 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:30.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:30.141 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:50:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:30.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:50:30.143 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:30.143 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.144 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:50:30.144 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:50:30.144 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:50:30.145 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:50:30.145 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:30.145 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:30.145 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:50:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.147 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:30.147 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.147 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.147 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:50:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.149 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.149 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:30.149 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.149 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.150 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.150 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.150 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.150 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:50:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.152 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.152 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.152 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.152 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:50:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.154 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:30.154 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.154 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.154 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.155 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.155 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.155 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:50:30.155 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.157 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.157 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.157 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.157 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.158 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:50:30.158 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.158 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.158 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.159 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:30.159 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.159 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.159 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:50:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:50:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:50:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:50:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:50:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:50:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:50:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:50:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:50:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:50:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:50:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:50:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:50:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:50:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:50:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:50:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:50:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:50:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:50:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:50:30.171 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:50:30.171 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:50:30.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9934053a40 msgr2=0x7f9934055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9934053a40 0x7f9934055f00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f994c00c000 tx=0x7f994c008040).stop 2022-07-12T08:50:30.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 msgr2=0x7f995813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 0x7f995813f670 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f9948001110 tx=0x7f9948007b20).stop 2022-07-12T08:50:30.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 shutdown_connections 2022-07-12T08:50:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9934053a40 0x7f9934055f00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99580b9eb0 0x7f995813fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99580b34a0 0x7f995813f670 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 --2- 172.21.15.61:0/3846298401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99580b08e0 0x7f9958146970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 >> 172.21.15.61:0/3846298401 conn(0x7f99580676e0 msgr2=0x7f99580b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:30.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 shutdown_connections 2022-07-12T08:50:30.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:29.800+0000 7f995759e700 1 -- 172.21.15.61:0/3846298401 wait complete. 2022-07-12T08:50:30.177 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:30.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 Processor -- start 2022-07-12T08:50:30.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 -- start start 2022-07-12T08:50:30.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300ad1b0 0x7ff4301434e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff4300b05e0 0x7ff4300b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4300aebc0 con 0x7ff4300b1900 2022-07-12T08:50:30.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300ad590 con 0x7ff4300b05e0 2022-07-12T08:50:30.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff43486b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff430060570 con 0x7ff4300ad1b0 2022-07-12T08:50:30.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff42ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff4300b1900 0x7ff43005ee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49508/0 (socket says 172.21.15.61:49508) 2022-07-12T08:50:30.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff42ed9d700 1 -- 172.21.15.61:0/3000470627 learned_addr learned my addr 172.21.15.61:0/3000470627 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:30.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/3000470627 >> v2:172.21.15.111:3300/0 conn(0x7ff4300b05e0 0x7ff4300b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.288+0000 7ff42e59c700 1 --2- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300ad1b0 0x7ff4301434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42d59a700 1 -- 172.21.15.61:0/3000470627 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371861001 0 0) 0x7ff4300aebc0 con 0x7ff4300b1900 2022-07-12T08:50:30.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42d59a700 1 -- 172.21.15.61:0/3000470627 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff414003260 con 0x7ff4300b1900 2022-07-12T08:50:30.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42dd9b700 1 -- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300ad1b0 msgr2=0x7ff4301434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:30.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300ad1b0 0x7ff4301434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42dd9b700 1 -- 172.21.15.61:0/3000470627 >> v1:172.21.15.61:6789/0 conn(0x7ff4300b1900 legacy=0x7ff43005ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42dd9b700 1 -- 172.21.15.61:0/3000470627 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430142da0 con 0x7ff4300b05e0 2022-07-12T08:50:30.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b05e0 0x7ff4300b1410 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7ff424002f70 tx=0x7ff424006e80).ready entity=mon.1 client_cookie=98f55309c06bcfa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:30.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42d59a700 1 -- 172.21.15.61:0/3000470627 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff424014070 con 0x7ff4300b05e0 2022-07-12T08:50:30.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42d59a700 1 -- 172.21.15.61:0/3000470627 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff424000e90 con 0x7ff4300b05e0 2022-07-12T08:50:30.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff42d59a700 1 -- 172.21.15.61:0/3000470627 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4240107f0 con 0x7ff4300b05e0 2022-07-12T08:50:30.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 -- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b05e0 msgr2=0x7ff4300b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 --2- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b05e0 0x7ff4300b1410 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7ff424002f70 tx=0x7ff424006e80).stop 2022-07-12T08:50:30.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 -- 172.21.15.61:0/3000470627 shutdown_connections 2022-07-12T08:50:30.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 --2- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b05e0 0x7ff4300b1410 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 --2- 172.21.15.61:0/3000470627 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300ad1b0 0x7ff4301434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 -- 172.21.15.61:0/3000470627 >> 172.21.15.61:0/3000470627 conn(0x7ff4300b5a30 msgr2=0x7ff4300b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:30.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 -- 172.21.15.61:0/3000470627 shutdown_connections 2022-07-12T08:50:30.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 -- 172.21.15.61:0/3000470627 wait complete. 2022-07-12T08:50:30.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.292+0000 7ff43486b700 1 Processor -- start 2022-07-12T08:50:30.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 -- start start 2022-07-12T08:50:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 0x7ff4300b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300b05e0 0x7ff430143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b1900 0x7ff43013e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430065240 con 0x7ff4300ad1b0 2022-07-12T08:50:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300b3830 con 0x7ff4300b1900 2022-07-12T08:50:30.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff430065470 con 0x7ff4300b05e0 2022-07-12T08:50:30.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 0x7ff4300b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 0x7ff4300b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33060/0 (socket says 172.21.15.61:33060) 2022-07-12T08:50:30.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 -- 172.21.15.61:0/729838896 learned_addr learned my addr 172.21.15.61:0/729838896 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:30.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300b05e0 0x7ff430143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42ed9d700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b1900 0x7ff43013e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 -- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300b05e0 msgr2=0x7ff430143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300b05e0 0x7ff430143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 -- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b1900 msgr2=0x7ff43013e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b1900 0x7ff43013e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 -- 172.21.15.61:0/729838896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42400c310 con 0x7ff4300ad1b0 2022-07-12T08:50:30.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42ed9d700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b1900 0x7ff43013e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:30.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300b05e0 0x7ff430143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:30.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff42e59c700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 0x7ff4300b3c90 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7ff42000c000 tx=0x7ff4200091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:30.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42000c850 con 0x7ff4300ad1b0 2022-07-12T08:50:30.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42000c9b0 con 0x7ff4300ad1b0 2022-07-12T08:50:30.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff430066de0 con 0x7ff4300ad1b0 2022-07-12T08:50:30.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42000cd10 con 0x7ff4300ad1b0 2022-07-12T08:50:30.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.296+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff430146040 con 0x7ff4300ad1b0 2022-07-12T08:50:30.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.300+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff430057460 con 0x7ff4300ad1b0 2022-07-12T08:50:30.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.300+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff420022070 con 0x7ff4300ad1b0 2022-07-12T08:50:30.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.300+0000 7ff41f7fe700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff40c053a40 0x7ff40c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.300+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff42006c560 con 0x7ff4300ad1b0 2022-07-12T08:50:30.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.300+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff40c053a40 0x7ff40c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.300+0000 7ff42dd9b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff40c053a40 0x7ff40c055f00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7ff42401a000 tx=0x7ff430061db0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:30.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.308+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4200434c0 con 0x7ff4300ad1b0 2022-07-12T08:50:30.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.468+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4300b1070 con 0x7ff4300ad1b0 2022-07-12T08:50:30.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff41f7fe700 1 -- 172.21.15.61:0/729838896 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2145 (secure 0 0 0) 0x7ff420046ae0 con 0x7ff4300ad1b0 2022-07-12T08:50:30.473 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:30.474 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:10.807048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8082},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8082":{"gid":8082,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2724832679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2724832679},{"type":"v1","addr":"172.21.15.61:6833","nonce":2724832679}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:30.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff40c053a40 msgr2=0x7ff40c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff40c053a40 0x7ff40c055f00 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7ff42401a000 tx=0x7ff430061db0).stop 2022-07-12T08:50:30.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 msgr2=0x7ff4300b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 0x7ff4300b3c90 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 rx=0x7ff42000c000 tx=0x7ff4200091d0).stop 2022-07-12T08:50:30.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 shutdown_connections 2022-07-12T08:50:30.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff40c053a40 0x7ff40c055f00 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4300b1900 0x7ff43013e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff4300b05e0 0x7ff430143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 --2- 172.21.15.61:0/729838896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4300ad1b0 0x7ff4300b3c90 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 >> 172.21.15.61:0/729838896 conn(0x7ff4300b5a30 msgr2=0x7ff430140b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:30.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 shutdown_connections 2022-07-12T08:50:30.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.472+0000 7ff43486b700 1 -- 172.21.15.61:0/729838896 wait complete. 2022-07-12T08:50:30.479 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-12T08:50:30.487 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:50:30.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34df4e2700 1 Processor -- start 2022-07-12T08:50:30.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34df4e2700 1 -- start start 2022-07-12T08:50:30.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34df4e2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f34d80b9d20 0x7f34d8143430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34df4e2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8142d60 con 0x7f34d80b3e10 2022-07-12T08:50:30.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34df4e2700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8147140 con 0x7f34d8143920 2022-07-12T08:50:30.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34df4e2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d8146220 con 0x7f34d80b9d20 2022-07-12T08:50:30.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34de4e0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f34d80b9d20 0x7f34d8143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34ddcdf700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f34d80b3e10 0x7f34d813c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49518/0 (socket says 172.21.15.61:49518) 2022-07-12T08:50:30.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.592+0000 7f34ddcdf700 1 -- 172.21.15.61:0/3395668113 learned_addr learned my addr 172.21.15.61:0/3395668113 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:30.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34dd4de700 1 -- 172.21.15.61:0/3395668113 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631454895 0 0) 0x7f34d8142d60 con 0x7f34d80b3e10 2022-07-12T08:50:30.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34dd4de700 1 -- 172.21.15.61:0/3395668113 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c0003260 con 0x7f34d80b3e10 2022-07-12T08:50:30.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34dd4de700 1 -- 172.21.15.61:0/3395668113 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1706697717 0 0) 0x7f34c0003260 con 0x7f34d80b3e10 2022-07-12T08:50:30.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34de4e0700 1 -- 172.21.15.61:0/3395668113 >> v1:172.21.15.111:6789/0 conn(0x7f34d8143920 legacy=0x7f34d80b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34de4e0700 1 -- 172.21.15.61:0/3395668113 >> v1:172.21.15.61:6789/0 conn(0x7f34d80b3e10 legacy=0x7f34d813c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34de4e0700 1 -- 172.21.15.61:0/3395668113 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8142f90 con 0x7f34d80b9d20 2022-07-12T08:50:30.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34de4e0700 1 --2- 172.21.15.61:0/3395668113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d8143430 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f34d4001800 tx=0x7f34c8006df0).ready entity=mon.0 client_cookie=451dec65ca9f9355 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:30.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34dd4de700 1 -- 172.21.15.61:0/3395668113 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d4001660 con 0x7f34d80b9d20 2022-07-12T08:50:30.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34dd4de700 1 -- 172.21.15.61:0/3395668113 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c8000e90 con 0x7f34d80b9d20 2022-07-12T08:50:30.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34dd4de700 1 -- 172.21.15.61:0/3395668113 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34c8011400 con 0x7f34d80b9d20 2022-07-12T08:50:30.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 -- 172.21.15.61:0/3395668113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 msgr2=0x7f34d8143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 --2- 172.21.15.61:0/3395668113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d8143430 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f34d4001800 tx=0x7f34c8006df0).stop 2022-07-12T08:50:30.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 -- 172.21.15.61:0/3395668113 shutdown_connections 2022-07-12T08:50:30.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 --2- 172.21.15.61:0/3395668113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d8143430 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 -- 172.21.15.61:0/3395668113 >> 172.21.15.61:0/3395668113 conn(0x7f34d8067620 msgr2=0x7f34d80b9470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:30.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 -- 172.21.15.61:0/3395668113 shutdown_connections 2022-07-12T08:50:30.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.596+0000 7f34df4e2700 1 -- 172.21.15.61:0/3395668113 wait complete. 2022-07-12T08:50:30.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 Processor -- start 2022-07-12T08:50:30.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 -- start start 2022-07-12T08:50:30.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34d80b3e10 0x7f34d813e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34d8143920 0x7f34d8152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80ad020 con 0x7f34d80b9d20 2022-07-12T08:50:30.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80ad2c0 con 0x7f34d80b3e10 2022-07-12T08:50:30.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d80ad560 con 0x7f34d8143920 2022-07-12T08:50:30.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d80b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33074/0 (socket says 172.21.15.61:33074) 2022-07-12T08:50:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 -- 172.21.15.61:0/647526977 learned_addr learned my addr 172.21.15.61:0/647526977 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34de4e0700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34d80b3e10 0x7f34d813e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34dece1700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34d8143920 0x7f34d8152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 -- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34d8143920 msgr2=0x7f34d8152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34d8143920 0x7f34d8152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 -- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34d80b3e10 msgr2=0x7f34d813e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:30.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34d80b3e10 0x7f34d813e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 -- 172.21.15.61:0/647526977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c800b180 con 0x7f34d80b9d20 2022-07-12T08:50:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34ddcdf700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d80b2480 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7f34d4001840 tx=0x7f34d4001790).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d4007170 con 0x7f34d80b9d20 2022-07-12T08:50:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d40072d0 con 0x7f34d80b9d20 2022-07-12T08:50:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d80acca0 con 0x7f34d80b9d20 2022-07-12T08:50:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34d4004920 con 0x7f34d80b9d20 2022-07-12T08:50:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.600+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d8151e30 con 0x7f34d80b9d20 2022-07-12T08:50:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.604+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d80ace90 con 0x7f34d80b9d20 2022-07-12T08:50:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.604+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f34d401b070 con 0x7f34d80b9d20 2022-07-12T08:50:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.604+0000 7f34c77fe700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34b8053a60 0x7f34b8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.604+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f34d406b5e0 con 0x7f34d80b9d20 2022-07-12T08:50:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.604+0000 7f34de4e0700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34b8053a60 0x7f34b8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.604+0000 7f34de4e0700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34b8053a60 0x7f34b8055f20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f34c8017000 tx=0x7f34d80b70e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.608+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f34d40424e0 con 0x7f34d80b9d20 2022-07-12T08:50:30.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:30.772+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f34d8155380 con 0x7f34d80b9d20 2022-07-12T08:50:31.255 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:50:31.252+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:50:31.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.260+0000 7f34c77fe700 1 -- 172.21.15.61:0/647526977 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7f34d4045b00 con 0x7f34d80b9d20 2022-07-12T08:50:31.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34b8053a60 msgr2=0x7f34b8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34b8053a60 0x7f34b8055f20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f34c8017000 tx=0x7f34d80b70e0).stop 2022-07-12T08:50:31.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 msgr2=0x7f34d80b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d80b2480 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 rx=0x7f34d4001840 tx=0x7f34d4001790).stop 2022-07-12T08:50:31.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 shutdown_connections 2022-07-12T08:50:31.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f34d8143920 0x7f34d8152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f34d80b9d20 0x7f34d80b2480 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f34b8053a60 0x7f34b8055f20 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 --2- 172.21.15.61:0/647526977 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34d80b3e10 0x7f34d813e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 >> 172.21.15.61:0/647526977 conn(0x7f34d8067620 msgr2=0x7f34d80b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:31.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 shutdown_connections 2022-07-12T08:50:31.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.264+0000 7f34df4e2700 1 -- 172.21.15.61:0/647526977 wait complete. 2022-07-12T08:50:31.276 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:50:31.286 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:31.301 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:50:31.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.388+0000 7fd0b68ec700 1 Processor -- start 2022-07-12T08:50:31.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.388+0000 7fd0b68ec700 1 -- start start 2022-07-12T08:50:31.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd0b014b350 0x7fd0b0147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00b8410 con 0x7fd0b00adfd0 2022-07-12T08:50:31.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00b2240 con 0x7fd0b00acca0 2022-07-12T08:50:31.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00b24e0 con 0x7fd0b014b350 2022-07-12T08:50:31.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0affff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd0b00adfd0 0x7fd0b00b7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49544/0 (socket says 172.21.15.61:49544) 2022-07-12T08:50:31.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0affff700 1 -- 172.21.15.61:0/746550501 learned_addr learned my addr 172.21.15.61:0/746550501 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:31.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b4e89700 1 --2- 172.21.15.61:0/746550501 >> v2:172.21.15.61:3300/0 conn(0x7fd0b014b350 0x7fd0b0147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0aeffd700 1 -- 172.21.15.61:0/746550501 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955795831 0 0) 0x7fd0b00b8410 con 0x7fd0b00adfd0 2022-07-12T08:50:31.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0aeffd700 1 -- 172.21.15.61:0/746550501 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd09c003260 con 0x7fd0b00adfd0 2022-07-12T08:50:31.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0aeffd700 1 -- 172.21.15.61:0/746550501 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081611022 0 0) 0x7fd0b00b2240 con 0x7fd0b00acca0 2022-07-12T08:50:31.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b4e89700 1 -- 172.21.15.61:0/746550501 >> v1:172.21.15.111:6789/0 conn(0x7fd0b00acca0 legacy=0x7fd0b01475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b4e89700 1 -- 172.21.15.61:0/746550501 >> v1:172.21.15.61:6789/0 conn(0x7fd0b00adfd0 legacy=0x7fd0b00b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b4e89700 1 -- 172.21.15.61:0/746550501 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00b2780 con 0x7fd0b014b350 2022-07-12T08:50:31.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b4e89700 1 --2- 172.21.15.61:0/746550501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b014b350 0x7fd0b0147cc0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fd0a8001b80 tx=0x7fd0a0006df0).ready entity=mon.0 client_cookie=1e35c1ccbdb64ce6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:31.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0aeffd700 1 -- 172.21.15.61:0/746550501 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a80019e0 con 0x7fd0b014b350 2022-07-12T08:50:31.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0aeffd700 1 -- 172.21.15.61:0/746550501 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0000e90 con 0x7fd0b014b350 2022-07-12T08:50:31.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0aeffd700 1 -- 172.21.15.61:0/746550501 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a0011400 con 0x7fd0b014b350 2022-07-12T08:50:31.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- 172.21.15.61:0/746550501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b014b350 msgr2=0x7fd0b0147cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/746550501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b014b350 0x7fd0b0147cc0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7fd0a8001b80 tx=0x7fd0a0006df0).stop 2022-07-12T08:50:31.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- 172.21.15.61:0/746550501 shutdown_connections 2022-07-12T08:50:31.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/746550501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b014b350 0x7fd0b0147cc0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- 172.21.15.61:0/746550501 >> 172.21.15.61:0/746550501 conn(0x7fd0b00657d0 msgr2=0x7fd0b00b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- 172.21.15.61:0/746550501 shutdown_connections 2022-07-12T08:50:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.392+0000 7fd0b68ec700 1 -- 172.21.15.61:0/746550501 wait complete. 2022-07-12T08:50:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 Processor -- start 2022-07-12T08:50:31.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 -- start start 2022-07-12T08:50:31.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd0b00acca0 0x7fd0b00af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 0x7fd0b0146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0b014b350 0x7fd0b0156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00afaa0 con 0x7fd0b00adfd0 2022-07-12T08:50:31.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00aea30 con 0x7fd0b014b350 2022-07-12T08:50:31.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b68ec700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00b1270 con 0x7fd0b00acca0 2022-07-12T08:50:31.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 0x7fd0b0146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 0x7fd0b0146250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33102/0 (socket says 172.21.15.61:33102) 2022-07-12T08:50:31.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 -- 172.21.15.61:0/2238795903 learned_addr learned my addr 172.21.15.61:0/2238795903 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:31.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0affff700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd0b00acca0 0x7fd0b00af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b4e89700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0b014b350 0x7fd0b0156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 -- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd0b00acca0 msgr2=0x7fd0b00af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd0b00acca0 0x7fd0b00af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 -- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0b014b350 msgr2=0x7fd0b0156370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0b014b350 0x7fd0b0156370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 -- 172.21.15.61:0/2238795903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a000b180 con 0x7fd0b00adfd0 2022-07-12T08:50:31.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0b4e89700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0b014b350 0x7fd0b0156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:31.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.396+0000 7fd0af7fe700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 0x7fd0b0146250 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7fd0b00b17e0 tx=0x7fd0a4007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:31.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a400ce40 con 0x7fd0b00adfd0 2022-07-12T08:50:31.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a400f470 con 0x7fd0b00adfd0 2022-07-12T08:50:31.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b00b1510 con 0x7fd0b00adfd0 2022-07-12T08:50:31.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a400f7d0 con 0x7fd0b00adfd0 2022-07-12T08:50:31.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b0067050 con 0x7fd0b00adfd0 2022-07-12T08:50:31.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b0057460 con 0x7fd0b00adfd0 2022-07-12T08:50:31.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd0a4019030 con 0x7fd0b00adfd0 2022-07-12T08:50:31.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0ad7fa700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd090053a40 0x7fd090055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd0a406b230 con 0x7fd0b00adfd0 2022-07-12T08:50:31.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0affff700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd090053a40 0x7fd090055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.400+0000 7fd0affff700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd090053a40 0x7fd090055f00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fd0a800d000 tx=0x7fd0a8000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:31.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.408+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd0a40421f0 con 0x7fd0b00adfd0 2022-07-12T08:50:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.572+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0b0067d30 con 0x7fd0b00adfd0 2022-07-12T08:50:31.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.572+0000 7fd0ad7fa700 1 -- 172.21.15.61:0/2238795903 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+1785 (secure 0 0 0) 0x7fd0a4045810 con 0x7fd0b00adfd0 2022-07-12T08:50:31.580 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:31.581 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":73,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71}],"filesystems":[{"mdsmap":{"epoch":73,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:09.803335+0000","modified":"2022-07-12T08:50:31.255439+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":129,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-12T08:50:31.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd090053a40 msgr2=0x7fd090055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd090053a40 0x7fd090055f00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fd0a800d000 tx=0x7fd0a8000f40).stop 2022-07-12T08:50:31.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 msgr2=0x7fd0b0146250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 0x7fd0b0146250 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7fd0b00b17e0 tx=0x7fd0a4007b50).stop 2022-07-12T08:50:31.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 shutdown_connections 2022-07-12T08:50:31.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd090053a40 0x7fd090055f00 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd0b014b350 0x7fd0b0156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd0b00adfd0 0x7fd0b0146250 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 --2- 172.21.15.61:0/2238795903 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd0b00acca0 0x7fd0b00af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 >> 172.21.15.61:0/2238795903 conn(0x7fd0b00657d0 msgr2=0x7fd0b014d190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:31.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 shutdown_connections 2022-07-12T08:50:31.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.576+0000 7fd0b68ec700 1 -- 172.21.15.61:0/2238795903 wait complete. 2022-07-12T08:50:31.589 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 73 2022-07-12T08:50:31.599 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:50:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 Processor -- start 2022-07-12T08:50:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- start start 2022-07-12T08:50:31.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f326805d190 0x7f32681465a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268146a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32680afb30 con 0x7f32680b5be0 2022-07-12T08:50:31.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326814a6f0 con 0x7f326805d190 2022-07-12T08:50:31.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32680b5fc0 con 0x7f326805c240 2022-07-12T08:50:31.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32667fc700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f326805d190 0x7f32681465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32677fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f3266ffd700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f32680b5be0 0x7f326814adf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:34928/0 (socket says 172.21.15.61:34928) 2022-07-12T08:50:31.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f3266ffd700 1 -- 172.21.15.61:0/3799130679 learned_addr learned my addr 172.21.15.61:0/3799130679 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:31.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32667fc700 1 -- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 msgr2=0x7f3268146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:31.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32667fc700 1 --2- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32667fc700 1 -- 172.21.15.61:0/3799130679 >> v1:172.21.15.111:6789/0 conn(0x7f32680b5be0 legacy=0x7f326814adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32667fc700 1 -- 172.21.15.61:0/3799130679 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32680afd60 con 0x7f326805d190 2022-07-12T08:50:31.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f3265ffb700 1 -- 172.21.15.61:0/3799130679 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538098269 0 0) 0x7f32680afb30 con 0x7f32680b5be0 2022-07-12T08:50:31.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f32667fc700 1 --2- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32681465a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f32500032f0 tx=0x7f32500091d0).ready entity=mon.1 client_cookie=62a9e3f6f4c4b633 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:31.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f3265ffb700 1 -- 172.21.15.61:0/3799130679 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325000b6c0 con 0x7f326805d190 2022-07-12T08:50:31.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f3265ffb700 1 -- 172.21.15.61:0/3799130679 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325000b820 con 0x7f326805d190 2022-07-12T08:50:31.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f3265ffb700 1 -- 172.21.15.61:0/3799130679 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f325000b990 con 0x7f326805d190 2022-07-12T08:50:31.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 msgr2=0x7f32681465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 --2- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32681465a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f32500032f0 tx=0x7f32500091d0).stop 2022-07-12T08:50:31.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- 172.21.15.61:0/3799130679 shutdown_connections 2022-07-12T08:50:31.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 --2- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 --2- 172.21.15.61:0/3799130679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32681465a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- 172.21.15.61:0/3799130679 >> 172.21.15.61:0/3799130679 conn(0x7f3268063450 msgr2=0x7f32680b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:31.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- 172.21.15.61:0/3799130679 shutdown_connections 2022-07-12T08:50:31.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.760+0000 7f326d9eb700 1 -- 172.21.15.61:0/3799130679 wait complete. 2022-07-12T08:50:31.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 Processor -- start 2022-07-12T08:50:31.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 -- start start 2022-07-12T08:50:31.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32680aca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 0x7f32680acf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32681493d0 con 0x7f32680b5be0 2022-07-12T08:50:31.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3268149670 con 0x7f326805d190 2022-07-12T08:50:31.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3268149910 con 0x7f326805c240 2022-07-12T08:50:31.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 0x7f32680acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f3266ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 0x7f32680acf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33118/0 (socket says 172.21.15.61:33118) 2022-07-12T08:50:31.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 -- 172.21.15.61:0/929324671 learned_addr learned my addr 172.21.15.61:0/929324671 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:31.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32667fc700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32680aca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 -- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 msgr2=0x7f3268144760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268144760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 -- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 msgr2=0x7f32680aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:31.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32680aca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:31.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 -- 172.21.15.61:0/929324671 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250009470 con 0x7f32680b5be0 2022-07-12T08:50:31.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f32677fe700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 0x7f32680acf40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f326000c000 tx=0x7f3260006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:31.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3260011750 con 0x7f32680b5be0 2022-07-12T08:50:31.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32600118b0 con 0x7f32680b5be0 2022-07-12T08:50:31.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3268156430 con 0x7f32680b5be0 2022-07-12T08:50:31.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3260011c10 con 0x7f32680b5be0 2022-07-12T08:50:31.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.764+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3268067700 con 0x7f32680b5be0 2022-07-12T08:50:31.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.768+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32680ac850 con 0x7f32680b5be0 2022-07-12T08:50:31.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.768+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3260011d70 con 0x7f32680b5be0 2022-07-12T08:50:31.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.768+0000 7f325ffff700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f324c053a40 0x7f324c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:31.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.768+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f326006c3f0 con 0x7f32680b5be0 2022-07-12T08:50:31.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.768+0000 7f3266ffd700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f324c053a40 0x7f324c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:31.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.772+0000 7f3266ffd700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f324c053a40 0x7f324c055f00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f325800d000 tx=0x7f3258007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:31.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.776+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f32600431f0 con 0x7f32680b5be0 2022-07-12T08:50:31.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:31.940+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f326814a4b0 con 0x7f32680b5be0 2022-07-12T08:50:32.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.272+0000 7f325ffff700 1 -- 172.21.15.61:0/929324671 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7f326814a4b0 con 0x7f32680b5be0 2022-07-12T08:50:32.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f324c053a40 msgr2=0x7f324c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:32.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f324c053a40 0x7f324c055f00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7f325800d000 tx=0x7f3258007c20).stop 2022-07-12T08:50:32.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 msgr2=0x7f32680acf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:32.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 0x7f32680acf40 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f326000c000 tx=0x7f3260006df0).stop 2022-07-12T08:50:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 shutdown_connections 2022-07-12T08:50:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32680b5be0 0x7f32680acf40 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f324c053a40 0x7f324c055f00 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f326805d190 0x7f32680aca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 --2- 172.21.15.61:0/929324671 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f326805c240 0x7f3268144760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 >> 172.21.15.61:0/929324671 conn(0x7f3268063450 msgr2=0x7f3268143640 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 shutdown_connections 2022-07-12T08:50:32.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.276+0000 7f326d9eb700 1 -- 172.21.15.61:0/929324671 wait complete. 2022-07-12T08:50:32.296 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:50:32.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.412+0000 7f2ff9bd9700 1 Processor -- start 2022-07-12T08:50:32.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.412+0000 7f2ff9bd9700 1 -- start start 2022-07-12T08:50:32.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f2ff405f870 0x7f2ff40603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff40b0840 0x7f2ff405fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2ff405ee50 0x7f2ff40b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff4064d00 con 0x7f2ff405ee50 2022-07-12T08:50:32.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff4066600 con 0x7f2ff405f870 2022-07-12T08:50:32.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff4066360 con 0x7f2ff40b0840 2022-07-12T08:50:32.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2ff405ee50 0x7f2ff40b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2ff405ee50 0x7f2ff40b2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33126/0 (socket says 172.21.15.61:33126) 2022-07-12T08:50:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 -- 172.21.15.61:0/1728407987 learned_addr learned my addr 172.21.15.61:0/1728407987 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff8bd7700 1 --2- 172.21.15.61:0/1728407987 >> v2:172.21.15.111:3300/0 conn(0x7f2ff405f870 0x7f2ff40603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff3fff700 1 --2- 172.21.15.61:0/1728407987 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff40b0840 0x7f2ff405fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 -- 172.21.15.61:0/1728407987 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff40b0840 msgr2=0x7f2ff405fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 --2- 172.21.15.61:0/1728407987 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff40b0840 0x7f2ff405fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 -- 172.21.15.61:0/1728407987 >> v2:172.21.15.111:3300/0 conn(0x7f2ff405f870 msgr2=0x7f2ff40603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 --2- 172.21.15.61:0/1728407987 >> v2:172.21.15.111:3300/0 conn(0x7f2ff405f870 0x7f2ff40603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 -- 172.21.15.61:0/1728407987 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff413f8d0 con 0x7f2ff405ee50 2022-07-12T08:50:32.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff93d8700 1 --2- 172.21.15.61:0/1728407987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405ee50 0x7f2ff40b2a80 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7f2fec0032f0 tx=0x7f2fec009290).ready entity=mon.0 client_cookie=99561e9ae49092a2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:32.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff37fe700 1 -- 172.21.15.61:0/1728407987 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fec00b680 con 0x7f2ff405ee50 2022-07-12T08:50:32.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff37fe700 1 -- 172.21.15.61:0/1728407987 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec00b7e0 con 0x7f2ff405ee50 2022-07-12T08:50:32.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff37fe700 1 -- 172.21.15.61:0/1728407987 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fec00b950 con 0x7f2ff405ee50 2022-07-12T08:50:32.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/1728407987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405ee50 msgr2=0x7f2ff40b2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:32.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/1728407987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405ee50 0x7f2ff40b2a80 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7f2fec0032f0 tx=0x7f2fec009290).stop 2022-07-12T08:50:32.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/1728407987 shutdown_connections 2022-07-12T08:50:32.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/1728407987 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff40b0840 0x7f2ff405fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/1728407987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff405ee50 0x7f2ff40b2a80 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/1728407987 >> v2:172.21.15.111:3300/0 conn(0x7f2ff405f870 0x7f2ff40603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.416+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/1728407987 >> 172.21.15.61:0/1728407987 conn(0x7f2ff4063950 msgr2=0x7f2ff4063d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:32.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/1728407987 shutdown_connections 2022-07-12T08:50:32.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/1728407987 wait complete. 2022-07-12T08:50:32.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 Processor -- start 2022-07-12T08:50:32.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 -- start start 2022-07-12T08:50:32.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff405ee50 0x7f2ff40b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff405f870 0x7f2ff41459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 0x7f2ff413be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff413b4d0 con 0x7f2ff40b0840 2022-07-12T08:50:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff413c2f0 con 0x7f2ff405f870 2022-07-12T08:50:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff9bd9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff413b6c0 con 0x7f2ff405ee50 2022-07-12T08:50:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 0x7f2ff413be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 0x7f2ff413be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33134/0 (socket says 172.21.15.61:33134) 2022-07-12T08:50:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 -- 172.21.15.61:0/2937697219 learned_addr learned my addr 172.21.15.61:0/2937697219 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:32.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff3fff700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff405f870 0x7f2ff41459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff8bd7700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff405ee50 0x7f2ff40b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 -- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff405ee50 msgr2=0x7f2ff40b0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:32.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff405ee50 0x7f2ff40b0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 -- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff405f870 msgr2=0x7f2ff41459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:32.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff405f870 0x7f2ff41459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:32.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 -- 172.21.15.61:0/2937697219 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec009530 con 0x7f2ff40b0840 2022-07-12T08:50:32.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff3fff700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff405f870 0x7f2ff41459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:32.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff93d8700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 0x7f2ff413be00 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f2fec0032f0 tx=0x7f2fec017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.420+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fec00b680 con 0x7f2ff40b0840 2022-07-12T08:50:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec011e50 con 0x7f2ff40b0840 2022-07-12T08:50:32.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff413b960 con 0x7f2ff40b0840 2022-07-12T08:50:32.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fec01b810 con 0x7f2ff40b0840 2022-07-12T08:50:32.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff413a170 con 0x7f2ff40b0840 2022-07-12T08:50:32.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff413a480 con 0x7f2ff40b0840 2022-07-12T08:50:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2fec00f070 con 0x7f2ff40b0840 2022-07-12T08:50:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff1ffb700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fd4053a40 0x7f2fd4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2fec06d620 con 0x7f2ff40b0840 2022-07-12T08:50:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.424+0000 7f2ff8bd7700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fd4053a40 0x7f2fd4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.428+0000 7f2ff8bd7700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fd4053a40 0x7f2fd4055f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f2fe800b000 tx=0x7f2fe8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:32.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.432+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2fec0445f0 con 0x7f2ff40b0840 2022-07-12T08:50:32.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:32.560+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ff413baf0 con 0x7f2ff40b0840 2022-07-12T08:50:33.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.292+0000 7f2ff1ffb700 1 -- 172.21.15.61:0/2937697219 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2fec01f020 con 0x7f2ff40b0840 2022-07-12T08:50:33.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.292+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fd4053a40 msgr2=0x7f2fd4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.292+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fd4053a40 0x7f2fd4055f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f2fe800b000 tx=0x7f2fe8006df0).stop 2022-07-12T08:50:33.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.292+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 msgr2=0x7f2ff413be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.292+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 0x7f2ff413be00 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 rx=0x7f2fec0032f0 tx=0x7f2fec017000).stop 2022-07-12T08:50:33.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 shutdown_connections 2022-07-12T08:50:33.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ff40b0840 0x7f2ff413be00 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2fd4053a40 0x7f2fd4055f00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2ff405f870 0x7f2ff41459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 --2- 172.21.15.61:0/2937697219 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2ff405ee50 0x7f2ff40b0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 >> 172.21.15.61:0/2937697219 conn(0x7f2ff4063950 msgr2=0x7f2ff40b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:33.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 shutdown_connections 2022-07-12T08:50:33.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.296+0000 7f2ff9bd9700 1 -- 172.21.15.61:0/2937697219 wait complete. 2022-07-12T08:50:33.301 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:50:33.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:50:33.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.436+0000 7f81cd10b700 1 Processor -- start 2022-07-12T08:50:33.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.436+0000 7f81cd10b700 1 -- start start 2022-07-12T08:50:33.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f81c805f870 0x7f81c80603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:33.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8066600 con 0x7f81c805fc50 2022-07-12T08:50:33.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c8066360 con 0x7f81c80b0840 2022-07-12T08:50:33.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c80600a0 con 0x7f81c805f870 2022-07-12T08:50:33.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cc90a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f81c805fc50 0x7f81c813fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49592/0 (socket says 172.21.15.61:49592) 2022-07-12T08:50:33.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cc90a700 1 -- 172.21.15.61:0/1692587521 learned_addr learned my addr 172.21.15.61:0/1692587521 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:33.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c7fff700 1 --2- 172.21.15.61:0/1692587521 >> v2:172.21.15.111:3300/0 conn(0x7f81c805f870 0x7f81c80603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:33.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143344770 0 0) 0x7f81c8066360 con 0x7f81c80b0840 2022-07-12T08:50:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a8003260 con 0x7f81c80b0840 2022-07-12T08:50:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969394241 0 0) 0x7f81c8066600 con 0x7f81c805fc50 2022-07-12T08:50:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c8066360 con 0x7f81c805fc50 2022-07-12T08:50:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c7fff700 1 -- 172.21.15.61:0/1692587521 >> v1:172.21.15.111:6789/0 conn(0x7f81c80b0840 legacy=0x7f81c80b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c7fff700 1 -- 172.21.15.61:0/1692587521 >> v1:172.21.15.61:6789/0 conn(0x7f81c805fc50 legacy=0x7f81c813fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c7fff700 1 -- 172.21.15.61:0/1692587521 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c813f8d0 con 0x7f81c805f870 2022-07-12T08:50:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c7fff700 1 --2- 172.21.15.61:0/1692587521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c805f870 0x7f81c80603f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f81b8002f70 tx=0x7f81b8006df0).ready entity=mon.1 client_cookie=92eb5711659781ce server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b800d540 con 0x7f81c805f870 2022-07-12T08:50:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b8000e90 con 0x7f81c805f870 2022-07-12T08:50:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81c77fe700 1 -- 172.21.15.61:0/1692587521 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b800d890 con 0x7f81c805f870 2022-07-12T08:50:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- 172.21.15.61:0/1692587521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c805f870 msgr2=0x7f81c80603f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 --2- 172.21.15.61:0/1692587521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c805f870 0x7f81c80603f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f81b8002f70 tx=0x7f81b8006df0).stop 2022-07-12T08:50:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- 172.21.15.61:0/1692587521 shutdown_connections 2022-07-12T08:50:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 --2- 172.21.15.61:0/1692587521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c805f870 0x7f81c80603f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- 172.21.15.61:0/1692587521 >> 172.21.15.61:0/1692587521 conn(0x7f81c8063950 msgr2=0x7f81c8063d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- 172.21.15.61:0/1692587521 shutdown_connections 2022-07-12T08:50:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.440+0000 7f81cd10b700 1 -- 172.21.15.61:0/1692587521 wait complete. 2022-07-12T08:50:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 Processor -- start 2022-07-12T08:50:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 -- start start 2022-07-12T08:50:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c805f870 0x7f81c8155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 0x7f81c8155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c80b0840 0x7f81c813d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8156150 con 0x7f81c805fc50 2022-07-12T08:50:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c8141f20 con 0x7f81c80b0840 2022-07-12T08:50:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c81421b0 con 0x7f81c805f870 2022-07-12T08:50:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 0x7f81c8155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 0x7f81c8155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33148/0 (socket says 172.21.15.61:33148) 2022-07-12T08:50:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3371598854 learned_addr learned my addr 172.21.15.61:0/3371598854 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cc90a700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c80b0840 0x7f81c813d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81c7fff700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c805f870 0x7f81c8155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:33.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c805f870 msgr2=0x7f81c8155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c805f870 0x7f81c8155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c80b0840 msgr2=0x7f81c813d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:33.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c80b0840 0x7f81c813d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:33.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 -- 172.21.15.61:0/3371598854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b800b180 con 0x7f81c805fc50 2022-07-12T08:50:33.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81bf7fe700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 0x7f81c8155c60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f81b0001110 tx=0x7f81b0007b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:33.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b0012400 con 0x7f81c805fc50 2022-07-12T08:50:33.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c8142450 con 0x7f81c805fc50 2022-07-12T08:50:33.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b0012560 con 0x7f81c805fc50 2022-07-12T08:50:33.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81b00128c0 con 0x7f81c805fc50 2022-07-12T08:50:33.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.444+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c8140c20 con 0x7f81c805fc50 2022-07-12T08:50:33.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.448+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c8140f30 con 0x7f81c805fc50 2022-07-12T08:50:33.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.448+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f81b0019030 con 0x7f81c805fc50 2022-07-12T08:50:33.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.448+0000 7f81c5ffb700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f81ac053a40 0x7f81ac055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:33.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.448+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f81b006b270 con 0x7f81c805fc50 2022-07-12T08:50:33.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.448+0000 7f81c7fff700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f81ac053a40 0x7f81ac055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:33.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.448+0000 7f81c7fff700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f81ac053a40 0x7f81ac055f00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f81b801a000 tx=0x7f81b8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:33.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.452+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f81b00421a0 con 0x7f81c805fc50 2022-07-12T08:50:33.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:33.584+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f81c8057460 con 0x7f81c805fc50 2022-07-12T08:50:33.653 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:33.656 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:33.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:33.663 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:33.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:50:34.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.300+0000 7f81c5ffb700 1 -- 172.21.15.61:0/3371598854 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 173+0+0 (secure 0 0 0) 0x7f81b00457c0 con 0x7f81c805fc50 2022-07-12T08:50:34.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f81ac053a40 msgr2=0x7f81ac055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f81ac053a40 0x7f81ac055f00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f81b801a000 tx=0x7f81b8014000).stop 2022-07-12T08:50:34.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 msgr2=0x7f81c8155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 0x7f81c8155c60 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7f81b0001110 tx=0x7f81b0007b00).stop 2022-07-12T08:50:34.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 shutdown_connections 2022-07-12T08:50:34.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f81ac053a40 0x7f81ac055f00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81c80b0840 0x7f81c813d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81c805fc50 0x7f81c8155c60 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 --2- 172.21.15.61:0/3371598854 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f81c805f870 0x7f81c8155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 >> 172.21.15.61:0/3371598854 conn(0x7f81c8063950 msgr2=0x7f81c80b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:34.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 shutdown_connections 2022-07-12T08:50:34.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.304+0000 7f81cd10b700 1 -- 172.21.15.61:0/3371598854 wait complete. 2022-07-12T08:50:34.313 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:50:34.322 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:34.325 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:34.329 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:34.334 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:34.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-12T08:50:34.358 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2022-07-12T08:50:34.359 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2022-07-12T08:50:34.359 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:34.360 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2022-07-12T08:50:34.360 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.361 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2022-07-12T08:50:34.361 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.362 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:34.362 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.362 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:34.363 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2022-07-12T08:50:34.363 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.364 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2022-07-12T08:50:34.364 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2022-07-12T08:50:34.364 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.365 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:34.365 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:34.366 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.366 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:34.366 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:34.367 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.367 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2022-07-12T08:50:34.367 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.367 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:34.367 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2022-07-12T08:50:34.368 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:34.368 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:34.368 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:34.368 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:34.369 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2022-07-12T08:50:34.369 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:34.370 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:34.370 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.371 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:34.371 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.371 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2022-07-12T08:50:34.371 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2022-07-12T08:50:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2022-07-12T08:50:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2022-07-12T08:50:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2022-07-12T08:50:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.374 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2022-07-12T08:50:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2022-07-12T08:50:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2022-07-12T08:50:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2022-07-12T08:50:34.376 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:34.376 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.377 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2022-07-12T08:50:34.377 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2022-07-12T08:50:34.378 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:34.378 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.378 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2022-07-12T08:50:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2022-07-12T08:50:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2022-07-12T08:50:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2022-07-12T08:50:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2022-07-12T08:50:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2022-07-12T08:50:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 20 2022-07-12T08:50:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = filestore 2022-07-12T08:50:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-12T08:50:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: osd sloppy crc = True 2022-07-12T08:50:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.384 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2022-07-12T08:50:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2022-07-12T08:50:34.385 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-12T08:50:34.385 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-12T08:50:34.386 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.386 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.387 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2022-07-12T08:50:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-12T08:50:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2022-07-12T08:50:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-12T08:50:34.389 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2022-07-12T08:50:34.389 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:34.389 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.390 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:34.390 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2022-07-12T08:50:34.391 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.391 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.391 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:34.392 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:34.392 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:34.392 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.393 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:34.393 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:34.394 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:34.394 INFO:teuthology.orchestra.run.smithi061.stdout: mon op complaint time = 120 2022-07-12T08:50:34.395 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:34.395 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2022-07-12T08:50:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2022-07-12T08:50:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2022-07-12T08:50:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2022-07-12T08:50:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: client mount timeout = 600 2022-07-12T08:50:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 20 2022-07-12T08:50:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.399 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2022-07-12T08:50:34.399 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.b] 2022-07-12T08:50:34.399 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.c] 2022-07-12T08:50:34.399 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-12T08:50:34.400 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-12T08:50:34.400 INFO:teuthology.orchestra.run.smithi061.stdout:[mds] 2022-07-12T08:50:34.400 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 20 2022-07-12T08:50:34.400 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-12T08:50:34.400 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal frag = True 2022-07-12T08:50:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal merge size = 5 2022-07-12T08:50:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split bits = 3 2022-07-12T08:50:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split size = 100 2022-07-12T08:50:34.402 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug frag = True 2022-07-12T08:50:34.402 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug scatterstat = True 2022-07-12T08:50:34.403 INFO:teuthology.orchestra.run.smithi061.stdout: mds op complaint time = 180 2022-07-12T08:50:34.403 INFO:teuthology.orchestra.run.smithi061.stdout: mds verify scatter = True 2022-07-12T08:50:34.404 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-12T08:50:34.404 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.404 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.412 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2022-07-12T08:50:34.413 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2022-07-12T08:50:34.413 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:34.414 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2022-07-12T08:50:34.414 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.415 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2022-07-12T08:50:34.415 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.415 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:34.416 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2022-07-12T08:50:34.417 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.417 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2022-07-12T08:50:34.417 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2022-07-12T08:50:34.417 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.418 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:34.418 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:34.418 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.419 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:34.419 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:34.419 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.420 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2022-07-12T08:50:34.420 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.420 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:34.421 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2022-07-12T08:50:34.421 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:34.422 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:34.422 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:34.423 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:34.423 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2022-07-12T08:50:34.423 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:34.424 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:34.424 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.425 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:34.425 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.426 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2022-07-12T08:50:34.426 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.427 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2022-07-12T08:50:34.427 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.427 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2022-07-12T08:50:34.428 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2022-07-12T08:50:34.428 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2022-07-12T08:50:34.428 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:34.429 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:34.429 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.429 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2022-07-12T08:50:34.430 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2022-07-12T08:50:34.430 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.431 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:34.431 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2022-07-12T08:50:34.431 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.432 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2022-07-12T08:50:34.432 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:34.432 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.433 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2022-07-12T08:50:34.433 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2022-07-12T08:50:34.434 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:34.434 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.434 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2022-07-12T08:50:34.435 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2022-07-12T08:50:34.435 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.436 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:34.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.436 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2022-07-12T08:50:34.436 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.437 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2022-07-12T08:50:34.437 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.437 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:34.438 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:34.438 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.439 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2022-07-12T08:50:34.439 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2022-07-12T08:50:34.440 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:34.440 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2022-07-12T08:50:34.440 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = filestore 2022-07-12T08:50:34.441 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2022-07-12T08:50:34.441 INFO:teuthology.orchestra.run.smithi137.stdout: osd sloppy crc = True 2022-07-12T08:50:34.441 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.441 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2022-07-12T08:50:34.442 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:34.442 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2022-07-12T08:50:34.442 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2022-07-12T08:50:34.442 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2022-07-12T08:50:34.443 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.443 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.443 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:34.443 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.444 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2022-07-12T08:50:34.444 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:34.444 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2022-07-12T08:50:34.444 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2022-07-12T08:50:34.444 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2022-07-12T08:50:34.445 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2022-07-12T08:50:34.445 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:34.445 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.445 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:34.446 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2022-07-12T08:50:34.446 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.446 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.446 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:34.447 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:34.447 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:34.448 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.448 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:34.449 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:34.449 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:34.450 INFO:teuthology.orchestra.run.smithi137.stdout: mon op complaint time = 120 2022-07-12T08:50:34.450 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-07-12T08:50:34.451 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2022-07-12T08:50:34.451 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2022-07-12T08:50:34.451 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2022-07-12T08:50:34.452 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2022-07-12T08:50:34.452 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:34.453 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:34.453 INFO:teuthology.orchestra.run.smithi137.stdout: client mount timeout = 600 2022-07-12T08:50:34.453 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2022-07-12T08:50:34.454 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:34.454 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.455 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.455 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2022-07-12T08:50:34.456 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2022-07-12T08:50:34.456 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2022-07-12T08:50:34.456 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2022-07-12T08:50:34.457 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2022-07-12T08:50:34.457 INFO:teuthology.orchestra.run.smithi137.stdout:[mds] 2022-07-12T08:50:34.458 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds = 20 2022-07-12T08:50:34.458 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2022-07-12T08:50:34.459 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal frag = True 2022-07-12T08:50:34.459 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:34.459 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal merge size = 5 2022-07-12T08:50:34.460 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal split bits = 3 2022-07-12T08:50:34.460 INFO:teuthology.orchestra.run.smithi137.stdout: mds bal split size = 100 2022-07-12T08:50:34.460 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug frag = True 2022-07-12T08:50:34.461 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug scatterstat = True 2022-07-12T08:50:34.461 INFO:teuthology.orchestra.run.smithi137.stdout: mds op complaint time = 180 2022-07-12T08:50:34.462 INFO:teuthology.orchestra.run.smithi137.stdout: mds verify scatter = True 2022-07-12T08:50:34.462 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2022-07-12T08:50:34.463 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.463 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.463 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2022-07-12T08:50:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2022-07-12T08:50:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:34.464 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2022-07-12T08:50:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.465 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2022-07-12T08:50:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:34.466 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:34.467 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2022-07-12T08:50:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.468 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2022-07-12T08:50:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2022-07-12T08:50:34.469 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:34.470 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.471 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.472 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2022-07-12T08:50:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2022-07-12T08:50:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:34.474 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:34.475 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2022-07-12T08:50:34.476 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:34.477 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:34.478 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2022-07-12T08:50:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.479 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2022-07-12T08:50:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.480 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2022-07-12T08:50:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2022-07-12T08:50:34.481 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2022-07-12T08:50:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:34.482 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.483 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2022-07-12T08:50:34.483 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2022-07-12T08:50:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.484 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2022-07-12T08:50:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.485 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2022-07-12T08:50:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:34.486 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2022-07-12T08:50:34.487 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2022-07-12T08:50:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:34.488 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2022-07-12T08:50:34.489 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2022-07-12T08:50:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.490 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:34.491 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.491 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2022-07-12T08:50:34.491 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.492 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2022-07-12T08:50:34.492 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.493 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:34.493 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:34.494 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.494 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2022-07-12T08:50:34.494 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2022-07-12T08:50:34.495 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:34.495 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2022-07-12T08:50:34.496 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = filestore 2022-07-12T08:50:34.496 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-07-12T08:50:34.497 INFO:teuthology.orchestra.run.smithi121.stdout: osd sloppy crc = True 2022-07-12T08:50:34.497 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.497 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2022-07-12T08:50:34.498 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:34.498 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2022-07-12T08:50:34.499 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-07-12T08:50:34.499 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-07-12T08:50:34.499 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.500 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.500 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:34.500 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.501 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2022-07-12T08:50:34.501 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:34.501 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2022-07-12T08:50:34.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2022-07-12T08:50:34.502 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2022-07-12T08:50:34.503 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2022-07-12T08:50:34.503 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:34.503 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.504 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:34.504 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2022-07-12T08:50:34.505 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.505 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.505 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:34.506 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:34.506 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:34.507 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.507 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:34.507 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:34.508 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:34.508 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2022-07-12T08:50:34.508 INFO:teuthology.orchestra.run.smithi121.stdout: 2022-07-12T08:50:34.509 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2022-07-12T08:50:34.509 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2022-07-12T08:50:34.510 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2022-07-12T08:50:34.510 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2022-07-12T08:50:34.511 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:34.511 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:34.511 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2022-07-12T08:50:34.512 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2022-07-12T08:50:34.513 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:34.514 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.514 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.515 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2022-07-12T08:50:34.515 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2022-07-12T08:50:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2022-07-12T08:50:34.516 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2022-07-12T08:50:34.517 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2022-07-12T08:50:34.517 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2022-07-12T08:50:34.518 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2022-07-12T08:50:34.518 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2022-07-12T08:50:34.519 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal frag = True 2022-07-12T08:50:34.519 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:34.519 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal merge size = 5 2022-07-12T08:50:34.520 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split bits = 3 2022-07-12T08:50:34.520 INFO:teuthology.orchestra.run.smithi121.stdout: mds bal split size = 100 2022-07-12T08:50:34.521 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2022-07-12T08:50:34.521 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2022-07-12T08:50:34.521 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2022-07-12T08:50:34.522 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2022-07-12T08:50:34.522 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2022-07-12T08:50:34.522 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.523 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.523 INFO:teuthology.orchestra.run.smithi130.stdout:[global] 2022-07-12T08:50:34.523 INFO:teuthology.orchestra.run.smithi130.stdout: chdir = "" 2022-07-12T08:50:34.523 INFO:teuthology.orchestra.run.smithi130.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:34.524 INFO:teuthology.orchestra.run.smithi130.stdout: auth supported = cephx 2022-07-12T08:50:34.524 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.524 INFO:teuthology.orchestra.run.smithi130.stdout: filestore xattr use omap = true 2022-07-12T08:50:34.524 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.524 INFO:teuthology.orchestra.run.smithi130.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:34.525 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.525 INFO:teuthology.orchestra.run.smithi130.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:34.525 INFO:teuthology.orchestra.run.smithi130.stdout: auth debug = true 2022-07-12T08:50:34.525 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.526 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on old message = true 2022-07-12T08:50:34.526 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on bug = true 2022-07-12T08:50:34.526 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.526 INFO:teuthology.orchestra.run.smithi130.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:34.526 INFO:teuthology.orchestra.run.smithi130.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:34.527 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.527 INFO:teuthology.orchestra.run.smithi130.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:34.527 INFO:teuthology.orchestra.run.smithi130.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:34.527 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.527 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default size = 2 2022-07-12T08:50:34.528 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.528 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:34.528 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow pg remap = true 2022-07-12T08:50:34.528 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:34.529 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:34.529 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:34.529 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:34.529 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on too few osds = false 2022-07-12T08:50:34.529 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:34.530 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:34.530 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.530 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:34.530 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.530 INFO:teuthology.orchestra.run.smithi130.stdout: osd default data pool replay window = 5 2022-07-12T08:50:34.531 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.531 INFO:teuthology.orchestra.run.smithi130.stdout: mon allow pool delete = true 2022-07-12T08:50:34.531 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.531 INFO:teuthology.orchestra.run.smithi130.stdout: mon cluster log file level = debug 2022-07-12T08:50:34.532 INFO:teuthology.orchestra.run.smithi130.stdout: debug asserts on shutdown = true 2022-07-12T08:50:34.532 INFO:teuthology.orchestra.run.smithi130.stdout: mon health detail to clog = false 2022-07-12T08:50:34.532 INFO:teuthology.orchestra.run.smithi130.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:34.532 INFO:teuthology.orchestra.run.smithi130.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:34.532 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.533 INFO:teuthology.orchestra.run.smithi130.stdout:[osd] 2022-07-12T08:50:34.533 INFO:teuthology.orchestra.run.smithi130.stdout: osd journal size = 100 2022-07-12T08:50:34.533 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.533 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:34.533 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub max interval = 600 2022-07-12T08:50:34.534 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.534 INFO:teuthology.orchestra.run.smithi130.stdout: osd recover clone overlap = true 2022-07-12T08:50:34.534 INFO:teuthology.orchestra.run.smithi130.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:34.534 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.535 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug shutdown = true 2022-07-12T08:50:34.535 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug op order = true 2022-07-12T08:50:34.535 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:34.535 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.536 INFO:teuthology.orchestra.run.smithi130.stdout: osd open classes on start = true 2022-07-12T08:50:34.536 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug pg log writeout = true 2022-07-12T08:50:34.536 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.536 INFO:teuthology.orchestra.run.smithi130.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:34.536 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.537 INFO:teuthology.orchestra.run.smithi130.stdout: osd map max advance = 10 2022-07-12T08:50:34.537 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.537 INFO:teuthology.orchestra.run.smithi130.stdout: journal zero on create = true 2022-07-12T08:50:34.537 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.537 INFO:teuthology.orchestra.run.smithi130.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:34.538 INFO:teuthology.orchestra.run.smithi130.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:34.538 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.538 INFO:teuthology.orchestra.run.smithi130.stdout: bdev debug aio = true 2022-07-12T08:50:34.538 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug misdirected ops = true 2022-07-12T08:50:34.539 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:34.539 INFO:teuthology.orchestra.run.smithi130.stdout: debug osd = 20 2022-07-12T08:50:34.539 INFO:teuthology.orchestra.run.smithi130.stdout: osd objectstore = filestore 2022-07-12T08:50:34.539 INFO:teuthology.orchestra.run.smithi130.stdout: osd op complaint time = 180 2022-07-12T08:50:34.539 INFO:teuthology.orchestra.run.smithi130.stdout: osd sloppy crc = True 2022-07-12T08:50:34.540 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.540 INFO:teuthology.orchestra.run.smithi130.stdout:[mgr] 2022-07-12T08:50:34.540 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:34.540 INFO:teuthology.orchestra.run.smithi130.stdout: debug mgr = 20 2022-07-12T08:50:34.540 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2022-07-12T08:50:34.541 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2022-07-12T08:50:34.541 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.541 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.541 INFO:teuthology.orchestra.run.smithi130.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:34.541 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.542 INFO:teuthology.orchestra.run.smithi130.stdout:[mon] 2022-07-12T08:50:34.542 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:34.542 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2022-07-12T08:50:34.542 INFO:teuthology.orchestra.run.smithi130.stdout: debug paxos = 20 2022-07-12T08:50:34.543 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2022-07-12T08:50:34.543 INFO:teuthology.orchestra.run.smithi130.stdout: mon data avail warn = 5 2022-07-12T08:50:34.543 INFO:teuthology.orchestra.run.smithi130.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:34.543 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.543 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:34.544 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd prime pg temp = true 2022-07-12T08:50:34.544 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.544 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.544 INFO:teuthology.orchestra.run.smithi130.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:34.545 INFO:teuthology.orchestra.run.smithi130.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:34.545 INFO:teuthology.orchestra.run.smithi130.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:34.545 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.545 INFO:teuthology.orchestra.run.smithi130.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:34.545 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:34.546 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:34.546 INFO:teuthology.orchestra.run.smithi130.stdout: mon op complaint time = 120 2022-07-12T08:50:34.546 INFO:teuthology.orchestra.run.smithi130.stdout: 2022-07-12T08:50:34.546 INFO:teuthology.orchestra.run.smithi130.stdout:[client] 2022-07-12T08:50:34.546 INFO:teuthology.orchestra.run.smithi130.stdout: rgw cache enabled = true 2022-07-12T08:50:34.547 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable ops log = true 2022-07-12T08:50:34.547 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable usage log = true 2022-07-12T08:50:34.547 INFO:teuthology.orchestra.run.smithi130.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:34.547 INFO:teuthology.orchestra.run.smithi130.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:34.547 INFO:teuthology.orchestra.run.smithi130.stdout: client mount timeout = 600 2022-07-12T08:50:34.548 INFO:teuthology.orchestra.run.smithi130.stdout: debug client = 20 2022-07-12T08:50:34.548 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:34.548 INFO:teuthology.orchestra.run.smithi130.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.548 INFO:teuthology.orchestra.run.smithi130.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.549 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.a] 2022-07-12T08:50:34.549 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.b] 2022-07-12T08:50:34.549 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.c] 2022-07-12T08:50:34.549 INFO:teuthology.orchestra.run.smithi130.stdout:[mds.a] 2022-07-12T08:50:34.549 INFO:teuthology.orchestra.run.smithi130.stdout:[mds.b] 2022-07-12T08:50:34.550 INFO:teuthology.orchestra.run.smithi130.stdout:[mds] 2022-07-12T08:50:34.550 INFO:teuthology.orchestra.run.smithi130.stdout: debug mds = 20 2022-07-12T08:50:34.550 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2022-07-12T08:50:34.550 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal frag = True 2022-07-12T08:50:34.550 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:34.551 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal merge size = 5 2022-07-12T08:50:34.551 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal split bits = 3 2022-07-12T08:50:34.551 INFO:teuthology.orchestra.run.smithi130.stdout: mds bal split size = 100 2022-07-12T08:50:34.551 INFO:teuthology.orchestra.run.smithi130.stdout: mds debug frag = True 2022-07-12T08:50:34.552 INFO:teuthology.orchestra.run.smithi130.stdout: mds debug scatterstat = True 2022-07-12T08:50:34.552 INFO:teuthology.orchestra.run.smithi130.stdout: mds op complaint time = 180 2022-07-12T08:50:34.552 INFO:teuthology.orchestra.run.smithi130.stdout: mds verify scatter = True 2022-07-12T08:50:34.552 INFO:teuthology.orchestra.run.smithi130.stdout: osd op complaint time = 180 2022-07-12T08:50:34.552 INFO:teuthology.orchestra.run.smithi130.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.553 INFO:teuthology.orchestra.run.smithi130.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.553 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2022-07-12T08:50:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2022-07-12T08:50:34.553 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-12T08:50:34.554 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2022-07-12T08:50:34.554 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.554 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2022-07-12T08:50:34.554 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.554 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2022-07-12T08:50:34.555 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.555 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2022-07-12T08:50:34.555 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2022-07-12T08:50:34.555 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.555 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2022-07-12T08:50:34.556 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2022-07-12T08:50:34.556 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.556 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-12T08:50:34.556 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2022-07-12T08:50:34.556 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.557 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2022-07-12T08:50:34.557 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-12T08:50:34.557 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.557 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2022-07-12T08:50:34.558 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.558 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2022-07-12T08:50:34.558 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2022-07-12T08:50:34.558 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2022-07-12T08:50:34.558 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2022-07-12T08:50:34.559 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2022-07-12T08:50:34.559 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2022-07-12T08:50:34.559 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2022-07-12T08:50:34.559 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-12T08:50:34.560 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-12T08:50:34.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.560 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-12T08:50:34.560 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.560 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2022-07-12T08:50:34.561 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.561 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2022-07-12T08:50:34.561 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.561 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2022-07-12T08:50:34.561 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2022-07-12T08:50:34.562 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2022-07-12T08:50:34.562 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.61,172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790]" 2022-07-12T08:50:34.562 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a 2022-07-12T08:50:34.562 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.562 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2022-07-12T08:50:34.563 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2022-07-12T08:50:34.563 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.563 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2022-07-12T08:50:34.563 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2022-07-12T08:50:34.564 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.564 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2022-07-12T08:50:34.564 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2022-07-12T08:50:34.564 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.564 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2022-07-12T08:50:34.565 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2022-07-12T08:50:34.565 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2022-07-12T08:50:34.565 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.565 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2022-07-12T08:50:34.565 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2022-07-12T08:50:34.566 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.566 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2022-07-12T08:50:34.566 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.566 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2022-07-12T08:50:34.566 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.567 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2022-07-12T08:50:34.567 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.567 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2022-07-12T08:50:34.567 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2022-07-12T08:50:34.568 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.568 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2022-07-12T08:50:34.568 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2022-07-12T08:50:34.568 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:34.568 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2022-07-12T08:50:34.569 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = filestore 2022-07-12T08:50:34.569 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2022-07-12T08:50:34.569 INFO:teuthology.orchestra.run.smithi111.stdout: osd sloppy crc = True 2022-07-12T08:50:34.569 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.569 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2022-07-12T08:50:34.570 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:34.570 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2022-07-12T08:50:34.570 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2022-07-12T08:50:34.571 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2022-07-12T08:50:34.571 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.571 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.571 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2022-07-12T08:50:34.571 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.572 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2022-07-12T08:50:34.572 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:34.572 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2022-07-12T08:50:34.572 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2022-07-12T08:50:34.572 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2022-07-12T08:50:34.573 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2022-07-12T08:50:34.573 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2022-07-12T08:50:34.573 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2022-07-12T08:50:34.573 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2022-07-12T08:50:34.574 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2022-07-12T08:50:34.574 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2022-07-12T08:50:34.574 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.574 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-12T08:50:34.574 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2022-07-12T08:50:34.575 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2022-07-12T08:50:34.575 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.575 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2022-07-12T08:50:34.575 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-12T08:50:34.575 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-12T08:50:34.576 INFO:teuthology.orchestra.run.smithi111.stdout: mon op complaint time = 120 2022-07-12T08:50:34.576 INFO:teuthology.orchestra.run.smithi111.stdout: 2022-07-12T08:50:34.576 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2022-07-12T08:50:34.576 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2022-07-12T08:50:34.576 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2022-07-12T08:50:34.577 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2022-07-12T08:50:34.577 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-12T08:50:34.577 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-12T08:50:34.577 INFO:teuthology.orchestra.run.smithi111.stdout: client mount timeout = 600 2022-07-12T08:50:34.578 INFO:teuthology.orchestra.run.smithi111.stdout: debug client = 20 2022-07-12T08:50:34.578 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:34.578 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.578 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.579 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2022-07-12T08:50:34.579 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2022-07-12T08:50:34.579 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2022-07-12T08:50:34.579 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.a] 2022-07-12T08:50:34.579 INFO:teuthology.orchestra.run.smithi111.stdout:[mds.b] 2022-07-12T08:50:34.580 INFO:teuthology.orchestra.run.smithi111.stdout:[mds] 2022-07-12T08:50:34.580 INFO:teuthology.orchestra.run.smithi111.stdout: debug mds = 20 2022-07-12T08:50:34.580 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2022-07-12T08:50:34.580 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal frag = True 2022-07-12T08:50:34.580 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal fragment size max = 10000 2022-07-12T08:50:34.581 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal merge size = 5 2022-07-12T08:50:34.581 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal split bits = 3 2022-07-12T08:50:34.581 INFO:teuthology.orchestra.run.smithi111.stdout: mds bal split size = 100 2022-07-12T08:50:34.581 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug frag = True 2022-07-12T08:50:34.581 INFO:teuthology.orchestra.run.smithi111.stdout: mds debug scatterstat = True 2022-07-12T08:50:34.582 INFO:teuthology.orchestra.run.smithi111.stdout: mds op complaint time = 180 2022-07-12T08:50:34.582 INFO:teuthology.orchestra.run.smithi111.stdout: mds verify scatter = True 2022-07-12T08:50:34.582 INFO:teuthology.orchestra.run.smithi111.stdout: osd op complaint time = 180 2022-07-12T08:50:34.582 INFO:teuthology.orchestra.run.smithi111.stdout: rados mon op timeout = 15m 2022-07-12T08:50:34.583 INFO:teuthology.orchestra.run.smithi111.stdout: rados osd op timeout = 15m 2022-07-12T08:50:34.583 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:50:34.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 Processor -- start 2022-07-12T08:50:34.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 -- start start 2022-07-12T08:50:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f205005cd20 0x7f20500b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2050098930 0x7f205005d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20500afe60 con 0x7f2050146690 2022-07-12T08:50:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2050143f40 con 0x7f2050098930 2022-07-12T08:50:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f2055c8b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2050147210 con 0x7f205005cd20 2022-07-12T08:50:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2050098930 0x7f205005d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2050146690 0x7f2050146b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49606/0 (socket says 172.21.15.61:49606) 2022-07-12T08:50:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204ffff700 1 -- 172.21.15.61:0/3375754400 learned_addr learned my addr 172.21.15.61:0/3375754400 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204f7fe700 1 --2- 172.21.15.61:0/3375754400 >> v2:172.21.15.111:3300/0 conn(0x7f205005cd20 0x7f20500b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204e7fc700 1 -- 172.21.15.61:0/3375754400 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837931619 0 0) 0x7f20500afe60 con 0x7f2050146690 2022-07-12T08:50:34.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204e7fc700 1 -- 172.21.15.61:0/3375754400 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c003260 con 0x7f2050146690 2022-07-12T08:50:34.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204effd700 1 -- 172.21.15.61:0/3375754400 >> v2:172.21.15.111:3300/0 conn(0x7f205005cd20 msgr2=0x7f20500b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:34.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204effd700 1 --2- 172.21.15.61:0/3375754400 >> v2:172.21.15.111:3300/0 conn(0x7f205005cd20 0x7f20500b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204effd700 1 -- 172.21.15.61:0/3375754400 >> v1:172.21.15.61:6789/0 conn(0x7f2050146690 legacy=0x7f2050146b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204e7fc700 1 -- 172.21.15.61:0/3375754400 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000032009 0 0) 0x7f203c003260 con 0x7f2050146690 2022-07-12T08:50:34.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.700+0000 7f204effd700 1 -- 172.21.15.61:0/3375754400 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2050146260 con 0x7f2050098930 2022-07-12T08:50:34.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f204effd700 1 --2- 172.21.15.61:0/3375754400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205005d100 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f2048001800 tx=0x7f2044006df0).ready entity=mon.0 client_cookie=bf052a8a5c45fcaa server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:34.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f204e7fc700 1 -- 172.21.15.61:0/3375754400 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2048001660 con 0x7f2050098930 2022-07-12T08:50:34.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f204e7fc700 1 -- 172.21.15.61:0/3375754400 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044000e90 con 0x7f2050098930 2022-07-12T08:50:34.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f204e7fc700 1 -- 172.21.15.61:0/3375754400 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f204400be80 con 0x7f2050098930 2022-07-12T08:50:34.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 -- 172.21.15.61:0/3375754400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 msgr2=0x7f205005d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 --2- 172.21.15.61:0/3375754400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205005d100 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f2048001800 tx=0x7f2044006df0).stop 2022-07-12T08:50:34.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 -- 172.21.15.61:0/3375754400 shutdown_connections 2022-07-12T08:50:34.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 --2- 172.21.15.61:0/3375754400 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205005d100 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 --2- 172.21.15.61:0/3375754400 >> v2:172.21.15.111:3300/0 conn(0x7f205005cd20 0x7f20500b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 -- 172.21.15.61:0/3375754400 >> 172.21.15.61:0/3375754400 conn(0x7f20500b69d0 msgr2=0x7f2050066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 -- 172.21.15.61:0/3375754400 shutdown_connections 2022-07-12T08:50:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.704+0000 7f2055c8b700 1 -- 172.21.15.61:0/3375754400 wait complete. 2022-07-12T08:50:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 Processor -- start 2022-07-12T08:50:34.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 -- start start 2022-07-12T08:50:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f205005cd20 0x7f20500b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205013d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2050146690 0x7f205013dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f205013eae0 con 0x7f2050098930 2022-07-12T08:50:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20500b0e70 con 0x7f2050146690 2022-07-12T08:50:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205014a320 con 0x7f205005cd20 2022-07-12T08:50:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205013d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205013d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33160/0 (socket says 172.21.15.61:33160) 2022-07-12T08:50:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 -- 172.21.15.61:0/1185605182 learned_addr learned my addr 172.21.15.61:0/1185605182 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204f7fe700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f205005cd20 0x7f20500b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204ffff700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2050146690 0x7f205013dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 -- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f205005cd20 msgr2=0x7f20500b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f205005cd20 0x7f20500b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 -- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2050146690 msgr2=0x7f205013dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2050146690 0x7f205013dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 -- 172.21.15.61:0/1185605182 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204400b120 con 0x7f2050098930 2022-07-12T08:50:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204f7fe700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f205005cd20 0x7f20500b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204ffff700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2050146690 0x7f205013dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:34.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204effd700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205013d6d0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f20500b1000 tx=0x7f2044014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:34.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f204401c5f0 con 0x7f2050098930 2022-07-12T08:50:34.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204401c750 con 0x7f2050098930 2022-07-12T08:50:34.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20500b2eb0 con 0x7f2050098930 2022-07-12T08:50:34.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f204401cab0 con 0x7f2050098930 2022-07-12T08:50:34.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.708+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20500b3720 con 0x7f2050098930 2022-07-12T08:50:34.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.712+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20500b3350 con 0x7f2050098930 2022-07-12T08:50:34.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.712+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f204400f070 con 0x7f2050098930 2022-07-12T08:50:34.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.712+0000 7f204cff9700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20300539d0 0x7f2030055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.712+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f204406cf00 con 0x7f2050098930 2022-07-12T08:50:34.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.712+0000 7f204f7fe700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20300539d0 0x7f2030055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.712+0000 7f204f7fe700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20300539d0 0x7f2030055e90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f204000b000 tx=0x7f2040008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:34.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.720+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2044044140 con 0x7f2050098930 2022-07-12T08:50:34.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.844+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f205013e900 con 0x7f2050098930 2022-07-12T08:50:34.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.852+0000 7f204cff9700 1 -- 172.21.15.61:0/1185605182 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2044047760 con 0x7f2050098930 2022-07-12T08:50:34.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20300539d0 msgr2=0x7f2030055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20300539d0 0x7f2030055e90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7f204000b000 tx=0x7f2040008040).stop 2022-07-12T08:50:34.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 msgr2=0x7f205013d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205013d6d0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 rx=0x7f20500b1000 tx=0x7f2044014000).stop 2022-07-12T08:50:34.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 shutdown_connections 2022-07-12T08:50:34.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f20300539d0 0x7f2030055e90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2050146690 0x7f205013dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2050098930 0x7f205013d6d0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 --2- 172.21.15.61:0/1185605182 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f205005cd20 0x7f20500b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:34.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 >> 172.21.15.61:0/1185605182 conn(0x7f20500b69d0 msgr2=0x7f20500b8400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:34.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 shutdown_connections 2022-07-12T08:50:34.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.856+0000 7f2055c8b700 1 -- 172.21.15.61:0/1185605182 wait complete. 2022-07-12T08:50:34.872 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-12T08:50:34.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.976+0000 7fa9a9e86700 1 Processor -- start 2022-07-12T08:50:34.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.976+0000 7fa9a9e86700 1 -- start start 2022-07-12T08:50:34.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.976+0000 7fa9a9e86700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9a40ad7a0 0x7fa9a40b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:34.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.976+0000 7fa9a9e86700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40adb80 con 0x7fa9a4142a20 2022-07-12T08:50:34.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.976+0000 7fa9a9e86700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a40b03d0 con 0x7fa9a4064300 2022-07-12T08:50:34.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.976+0000 7fa9a9e86700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a4146ad0 con 0x7fa9a40ad7a0 2022-07-12T08:50:34.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa9a4142a20 0x7fa9a40b2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49616/0 (socket says 172.21.15.61:49616) 2022-07-12T08:50:34.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a8e84700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9a40ad7a0 0x7fa9a40b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:34.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a3fff700 1 -- 172.21.15.61:0/415681995 learned_addr learned my addr 172.21.15.61:0/415681995 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a37fe700 1 -- 172.21.15.61:0/415681995 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427410461 0 0) 0x7fa9a40adb80 con 0x7fa9a4142a20 2022-07-12T08:50:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a37fe700 1 -- 172.21.15.61:0/415681995 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa990003260 con 0x7fa9a4142a20 2022-07-12T08:50:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a8e84700 1 -- 172.21.15.61:0/415681995 >> v1:172.21.15.111:6789/0 conn(0x7fa9a4064300 legacy=0x7fa9a4141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:34.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a8e84700 1 -- 172.21.15.61:0/415681995 >> v1:172.21.15.61:6789/0 conn(0x7fa9a4142a20 legacy=0x7fa9a40b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a8e84700 1 -- 172.21.15.61:0/415681995 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a4146d70 con 0x7fa9a40ad7a0 2022-07-12T08:50:35.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a8e84700 1 --2- 172.21.15.61:0/415681995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a40b0ed0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7fa998002f70 tx=0x7fa998006df0).ready entity=mon.0 client_cookie=f81cdb39fc89dd67 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:35.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a37fe700 1 -- 172.21.15.61:0/415681995 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa99800d610 con 0x7fa9a40ad7a0 2022-07-12T08:50:35.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a37fe700 1 -- 172.21.15.61:0/415681995 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998000e90 con 0x7fa9a40ad7a0 2022-07-12T08:50:35.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a37fe700 1 -- 172.21.15.61:0/415681995 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa99800d960 con 0x7fa9a40ad7a0 2022-07-12T08:50:35.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 -- 172.21.15.61:0/415681995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a40ad7a0 msgr2=0x7fa9a40b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/415681995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a40b0ed0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7fa998002f70 tx=0x7fa998006df0).stop 2022-07-12T08:50:35.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 -- 172.21.15.61:0/415681995 shutdown_connections 2022-07-12T08:50:35.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/415681995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a40b0ed0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 -- 172.21.15.61:0/415681995 >> 172.21.15.61:0/415681995 conn(0x7fa9a4061e70 msgr2=0x7fa9a4063a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:35.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 -- 172.21.15.61:0/415681995 shutdown_connections 2022-07-12T08:50:35.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.980+0000 7fa9a9e86700 1 -- 172.21.15.61:0/415681995 wait complete. 2022-07-12T08:50:35.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 Processor -- start 2022-07-12T08:50:35.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- start start 2022-07-12T08:50:35.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 0x7fa9a40ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a4155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a4142a20 0x7fa9a4114ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a41146b0 con 0x7fa9a4064300 2022-07-12T08:50:35.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a4066750 con 0x7fa9a40ad7a0 2022-07-12T08:50:35.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a40669f0 con 0x7fa9a4142a20 2022-07-12T08:50:35.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 0x7fa9a40ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 0x7fa9a40ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33172/0 (socket says 172.21.15.61:33172) 2022-07-12T08:50:35.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 -- 172.21.15.61:0/1697492983 learned_addr learned my addr 172.21.15.61:0/1697492983 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:35.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a3fff700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a4155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9685700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a4142a20 0x7fa9a4114ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 -- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a4142a20 msgr2=0x7fa9a4114ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a4142a20 0x7fa9a4114ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 -- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a40ad7a0 msgr2=0x7fa9a4155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a4155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 -- 172.21.15.61:0/1697492983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99800b180 con 0x7fa9a4064300 2022-07-12T08:50:35.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9685700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a4142a20 0x7fa9a4114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:35.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a8e84700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 0x7fa9a40ae8f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7fa998002f70 tx=0x7fa998014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:35.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa998016070 con 0x7fa9a4064300 2022-07-12T08:50:35.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99800de90 con 0x7fa9a4064300 2022-07-12T08:50:35.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a40663d0 con 0x7fa9a4064300 2022-07-12T08:50:35.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa998013e50 con 0x7fa9a4064300 2022-07-12T08:50:35.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a4067920 con 0x7fa9a4064300 2022-07-12T08:50:35.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.984+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a4057460 con 0x7fa9a4064300 2022-07-12T08:50:35.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.988+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa998025030 con 0x7fa9a4064300 2022-07-12T08:50:35.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.988+0000 7fa9a1ffb700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa984053920 0x7fa984055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.988+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa99806cc50 con 0x7fa9a4064300 2022-07-12T08:50:35.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.988+0000 7fa9a3fff700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa984053920 0x7fa984055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.988+0000 7fa9a3fff700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa984053920 0x7fa984055de0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7fa99400d000 tx=0x7fa994009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:35.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:34.992+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa9980478c0 con 0x7fa9a4064300 2022-07-12T08:50:35.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.160+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7fa9a41147f0 con 0x7fa9a4064300 2022-07-12T08:50:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.304+0000 7fa9a1ffb700 1 -- 172.21.15.61:0/1697492983 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v437) v1 ==== 128+0+0 (secure 0 0 0) 0x7fa998023070 con 0x7fa9a4064300 2022-07-12T08:50:35.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa984053920 msgr2=0x7fa984055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa984053920 0x7fa984055de0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7fa99400d000 tx=0x7fa994009040).stop 2022-07-12T08:50:35.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 msgr2=0x7fa9a40ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 0x7fa9a40ae8f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7fa998002f70 tx=0x7fa998014000).stop 2022-07-12T08:50:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 shutdown_connections 2022-07-12T08:50:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9a4142a20 0x7fa9a4114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa984053920 0x7fa984055de0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9a40ad7a0 0x7fa9a4155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 --2- 172.21.15.61:0/1697492983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9a4064300 0x7fa9a40ae8f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 >> 172.21.15.61:0/1697492983 conn(0x7fa9a4061e70 msgr2=0x7fa9a405f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 shutdown_connections 2022-07-12T08:50:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.308+0000 7fa9a9e86700 1 -- 172.21.15.61:0/1697492983 wait complete. 2022-07-12T08:50:35.326 INFO:tasks.cephfs_test_runner:test_getattr_caps (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:50:35.327 INFO:tasks.cephfs_test_runner:Starting test: test_root_rctime (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:50:35.327 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-12T08:50:35.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 Processor -- start 2022-07-12T08:50:35.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 -- start start 2022-07-12T08:50:35.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe1f00b5780 0x7fe1f0146540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f00abdf0 0x7fe1f0146a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe1f005c730 0x7fe1f005faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f00ab3f0 con 0x7fe1f00b5780 2022-07-12T08:50:35.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f00ab950 con 0x7fe1f005c730 2022-07-12T08:50:35.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6958700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00ab690 con 0x7fe1f00abdf0 2022-07-12T08:50:35.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe1f00b5780 0x7fe1f0146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe1f00b5780 0x7fe1f0146540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33182/0 (socket says 172.21.15.61:33182) 2022-07-12T08:50:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 -- 172.21.15.61:0/4283381887 learned_addr learned my addr 172.21.15.61:0/4283381887 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5155700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f00abdf0 0x7fe1f0146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f6157700 1 --2- 172.21.15.61:0/4283381887 >> v2:172.21.15.111:3300/0 conn(0x7fe1f005c730 0x7fe1f005faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 -- 172.21.15.61:0/4283381887 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f00abdf0 msgr2=0x7fe1f0146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f00abdf0 0x7fe1f0146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 -- 172.21.15.61:0/4283381887 >> v2:172.21.15.111:3300/0 conn(0x7fe1f005c730 msgr2=0x7fe1f005faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:35.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 --2- 172.21.15.61:0/4283381887 >> v2:172.21.15.111:3300/0 conn(0x7fe1f005c730 0x7fe1f005faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 -- 172.21.15.61:0/4283381887 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f00b5b60 con 0x7fe1f00b5780 2022-07-12T08:50:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5155700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f00abdf0 0x7fe1f0146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.448+0000 7fe1f5956700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f0146540 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7fe1f00abcd0 tx=0x7fe1e0006eb0).ready entity=mon.0 client_cookie=9975f61d390ca5fc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f4954700 1 -- 172.21.15.61:0/4283381887 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e000d570 con 0x7fe1f00b5780 2022-07-12T08:50:35.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f4954700 1 -- 172.21.15.61:0/4283381887 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e0000e90 con 0x7fe1f00b5780 2022-07-12T08:50:35.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f4954700 1 -- 172.21.15.61:0/4283381887 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e000d900 con 0x7fe1f00b5780 2022-07-12T08:50:35.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 -- 172.21.15.61:0/4283381887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 msgr2=0x7fe1f0146540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f0146540 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7fe1f00abcd0 tx=0x7fe1e0006eb0).stop 2022-07-12T08:50:35.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 -- 172.21.15.61:0/4283381887 shutdown_connections 2022-07-12T08:50:35.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f00abdf0 0x7fe1f0146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 --2- 172.21.15.61:0/4283381887 >> v2:172.21.15.111:3300/0 conn(0x7fe1f005c730 0x7fe1f005faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 --2- 172.21.15.61:0/4283381887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f0146540 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 -- 172.21.15.61:0/4283381887 >> 172.21.15.61:0/4283381887 conn(0x7fe1f0063380 msgr2=0x7fe1f00b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:35.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 -- 172.21.15.61:0/4283381887 shutdown_connections 2022-07-12T08:50:35.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 -- 172.21.15.61:0/4283381887 wait complete. 2022-07-12T08:50:35.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 Processor -- start 2022-07-12T08:50:35.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.452+0000 7fe1f6958700 1 -- start start 2022-07-12T08:50:35.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f005c730 0x7fe1f00aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1f00abdf0 0x7fe1f01564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f01569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0157310 con 0x7fe1f00b5780 2022-07-12T08:50:35.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0156090 con 0x7fe1f00abdf0 2022-07-12T08:50:35.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f00677e0 con 0x7fe1f005c730 2022-07-12T08:50:35.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f01569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f01569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33190/0 (socket says 172.21.15.61:33190) 2022-07-12T08:50:35.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 -- 172.21.15.61:0/1024187737 learned_addr learned my addr 172.21.15.61:0/1024187737 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:35.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f5956700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f005c730 0x7fe1f00aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f5155700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1f00abdf0 0x7fe1f01564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 -- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f005c730 msgr2=0x7fe1f00aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f005c730 0x7fe1f00aece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 -- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1f00abdf0 msgr2=0x7fe1f01564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:35.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1f00abdf0 0x7fe1f01564f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:35.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 -- 172.21.15.61:0/1024187737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e000b1e0 con 0x7fe1f00b5780 2022-07-12T08:50:35.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6157700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f01569e0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7fe1e800c000 tx=0x7fe1e8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:35.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e8011750 con 0x7fe1f00b5780 2022-07-12T08:50:35.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e80118b0 con 0x7fe1f00b5780 2022-07-12T08:50:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f0067a80 con 0x7fe1f00b5780 2022-07-12T08:50:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1e8011c10 con 0x7fe1f00b5780 2022-07-12T08:50:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.456+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f0099d40 con 0x7fe1f00b5780 2022-07-12T08:50:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.460+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f01487a0 con 0x7fe1f00b5780 2022-07-12T08:50:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.460+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe1e8011d70 con 0x7fe1f00b5780 2022-07-12T08:50:35.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.460+0000 7fe1deffd700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1d00539b0 0x7fe1d0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:35.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.460+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe1e806bec0 con 0x7fe1f00b5780 2022-07-12T08:50:35.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.460+0000 7fe1f5956700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1d00539b0 0x7fe1d0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:35.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.460+0000 7fe1f5956700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1d00539b0 0x7fe1d0055e70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fe1e001c000 tx=0x7fe1e0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:35.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.464+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe1e8043100 con 0x7fe1f00b5780 2022-07-12T08:50:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:35.636+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7fe1f0067c10 con 0x7fe1f00b5780 2022-07-12T08:50:36.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.304+0000 7fe1deffd700 1 -- 172.21.15.61:0/1024187737 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v438) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe1e8046720 con 0x7fe1f00b5780 2022-07-12T08:50:36.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1d00539b0 msgr2=0x7fe1d0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1d00539b0 0x7fe1d0055e70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7fe1e001c000 tx=0x7fe1e0015000).stop 2022-07-12T08:50:36.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 msgr2=0x7fe1f01569e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f01569e0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 rx=0x7fe1e800c000 tx=0x7fe1e8006df0).stop 2022-07-12T08:50:36.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 shutdown_connections 2022-07-12T08:50:36.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1f00b5780 0x7fe1f01569e0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe1d00539b0 0x7fe1d0055e70 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe1f00abdf0 0x7fe1f01564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 --2- 172.21.15.61:0/1024187737 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe1f005c730 0x7fe1f00aece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 >> 172.21.15.61:0/1024187737 conn(0x7fe1f0063380 msgr2=0x7fe1f00b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:36.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 shutdown_connections 2022-07-12T08:50:36.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.308+0000 7fe1f6958700 1 -- 172.21.15.61:0/1024187737 wait complete. 2022-07-12T08:50:36.329 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:50:36.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 Processor -- start 2022-07-12T08:50:36.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- start start 2022-07-12T08:50:36.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd8fc14a650 0x7fd8fc146460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8fc146950 0x7fd8fc0ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0b5e80 con 0x7fd8fc0b5aa0 2022-07-12T08:50:36.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc05c540 con 0x7fd8fc146950 2022-07-12T08:50:36.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc05d2d0 con 0x7fd8fc14a650 2022-07-12T08:50:36.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8fc146950 0x7fd8fc0ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8fc146950 0x7fd8fc0ac9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33198/0 (socket says 172.21.15.61:33198) 2022-07-12T08:50:36.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 -- 172.21.15.61:0/3625994695 learned_addr learned my addr 172.21.15.61:0/3625994695 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:36.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8faffd700 1 --2- 172.21.15.61:0/3625994695 >> v2:172.21.15.111:3300/0 conn(0x7fd8fc14a650 0x7fd8fc146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 -- 172.21.15.61:0/3625994695 >> v2:172.21.15.111:3300/0 conn(0x7fd8fc14a650 msgr2=0x7fd8fc146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:36.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 --2- 172.21.15.61:0/3625994695 >> v2:172.21.15.111:3300/0 conn(0x7fd8fc14a650 0x7fd8fc146460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 -- 172.21.15.61:0/3625994695 >> v1:172.21.15.111:6789/0 conn(0x7fd8fc0b5aa0 legacy=0x7fd8fc05be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 -- 172.21.15.61:0/3625994695 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0b05a0 con 0x7fd8fc146950 2022-07-12T08:50:36.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fbfff700 1 --2- 172.21.15.61:0/3625994695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0ac9d0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7fd8f40032f0 tx=0x7fd8f4009290).ready entity=mon.0 client_cookie=ad154d3ceeb1495d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:36.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fa7fc700 1 -- 172.21.15.61:0/3625994695 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f400b870 con 0x7fd8fc146950 2022-07-12T08:50:36.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fa7fc700 1 -- 172.21.15.61:0/3625994695 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f400b9d0 con 0x7fd8fc146950 2022-07-12T08:50:36.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd8fa7fc700 1 -- 172.21.15.61:0/3625994695 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8f400bb40 con 0x7fd8fc146950 2022-07-12T08:50:36.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- 172.21.15.61:0/3625994695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 msgr2=0x7fd8fc0ac9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 --2- 172.21.15.61:0/3625994695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0ac9d0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7fd8f40032f0 tx=0x7fd8f4009290).stop 2022-07-12T08:50:36.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- 172.21.15.61:0/3625994695 shutdown_connections 2022-07-12T08:50:36.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 --2- 172.21.15.61:0/3625994695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0ac9d0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 --2- 172.21.15.61:0/3625994695 >> v2:172.21.15.111:3300/0 conn(0x7fd8fc14a650 0x7fd8fc146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- 172.21.15.61:0/3625994695 >> 172.21.15.61:0/3625994695 conn(0x7fd8fc0634d0 msgr2=0x7fd8fc0b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:36.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- 172.21.15.61:0/3625994695 shutdown_connections 2022-07-12T08:50:36.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.444+0000 7fd902182700 1 -- 172.21.15.61:0/3625994695 wait complete. 2022-07-12T08:50:36.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 Processor -- start 2022-07-12T08:50:36.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 -- start start 2022-07-12T08:50:36.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8fc0b5aa0 0x7fd8fc0b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0673a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8fc14a650 0x7fd8fc063a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc14a4b0 con 0x7fd8fc146950 2022-07-12T08:50:36.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc09e410 con 0x7fd8fc14a650 2022-07-12T08:50:36.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd902182700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc15a7a0 con 0x7fd8fc0b5aa0 2022-07-12T08:50:36.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd8faffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0673a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd8faffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0673a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33202/0 (socket says 172.21.15.61:33202) 2022-07-12T08:50:36.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd8fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8fc14a650 0x7fd8fc063a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd8faffd700 1 -- 172.21.15.61:0/985429576 learned_addr learned my addr 172.21.15.61:0/985429576 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:36.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.448+0000 7fd8fb7fe700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8fc0b5aa0 0x7fd8fc0b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8faffd700 1 -- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8fc0b5aa0 msgr2=0x7fd8fc0b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8faffd700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8fc0b5aa0 0x7fd8fc0b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8faffd700 1 -- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8fc14a650 msgr2=0x7fd8fc063a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8faffd700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8fc14a650 0x7fd8fc063a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8faffd700 1 -- 172.21.15.61:0/985429576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f4009530 con 0x7fd8fc146950 2022-07-12T08:50:36.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8fbfff700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8fc14a650 0x7fd8fc063a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:36.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8faffd700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0673a0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7fd8ec00c000 tx=0x7fd8ec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:36.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec011750 con 0x7fd8fc146950 2022-07-12T08:50:36.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0118b0 con 0x7fd8fc146950 2022-07-12T08:50:36.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8ec011c10 con 0x7fd8fc146950 2022-07-12T08:50:36.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc15aa40 con 0x7fd8fc146950 2022-07-12T08:50:36.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc156720 con 0x7fd8fc146950 2022-07-12T08:50:36.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc057460 con 0x7fd8fc146950 2022-07-12T08:50:36.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd8ec011d70 con 0x7fd8fc146950 2022-07-12T08:50:36.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8f8ff9700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8dc053ad0 0x7fd8dc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.452+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd8ec06be90 con 0x7fd8fc146950 2022-07-12T08:50:36.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.456+0000 7fd8fb7fe700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8dc053ad0 0x7fd8dc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.456+0000 7fd8fb7fe700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8dc053ad0 0x7fd8dc055f90 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fd8f000d000 tx=0x7fd8f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:36.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.460+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd8ec0430d0 con 0x7fd8fc146950 2022-07-12T08:50:36.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.596+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd8fc15abd0 con 0x7fd8fc146950 2022-07-12T08:50:36.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd8f8ff9700 1 -- 172.21.15.61:0/985429576 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd8fc15abd0 con 0x7fd8fc146950 2022-07-12T08:50:36.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8dc053ad0 msgr2=0x7fd8dc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8dc053ad0 0x7fd8dc055f90 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7fd8f000d000 tx=0x7fd8f0009040).stop 2022-07-12T08:50:36.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 msgr2=0x7fd8fc0673a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0673a0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7fd8ec00c000 tx=0x7fd8ec006df0).stop 2022-07-12T08:50:36.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 shutdown_connections 2022-07-12T08:50:36.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd8dc053ad0 0x7fd8dc055f90 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8fc14a650 0x7fd8fc063a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8fc146950 0x7fd8fc0673a0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 --2- 172.21.15.61:0/985429576 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd8fc0b5aa0 0x7fd8fc0b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 >> 172.21.15.61:0/985429576 conn(0x7fd8fc0634d0 msgr2=0x7fd8fc0b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:36.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 shutdown_connections 2022-07-12T08:50:36.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.604+0000 7fd902182700 1 -- 172.21.15.61:0/985429576 wait complete. 2022-07-12T08:50:36.619 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:50:36.620 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:36.641 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:36.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:36.687 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:36.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:36.734 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:50:36.735 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:36.735 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:36.736 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:36.746 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:50:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:36.749 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:36.750 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:36.751 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:36.752 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:36.760 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:50:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:36.764 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:36.765 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:36.766 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:36.767 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:50:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:36.784 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:50:36.785 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:50:36.787 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:50:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0f59e700 1 Processor -- start 2022-07-12T08:50:36.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0f59e700 1 -- start start 2022-07-12T08:50:36.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0f59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca100adbb0 0x7fca101446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100ae080 con 0x7fca10061f50 2022-07-12T08:50:36.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0f59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca10143f60 con 0x7fca10144b90 2022-07-12T08:50:36.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0f59e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100b2690 con 0x7fca100adbb0 2022-07-12T08:50:36.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0e59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fca10061f50 0x7fca100b1f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49658/0 (socket says 172.21.15.61:49658) 2022-07-12T08:50:36.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0e59c700 1 -- 172.21.15.61:0/1782831704 learned_addr learned my addr 172.21.15.61:0/1782831704 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:36.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0dd9b700 1 --2- 172.21.15.61:0/1782831704 >> v2:172.21.15.111:3300/0 conn(0x7fca100adbb0 0x7fca101446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742534573 0 0) 0x7fca100ae080 con 0x7fca10061f50 2022-07-12T08:50:36.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.896+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f0003260 con 0x7fca10061f50 2022-07-12T08:50:36.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562307043 0 0) 0x7fca10143f60 con 0x7fca10144b90 2022-07-12T08:50:36.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca100ae080 con 0x7fca10144b90 2022-07-12T08:50:36.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435832824 0 0) 0x7fc9f0003260 con 0x7fca10061f50 2022-07-12T08:50:36.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 >> v2:172.21.15.111:3300/0 conn(0x7fca100adbb0 msgr2=0x7fca101446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:36.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 --2- 172.21.15.61:0/1782831704 >> v2:172.21.15.111:3300/0 conn(0x7fca100adbb0 0x7fca101446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 >> v1:172.21.15.111:6789/0 conn(0x7fca10144b90 legacy=0x7fca1013ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10144200 con 0x7fca10061f50 2022-07-12T08:50:36.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fca000019e0 con 0x7fca10061f50 2022-07-12T08:50:36.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 >> v1:172.21.15.61:6789/0 conn(0x7fca10061f50 legacy=0x7fca100b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc9f00038e0 0x7fc9f0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f0004500 0x7fc9f0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9f0004e70 0x7fc9f000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca100b2690 con 0x7fc9f0004e70 2022-07-12T08:50:36.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f000bcf0 con 0x7fc9f0004500 2022-07-12T08:50:36.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f000beb0 con 0x7fc9f00038e0 2022-07-12T08:50:36.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc9f00038e0 0x7fc9f0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0dd9b700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f0004500 0x7fc9f0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0ed9d700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9f0004e70 0x7fc9f000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 -- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f0004500 msgr2=0x7fc9f0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:36.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f0004500 0x7fc9f0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 -- 172.21.15.61:0/1782831704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9f0004e70 msgr2=0x7fc9f000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:36.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9f0004e70 0x7fc9f000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 -- 172.21.15.61:0/1782831704 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f0003260 con 0x7fc9f00038e0 2022-07-12T08:50:36.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0ed9d700 1 -- 172.21.15.61:0/1782831704 reap_dead start 2022-07-12T08:50:36.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc9f00038e0 0x7fc9f0003ce0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc9f00054d0 tx=0x7fca101440f0).ready entity=mon.2 client_cookie=a0ad4dc3a0e63704 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:36.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca0000cc60 con 0x7fc9f00038e0 2022-07-12T08:50:36.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0000cdc0 con 0x7fc9f00038e0 2022-07-12T08:50:36.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0d59a700 1 -- 172.21.15.61:0/1782831704 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca00005b80 con 0x7fc9f00038e0 2022-07-12T08:50:36.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 -- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc9f00038e0 msgr2=0x7fc9f0003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc9f00038e0 0x7fc9f0003ce0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fc9f00054d0 tx=0x7fca101440f0).stop 2022-07-12T08:50:36.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 -- 172.21.15.61:0/1782831704 shutdown_connections 2022-07-12T08:50:36.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1782831704 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc9f00038e0 0x7fc9f0003ce0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 -- 172.21.15.61:0/1782831704 >> 172.21.15.61:0/1782831704 conn(0x7fca100b70d0 msgr2=0x7fca100b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:36.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 -- 172.21.15.61:0/1782831704 shutdown_connections 2022-07-12T08:50:36.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.900+0000 7fca0f59e700 1 -- 172.21.15.61:0/1782831704 wait complete. 2022-07-12T08:50:36.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 Processor -- start 2022-07-12T08:50:36.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 -- start start 2022-07-12T08:50:36.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00038e0 0x7fca101463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca100af700 0x7fca10153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 0x7fca1013f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1009a3a0 con 0x7fca10153830 2022-07-12T08:50:36.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1009a9f0 con 0x7fc9f00038e0 2022-07-12T08:50:36.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1009ac90 con 0x7fca100af700 2022-07-12T08:50:36.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 0x7fca1013f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 0x7fca1013f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33222/0 (socket says 172.21.15.61:33222) 2022-07-12T08:50:36.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 -- 172.21.15.61:0/1104465401 learned_addr learned my addr 172.21.15.61:0/1104465401 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:36.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0dd9b700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca100af700 0x7fca10153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00038e0 0x7fca101463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 -- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca100af700 msgr2=0x7fca10153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca100af700 0x7fca10153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 -- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00038e0 msgr2=0x7fca101463e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:36.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00038e0 0x7fca101463e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:36.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 -- 172.21.15.61:0/1104465401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10152ea0 con 0x7fca10153830 2022-07-12T08:50:36.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0dd9b700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca100af700 0x7fca10153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:36.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0ed9d700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 0x7fca1013f7c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fca100b4ec0 tx=0x7fca08007a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:36.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca08007240 con 0x7fca10153830 2022-07-12T08:50:36.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca080073a0 con 0x7fca10153830 2022-07-12T08:50:36.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca10153cb0 con 0x7fca10153830 2022-07-12T08:50:36.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca08006750 con 0x7fca10153830 2022-07-12T08:50:36.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.904+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1009a5c0 con 0x7fca10153830 2022-07-12T08:50:36.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.908+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fca080068b0 con 0x7fca10153830 2022-07-12T08:50:36.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.908+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1009a840 con 0x7fca10153830 2022-07-12T08:50:36.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.908+0000 7fca077fe700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc9f40539b0 0x7fc9f4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:36.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.908+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fca0806ba70 con 0x7fca10153830 2022-07-12T08:50:36.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.908+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc9f40539b0 0x7fc9f4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:36.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.908+0000 7fca0e59c700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc9f40539b0 0x7fc9f4055e70 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fca00018000 tx=0x7fca00013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:36.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:36.912+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca08042cb0 con 0x7fca10153830 2022-07-12T08:50:37.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.044+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fca10153030 con 0x7fca10153830 2022-07-12T08:50:37.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.044+0000 7fca077fe700 1 -- 172.21.15.61:0/1104465401 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v131) v1 ==== 81+0+28885 (secure 0 0 0) 0x7fca0806be60 con 0x7fca10153830 2022-07-12T08:50:37.049 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:37.049 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:50:37.050 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 131, 2022-07-12T08:50:37.050 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:50:37.050 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:50:37.051 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:50:34.288043+0000", 2022-07-12T08:50:37.051 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:37.051 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:37.052 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:50:37.052 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:50:37.052 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:50:37.053 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:50:37.053 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:50:37.054 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:50:37.054 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:50:37.055 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:37.055 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:50:37.055 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:50:37.055 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:50:37.056 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:50:37.056 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:50:37.056 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 26, 2022-07-12T08:50:37.056 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:50:37.056 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:50:37.057 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:50:37.057 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:50:37.057 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:50:37.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.059 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:50:37.060 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:50:37.060 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:50:37.060 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:37.060 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:37.061 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:37.061 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:37.061 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:37.061 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:37.061 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:37.062 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:37.062 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:50:37.062 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:50:37.062 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:50:37.063 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:50:37.063 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:50:37.063 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:37.063 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:37.064 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:37.064 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:37.064 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:37.065 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:37.065 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:37.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.066 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:50:37.066 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:37.066 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:37.066 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:37.067 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:37.067 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:37.067 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:37.067 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:37.068 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:37.068 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:37.068 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:37.068 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:37.069 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:37.069 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:37.069 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:37.069 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:37.070 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:37.070 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:37.070 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:37.070 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:37.070 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:37.071 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:37.071 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:37.071 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:37.072 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:37.072 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:37.072 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:37.073 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.073 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:37.073 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:37.073 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:37.074 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:37.074 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:37.074 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:37.074 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:37.075 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:37.075 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:37.076 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:37.076 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:37.076 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:50:37.076 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:50:37.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.077 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:37.077 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:50:37.077 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.078 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.078 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.078 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:50:37.078 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:50:37.079 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:50:37.079 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:37.079 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:37.080 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:37.080 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:37.080 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:37.081 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:37.081 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:37.081 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:37.081 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:37.082 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:37.082 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:37.082 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:37.082 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:37.083 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:37.083 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:37.083 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:37.083 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:37.084 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:37.084 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:37.084 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:37.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.085 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:50:37.085 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:37.085 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:37.085 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:37.085 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:37.086 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:37.086 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:37.086 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:37.086 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:37.088 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:37.088 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:37.088 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:37.088 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:37.089 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:37.089 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:37.089 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:37.089 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:37.090 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:37.090 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:37.090 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:37.090 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:37.091 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:37.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.091 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:37.091 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:37.091 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:37.092 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:37.092 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:37.092 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:37.092 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:37.093 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:37.093 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:37.093 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:37.093 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:37.094 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:50:37.094 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:37.094 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:50:37.094 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.094 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.095 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:37.095 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:50:37.095 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.095 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:37.096 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:50:37.096 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.096 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.097 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.097 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.097 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.098 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.098 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.098 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 125, 2022-07-12T08:50:37.098 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.099 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.099 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:50:37.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.102 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.102 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:50:37.103 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.103 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.103 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.104 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.105 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.105 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:50:37.106 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.106 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.107 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:50:37.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.108 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.108 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.108 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.109 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:50:37.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.111 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:50:37.112 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.112 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.112 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.113 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.114 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.114 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:50:37.114 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.115 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.115 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.115 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:50:37.116 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:37.116 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.116 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.117 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:50:37.117 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:50:37.118 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:50:37.118 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:50:37.118 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.118 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.119 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.119 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.120 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:37.120 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:50:37.120 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.121 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.121 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.121 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.121 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.122 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.122 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.122 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:37.123 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.123 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.123 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.125 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:50:37.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.126 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.126 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:50:37.126 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.127 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.128 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.128 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.129 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:50:37.130 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.130 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.131 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.131 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:50:37.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.132 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.132 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.132 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.132 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.133 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.133 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:50:37.134 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:50:37.136 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.136 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.136 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.137 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:50:37.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:50:37.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:37.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.141 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.141 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:50:37.141 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:50:37.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:50:37.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:50:37.142 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.143 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.143 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.144 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:37.145 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:50:37.145 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.145 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.146 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.146 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.146 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:37.147 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.148 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.148 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.149 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.149 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:50:37.150 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.150 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.150 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.150 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:50:37.151 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.151 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.151 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.152 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.152 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.152 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.153 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.153 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:50:37.153 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.154 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.154 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.154 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:50:37.155 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.155 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.155 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.156 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.156 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.157 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:50:37.157 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.158 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.158 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.158 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:50:37.158 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.159 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.159 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.159 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.159 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.159 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.160 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.161 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.161 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:50:37.161 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.162 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.162 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.162 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:50:37.162 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:37.163 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.163 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.163 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:50:37.164 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:50:37.164 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:50:37.164 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:50:37.164 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.165 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.165 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.166 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.166 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.166 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:37.167 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:50:37.167 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.167 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.167 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.168 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.168 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.168 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.169 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.169 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 125, 2022-07-12T08:50:37.169 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.169 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.170 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.170 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.170 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:50:37.171 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.172 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:50:37.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.173 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.173 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.173 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.174 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.174 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:50:37.174 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.175 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.175 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:50:37.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.176 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.176 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.176 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:50:37.178 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.178 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:50:37.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.179 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.179 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.180 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.180 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.180 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:50:37.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.181 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.181 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:50:37.182 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:37.182 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.182 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.183 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:50:37.183 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:50:37.183 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:50:37.184 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:50:37.184 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.184 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.184 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.184 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.185 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.185 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.185 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:37.185 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:50:37.186 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.186 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.186 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.186 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.187 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.187 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:37.187 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.188 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.188 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:50:37.189 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.189 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.190 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.190 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.190 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:50:37.191 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.191 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.191 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.192 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.192 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.192 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.192 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.193 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.193 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:50:37.193 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.194 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.194 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.195 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:50:37.195 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.195 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.195 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.196 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.196 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.197 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.197 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:50:37.197 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.198 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.198 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.198 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:50:37.199 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.199 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.200 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.200 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.200 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.201 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:50:37.201 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.201 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.201 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.202 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:50:37.202 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:37.202 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.202 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.203 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.203 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:50:37.203 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:50:37.203 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:50:37.204 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:50:37.204 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.204 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.204 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.204 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.205 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:37.205 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:50:37.206 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.206 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.206 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.206 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.206 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.207 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.207 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.207 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 119, 2022-07-12T08:50:37.207 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.208 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.208 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:50:37.209 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.210 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:50:37.210 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.211 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.211 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.211 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.211 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.212 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.212 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:50:37.213 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.213 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:50:37.214 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.214 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.215 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.215 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.216 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:50:37.216 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.217 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.217 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:50:37.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.218 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.218 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.219 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:50:37.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:50:37.221 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:37.221 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.222 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.222 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:50:37.222 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:50:37.223 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:50:37.223 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:50:37.223 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.223 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.224 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.224 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.225 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:37.225 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:50:37.225 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.226 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.226 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.226 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.227 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:37.227 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.228 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.228 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:50:37.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:50:37.230 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.231 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.231 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.231 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.231 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.232 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:50:37.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.233 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.233 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.233 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.233 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:50:37.234 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.234 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.234 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:50:37.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.236 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:50:37.237 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.238 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.239 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.239 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.240 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:50:37.240 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.241 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.241 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.241 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:50:37.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:37.242 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.242 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.243 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:50:37.243 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:50:37.243 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:50:37.243 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:50:37.244 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.244 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.244 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.245 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.246 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.246 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:37.247 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:50:37.247 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:37.248 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:37.248 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:37.248 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:37.249 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:37.249 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 126, 2022-07-12T08:50:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:37.251 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.251 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:50:37.252 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.252 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:50:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.254 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:37.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.254 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.255 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.255 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:50:37.255 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.255 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.256 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.256 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:50:37.257 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.257 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.258 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:37.258 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.258 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:50:37.259 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:50:37.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.261 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.261 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.261 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.261 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:37.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:50:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:37.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:50:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:50:37.266 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:50:37.267 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:50:37.267 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:50:37.268 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:37.268 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:37.269 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:37.269 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:37.269 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.270 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:37.270 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:50:37.270 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.270 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:37.271 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.271 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.271 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.271 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.271 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:50:37.272 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.273 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:37.273 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.273 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.274 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.274 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.275 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:50:37.275 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.276 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.276 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:37.276 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.277 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.277 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.277 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.277 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.278 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:50:37.278 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.279 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:37.279 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.279 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.279 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.279 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.280 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.280 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:50:37.280 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.281 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:37.281 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.281 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.282 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.282 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.282 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.282 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:50:37.282 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.283 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:37.283 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.284 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.284 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.284 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.284 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.285 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:50:37.285 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.285 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.286 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:37.286 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.286 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.286 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.287 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.287 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.287 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:50:37.287 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.288 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:37.288 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:37.288 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:37.289 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:37.289 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:37.289 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:37.289 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:37.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:50:37.290 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:37.290 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.290 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:37.290 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:50:37.291 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:50:37.291 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:50:37.291 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:50:37.291 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:50:37.292 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2724832679": "2022-07-13T08:50:30.776412+0000", 2022-07-12T08:50:37.292 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2724832679": "2022-07-13T08:50:30.776412+0000" 2022-07-12T08:50:37.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.292 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:50:37.293 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:50:37.293 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:50:37.293 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:50:37.293 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:50:37.293 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:50:37.294 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:50:37.294 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:50:37.294 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:37.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:37.295 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:50:37.295 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:50:37.295 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:50:37.295 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:50:37.295 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:50:37.296 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:50:37.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc9f40539b0 msgr2=0x7fc9f4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc9f40539b0 0x7fc9f4055e70 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fca00018000 tx=0x7fca00013000).stop 2022-07-12T08:50:37.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 msgr2=0x7fca1013f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 0x7fca1013f7c0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 rx=0x7fca100b4ec0 tx=0x7fca08007a50).stop 2022-07-12T08:50:37.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 shutdown_connections 2022-07-12T08:50:37.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca10153830 0x7fca1013f7c0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca100af700 0x7fca10153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc9f40539b0 0x7fc9f4055e70 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 --2- 172.21.15.61:0/1104465401 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9f00038e0 0x7fca101463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 >> 172.21.15.61:0/1104465401 conn(0x7fca100b70d0 msgr2=0x7fca10141210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 shutdown_connections 2022-07-12T08:50:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.048+0000 7fca0f59e700 1 -- 172.21.15.61:0/1104465401 wait complete. 2022-07-12T08:50:37.303 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:37.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb6ca8700 1 Processor -- start 2022-07-12T08:50:37.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb6ca8700 1 -- start start 2022-07-12T08:50:37.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb6ca8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7bb00ab670 0x7f7bb005f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb6ca8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00abc70 con 0x7f7bb00ab290 2022-07-12T08:50:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb6ca8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb005c720 con 0x7f7bb00afa60 2022-07-12T08:50:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb6ca8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb00aba90 con 0x7f7bb00ab670 2022-07-12T08:50:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb5ca6700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7bb00ab290 0x7f7bb005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49678/0 (socket says 172.21.15.61:49678) 2022-07-12T08:50:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.408+0000 7f7bb5ca6700 1 -- 172.21.15.61:0/795091466 learned_addr learned my addr 172.21.15.61:0/795091466 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/795091466 >> v2:172.21.15.111:3300/0 conn(0x7f7bb00ab670 0x7f7bb005f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905982682 0 0) 0x7f7bb00abc70 con 0x7f7bb00ab290 2022-07-12T08:50:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98003260 con 0x7f7bb00ab290 2022-07-12T08:50:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 206099756 0 0) 0x7f7b98003260 con 0x7f7bb00ab290 2022-07-12T08:50:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 >> v2:172.21.15.111:3300/0 conn(0x7f7bb00ab670 msgr2=0x7f7bb005f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 --2- 172.21.15.61:0/795091466 >> v2:172.21.15.111:3300/0 conn(0x7f7bb00ab670 0x7f7bb005f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 >> v1:172.21.15.111:6789/0 conn(0x7f7bb00afa60 legacy=0x7f7bb0147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb005c9c0 con 0x7f7bb00ab290 2022-07-12T08:50:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f7ba80019e0 con 0x7f7bb00ab290 2022-07-12T08:50:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 >> v1:172.21.15.61:6789/0 conn(0x7f7bb00ab290 legacy=0x7f7bb005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b98003750 0x7f7b98003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7b98004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b98004e70 0x7f7b9800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb00aba90 con 0x7f7b98004500 2022-07-12T08:50:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9800bc70 con 0x7f7b98004e70 2022-07-12T08:50:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9800be30 con 0x7f7b98003750 2022-07-12T08:50:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7b98004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb64a7700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b98004e70 0x7f7b9800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb5ca6700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b98003750 0x7f7b98003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 -- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b98003750 msgr2=0x7f7b98003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:37.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b98003750 0x7f7b98003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 -- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b98004e70 msgr2=0x7f7b9800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:37.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b98004e70 0x7f7b9800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 -- 172.21.15.61:0/795091466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b98003260 con 0x7f7b98004500 2022-07-12T08:50:37.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb64a7700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b98004e70 0x7f7b9800b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:37.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb64a7700 1 -- 172.21.15.61:0/795091466 reap_dead start 2022-07-12T08:50:37.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7b98004980 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f7ba000c000 tx=0x7f7ba0006df0).ready entity=mon.0 client_cookie=e2d01d5d8f331051 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:37.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba000e830 con 0x7f7b98004500 2022-07-12T08:50:37.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba000e990 con 0x7f7b98004500 2022-07-12T08:50:37.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb4ca4700 1 -- 172.21.15.61:0/795091466 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba000ecf0 con 0x7f7b98004500 2022-07-12T08:50:37.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/795091466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 msgr2=0x7f7b98004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7b98004980 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7f7ba000c000 tx=0x7f7ba0006df0).stop 2022-07-12T08:50:37.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/795091466 shutdown_connections 2022-07-12T08:50:37.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/795091466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7b98004980 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/795091466 >> 172.21.15.61:0/795091466 conn(0x7f7bb00b1950 msgr2=0x7f7bb00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:37.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/795091466 shutdown_connections 2022-07-12T08:50:37.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.412+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/795091466 wait complete. 2022-07-12T08:50:37.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 Processor -- start 2022-07-12T08:50:37.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 -- start start 2022-07-12T08:50:37.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7bb0066720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb0067500 0x7f7bb0159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb014ad70 0x7f7bb014b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb014a940 con 0x7f7b98004500 2022-07-12T08:50:37.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb014b6e0 con 0x7f7bb014ad70 2022-07-12T08:50:37.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb0063c40 con 0x7f7bb0067500 2022-07-12T08:50:37.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7bb0066720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7bb0066720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33240/0 (socket says 172.21.15.61:33240) 2022-07-12T08:50:37.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 -- 172.21.15.61:0/77108710 learned_addr learned my addr 172.21.15.61:0/77108710 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:37.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb0067500 0x7f7bb0159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb64a7700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb014ad70 0x7f7bb014b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 -- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb0067500 msgr2=0x7f7bb0159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb0067500 0x7f7bb0159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 -- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb014ad70 msgr2=0x7f7bb014b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb014ad70 0x7f7bb014b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 -- 172.21.15.61:0/77108710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0063ee0 con 0x7f7b98004500 2022-07-12T08:50:37.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb5ca6700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7bb0066720 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f7ba800d000 tx=0x7f7bb00aadc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:37.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba800fac0 con 0x7f7b98004500 2022-07-12T08:50:37.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba800fc20 con 0x7f7b98004500 2022-07-12T08:50:37.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba8012350 con 0x7f7b98004500 2022-07-12T08:50:37.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb00638c0 con 0x7f7b98004500 2022-07-12T08:50:37.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.416+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0149470 con 0x7f7b98004500 2022-07-12T08:50:37.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.420+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb0057460 con 0x7f7b98004500 2022-07-12T08:50:37.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.420+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7ba801b030 con 0x7f7b98004500 2022-07-12T08:50:37.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.420+0000 7f7b9effd700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053af0 0x7f7b90055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.420+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7ba806ba70 con 0x7f7b98004500 2022-07-12T08:50:37.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.420+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053af0 0x7f7b90055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.420+0000 7f7bb54a5700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053af0 0x7f7b90055fb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f7ba001d000 tx=0x7f7ba0019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:37.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.424+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7ba8042cb0 con 0x7f7b98004500 2022-07-12T08:50:37.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb0063a50 con 0x7f7b98004500 2022-07-12T08:50:37.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7b9effd700 1 -- 172.21.15.61:0/77108710 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f7ba806be60 con 0x7f7b98004500 2022-07-12T08:50:37.595 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:37.596 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71},{"gid":8364,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1967615971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1967615971},{"type":"v1","addr":"172.21.15.61:6833","nonce":1967615971}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":74}],"filesystems":[]} 2022-07-12T08:50:37.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053af0 msgr2=0x7f7b90055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053af0 0x7f7b90055fb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f7ba001d000 tx=0x7f7ba0019000).stop 2022-07-12T08:50:37.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 msgr2=0x7f7bb0066720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7bb0066720 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 rx=0x7f7ba800d000 tx=0x7f7bb00aadc0).stop 2022-07-12T08:50:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 shutdown_connections 2022-07-12T08:50:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053af0 0x7f7b90055fb0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb014ad70 0x7f7bb014b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb0067500 0x7f7bb0159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 --2- 172.21.15.61:0/77108710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b98004500 0x7f7bb0066720 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 >> 172.21.15.61:0/77108710 conn(0x7f7bb00b1950 msgr2=0x7f7bb00afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:37.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 shutdown_connections 2022-07-12T08:50:37.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.592+0000 7f7bb6ca8700 1 -- 172.21.15.61:0/77108710 wait complete. 2022-07-12T08:50:37.601 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 74 2022-07-12T08:50:37.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.716+0000 7f2089279700 1 Processor -- start 2022-07-12T08:50:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.716+0000 7f2089279700 1 -- start start 2022-07-12T08:50:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20840af280 0x7f2084060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f208409bce0 0x7f20840af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840b3f60 0x7f20840b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2084143b00 con 0x7f20840af280 2022-07-12T08:50:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20840aecb0 con 0x7f208409bce0 2022-07-12T08:50:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20841435c0 con 0x7f20840b3f60 2022-07-12T08:50:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20840af280 0x7f2084060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20840af280 0x7f2084060680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33250/0 (socket says 172.21.15.61:33250) 2022-07-12T08:50:37.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 -- 172.21.15.61:0/3116920175 learned_addr learned my addr 172.21.15.61:0/3116920175 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:37.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2088a78700 1 --2- 172.21.15.61:0/3116920175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840b3f60 0x7f20840b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f207b7fe700 1 --2- 172.21.15.61:0/3116920175 >> v2:172.21.15.111:3300/0 conn(0x7f208409bce0 0x7f20840af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 -- 172.21.15.61:0/3116920175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840b3f60 msgr2=0x7f20840b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:37.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 --2- 172.21.15.61:0/3116920175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840b3f60 0x7f20840b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 -- 172.21.15.61:0/3116920175 >> v2:172.21.15.111:3300/0 conn(0x7f208409bce0 msgr2=0x7f20840af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:37.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 --2- 172.21.15.61:0/3116920175 >> v2:172.21.15.111:3300/0 conn(0x7f208409bce0 0x7f20840af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 -- 172.21.15.61:0/3116920175 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084143860 con 0x7f20840af280 2022-07-12T08:50:37.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2083fff700 1 --2- 172.21.15.61:0/3116920175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840af280 0x7f2084060680 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f2074002f70 tx=0x7f2074006e50).ready entity=mon.0 client_cookie=f5a2e44edbee39ef server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:37.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f20837fe700 1 -- 172.21.15.61:0/3116920175 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f207400d570 con 0x7f20840af280 2022-07-12T08:50:37.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f20837fe700 1 -- 172.21.15.61:0/3116920175 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2074000e90 con 0x7f20840af280 2022-07-12T08:50:37.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f20837fe700 1 -- 172.21.15.61:0/3116920175 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f207400d900 con 0x7f20840af280 2022-07-12T08:50:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- 172.21.15.61:0/3116920175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840af280 msgr2=0x7f2084060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- 172.21.15.61:0/3116920175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840af280 0x7f2084060680 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 rx=0x7f2074002f70 tx=0x7f2074006e50).stop 2022-07-12T08:50:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- 172.21.15.61:0/3116920175 shutdown_connections 2022-07-12T08:50:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- 172.21.15.61:0/3116920175 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840b3f60 0x7f20840b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- 172.21.15.61:0/3116920175 >> v2:172.21.15.111:3300/0 conn(0x7f208409bce0 0x7f20840af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 --2- 172.21.15.61:0/3116920175 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840af280 0x7f2084060680 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- 172.21.15.61:0/3116920175 >> 172.21.15.61:0/3116920175 conn(0x7f20840b92c0 msgr2=0x7f20840b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:37.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- 172.21.15.61:0/3116920175 shutdown_connections 2022-07-12T08:50:37.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.720+0000 7f2089279700 1 -- 172.21.15.61:0/3116920175 wait complete. 2022-07-12T08:50:37.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 Processor -- start 2022-07-12T08:50:37.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 -- start start 2022-07-12T08:50:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f208409bce0 0x7f208405c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840af280 0x7f2084155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 0x7f2084143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20841561b0 con 0x7f20840b3f60 2022-07-12T08:50:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20840b07a0 con 0x7f208409bce0 2022-07-12T08:50:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20840b0a40 con 0x7f20840af280 2022-07-12T08:50:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 0x7f2084143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 0x7f2084143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33258/0 (socket says 172.21.15.61:33258) 2022-07-12T08:50:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 -- 172.21.15.61:0/3872923538 learned_addr learned my addr 172.21.15.61:0/3872923538 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2083fff700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f208409bce0 0x7f208405c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f207b7fe700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840af280 0x7f2084155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 -- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840af280 msgr2=0x7f2084155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840af280 0x7f2084155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 -- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f208409bce0 msgr2=0x7f208405c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f208409bce0 0x7f208405c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 -- 172.21.15.61:0/3872923538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f207400b1e0 con 0x7f20840b3f60 2022-07-12T08:50:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2083fff700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f208409bce0 0x7f208405c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2088a78700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 0x7f2084143f70 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f207c00c000 tx=0x7f207c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:37.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f207c00c670 con 0x7f20840b3f60 2022-07-12T08:50:37.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f207c00c7d0 con 0x7f20840b3f60 2022-07-12T08:50:37.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20840b0ce0 con 0x7f20840b3f60 2022-07-12T08:50:37.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f207c00cb30 con 0x7f20840b3f60 2022-07-12T08:50:37.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.724+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20841450b0 con 0x7f20840b3f60 2022-07-12T08:50:37.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.728+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2084057460 con 0x7f20840b3f60 2022-07-12T08:50:37.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.728+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f207c022070 con 0x7f20840b3f60 2022-07-12T08:50:37.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.728+0000 7f2081ffb700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2068053920 0x7f2068055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:37.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.728+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f207c06c2a0 con 0x7f20840b3f60 2022-07-12T08:50:37.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.728+0000 7f2083fff700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2068053920 0x7f2068055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:37.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.728+0000 7f2083fff700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2068053920 0x7f2068055de0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f2074018000 tx=0x7f2074014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:37.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.732+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f207c0433d0 con 0x7f20840b3f60 2022-07-12T08:50:37.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.896+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f208406b2b0 con 0x7f20840b3f60 2022-07-12T08:50:37.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.896+0000 7f2081ffb700 1 -- 172.21.15.61:0/3872923538 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f207c016090 con 0x7f20840b3f60 2022-07-12T08:50:37.902 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:37.902 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6989,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.111:6835/3762454803","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":3762454803},{"type":"v1","addr":"172.21.15.111:6835","nonce":3762454803}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":71},{"gid":8364,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1967615971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1967615971},{"type":"v1","addr":"172.21.15.61:6833","nonce":1967615971}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":74}],"filesystems":[]} 2022-07-12T08:50:37.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2068053920 msgr2=0x7f2068055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2068053920 0x7f2068055de0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7f2074018000 tx=0x7f2074014000).stop 2022-07-12T08:50:37.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 msgr2=0x7f2084143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 0x7f2084143f70 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 rx=0x7f207c00c000 tx=0x7f207c0091d0).stop 2022-07-12T08:50:37.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 shutdown_connections 2022-07-12T08:50:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20840b3f60 0x7f2084143f70 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20840af280 0x7f2084155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2068053920 0x7f2068055de0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 --2- 172.21.15.61:0/3872923538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f208409bce0 0x7f208405c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:37.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 >> 172.21.15.61:0/3872923538 conn(0x7f20840b92c0 msgr2=0x7f208413ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 shutdown_connections 2022-07-12T08:50:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:37.900+0000 7f2089279700 1 -- 172.21.15.61:0/3872923538 wait complete. 2022-07-12T08:50:37.911 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 74 2022-07-12T08:50:37.914 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:50:37.915 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:50:37.915 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:50:37.915 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:50:37.916 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:50:37.916 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:50:37.916 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:50:37.917 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:50:37.932 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:50:37.932 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:50:37.934 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:50:37.934 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:50:37.937 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:50:37.938 DEBUG:teuthology.parallel:result is None 2022-07-12T08:50:37.982 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:50:37.984 DEBUG:teuthology.parallel:result is None 2022-07-12T08:50:37.985 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:50:38.013 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:50:38.055 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:50:38.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 Processor -- start 2022-07-12T08:50:38.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 -- start start 2022-07-12T08:50:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe4e00b35a0 0x7fe4e0096d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4e00b2d20 0x7fe4e00b2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e00b2900 con 0x7fe4e00ac6e0 2022-07-12T08:50:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e00b3980 con 0x7fe4e00b2d20 2022-07-12T08:50:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e651c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e0144df0 con 0x7fe4e00b35a0 2022-07-12T08:50:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4df7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4e00b2d20 0x7fe4e00b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.096+0000 7fe4e4ab9700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe4e00ac6e0 0x7fe4e00acb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49728/0 (socket says 172.21.15.61:49728) 2022-07-12T08:50:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e4ab9700 1 -- 172.21.15.61:0/4013733994 learned_addr learned my addr 172.21.15.61:0/4013733994 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:38.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4dffff700 1 --2- 172.21.15.61:0/4013733994 >> v2:172.21.15.111:3300/0 conn(0x7fe4e00b35a0 0x7fe4e0096d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 -- 172.21.15.61:0/4013733994 >> v2:172.21.15.111:3300/0 conn(0x7fe4e00b35a0 msgr2=0x7fe4e0096d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 --2- 172.21.15.61:0/4013733994 >> v2:172.21.15.111:3300/0 conn(0x7fe4e00b35a0 0x7fe4e0096d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 -- 172.21.15.61:0/4013733994 >> v1:172.21.15.61:6789/0 conn(0x7fe4e00ac6e0 legacy=0x7fe4e00acb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:38.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 -- 172.21.15.61:0/4013733994 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0145090 con 0x7fe4e00b2d20 2022-07-12T08:50:38.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4deffd700 1 -- 172.21.15.61:0/4013733994 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532784145 0 0) 0x7fe4e00b2900 con 0x7fe4e00ac6e0 2022-07-12T08:50:38.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4dffff700 1 --2- 172.21.15.61:0/4013733994 >> v2:172.21.15.111:3300/0 conn(0x7fe4e00b35a0 0x7fe4e0096d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:38.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 --2- 172.21.15.61:0/4013733994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e00b2410 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7fe4c8002f70 tx=0x7fe4c8006df0).ready entity=mon.0 client_cookie=864a15bd3a8c54ec server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:38.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4deffd700 1 -- 172.21.15.61:0/4013733994 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4c800d5c0 con 0x7fe4e00b2d20 2022-07-12T08:50:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4deffd700 1 -- 172.21.15.61:0/4013733994 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c8000e90 con 0x7fe4e00b2d20 2022-07-12T08:50:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4deffd700 1 -- 172.21.15.61:0/4013733994 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4c800d910 con 0x7fe4e00b2d20 2022-07-12T08:50:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/4013733994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 msgr2=0x7fe4e00b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:38.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/4013733994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e00b2410 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7fe4c8002f70 tx=0x7fe4c8006df0).stop 2022-07-12T08:50:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/4013733994 shutdown_connections 2022-07-12T08:50:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/4013733994 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e00b2410 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/4013733994 >> v2:172.21.15.111:3300/0 conn(0x7fe4e00b35a0 0x7fe4e0096d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/4013733994 >> 172.21.15.61:0/4013733994 conn(0x7fe4e00b47e0 msgr2=0x7fe4e005bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:38.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/4013733994 shutdown_connections 2022-07-12T08:50:38.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/4013733994 wait complete. 2022-07-12T08:50:38.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 Processor -- start 2022-07-12T08:50:38.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- start start 2022-07-12T08:50:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4e00ac6e0 0x7fe4e005f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e0060720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4e00b35a0 0x7fe4e013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0094dc0 con 0x7fe4e00b2d20 2022-07-12T08:50:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0060c10 con 0x7fe4e00b35a0 2022-07-12T08:50:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e651c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0060e50 con 0x7fe4e00ac6e0 2022-07-12T08:50:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e0060720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:38.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e0060720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33282/0 (socket says 172.21.15.61:33282) 2022-07-12T08:50:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4df7fe700 1 -- 172.21.15.61:0/3070620349 learned_addr learned my addr 172.21.15.61:0/3070620349 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4dffff700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4e00ac6e0 0x7fe4e005f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.100+0000 7fe4e4ab9700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4e00b35a0 0x7fe4e013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4df7fe700 1 -- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4e00ac6e0 msgr2=0x7fe4e005f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:38.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4df7fe700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4e00ac6e0 0x7fe4e005f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4df7fe700 1 -- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4e00b35a0 msgr2=0x7fe4e013faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4df7fe700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4e00b35a0 0x7fe4e013faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4df7fe700 1 -- 172.21.15.61:0/3070620349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c800b180 con 0x7fe4e00b2d20 2022-07-12T08:50:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4df7fe700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e0060720 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7fe4c8002f70 tx=0x7fe4c8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4c8013de0 con 0x7fe4e00b2d20 2022-07-12T08:50:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c801e400 con 0x7fe4e00b2d20 2022-07-12T08:50:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e00610f0 con 0x7fe4e00b2d20 2022-07-12T08:50:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4c801e760 con 0x7fe4e00b2d20 2022-07-12T08:50:38.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e00b1110 con 0x7fe4e00b2d20 2022-07-12T08:50:38.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e0057460 con 0x7fe4e00b2d20 2022-07-12T08:50:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe4c8025030 con 0x7fe4e00b2d20 2022-07-12T08:50:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dd7fa700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4c4053a40 0x7fe4c4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe4c806c8f0 con 0x7fe4e00b2d20 2022-07-12T08:50:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.104+0000 7fe4dffff700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4c4053a40 0x7fe4c4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:38.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.108+0000 7fe4dffff700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4c4053a40 0x7fe4c4055f00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fe4d000c000 tx=0x7fe4d0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:38.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.112+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4c8043b30 con 0x7fe4e00b2d20 2022-07-12T08:50:38.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:38.248+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fe4e0099f30 con 0x7fe4e00b2d20 2022-07-12T08:50:39.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.096+0000 7fe4dd7fa700 1 -- 172.21.15.61:0/3070620349 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v134) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe4c8023020 con 0x7fe4e00b2d20 2022-07-12T08:50:39.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4c4053a40 msgr2=0x7fe4c4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4c4053a40 0x7fe4c4055f00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7fe4d000c000 tx=0x7fe4d0008040).stop 2022-07-12T08:50:39.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 msgr2=0x7fe4e0060720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e0060720 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7fe4c8002f70 tx=0x7fe4c8014000).stop 2022-07-12T08:50:39.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 shutdown_connections 2022-07-12T08:50:39.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe4c4053a40 0x7fe4c4055f00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe4e00b35a0 0x7fe4e013faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4e00b2d20 0x7fe4e0060720 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 --2- 172.21.15.61:0/3070620349 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe4e00ac6e0 0x7fe4e005f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 >> 172.21.15.61:0/3070620349 conn(0x7fe4e00b47e0 msgr2=0x7fe4e0098b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:39.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 shutdown_connections 2022-07-12T08:50:39.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.100+0000 7fe4e651c700 1 -- 172.21.15.61:0/3070620349 wait complete. 2022-07-12T08:50:39.127 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:50:39.129 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:50:39.130 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:50:39.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 Processor -- start 2022-07-12T08:50:39.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- start start 2022-07-12T08:50:39.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c340b5aa0 0x7f0c340b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3405d3d0 con 0x7f0c34147000 2022-07-12T08:50:39.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340b5e80 con 0x7f0c3405bfc0 2022-07-12T08:50:39.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3414b590 con 0x7f0c340b5aa0 2022-07-12T08:50:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39d93700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0c34147000 0x7f0c34144140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49742/0 (socket says 172.21.15.61:49742) 2022-07-12T08:50:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39d93700 1 -- 172.21.15.61:0/3408833140 learned_addr learned my addr 172.21.15.61:0/3408833140 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39592700 1 --2- 172.21.15.61:0/3408833140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c340b5aa0 0x7f0c340b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185406225 0 0) 0x7f0c3405d3d0 con 0x7f0c34147000 2022-07-12T08:50:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c18003260 con 0x7f0c34147000 2022-07-12T08:50:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764534170 0 0) 0x7f0c340b5e80 con 0x7f0c3405bfc0 2022-07-12T08:50:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c3405d3d0 con 0x7f0c3405bfc0 2022-07-12T08:50:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39592700 1 -- 172.21.15.61:0/3408833140 >> v1:172.21.15.111:6789/0 conn(0x7f0c3405bfc0 legacy=0x7f0c3414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39592700 1 -- 172.21.15.61:0/3408833140 >> v1:172.21.15.61:6789/0 conn(0x7f0c34147000 legacy=0x7f0c34144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39592700 1 -- 172.21.15.61:0/3408833140 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3414a740 con 0x7f0c340b5aa0 2022-07-12T08:50:39.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c39592700 1 --2- 172.21.15.61:0/3408833140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c340b5aa0 0x7f0c340b0380 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0c24002f70 tx=0x7f0c24006df0).ready entity=mon.2 client_cookie=ebe5d4fccc708a98 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:39.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c2400d5c0 con 0x7f0c340b5aa0 2022-07-12T08:50:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c24000e90 con 0x7f0c340b5aa0 2022-07-12T08:50:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c2bfff700 1 -- 172.21.15.61:0/3408833140 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c2400d910 con 0x7f0c340b5aa0 2022-07-12T08:50:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/3408833140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c340b5aa0 msgr2=0x7f0c340b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/3408833140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c340b5aa0 0x7f0c340b0380 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0c24002f70 tx=0x7f0c24006df0).stop 2022-07-12T08:50:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/3408833140 shutdown_connections 2022-07-12T08:50:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/3408833140 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c340b5aa0 0x7f0c340b0380 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/3408833140 >> 172.21.15.61:0/3408833140 conn(0x7f0c34063520 msgr2=0x7f0c340b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/3408833140 shutdown_connections 2022-07-12T08:50:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.272+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/3408833140 wait complete. 2022-07-12T08:50:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 Processor -- start 2022-07-12T08:50:39.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 -- start start 2022-07-12T08:50:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c3405bfc0 0x7f0c340aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 0x7f0c34141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c34147000 0x7f0c34142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340ae810 con 0x7f0c340b5aa0 2022-07-12T08:50:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340ae3b0 con 0x7f0c3405bfc0 2022-07-12T08:50:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c340ae070 con 0x7f0c34147000 2022-07-12T08:50:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 0x7f0c34141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 0x7f0c34141d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33296/0 (socket says 172.21.15.61:33296) 2022-07-12T08:50:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 -- 172.21.15.61:0/866817673 learned_addr learned my addr 172.21.15.61:0/866817673 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c39592700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c3405bfc0 0x7f0c340aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c39d93700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c34147000 0x7f0c34142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 -- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c34147000 msgr2=0x7f0c34142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c34147000 0x7f0c34142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 -- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c3405bfc0 msgr2=0x7f0c340aee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c3405bfc0 0x7f0c340aee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 -- 172.21.15.61:0/866817673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2400b180 con 0x7f0c340b5aa0 2022-07-12T08:50:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c38d91700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 0x7f0c34141d30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f0c340ae200 tx=0x7f0c30007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c30012400 con 0x7f0c340b5aa0 2022-07-12T08:50:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c30012560 con 0x7f0c340b5aa0 2022-07-12T08:50:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c34142710 con 0x7f0c340b5aa0 2022-07-12T08:50:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c300128c0 con 0x7f0c340b5aa0 2022-07-12T08:50:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.276+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c340acec0 con 0x7f0c340b5aa0 2022-07-12T08:50:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.280+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c34057460 con 0x7f0c340b5aa0 2022-07-12T08:50:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.280+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0c30010070 con 0x7f0c340b5aa0 2022-07-12T08:50:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.284+0000 7f0c2a7fc700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c1c0539b0 0x7f0c1c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.284+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0c3006b120 con 0x7f0c340b5aa0 2022-07-12T08:50:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.284+0000 7f0c39592700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c1c0539b0 0x7f0c1c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.284+0000 7f0c39592700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c1c0539b0 0x7f0c1c055e70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f0c24018000 tx=0x7f0c24014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.288+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c300423b0 con 0x7f0c340b5aa0 2022-07-12T08:50:39.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.456+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0c34067a80 con 0x7f0c340b5aa0 2022-07-12T08:50:39.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c2a7fc700 1 -- 172.21.15.61:0/866817673 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f0c300459d0 con 0x7f0c340b5aa0 2022-07-12T08:50:39.466 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:39.466 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:50:39.467 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:50:39.467 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.480 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:50:39.480 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:50:39.481 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.481 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:39.482 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.482 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.483 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.483 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.484 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.485 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.485 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:50:39.486 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:50:39.486 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.487 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:39.487 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.488 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.488 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.489 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.489 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.491 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.491 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:50:39.491 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:50:39.492 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.493 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.493 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.493 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.495 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.495 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.497 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:50:39.498 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:50:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.499 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.500 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.501 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:50:39.502 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:50:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.503 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.504 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.505 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:50:39.506 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:50:39.507 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.508 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.508 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.508 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.509 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.512 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.516 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.516 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:50:39.518 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:50:39.518 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.519 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.519 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.519 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.520 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.520 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.521 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.521 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:50:39.521 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:50:39.522 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.522 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.523 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.523 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.523 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.524 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.524 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.524 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:50:39.524 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:50:39.524 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.525 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.525 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.526 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.526 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.527 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.527 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.528 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:50:39.529 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:50:39.530 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.534 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.536 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.536 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.536 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.537 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.537 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:50:39.538 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:50:39.538 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:39.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:50:39.540 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:50:39.541 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.542 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.542 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.543 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:50:39.543 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:50:39.544 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:39.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:50:39.546 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:50:39.546 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.547 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.548 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:50:39.548 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:50:39.549 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.549 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:50:39.549 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:50:39.550 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:50:39.550 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.552 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:50:39.552 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:50:39.553 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.553 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:50:39.553 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:50:39.554 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:50:39.554 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.558 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.559 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.559 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.564 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:50:39.564 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:50:39.564 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.565 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:50:39.565 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:50:39.565 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.566 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.566 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:39.567 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:39.567 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:50:39.567 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:50:39.568 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:50:39.568 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:50:39.568 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:50:39.568 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:50:39.569 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.569 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:39.569 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:39.570 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:50:39.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c1c0539b0 msgr2=0x7f0c1c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c1c0539b0 0x7f0c1c055e70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f0c24018000 tx=0x7f0c24014000).stop 2022-07-12T08:50:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 msgr2=0x7f0c34141d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 0x7f0c34141d30 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f0c340ae200 tx=0x7f0c30007af0).stop 2022-07-12T08:50:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 shutdown_connections 2022-07-12T08:50:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0c34147000 0x7f0c34142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c340b5aa0 0x7f0c34141d30 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0c1c0539b0 0x7f0c1c055e70 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 --2- 172.21.15.61:0/866817673 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0c3405bfc0 0x7f0c340aee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.460+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 >> 172.21.15.61:0/866817673 conn(0x7f0c34063520 msgr2=0x7f0c34145f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.464+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 shutdown_connections 2022-07-12T08:50:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.464+0000 7f0c3b7f6700 1 -- 172.21.15.61:0/866817673 wait complete. 2022-07-12T08:50:39.576 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:39.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.692+0000 7f7507031700 1 Processor -- start 2022-07-12T08:50:39.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.692+0000 7f7507031700 1 -- start start 2022-07-12T08:50:39.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f75000b15a0 0x7f75000b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750013d240 con 0x7f7500142d50 2022-07-12T08:50:39.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75001431f0 con 0x7f750013ce60 2022-07-12T08:50:39.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75000b24c0 con 0x7f75000b15a0 2022-07-12T08:50:39.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7500142d50 0x7f75000b47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49752/0 (socket says 172.21.15.61:49752) 2022-07-12T08:50:39.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f7fff700 1 -- 172.21.15.61:0/1004217755 learned_addr learned my addr 172.21.15.61:0/1004217755 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:39.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 --2- 172.21.15.61:0/1004217755 >> v2:172.21.15.111:3300/0 conn(0x7f75000b15a0 0x7f75000b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434758003 0 0) 0x7f750013d240 con 0x7f7500142d50 2022-07-12T08:50:39.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ec003260 con 0x7f7500142d50 2022-07-12T08:50:39.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328611223 0 0) 0x7f74ec003260 con 0x7f7500142d50 2022-07-12T08:50:39.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 >> v2:172.21.15.111:3300/0 conn(0x7f75000b15a0 msgr2=0x7f75000b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:39.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 --2- 172.21.15.61:0/1004217755 >> v2:172.21.15.111:3300/0 conn(0x7f75000b15a0 0x7f75000b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 >> v1:172.21.15.111:6789/0 conn(0x7f750013ce60 legacy=0x7f75000b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500143490 con 0x7f7500142d50 2022-07-12T08:50:39.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f74f0001660 con 0x7f7500142d50 2022-07-12T08:50:39.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 >> v1:172.21.15.61:6789/0 conn(0x7f7500142d50 legacy=0x7f75000b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec0037a0 0x7f74ec003bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74ec004500 0x7f74ec004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74ec004e70 0x7f74ec00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75000b24c0 con 0x7f74ec0037a0 2022-07-12T08:50:39.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec00bda0 con 0x7f74ec004500 2022-07-12T08:50:39.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec00bf60 con 0x7f74ec004e70 2022-07-12T08:50:39.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f7fff700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74ec004500 0x7f74ec004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7504dcd700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec0037a0 0x7f74ec003bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74ec004e70 0x7f74ec00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 -- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74ec004500 msgr2=0x7f74ec004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:39.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74ec004500 0x7f74ec004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 -- 172.21.15.61:0/1004217755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec0037a0 msgr2=0x7f74ec003bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:39.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec0037a0 0x7f74ec003bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 -- 172.21.15.61:0/1004217755 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec003260 con 0x7f74ec004e70 2022-07-12T08:50:39.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 -- 172.21.15.61:0/1004217755 reap_dead start 2022-07-12T08:50:39.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f75055ce700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74ec004e70 0x7f74ec00b750 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f74f800c000 tx=0x7f74f80091d0).ready entity=mon.2 client_cookie=5d084fbda536acf3 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:39.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74f800e7e0 con 0x7f74ec004e70 2022-07-12T08:50:39.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f800e940 con 0x7f74ec004e70 2022-07-12T08:50:39.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f74f77fe700 1 -- 172.21.15.61:0/1004217755 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74f800eca0 con 0x7f74ec004e70 2022-07-12T08:50:39.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74ec004e70 msgr2=0x7f74ec00b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74ec004e70 0x7f74ec00b750 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f74f800c000 tx=0x7f74f80091d0).stop 2022-07-12T08:50:39.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- 172.21.15.61:0/1004217755 shutdown_connections 2022-07-12T08:50:39.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 --2- 172.21.15.61:0/1004217755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f74ec004e70 0x7f74ec00b750 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- 172.21.15.61:0/1004217755 >> 172.21.15.61:0/1004217755 conn(0x7f750005f9f0 msgr2=0x7f7500066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:39.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- 172.21.15.61:0/1004217755 shutdown_connections 2022-07-12T08:50:39.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.696+0000 7f7507031700 1 -- 172.21.15.61:0/1004217755 wait complete. 2022-07-12T08:50:39.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 Processor -- start 2022-07-12T08:50:39.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 -- start start 2022-07-12T08:50:39.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 0x7f75000b7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75000ae690 0x7f750013acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7500151e10 0x7f750013b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75000af9b0 con 0x7f74ec004e70 2022-07-12T08:50:39.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75000af710 con 0x7f75000ae690 2022-07-12T08:50:39.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750013b6a0 con 0x7f7500151e10 2022-07-12T08:50:39.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 0x7f75000b7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 0x7f75000b7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33314/0 (socket says 172.21.15.61:33314) 2022-07-12T08:50:39.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 -- 172.21.15.61:0/115818043 learned_addr learned my addr 172.21.15.61:0/115818043 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:39.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f74f7fff700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75000ae690 0x7f750013acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f75055ce700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7500151e10 0x7f750013b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 -- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7500151e10 msgr2=0x7f750013b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7500151e10 0x7f750013b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 -- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75000ae690 msgr2=0x7f750013acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75000ae690 0x7f750013acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 -- 172.21.15.61:0/115818043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75000adbe0 con 0x7f74ec004e70 2022-07-12T08:50:39.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7504dcd700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 0x7f75000b7930 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f74fc00d000 tx=0x7f74fc007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:39.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74fc007e70 con 0x7f74ec004e70 2022-07-12T08:50:39.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0114c0 con 0x7f74ec004e70 2022-07-12T08:50:39.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75000ad340 con 0x7f74ec004e70 2022-07-12T08:50:39.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74fc011820 con 0x7f74ec004e70 2022-07-12T08:50:39.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.700+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75000b8e00 con 0x7f74ec004e70 2022-07-12T08:50:39.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.704+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75000af550 con 0x7f74ec004e70 2022-07-12T08:50:39.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.704+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f74fc01a030 con 0x7f74ec004e70 2022-07-12T08:50:39.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.704+0000 7f74f5ffb700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f74e0053a40 0x7f74e0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:39.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.704+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f74fc06b4b0 con 0x7f74ec004e70 2022-07-12T08:50:39.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.704+0000 7f74f7fff700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f74e0053a40 0x7f74e0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:39.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.704+0000 7f74f7fff700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f74e0053a40 0x7f74e0055f00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f74f0005000 tx=0x7f7500143620).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:39.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.712+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74fc0426e0 con 0x7f74ec004e70 2022-07-12T08:50:39.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.840+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7500057460 con 0x7f74ec004e70 2022-07-12T08:50:39.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.840+0000 7f74f5ffb700 1 -- 172.21.15.61:0/115818043 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f74fc0253d0 con 0x7f74ec004e70 2022-07-12T08:50:39.849 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:39.856 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":134,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:39.088085+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":119,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:39.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f74e0053a40 msgr2=0x7f74e0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f74e0053a40 0x7f74e0055f00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f74f0005000 tx=0x7f7500143620).stop 2022-07-12T08:50:39.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 msgr2=0x7f75000b7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:39.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 0x7f75000b7930 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f74fc00d000 tx=0x7f74fc007a00).stop 2022-07-12T08:50:39.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 shutdown_connections 2022-07-12T08:50:39.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7500151e10 0x7f750013b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f74e0053a40 0x7f74e0055f00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f75000ae690 0x7f750013acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 --2- 172.21.15.61:0/115818043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74ec004e70 0x7f75000b7930 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:39.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 >> 172.21.15.61:0/115818043 conn(0x7f750005f9f0 msgr2=0x7f750005c690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:39.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 shutdown_connections 2022-07-12T08:50:39.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.844+0000 7f7507031700 1 -- 172.21.15.61:0/115818043 wait complete. 2022-07-12T08:50:39.867 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:50:39.868 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:39.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 Processor -- start 2022-07-12T08:50:39.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 -- start start 2022-07-12T08:50:40.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f46b80b9db0 0x7f46b80b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b80b3ea0 0x7f46b805c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f46b805c5f0 0x7f46b80b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b805bd10 con 0x7f46b80b9db0 2022-07-12T08:50:40.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46b80b3120 con 0x7f46b805c5f0 2022-07-12T08:50:40.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bf00e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80b33c0 con 0x7f46b80b3ea0 2022-07-12T08:50:40.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f46b80b9db0 0x7f46b80b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f46b80b9db0 0x7f46b80b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33322/0 (socket says 172.21.15.61:33322) 2022-07-12T08:50:40.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 -- 172.21.15.61:0/2551541180 learned_addr learned my addr 172.21.15.61:0/2551541180 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:40.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46b7fff700 1 --2- 172.21.15.61:0/2551541180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b80b3ea0 0x7f46b805c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bd5ab700 1 --2- 172.21.15.61:0/2551541180 >> v2:172.21.15.111:3300/0 conn(0x7f46b805c5f0 0x7f46b80b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 -- 172.21.15.61:0/2551541180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b80b3ea0 msgr2=0x7f46b805c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 --2- 172.21.15.61:0/2551541180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b80b3ea0 0x7f46b805c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 -- 172.21.15.61:0/2551541180 >> v2:172.21.15.111:3300/0 conn(0x7f46b805c5f0 msgr2=0x7f46b80b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 --2- 172.21.15.61:0/2551541180 >> v2:172.21.15.111:3300/0 conn(0x7f46b805c5f0 0x7f46b80b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 -- 172.21.15.61:0/2551541180 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b80ba190 con 0x7f46b80b9db0 2022-07-12T08:50:40.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46bcdaa700 1 --2- 172.21.15.61:0/2551541180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b9db0 0x7f46b80b2a40 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f46a80032f0 tx=0x7f46a8011b30).ready entity=mon.0 client_cookie=d4b91bcbb6bed7a9 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46b77fe700 1 -- 172.21.15.61:0/2551541180 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46a8016070 con 0x7f46b80b9db0 2022-07-12T08:50:40.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46b77fe700 1 -- 172.21.15.61:0/2551541180 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a800fa40 con 0x7f46b80b9db0 2022-07-12T08:50:40.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.988+0000 7f46b77fe700 1 -- 172.21.15.61:0/2551541180 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46a800fda0 con 0x7f46b80b9db0 2022-07-12T08:50:40.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- 172.21.15.61:0/2551541180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b9db0 msgr2=0x7f46b80b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- 172.21.15.61:0/2551541180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b9db0 0x7f46b80b2a40 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f46a80032f0 tx=0x7f46a8011b30).stop 2022-07-12T08:50:40.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- 172.21.15.61:0/2551541180 shutdown_connections 2022-07-12T08:50:40.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- 172.21.15.61:0/2551541180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b80b3ea0 0x7f46b805c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- 172.21.15.61:0/2551541180 >> v2:172.21.15.111:3300/0 conn(0x7f46b805c5f0 0x7f46b80b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- 172.21.15.61:0/2551541180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b9db0 0x7f46b80b2a40 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- 172.21.15.61:0/2551541180 >> 172.21.15.61:0/2551541180 conn(0x7f46b8067960 msgr2=0x7f46b80b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- 172.21.15.61:0/2551541180 shutdown_connections 2022-07-12T08:50:40.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- 172.21.15.61:0/2551541180 wait complete. 2022-07-12T08:50:40.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 Processor -- start 2022-07-12T08:50:40.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- start start 2022-07-12T08:50:40.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b805c5f0 0x7f46b8143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 0x7f46b80af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80b9db0 0x7f46b8146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80aeaf0 con 0x7f46b80b3ea0 2022-07-12T08:50:40.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b80acc00 con 0x7f46b80b9db0 2022-07-12T08:50:40.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bf00e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46b8061320 con 0x7f46b805c5f0 2022-07-12T08:50:40.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 0x7f46b80af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 0x7f46b80af580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33330/0 (socket says 172.21.15.61:33330) 2022-07-12T08:50:40.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 -- 172.21.15.61:0/513774201 learned_addr learned my addr 172.21.15.61:0/513774201 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:40.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bcdaa700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b805c5f0 0x7f46b8143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46bd5ab700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80b9db0 0x7f46b8146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 -- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b805c5f0 msgr2=0x7f46b8143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b805c5f0 0x7f46b8143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 -- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80b9db0 msgr2=0x7f46b8146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80b9db0 0x7f46b8146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 -- 172.21.15.61:0/513774201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8011dd0 con 0x7f46b80b3ea0 2022-07-12T08:50:40.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.992+0000 7f46b7fff700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 0x7f46b80af580 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f46a00030b0 tx=0x7f46a0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46a0011780 con 0x7f46b80b3ea0 2022-07-12T08:50:40.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a00118e0 con 0x7f46b80b3ea0 2022-07-12T08:50:40.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b80ace30 con 0x7f46b80b3ea0 2022-07-12T08:50:40.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b80afa70 con 0x7f46b80b3ea0 2022-07-12T08:50:40.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46a0011c40 con 0x7f46b80b3ea0 2022-07-12T08:50:40.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f46a001c460 con 0x7f46b80b3ea0 2022-07-12T08:50:40.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46b5ffb700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f469c053ad0 0x7f469c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f46a006be20 con 0x7f46b80b3ea0 2022-07-12T08:50:40.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b80ae260 con 0x7f46b80b3ea0 2022-07-12T08:50:40.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46bcdaa700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f469c053ad0 0x7f469c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:39.996+0000 7f46bcdaa700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f469c053ad0 0x7f469c055f90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f46a8021000 tx=0x7f46a8011b30).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.004+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f46a0042fb0 con 0x7f46b80b3ea0 2022-07-12T08:50:40.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.136+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f46b8057460 con 0x7f46b80b3ea0 2022-07-12T08:50:40.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46b5ffb700 1 -- 172.21.15.61:0/513774201 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v134) v1 ==== 110+0+10 (secure 0 0 0) 0x7f46a00465d0 con 0x7f46b80b3ea0 2022-07-12T08:50:40.147 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:40.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f469c053ad0 msgr2=0x7f469c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f469c053ad0 0x7f469c055f90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f46a8021000 tx=0x7f46a8011b30).stop 2022-07-12T08:50:40.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 msgr2=0x7f46b80af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 0x7f46b80af580 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f46a00030b0 tx=0x7f46a0008040).stop 2022-07-12T08:50:40.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 shutdown_connections 2022-07-12T08:50:40.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f469c053ad0 0x7f469c055f90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80b9db0 0x7f46b8146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46b80b3ea0 0x7f46b80af580 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 --2- 172.21.15.61:0/513774201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f46b805c5f0 0x7f46b8143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 >> 172.21.15.61:0/513774201 conn(0x7f46b8067960 msgr2=0x7f46b80b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 shutdown_connections 2022-07-12T08:50:40.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.140+0000 7f46bf00e700 1 -- 172.21.15.61:0/513774201 wait complete. 2022-07-12T08:50:40.156 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:40.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f850359e700 1 Processor -- start 2022-07-12T08:50:40.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f850359e700 1 -- start start 2022-07-12T08:50:40.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f850359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f85040ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f850359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85040ab170 con 0x7f850405bd10 2022-07-12T08:50:40.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f850359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8504061b90 con 0x7f85040aad90 2022-07-12T08:50:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f850359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85041435c0 con 0x7f85040b2e70 2022-07-12T08:50:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f8502d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f850405bd10 0x7f8504141a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49790/0 (socket says 172.21.15.61:49790) 2022-07-12T08:50:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f8502d9d700 1 -- 172.21.15.61:0/989030382 learned_addr learned my addr 172.21.15.61:0/989030382 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f84fbfff700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f85040ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895235332 0 0) 0x7f85040ab170 con 0x7f850405bd10 2022-07-12T08:50:40.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.264+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e4003260 con 0x7f850405bd10 2022-07-12T08:50:40.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919760752 0 0) 0x7f84e4003260 con 0x7f850405bd10 2022-07-12T08:50:40.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 msgr2=0x7f85040ab490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f85040ab490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 >> v1:172.21.15.111:6789/0 conn(0x7f85040aad90 legacy=0x7f85040603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8504143860 con 0x7f850405bd10 2022-07-12T08:50:40.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f84fc0019e0 con 0x7f850405bd10 2022-07-12T08:50:40.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 >> v1:172.21.15.61:6789/0 conn(0x7f850405bd10 legacy=0x7f8504141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84e4003780 0x7f84e4003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84e4004500 0x7f84e4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f84e4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85041435c0 con 0x7f84e4003780 2022-07-12T08:50:40.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e400b780 con 0x7f84e4004500 2022-07-12T08:50:40.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e400b940 con 0x7f85040b2e70 2022-07-12T08:50:40.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84e4003780 0x7f84e4003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8502d9d700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f84e4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f84fbfff700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84e4004500 0x7f84e4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 -- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 msgr2=0x7f84e4004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f84e4004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 -- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84e4004500 msgr2=0x7f84e4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84e4004500 0x7f84e4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 -- 172.21.15.61:0/989030382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4003260 con 0x7f84e4003780 2022-07-12T08:50:40.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f84fbfff700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f84e4004500 0x7f84e4004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:40.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8502d9d700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f85040b2e70 0x7f84e4004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:40.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850259c700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84e4003780 0x7f84e4003ba0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f8504062bd0 tx=0x7f84f4007a00).ready entity=mon.0 client_cookie=1f4a69db9fae8f58 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f4007d10 con 0x7f84e4003780 2022-07-12T08:50:40.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4007030 con 0x7f84e4003780 2022-07-12T08:50:40.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8501d9b700 1 -- 172.21.15.61:0/989030382 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f4007390 con 0x7f84e4003780 2022-07-12T08:50:40.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 -- 172.21.15.61:0/989030382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84e4003780 msgr2=0x7f84e4003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 --2- 172.21.15.61:0/989030382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84e4003780 0x7f84e4003ba0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f8504062bd0 tx=0x7f84f4007a00).stop 2022-07-12T08:50:40.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f8502d9d700 1 -- 172.21.15.61:0/989030382 reap_dead start 2022-07-12T08:50:40.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 -- 172.21.15.61:0/989030382 shutdown_connections 2022-07-12T08:50:40.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 -- 172.21.15.61:0/989030382 >> 172.21.15.61:0/989030382 conn(0x7f85040b57c0 msgr2=0x7f85040b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 -- 172.21.15.61:0/989030382 shutdown_connections 2022-07-12T08:50:40.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 -- 172.21.15.61:0/989030382 wait complete. 2022-07-12T08:50:40.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 Processor -- start 2022-07-12T08:50:40.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.268+0000 7f850359e700 1 -- start start 2022-07-12T08:50:40.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f850413fcd0 0x7f85040b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 0x7f85040b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8504152be0 0x7f85040af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504152200 con 0x7f8504151cf0 2022-07-12T08:50:40.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85041524a0 con 0x7f8504152be0 2022-07-12T08:50:40.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8504152740 con 0x7f850413fcd0 2022-07-12T08:50:40.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 0x7f85040b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 0x7f85040b0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33348/0 (socket says 172.21.15.61:33348) 2022-07-12T08:50:40.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 -- 172.21.15.61:0/1923573195 learned_addr learned my addr 172.21.15.61:0/1923573195 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:40.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850259c700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f850413fcd0 0x7f85040b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f8502d9d700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8504152be0 0x7f85040af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 -- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f850413fcd0 msgr2=0x7f85040b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f850413fcd0 0x7f85040b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 -- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8504152be0 msgr2=0x7f85040af060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8504152be0 0x7f85040af060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 -- 172.21.15.61:0/1923573195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85040b2210 con 0x7f8504151cf0 2022-07-12T08:50:40.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fbfff700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 0x7f85040b0920 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f84ec00b000 tx=0x7f84ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84ec00e6c0 con 0x7f8504151cf0 2022-07-12T08:50:40.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec00e820 con 0x7f8504151cf0 2022-07-12T08:50:40.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85040b0e10 con 0x7f8504151cf0 2022-07-12T08:50:40.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84ec00eb80 con 0x7f8504151cf0 2022-07-12T08:50:40.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8504140e10 con 0x7f8504151cf0 2022-07-12T08:50:40.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8504057460 con 0x7f8504151cf0 2022-07-12T08:50:40.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.272+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f84ec024070 con 0x7f8504151cf0 2022-07-12T08:50:40.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.276+0000 7f84fb7fe700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84e8053a60 0x7f84e8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.276+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f84ec06c4c0 con 0x7f8504151cf0 2022-07-12T08:50:40.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.276+0000 7f850259c700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84e8053a60 0x7f84e8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.276+0000 7f850259c700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84e8053a60 0x7f84e8055f20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f84f4019000 tx=0x7f84f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.280+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84ec0435c0 con 0x7f8504151cf0 2022-07-12T08:50:40.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.404+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8504152900 con 0x7f8504151cf0 2022-07-12T08:50:40.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.404+0000 7f84fb7fe700 1 -- 172.21.15.61:0/1923573195 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v134) v1 ==== 92+0+10 (secure 0 0 0) 0x7f84ec01b310 con 0x7f8504151cf0 2022-07-12T08:50:40.414 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:40.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84e8053a60 msgr2=0x7f84e8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84e8053a60 0x7f84e8055f20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f84f4019000 tx=0x7f84f4015000).stop 2022-07-12T08:50:40.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 msgr2=0x7f85040b0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 0x7f85040b0920 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f84ec00b000 tx=0x7f84ec008040).stop 2022-07-12T08:50:40.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 shutdown_connections 2022-07-12T08:50:40.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f84e8053a60 0x7f84e8055f20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8504152be0 0x7f85040af060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8504151cf0 0x7f85040b0920 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 --2- 172.21.15.61:0/1923573195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f850413fcd0 0x7f85040b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 >> 172.21.15.61:0/1923573195 conn(0x7f85040b57c0 msgr2=0x7f850413eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 shutdown_connections 2022-07-12T08:50:40.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.408+0000 7f850359e700 1 -- 172.21.15.61:0/1923573195 wait complete. 2022-07-12T08:50:40.433 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:50:40.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 Processor -- start 2022-07-12T08:50:40.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 -- start start 2022-07-12T08:50:40.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c05f550 0x7f151c146220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f151c1420b0 0x7f151c142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151c0b6ea0 con 0x7f151c0b6ac0 2022-07-12T08:50:40.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c060b40 con 0x7f151c1420b0 2022-07-12T08:50:40.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f1520cb3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c146710 con 0x7f151c05f550 2022-07-12T08:50:40.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f151c0b6ac0 0x7f151c060430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49806/0 (socket says 172.21.15.61:49806) 2022-07-12T08:50:40.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151b7fe700 1 -- 172.21.15.61:0/155480909 learned_addr learned my addr 172.21.15.61:0/155480909 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:40.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151bfff700 1 --2- 172.21.15.61:0/155480909 >> v2:172.21.15.61:3300/0 conn(0x7f151c1420b0 0x7f151c142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151affd700 1 --2- 172.21.15.61:0/155480909 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c05f550 0x7f151c146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151a7fc700 1 -- 172.21.15.61:0/155480909 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745086787 0 0) 0x7f151c0b6ea0 con 0x7f151c0b6ac0 2022-07-12T08:50:40.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151a7fc700 1 -- 172.21.15.61:0/155480909 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1508003260 con 0x7f151c0b6ac0 2022-07-12T08:50:40.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151affd700 1 -- 172.21.15.61:0/155480909 >> v2:172.21.15.61:3300/0 conn(0x7f151c1420b0 msgr2=0x7f151c142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151affd700 1 --2- 172.21.15.61:0/155480909 >> v2:172.21.15.61:3300/0 conn(0x7f151c1420b0 0x7f151c142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151affd700 1 -- 172.21.15.61:0/155480909 >> v1:172.21.15.61:6789/0 conn(0x7f151c0b6ac0 legacy=0x7f151c060430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151affd700 1 -- 172.21.15.61:0/155480909 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c060200 con 0x7f151c05f550 2022-07-12T08:50:40.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.540+0000 7f151affd700 1 --2- 172.21.15.61:0/155480909 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c05f550 0x7f151c146220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f150c002f70 tx=0x7f150c006df0).ready entity=mon.2 client_cookie=1688ea8f139b531a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f151a7fc700 1 -- 172.21.15.61:0/155480909 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150c00d4a0 con 0x7f151c05f550 2022-07-12T08:50:40.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f151a7fc700 1 -- 172.21.15.61:0/155480909 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c000e90 con 0x7f151c05f550 2022-07-12T08:50:40.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f151a7fc700 1 -- 172.21.15.61:0/155480909 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f150c00d7f0 con 0x7f151c05f550 2022-07-12T08:50:40.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 -- 172.21.15.61:0/155480909 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c05f550 msgr2=0x7f151c146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 --2- 172.21.15.61:0/155480909 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c05f550 0x7f151c146220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f150c002f70 tx=0x7f150c006df0).stop 2022-07-12T08:50:40.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 -- 172.21.15.61:0/155480909 shutdown_connections 2022-07-12T08:50:40.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 --2- 172.21.15.61:0/155480909 >> v2:172.21.15.61:3300/0 conn(0x7f151c1420b0 0x7f151c142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 --2- 172.21.15.61:0/155480909 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c05f550 0x7f151c146220 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 -- 172.21.15.61:0/155480909 >> 172.21.15.61:0/155480909 conn(0x7f151c064570 msgr2=0x7f151c0b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 -- 172.21.15.61:0/155480909 shutdown_connections 2022-07-12T08:50:40.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.544+0000 7f1520cb3700 1 -- 172.21.15.61:0/155480909 wait complete. 2022-07-12T08:50:40.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 Processor -- start 2022-07-12T08:50:40.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 -- start start 2022-07-12T08:50:40.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 0x7f151c148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c0b6ac0 0x7f151c0b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c1420b0 0x7f151c0adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1497e0 con 0x7f151c05f550 2022-07-12T08:50:40.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c1492a0 con 0x7f151c1420b0 2022-07-12T08:50:40.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c149540 con 0x7f151c0b6ac0 2022-07-12T08:50:40.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 0x7f151c148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 0x7f151c148b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33362/0 (socket says 172.21.15.61:33362) 2022-07-12T08:50:40.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 -- 172.21.15.61:0/3783108413 learned_addr learned my addr 172.21.15.61:0/3783108413 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:40.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151affd700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c0b6ac0 0x7f151c0b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151bfff700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c1420b0 0x7f151c0adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 -- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c0b6ac0 msgr2=0x7f151c0b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c0b6ac0 0x7f151c0b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 -- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c1420b0 msgr2=0x7f151c0adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c1420b0 0x7f151c0adcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 -- 172.21.15.61:0/3783108413 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c00b180 con 0x7f151c05f550 2022-07-12T08:50:40.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f151b7fe700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 0x7f151c148b10 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7f1510001110 tx=0x7f1510007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15100070d0 con 0x7f151c05f550 2022-07-12T08:50:40.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1510007230 con 0x7f151c05f550 2022-07-12T08:50:40.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c0b0460 con 0x7f151c05f550 2022-07-12T08:50:40.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1510005610 con 0x7f151c05f550 2022-07-12T08:50:40.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.548+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c0ae1c0 con 0x7f151c05f550 2022-07-12T08:50:40.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.552+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c057460 con 0x7f151c05f550 2022-07-12T08:50:40.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.552+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1510010070 con 0x7f151c05f550 2022-07-12T08:50:40.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.552+0000 7f1518ff9700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14fc053940 0x7f14fc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.552+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f151006afe0 con 0x7f151c05f550 2022-07-12T08:50:40.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.552+0000 7f151affd700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14fc053940 0x7f14fc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.552+0000 7f151affd700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14fc053940 0x7f14fc055e00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f150c01a000 tx=0x7f150c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.556+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1510042270 con 0x7f151c05f550 2022-07-12T08:50:40.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.688+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f151c068350 con 0x7f151c05f550 2022-07-12T08:50:40.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.688+0000 7f1518ff9700 1 -- 172.21.15.61:0/3783108413 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v134) v1 ==== 84+0+143 (secure 0 0 0) 0x7f1510018030 con 0x7f151c05f550 2022-07-12T08:50:40.697 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:40.706 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:50:40.707 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:40.707 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:50:40.708 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:50:40.708 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:40.709 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:40.709 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:50:40.710 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:50:40.710 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:40.710 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:50:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.688+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14fc053940 msgr2=0x7f14fc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.688+0000 7f1520cb3700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14fc053940 0x7f14fc055e00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f150c01a000 tx=0x7f150c014000).stop 2022-07-12T08:50:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 msgr2=0x7f151c148b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 0x7f151c148b10 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7f1510001110 tx=0x7f1510007b20).stop 2022-07-12T08:50:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 shutdown_connections 2022-07-12T08:50:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f14fc053940 0x7f14fc055e00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c1420b0 0x7f151c0adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f151c0b6ac0 0x7f151c0b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 --2- 172.21.15.61:0/3783108413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f151c05f550 0x7f151c148b10 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 >> 172.21.15.61:0/3783108413 conn(0x7f151c064570 msgr2=0x7f151c060430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 shutdown_connections 2022-07-12T08:50:40.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.692+0000 7f1520cb3700 1 -- 172.21.15.61:0/3783108413 wait complete. 2022-07-12T08:50:40.720 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:50:40.721 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:50:40.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.828+0000 7f4a21743700 1 Processor -- start 2022-07-12T08:50:40.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.828+0000 7f4a21743700 1 -- start start 2022-07-12T08:50:40.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4a1c142eb0 0x7f4a1c05c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f4a1c05d130 0x7f4a1c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a1c143500 con 0x7f4a1c0b5f10 2022-07-12T08:50:40.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c143290 con 0x7f4a1c142eb0 2022-07-12T08:50:40.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a1c143760 con 0x7f4a1c05d130 2022-07-12T08:50:40.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4a1c142eb0 0x7f4a1c05c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a20f42700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4a1c0b5f10 0x7f4a1c146090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49824/0 (socket says 172.21.15.61:49824) 2022-07-12T08:50:40.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a20f42700 1 -- 172.21.15.61:0/1598683148 learned_addr learned my addr 172.21.15.61:0/1598683148 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:40.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1b7fe700 1 --2- 172.21.15.61:0/1598683148 >> v2:172.21.15.111:3300/0 conn(0x7f4a1c05d130 0x7f4a1c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1affd700 1 -- 172.21.15.61:0/1598683148 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295921848 0 0) 0x7f4a1c143500 con 0x7f4a1c0b5f10 2022-07-12T08:50:40.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1affd700 1 -- 172.21.15.61:0/1598683148 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49fc003260 con 0x7f4a1c0b5f10 2022-07-12T08:50:40.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1bfff700 1 -- 172.21.15.61:0/1598683148 >> v2:172.21.15.111:3300/0 conn(0x7f4a1c05d130 msgr2=0x7f4a1c0b5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:40.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1bfff700 1 --2- 172.21.15.61:0/1598683148 >> v2:172.21.15.111:3300/0 conn(0x7f4a1c05d130 0x7f4a1c0b5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1bfff700 1 -- 172.21.15.61:0/1598683148 >> v1:172.21.15.61:6789/0 conn(0x7f4a1c0b5f10 legacy=0x7f4a1c146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1bfff700 1 -- 172.21.15.61:0/1598683148 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c0b4030 con 0x7f4a1c142eb0 2022-07-12T08:50:40.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1bfff700 1 --2- 172.21.15.61:0/1598683148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c05c6d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f4a1c143c30 tx=0x7f4a0c006df0).ready entity=mon.0 client_cookie=996040ff1717af94 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1affd700 1 -- 172.21.15.61:0/1598683148 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0c00d4f0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1affd700 1 -- 172.21.15.61:0/1598683148 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c000e90 con 0x7f4a1c142eb0 2022-07-12T08:50:40.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a1affd700 1 -- 172.21.15.61:0/1598683148 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a0c00d840 con 0x7f4a1c142eb0 2022-07-12T08:50:40.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- 172.21.15.61:0/1598683148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 msgr2=0x7f4a1c05c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 --2- 172.21.15.61:0/1598683148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c05c6d0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 rx=0x7f4a1c143c30 tx=0x7f4a0c006df0).stop 2022-07-12T08:50:40.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- 172.21.15.61:0/1598683148 shutdown_connections 2022-07-12T08:50:40.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 --2- 172.21.15.61:0/1598683148 >> v2:172.21.15.111:3300/0 conn(0x7f4a1c05d130 0x7f4a1c0b5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 --2- 172.21.15.61:0/1598683148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c05c6d0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- 172.21.15.61:0/1598683148 >> 172.21.15.61:0/1598683148 conn(0x7f4a1c066d70 msgr2=0x7f4a1c067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:40.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- 172.21.15.61:0/1598683148 shutdown_connections 2022-07-12T08:50:40.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.832+0000 7f4a21743700 1 -- 172.21.15.61:0/1598683148 wait complete. 2022-07-12T08:50:40.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 Processor -- start 2022-07-12T08:50:40.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 -- start start 2022-07-12T08:50:40.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a1c05d130 0x7f4a1c152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1c0b5f10 0x7f4a1c0af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c146390 con 0x7f4a1c142eb0 2022-07-12T08:50:40.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c145a80 con 0x7f4a1c05d130 2022-07-12T08:50:40.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c152a00 con 0x7f4a1c0b5f10 2022-07-12T08:50:40.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c145590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33380/0 (socket says 172.21.15.61:33380) 2022-07-12T08:50:40.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 -- 172.21.15.61:0/3481284338 learned_addr learned my addr 172.21.15.61:0/3481284338 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a1bfff700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a1c05d130 0x7f4a1c152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a1b7fe700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1c0b5f10 0x7f4a1c0af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 -- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1c0b5f10 msgr2=0x7f4a1c0af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1c0b5f10 0x7f4a1c0af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 -- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a1c05d130 msgr2=0x7f4a1c152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:40.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a1c05d130 0x7f4a1c152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:40.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 -- 172.21.15.61:0/3481284338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c00b180 con 0x7f4a1c142eb0 2022-07-12T08:50:40.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a20f42700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c145590 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f4a14001490 tx=0x7f4a14007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a14005ba0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14005d00 con 0x7f4a1c142eb0 2022-07-12T08:50:40.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4a140072c0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a1c1450f0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.836+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c144aa0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.840+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1c152bc0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.840+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4a14010070 con 0x7f4a1c142eb0 2022-07-12T08:50:40.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.840+0000 7f4a197fa700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a00053a60 0x7f4a00055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:40.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.840+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4a1406b560 con 0x7f4a1c142eb0 2022-07-12T08:50:40.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.840+0000 7f4a1bfff700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a00053a60 0x7f4a00055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:40.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.840+0000 7f4a1bfff700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a00053a60 0x7f4a00055f20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f4a0c01a000 tx=0x7f4a0c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:40.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.844+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4a140427f0 con 0x7f4a1c142eb0 2022-07-12T08:50:40.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:40.972+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f4a1c144f30 con 0x7f4a1c142eb0 2022-07-12T08:50:41.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.628+0000 7f4a197fa700 1 -- 172.21.15.61:0/3481284338 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v135) v1 ==== 133+0+0 (secure 0 0 0) 0x7f4a14018020 con 0x7f4a1c142eb0 2022-07-12T08:50:41.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a00053a60 msgr2=0x7f4a00055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:41.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a00053a60 0x7f4a00055f20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f4a0c01a000 tx=0x7f4a0c014000).stop 2022-07-12T08:50:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 msgr2=0x7f4a1c145590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c145590 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7f4a14001490 tx=0x7f4a14007e70).stop 2022-07-12T08:50:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 shutdown_connections 2022-07-12T08:50:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4a1c142eb0 0x7f4a1c145590 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4a1c0b5f10 0x7f4a1c0af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4a00053a60 0x7f4a00055f20 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 --2- 172.21.15.61:0/3481284338 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4a1c05d130 0x7f4a1c152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 >> 172.21.15.61:0/3481284338 conn(0x7f4a1c066d70 msgr2=0x7f4a1c0b9880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:41.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 shutdown_connections 2022-07-12T08:50:41.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.632+0000 7f4a21743700 1 -- 172.21.15.61:0/3481284338 wait complete. 2022-07-12T08:50:41.644 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:50:41.650 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:50:41.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.760+0000 7f2567343700 1 Processor -- start 2022-07-12T08:50:41.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.760+0000 7f2567343700 1 -- start start 2022-07-12T08:50:41.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f256005c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:41.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f256005d130 0x7f25600b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:41.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2560143500 con 0x7f25600b5f10 2022-07-12T08:50:41.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2560143290 con 0x7f256005d130 2022-07-12T08:50:41.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2560143760 con 0x7f2560142eb0 2022-07-12T08:50:41.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2565b40700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f256005d130 0x7f25600b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:41.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2566b42700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f25600b5f10 0x7f2560146090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49838/0 (socket says 172.21.15.61:49838) 2022-07-12T08:50:41.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2566b42700 1 -- 172.21.15.61:0/968796922 learned_addr learned my addr 172.21.15.61:0/968796922 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:41.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2566341700 1 --2- 172.21.15.61:0/968796922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f256005c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:41.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f256533f700 1 -- 172.21.15.61:0/968796922 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849200087 0 0) 0x7f2560143500 con 0x7f25600b5f10 2022-07-12T08:50:41.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f256533f700 1 -- 172.21.15.61:0/968796922 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2548003260 con 0x7f25600b5f10 2022-07-12T08:50:41.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2565b40700 1 -- 172.21.15.61:0/968796922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 msgr2=0x7f256005c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:41.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2565b40700 1 --2- 172.21.15.61:0/968796922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f256005c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2565b40700 1 -- 172.21.15.61:0/968796922 >> v1:172.21.15.61:6789/0 conn(0x7f25600b5f10 legacy=0x7f2560146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:41.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2565b40700 1 -- 172.21.15.61:0/968796922 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25600b4030 con 0x7f256005d130 2022-07-12T08:50:41.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2565b40700 1 --2- 172.21.15.61:0/968796922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f256005d130 0x7f25600b5a20 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7f2550002f70 tx=0x7f2550006df0).ready entity=mon.0 client_cookie=2dc8f1df8fdbfb5e server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:41.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f256533f700 1 -- 172.21.15.61:0/968796922 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255000d4a0 con 0x7f256005d130 2022-07-12T08:50:41.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f256533f700 1 -- 172.21.15.61:0/968796922 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2550000e90 con 0x7f256005d130 2022-07-12T08:50:41.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f256533f700 1 -- 172.21.15.61:0/968796922 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255000d7f0 con 0x7f256005d130 2022-07-12T08:50:41.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- 172.21.15.61:0/968796922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f256005d130 msgr2=0x7f25600b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:41.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 --2- 172.21.15.61:0/968796922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f256005d130 0x7f25600b5a20 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 rx=0x7f2550002f70 tx=0x7f2550006df0).stop 2022-07-12T08:50:41.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- 172.21.15.61:0/968796922 shutdown_connections 2022-07-12T08:50:41.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 --2- 172.21.15.61:0/968796922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f256005d130 0x7f25600b5a20 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 --2- 172.21.15.61:0/968796922 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f256005c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- 172.21.15.61:0/968796922 >> 172.21.15.61:0/968796922 conn(0x7f2560066d70 msgr2=0x7f2560067180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:41.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- 172.21.15.61:0/968796922 shutdown_connections 2022-07-12T08:50:41.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.764+0000 7f2567343700 1 -- 172.21.15.61:0/968796922 wait complete. 2022-07-12T08:50:41.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 Processor -- start 2022-07-12T08:50:41.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 -- start start 2022-07-12T08:50:41.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256005d130 0x7f25600acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:41.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 0x7f25601477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:41.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f25600ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:41.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25600acb60 con 0x7f25600b5f10 2022-07-12T08:50:41.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25600afcf0 con 0x7f256005d130 2022-07-12T08:50:41.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2567343700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2560146540 con 0x7f2560142eb0 2022-07-12T08:50:41.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 0x7f25601477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:41.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 0x7f25601477c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33392/0 (socket says 172.21.15.61:33392) 2022-07-12T08:50:41.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 -- 172.21.15.61:0/1885825286 learned_addr learned my addr 172.21.15.61:0/1885825286 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:41.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2566341700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256005d130 0x7f25600acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:41.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2566b42700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f25600ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:41.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 -- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 msgr2=0x7f25600ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:41.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f25600ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 -- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256005d130 msgr2=0x7f25600acf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:41.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256005d130 0x7f25600acf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:41.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 -- 172.21.15.61:0/1885825286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255000b180 con 0x7f25600b5f10 2022-07-12T08:50:41.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.768+0000 7f2565b40700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 0x7f25601477c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f2550002f70 tx=0x7f2550014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:41.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2550012930 con 0x7f25600b5f10 2022-07-12T08:50:41.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2550012a90 con 0x7f25600b5f10 2022-07-12T08:50:41.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25601467e0 con 0x7f25600b5f10 2022-07-12T08:50:41.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2550012df0 con 0x7f25600b5f10 2022-07-12T08:50:41.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2560149040 con 0x7f25600b5f10 2022-07-12T08:50:41.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2560146190 con 0x7f25600b5f10 2022-07-12T08:50:41.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f255001c070 con 0x7f25600b5f10 2022-07-12T08:50:41.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f254f7fe700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25400539b0 0x7f2540055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:41.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f255006c7f0 con 0x7f25600b5f10 2022-07-12T08:50:41.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.772+0000 7f2566341700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25400539b0 0x7f2540055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:41.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.776+0000 7f2566341700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25400539b0 0x7f2540055e70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f255c00b000 tx=0x7f255c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:41.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.780+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2560146190 con 0x7f25600b5f10 2022-07-12T08:50:41.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:41.912+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f2560146190 con 0x7f25600b5f10 2022-07-12T08:50:42.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.640+0000 7f254f7fe700 1 -- 172.21.15.61:0/1885825286 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v136) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2560146190 con 0x7f25600b5f10 2022-07-12T08:50:42.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25400539b0 msgr2=0x7f2540055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25400539b0 0x7f2540055e70 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f255c00b000 tx=0x7f255c006df0).stop 2022-07-12T08:50:42.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 msgr2=0x7f25601477c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 0x7f25601477c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7f2550002f70 tx=0x7f2550014000).stop 2022-07-12T08:50:42.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 shutdown_connections 2022-07-12T08:50:42.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f2560142eb0 0x7f25600ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b5f10 0x7f25601477c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f25400539b0 0x7f2540055e70 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 --2- 172.21.15.61:0/1885825286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f256005d130 0x7f25600acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.644+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 >> 172.21.15.61:0/1885825286 conn(0x7f2560066d70 msgr2=0x7f25600b8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:42.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.648+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 shutdown_connections 2022-07-12T08:50:42.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.648+0000 7f2567343700 1 -- 172.21.15.61:0/1885825286 wait complete. 2022-07-12T08:50:42.652 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:50:42.665 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:50:42.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdefde6b700 1 Processor -- start 2022-07-12T08:50:42.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdefde6b700 1 -- start start 2022-07-12T08:50:42.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdefde6b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdef80b2300 0x7fdef80aee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:42.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdefde6b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef8140020 con 0x7fdef80b2db0 2022-07-12T08:50:42.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdefde6b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef80b2810 con 0x7fdef813fb70 2022-07-12T08:50:42.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdefde6b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef81432c0 con 0x7fdef80b2300 2022-07-12T08:50:42.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdef6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdef80b2300 0x7fdef80aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:42.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdef6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdef80b2300 0x7fdef80aee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33400/0 (socket says 172.21.15.61:33400) 2022-07-12T08:50:42.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdef6ffd700 1 -- 172.21.15.61:0/434548239 learned_addr learned my addr 172.21.15.61:0/434548239 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:42.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.772+0000 7fdef77fe700 1 --1- 172.21.15.61:0/434548239 >> v1:172.21.15.61:6789/0 conn(0x7fdef80b2db0 0x7fdef8139810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49848/0 (socket says 172.21.15.61:49848) 2022-07-12T08:50:42.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef6ffd700 1 -- 172.21.15.61:0/434548239 >> v1:172.21.15.111:6789/0 conn(0x7fdef813fb70 legacy=0x7fdef8143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:42.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef6ffd700 1 -- 172.21.15.61:0/434548239 >> v1:172.21.15.61:6789/0 conn(0x7fdef80b2db0 legacy=0x7fdef8139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef6ffd700 1 -- 172.21.15.61:0/434548239 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef8139f20 con 0x7fdef80b2300 2022-07-12T08:50:42.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef6ffd700 1 --2- 172.21.15.61:0/434548239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2300 0x7fdef80aee80 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7fdee8002f70 tx=0x7fdee8006f40).ready entity=mon.0 client_cookie=271f307944d29967 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:42.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef67fc700 1 -- 172.21.15.61:0/434548239 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee800dc70 con 0x7fdef80b2300 2022-07-12T08:50:42.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef67fc700 1 -- 172.21.15.61:0/434548239 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8000e90 con 0x7fdef80b2300 2022-07-12T08:50:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdef67fc700 1 -- 172.21.15.61:0/434548239 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee800be90 con 0x7fdef80b2300 2022-07-12T08:50:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 -- 172.21.15.61:0/434548239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2300 msgr2=0x7fdef80aee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 --2- 172.21.15.61:0/434548239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2300 0x7fdef80aee80 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 rx=0x7fdee8002f70 tx=0x7fdee8006f40).stop 2022-07-12T08:50:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 -- 172.21.15.61:0/434548239 shutdown_connections 2022-07-12T08:50:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 --2- 172.21.15.61:0/434548239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2300 0x7fdef80aee80 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 -- 172.21.15.61:0/434548239 >> 172.21.15.61:0/434548239 conn(0x7fdef8063dc0 msgr2=0x7fdef80641a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 -- 172.21.15.61:0/434548239 shutdown_connections 2022-07-12T08:50:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 -- 172.21.15.61:0/434548239 wait complete. 2022-07-12T08:50:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 Processor -- start 2022-07-12T08:50:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.776+0000 7fdefde6b700 1 -- start start 2022-07-12T08:50:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef80b2300 0x7fdef813b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 0x7fdef81377f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef813fb70 0x7fdef8137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8144160 con 0x7fdef80b2db0 2022-07-12T08:50:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef8144400 con 0x7fdef813fb70 2022-07-12T08:50:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef81446a0 con 0x7fdef80b2300 2022-07-12T08:50:42.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 0x7fdef81377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef813fb70 0x7fdef8137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef77fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef80b2300 0x7fdef813b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 0x7fdef81377f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33408/0 (socket says 172.21.15.61:33408) 2022-07-12T08:50:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef813fb70 0x7fdef8137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:40212/0 (socket says 172.21.15.61:40212) 2022-07-12T08:50:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 -- 172.21.15.61:0/3495391831 learned_addr learned my addr 172.21.15.61:0/3495391831 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 -- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef80b2300 msgr2=0x7fdef813b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef80b2300 0x7fdef813b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 -- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef813fb70 msgr2=0x7fdef8137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef813fb70 0x7fdef8137ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 -- 172.21.15.61:0/3495391831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee800b460 con 0x7fdef80b2db0 2022-07-12T08:50:42.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef6ffd700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 0x7fdef81377f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7fdee8002f70 tx=0x7fdee8006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:42.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee8010dd0 con 0x7fdef80b2db0 2022-07-12T08:50:42.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee8024450 con 0x7fdef80b2db0 2022-07-12T08:50:42.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef813bbd0 con 0x7fdef80b2db0 2022-07-12T08:50:42.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdee80247b0 con 0x7fdef80b2db0 2022-07-12T08:50:42.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef8143d30 con 0x7fdef80b2db0 2022-07-12T08:50:42.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.780+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef8057460 con 0x7fdef80b2db0 2022-07-12T08:50:42.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.784+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdee8021070 con 0x7fdef80b2db0 2022-07-12T08:50:42.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.784+0000 7fdef4ff9700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fded8053ad0 0x7fded8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:42.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.784+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fdee806f960 con 0x7fdef80b2db0 2022-07-12T08:50:42.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.784+0000 7fdef77fe700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fded8053ad0 0x7fded8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:42.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.784+0000 7fdef77fe700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fded8053ad0 0x7fded8055f90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fdeec00d000 tx=0x7fdeec007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:42.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.792+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdee8046820 con 0x7fdef80b2db0 2022-07-12T08:50:42.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.956+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fdef80447a0 con 0x7fdef80b2db0 2022-07-12T08:50:42.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.968+0000 7fdef4ff9700 1 -- 172.21.15.61:0/3495391831 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 27 and data pool 28 v76) v1 ==== 189+0+0 (secure 0 0 0) 0x7fdee8029030 con 0x7fdef80b2db0 2022-07-12T08:50:42.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.968+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fded8053ad0 msgr2=0x7fded8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.968+0000 7fdefde6b700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fded8053ad0 0x7fded8055f90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fdeec00d000 tx=0x7fdeec007b80).stop 2022-07-12T08:50:42.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.968+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 msgr2=0x7fdef81377f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:42.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.968+0000 7fdefde6b700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 0x7fdef81377f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7fdee8002f70 tx=0x7fdee8006f40).stop 2022-07-12T08:50:42.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 shutdown_connections 2022-07-12T08:50:42.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fded8053ad0 0x7fded8055f90 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdef813fb70 0x7fdef8137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdef80b2db0 0x7fdef81377f0 secure :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0x7fdee8002f70 tx=0x7fdee8006f40).stop 2022-07-12T08:50:42.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 --2- 172.21.15.61:0/3495391831 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdef80b2300 0x7fdef813b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:42.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 >> 172.21.15.61:0/3495391831 conn(0x7fdef8063dc0 msgr2=0x7fdef80b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:42.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 shutdown_connections 2022-07-12T08:50:42.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:42.972+0000 7fdefde6b700 1 -- 172.21.15.61:0/3495391831 wait complete. 2022-07-12T08:50:42.995 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 27 and data pool 28 2022-07-12T08:50:42.996 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:43.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 Processor -- start 2022-07-12T08:50:43.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 -- start start 2022-07-12T08:50:43.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c142f80 0x7f425c143740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f425c0615a0 0x7f425c0af130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f425c0af620 0x7f425c13f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425c139cf0 con 0x7f425c0615a0 2022-07-12T08:50:43.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425c143c30 con 0x7f425c0af620 2022-07-12T08:50:43.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f4261062700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425c0b0cb0 con 0x7f425c142f80 2022-07-12T08:50:43.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f425a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f425c0615a0 0x7f425c0af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f425ad9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c142f80 0x7f425c143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f425b59e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f425c0af620 0x7f425c13f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f425ad9d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c142f80 0x7f425c143740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:43850/0 (socket says 172.21.15.61:43850) 2022-07-12T08:50:43.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f425ad9d700 1 -- 172.21.15.61:0/3513316130 learned_addr learned my addr 172.21.15.61:0/3513316130 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:43.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.092+0000 7f425a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f425c0615a0 0x7f425c0af130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33424/0 (socket says 172.21.15.61:33424) 2022-07-12T08:50:43.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f425a59c700 1 -- 172.21.15.61:0/3513316130 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c142f80 msgr2=0x7f425c143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f425a59c700 1 --2- 172.21.15.61:0/3513316130 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c142f80 0x7f425c143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f425a59c700 1 -- 172.21.15.61:0/3513316130 >> v2:172.21.15.111:3300/0 conn(0x7f425c0af620 msgr2=0x7f425c13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f425a59c700 1 --2- 172.21.15.61:0/3513316130 >> v2:172.21.15.111:3300/0 conn(0x7f425c0af620 0x7f425c13f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f425a59c700 1 -- 172.21.15.61:0/3513316130 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425c0b0f50 con 0x7f425c0615a0 2022-07-12T08:50:43.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f425a59c700 1 --2- 172.21.15.61:0/3513316130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0615a0 0x7f425c0af130 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f4250002f70 tx=0x7f4250006df0).ready entity=mon.0 client_cookie=312b3822d438d8da server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4259d9b700 1 -- 172.21.15.61:0/3513316130 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f425000d4a0 con 0x7f425c0615a0 2022-07-12T08:50:43.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4259d9b700 1 -- 172.21.15.61:0/3513316130 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250000e90 con 0x7f425c0615a0 2022-07-12T08:50:43.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4259d9b700 1 -- 172.21.15.61:0/3513316130 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f425000d7f0 con 0x7f425c0615a0 2022-07-12T08:50:43.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 -- 172.21.15.61:0/3513316130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0615a0 msgr2=0x7f425c0af130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 --2- 172.21.15.61:0/3513316130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0615a0 0x7f425c0af130 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f4250002f70 tx=0x7f4250006df0).stop 2022-07-12T08:50:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 -- 172.21.15.61:0/3513316130 shutdown_connections 2022-07-12T08:50:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 --2- 172.21.15.61:0/3513316130 >> v2:172.21.15.111:3300/0 conn(0x7f425c0af620 0x7f425c13f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 --2- 172.21.15.61:0/3513316130 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0615a0 0x7f425c0af130 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 --2- 172.21.15.61:0/3513316130 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c142f80 0x7f425c143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 -- 172.21.15.61:0/3513316130 >> 172.21.15.61:0/3513316130 conn(0x7f425c05f140 msgr2=0x7f425c05f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 -- 172.21.15.61:0/3513316130 shutdown_connections 2022-07-12T08:50:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.096+0000 7f4261062700 1 -- 172.21.15.61:0/3513316130 wait complete. 2022-07-12T08:50:43.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 Processor -- start 2022-07-12T08:50:43.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 -- start start 2022-07-12T08:50:43.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c0615a0 0x7f425c13e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 0x7f425c0ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f425c142f80 0x7f425c1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c13d640 con 0x7f425c0af620 2022-07-12T08:50:43.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f425c064df0 con 0x7f425c142f80 2022-07-12T08:50:43.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f4261062700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425c1520d0 con 0x7f425c0615a0 2022-07-12T08:50:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f425a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 0x7f425c0ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f425a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 0x7f425c0ab1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33430/0 (socket says 172.21.15.61:33430) 2022-07-12T08:50:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f425a59c700 1 -- 172.21.15.61:0/1821120500 learned_addr learned my addr 172.21.15.61:0/1821120500 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f425ad9d700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c0615a0 0x7f425c13e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.100+0000 7f425b59e700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f425c142f80 0x7f425c1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425a59c700 1 -- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c0615a0 msgr2=0x7f425c13e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425a59c700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c0615a0 0x7f425c13e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425a59c700 1 -- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f425c142f80 msgr2=0x7f425c1527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425a59c700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f425c142f80 0x7f425c1527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425a59c700 1 -- 172.21.15.61:0/1821120500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f425000b180 con 0x7f425c0af620 2022-07-12T08:50:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425a59c700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 0x7f425c0ab1a0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f425c13d910 tx=0x7f4250014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4250013c60 con 0x7f425c0af620 2022-07-12T08:50:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250013dc0 con 0x7f425c0af620 2022-07-12T08:50:43.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425c063f70 con 0x7f425c0af620 2022-07-12T08:50:43.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f425001e5f0 con 0x7f425c0af620 2022-07-12T08:50:43.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f425c064210 con 0x7f425c0af620 2022-07-12T08:50:43.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f425c0b11f0 con 0x7f425c0af620 2022-07-12T08:50:43.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4250025030 con 0x7f425c0af620 2022-07-12T08:50:43.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f424bfff700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4238055bc0 0x7f4238058080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f425006cc60 con 0x7f425c0af620 2022-07-12T08:50:43.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.104+0000 7f425ad9d700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4238055bc0 0x7f4238058080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.108+0000 7f425ad9d700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4238055bc0 0x7f4238058080 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f424c002000 tx=0x7f424c006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.112+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4250043ab0 con 0x7f425c0af620 2022-07-12T08:50:43.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f425c057460 con 0x7f425c0af620 2022-07-12T08:50:43.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f424bfff700 1 -- 172.21.15.61:0/1821120500 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f4250023070 con 0x7f425c0af620 2022-07-12T08:50:43.247 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:43.248 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":137,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:42.959095+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:41.024516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:41.962174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:43.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4238055bc0 msgr2=0x7f4238058080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4238055bc0 0x7f4238058080 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f424c002000 tx=0x7f424c006eb0).stop 2022-07-12T08:50:43.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 msgr2=0x7f425c0ab1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 0x7f425c0ab1a0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f425c13d910 tx=0x7f4250014000).stop 2022-07-12T08:50:43.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 shutdown_connections 2022-07-12T08:50:43.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4238055bc0 0x7f4238058080 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f425c142f80 0x7f425c1527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f425c0af620 0x7f425c0ab1a0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 --2- 172.21.15.61:0/1821120500 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f425c0615a0 0x7f425c13e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 >> 172.21.15.61:0/1821120500 conn(0x7f425c05f140 msgr2=0x7f425c0abbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:43.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 shutdown_connections 2022-07-12T08:50:43.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.244+0000 7f4261062700 1 -- 172.21.15.61:0/1821120500 wait complete. 2022-07-12T08:50:43.261 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:43.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.360+0000 7ff8fd31e700 1 Processor -- start 2022-07-12T08:50:43.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.360+0000 7ff8fd31e700 1 -- start start 2022-07-12T08:50:43.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8f80ab290 0x7ff8f80ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff8f805f140 0x7ff8f805ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80abc70 con 0x7ff8f80afa60 2022-07-12T08:50:43.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f805f580 con 0x7ff8f80ab290 2022-07-12T08:50:43.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f805c720 con 0x7ff8f805f140 2022-07-12T08:50:43.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fcb1d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff8f80afa60 0x7ff8f814a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49888/0 (socket says 172.21.15.61:49888) 2022-07-12T08:50:43.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fcb1d700 1 -- 172.21.15.61:0/1596043368 learned_addr learned my addr 172.21.15.61:0/1596043368 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:43.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f7fff700 1 --2- 172.21.15.61:0/1596043368 >> v2:172.21.15.61:3300/0 conn(0x7ff8f80ab290 0x7ff8f80ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f77fe700 1 --2- 172.21.15.61:0/1596043368 >> v2:172.21.15.111:3300/0 conn(0x7ff8f805f140 0x7ff8f805ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f6ffd700 1 -- 172.21.15.61:0/1596043368 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726055695 0 0) 0x7ff8f80abc70 con 0x7ff8f80afa60 2022-07-12T08:50:43.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f6ffd700 1 -- 172.21.15.61:0/1596043368 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e4003260 con 0x7ff8f80afa60 2022-07-12T08:50:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f7fff700 1 -- 172.21.15.61:0/1596043368 >> v2:172.21.15.111:3300/0 conn(0x7ff8f805f140 msgr2=0x7ff8f805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f6ffd700 1 -- 172.21.15.61:0/1596043368 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2827503599 0 0) 0x7ff8e4003260 con 0x7ff8f80afa60 2022-07-12T08:50:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f7fff700 1 --2- 172.21.15.61:0/1596043368 >> v2:172.21.15.111:3300/0 conn(0x7ff8f805f140 0x7ff8f805ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f7fff700 1 -- 172.21.15.61:0/1596043368 >> v1:172.21.15.61:6789/0 conn(0x7ff8f80afa60 legacy=0x7ff8f814a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f7fff700 1 -- 172.21.15.61:0/1596043368 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f805c9c0 con 0x7ff8f80ab290 2022-07-12T08:50:43.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f77fe700 1 --2- 172.21.15.61:0/1596043368 >> v2:172.21.15.111:3300/0 conn(0x7ff8f805f140 0x7ff8f805ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:43.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f7fff700 1 --2- 172.21.15.61:0/1596043368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80ab290 0x7ff8f80ab670 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7ff8f0001b80 tx=0x7ff8ec006df0).ready entity=mon.0 client_cookie=2e96a0858baf6a2d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f6ffd700 1 -- 172.21.15.61:0/1596043368 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f00019e0 con 0x7ff8f80ab290 2022-07-12T08:50:43.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f6ffd700 1 -- 172.21.15.61:0/1596043368 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec000e90 con 0x7ff8f80ab290 2022-07-12T08:50:43.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8f6ffd700 1 -- 172.21.15.61:0/1596043368 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8ec012460 con 0x7ff8f80ab290 2022-07-12T08:50:43.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1596043368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80ab290 msgr2=0x7ff8f80ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1596043368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80ab290 0x7ff8f80ab670 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 rx=0x7ff8f0001b80 tx=0x7ff8ec006df0).stop 2022-07-12T08:50:43.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1596043368 shutdown_connections 2022-07-12T08:50:43.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1596043368 >> v2:172.21.15.111:3300/0 conn(0x7ff8f805f140 0x7ff8f805ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1596043368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80ab290 0x7ff8f80ab670 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1596043368 >> 172.21.15.61:0/1596043368 conn(0x7ff8f80b1950 msgr2=0x7ff8f80b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:43.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1596043368 shutdown_connections 2022-07-12T08:50:43.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1596043368 wait complete. 2022-07-12T08:50:43.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 Processor -- start 2022-07-12T08:50:43.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.364+0000 7ff8fd31e700 1 -- start start 2022-07-12T08:50:43.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f805f140 0x7ff8f8156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8f80ab290 0x7ff8f8066750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 0x7ff8f8148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8067980 con 0x7ff8f80afa60 2022-07-12T08:50:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8156e50 con 0x7ff8f805f140 2022-07-12T08:50:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8156220 con 0x7ff8f80ab290 2022-07-12T08:50:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 0x7ff8f8148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 0x7ff8f8148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33446/0 (socket says 172.21.15.61:33446) 2022-07-12T08:50:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 -- 172.21.15.61:0/1581431603 learned_addr learned my addr 172.21.15.61:0/1581431603 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8f7fff700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f805f140 0x7ff8f8156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8f77fe700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8f80ab290 0x7ff8f8066750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 -- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8f80ab290 msgr2=0x7ff8f8066750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8f80ab290 0x7ff8f8066750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 -- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f805f140 msgr2=0x7ff8f8156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f805f140 0x7ff8f8156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 -- 172.21.15.61:0/1581431603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e800b180 con 0x7ff8f80afa60 2022-07-12T08:50:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fcb1d700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 0x7ff8f8148610 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7ff8f0001c40 tx=0x7ff8f0002190).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f0004ab0 con 0x7ff8f80afa60 2022-07-12T08:50:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f0004c10 con 0x7ff8f80afa60 2022-07-12T08:50:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec00b180 con 0x7ff8f80afa60 2022-07-12T08:50:43.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f0007170 con 0x7ff8f80afa60 2022-07-12T08:50:43.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.368+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f8066c40 con 0x7ff8f80afa60 2022-07-12T08:50:43.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.372+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f80675b0 con 0x7ff8f80afa60 2022-07-12T08:50:43.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.372+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff8f00072d0 con 0x7ff8f80afa60 2022-07-12T08:50:43.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.372+0000 7ff8f57fa700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8d8053ad0 0x7ff8d8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.372+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff8f006bd00 con 0x7ff8f80afa60 2022-07-12T08:50:43.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.372+0000 7ff8f7fff700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8d8053ad0 0x7ff8d8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.372+0000 7ff8f7fff700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8d8053ad0 0x7ff8d8055f90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7ff8ec01a000 tx=0x7ff8ec014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.380+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8f80675b0 con 0x7ff8f80afa60 2022-07-12T08:50:43.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.508+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f80675b0 con 0x7ff8f80afa60 2022-07-12T08:50:43.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.508+0000 7ff8f57fa700 1 -- 172.21.15.61:0/1581431603 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+16113 (secure 0 0 0) 0x7ff8f80675b0 con 0x7ff8f80afa60 2022-07-12T08:50:43.515 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:43.515 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":137,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:42.959095+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:41.024516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:41.962174+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:43.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8d8053ad0 msgr2=0x7ff8d8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8d8053ad0 0x7ff8d8055f90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7ff8ec01a000 tx=0x7ff8ec014000).stop 2022-07-12T08:50:43.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 msgr2=0x7ff8f8148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 0x7ff8f8148610 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7ff8f0001c40 tx=0x7ff8f0002190).stop 2022-07-12T08:50:43.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 shutdown_connections 2022-07-12T08:50:43.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8f80afa60 0x7ff8f8148610 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff8f80ab290 0x7ff8f8066750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff8d8053ad0 0x7ff8d8055f90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 --2- 172.21.15.61:0/1581431603 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8f805f140 0x7ff8f8156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 >> 172.21.15.61:0/1581431603 conn(0x7ff8f80b1950 msgr2=0x7ff8f80b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:43.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 shutdown_connections 2022-07-12T08:50:43.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.512+0000 7ff8fd31e700 1 -- 172.21.15.61:0/1581431603 wait complete. 2022-07-12T08:50:43.532 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:50:43.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 Processor -- start 2022-07-12T08:50:43.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 -- start start 2022-07-12T08:50:43.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 0x7ffb680b0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ffb68142ab0 0x7ffb680afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6805d440 con 0x7ffb68142f10 2022-07-12T08:50:43.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb68143d80 con 0x7ffb68142ab0 2022-07-12T08:50:43.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6f0e1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68143ab0 con 0x7ffb6805d060 2022-07-12T08:50:43.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6e8e0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ffb68142f10 0x7ffb681433a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49904/0 (socket says 172.21.15.61:49904) 2022-07-12T08:50:43.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6e8e0700 1 -- 172.21.15.61:0/3752371796 learned_addr learned my addr 172.21.15.61:0/3752371796 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:43.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 0x7ffb680b0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6d8de700 1 --2- 172.21.15.61:0/3752371796 >> v2:172.21.15.111:3300/0 conn(0x7ffb68142ab0 0x7ffb680afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380341435 0 0) 0x7ffb6805d440 con 0x7ffb68142f10 2022-07-12T08:50:43.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.640+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50003260 con 0x7ffb68142f10 2022-07-12T08:50:43.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675471109 0 0) 0x7ffb50003260 con 0x7ffb68142f10 2022-07-12T08:50:43.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 msgr2=0x7ffb680b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 0x7ffb680b0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 >> v2:172.21.15.111:3300/0 conn(0x7ffb68142ab0 msgr2=0x7ffb680afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 --2- 172.21.15.61:0/3752371796 >> v2:172.21.15.111:3300/0 conn(0x7ffb68142ab0 0x7ffb680afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680674b0 con 0x7ffb68142f10 2022-07-12T08:50:43.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7ffb600019e0 con 0x7ffb68142f10 2022-07-12T08:50:43.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 >> v1:172.21.15.61:6789/0 conn(0x7ffb68142f10 legacy=0x7ffb681433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb50003ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 0x7ffb50004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500049f0 0x7ffb50004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68143d80 con 0x7ffb50003780 2022-07-12T08:50:43.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50004360 con 0x7ffb500049f0 2022-07-12T08:50:43.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5000b7d0 con 0x7ffb6805d060 2022-07-12T08:50:43.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb50003ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e8e0700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500049f0 0x7ffb50004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d8de700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 0x7ffb50004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 msgr2=0x7ffb50004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6805d060 0x7ffb50004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500049f0 msgr2=0x7ffb50004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb500049f0 0x7ffb50004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3752371796 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50003260 con 0x7ffb50003780 2022-07-12T08:50:43.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb50003ba0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7ffb6400c000 tx=0x7ffb64006df0).ready entity=mon.0 client_cookie=31be5ed6cb902579 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6400e830 con 0x7ffb50003780 2022-07-12T08:50:43.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6400e990 con 0x7ffb50003780 2022-07-12T08:50:43.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6d0dd700 1 -- 172.21.15.61:0/3752371796 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6400ecf0 con 0x7ffb50003780 2022-07-12T08:50:43.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3752371796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 msgr2=0x7ffb50003ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3752371796 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb50003ba0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 rx=0x7ffb6400c000 tx=0x7ffb64006df0).stop 2022-07-12T08:50:43.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6e8e0700 1 -- 172.21.15.61:0/3752371796 reap_dead start 2022-07-12T08:50:43.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3752371796 shutdown_connections 2022-07-12T08:50:43.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3752371796 >> 172.21.15.61:0/3752371796 conn(0x7ffb680b6900 msgr2=0x7ffb68066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:43.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3752371796 shutdown_connections 2022-07-12T08:50:43.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.644+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3752371796 wait complete. 2022-07-12T08:50:43.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 Processor -- start 2022-07-12T08:50:43.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 -- start start 2022-07-12T08:50:43.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb6809e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6809ca40 0x7ffb681552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68144e30 0x7ffb681452e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68156020 con 0x7ffb50003780 2022-07-12T08:50:43.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68155ae0 con 0x7ffb68144e30 2022-07-12T08:50:43.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68155d80 con 0x7ffb6809ca40 2022-07-12T08:50:43.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb6809e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6d8de700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6809ca40 0x7ffb681552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb6809e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33462/0 (socket says 172.21.15.61:33462) 2022-07-12T08:50:43.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3388848727 learned_addr learned my addr 172.21.15.61:0/3388848727 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:43.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6d8de700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6809ca40 0x7ffb681552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:43892/0 (socket says 172.21.15.61:43892) 2022-07-12T08:50:43.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e8e0700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68144e30 0x7ffb681452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6809ca40 msgr2=0x7ffb681552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6809ca40 0x7ffb681552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68144e30 msgr2=0x7ffb681452e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68144e30 0x7ffb681452e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 -- 172.21.15.61:0/3388848727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb681557e0 con 0x7ffb50003780 2022-07-12T08:50:43.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e8e0700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68144e30 0x7ffb681452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:43.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6e0df700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb6809e020 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7ffb6400c000 tx=0x7ffb64019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6401b820 con 0x7ffb50003780 2022-07-12T08:50:43.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64000e90 con 0x7ffb50003780 2022-07-12T08:50:43.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6809e8a0 con 0x7ffb50003780 2022-07-12T08:50:43.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6401bb70 con 0x7ffb50003780 2022-07-12T08:50:43.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.648+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb68143fc0 con 0x7ffb50003780 2022-07-12T08:50:43.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.652+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb68057460 con 0x7ffb50003780 2022-07-12T08:50:43.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.652+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffb6401bcd0 con 0x7ffb50003780 2022-07-12T08:50:43.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.652+0000 7ffb577fe700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffb48053a40 0x7ffb48055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:43.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.652+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffb6406d3f0 con 0x7ffb50003780 2022-07-12T08:50:43.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.652+0000 7ffb6d8de700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffb48053a40 0x7ffb48055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:43.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.652+0000 7ffb6d8de700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffb48053a40 0x7ffb48055f00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7ffb5800b000 tx=0x7ffb58008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:43.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.656+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffb64047d80 con 0x7ffb50003780 2022-07-12T08:50:43.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.824+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ffb68068350 con 0x7ffb50003780 2022-07-12T08:50:43.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.976+0000 7ffb577fe700 1 -- 172.21.15.61:0/3388848727 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffb64047970 con 0x7ffb50003780 2022-07-12T08:50:43.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffb48053a40 msgr2=0x7ffb48055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffb48053a40 0x7ffb48055f00 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7ffb5800b000 tx=0x7ffb58008040).stop 2022-07-12T08:50:43.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 msgr2=0x7ffb6809e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:43.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb6809e020 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7ffb6400c000 tx=0x7ffb64019000).stop 2022-07-12T08:50:43.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 shutdown_connections 2022-07-12T08:50:43.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ffb48053a40 0x7ffb48055f00 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffb68144e30 0x7ffb681452e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ffb6809ca40 0x7ffb681552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 --2- 172.21.15.61:0/3388848727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb50003780 0x7ffb6809e020 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:43.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 >> 172.21.15.61:0/3388848727 conn(0x7ffb680b6900 msgr2=0x7ffb6805f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:43.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 shutdown_connections 2022-07-12T08:50:43.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:43.980+0000 7ffb6f0e1700 1 -- 172.21.15.61:0/3388848727 wait complete. 2022-07-12T08:50:43.999 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:44.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 Processor -- start 2022-07-12T08:50:44.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 -- start start 2022-07-12T08:50:44.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f61bc0af280 0x7f61bc060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f61bc09bce0 0x7f61bc0af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61bc0aecb0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc143b00 con 0x7f61bc09bce0 2022-07-12T08:50:44.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61bc1435c0 con 0x7f61bc0af280 2022-07-12T08:50:44.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f61bc09bce0 0x7f61bc0af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f61bc09bce0 0x7f61bc0af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33470/0 (socket says 172.21.15.61:33470) 2022-07-12T08:50:44.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 -- 172.21.15.61:0/1046076300 learned_addr learned my addr 172.21.15.61:0/1046076300 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:44.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c12f4700 1 --2- 172.21.15.61:0/1046076300 >> v2:172.21.15.111:3300/0 conn(0x7f61bc0af280 0x7f61bc060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 -- 172.21.15.61:0/1046076300 >> v2:172.21.15.111:3300/0 conn(0x7f61bc0af280 msgr2=0x7f61bc060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:44.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 --2- 172.21.15.61:0/1046076300 >> v2:172.21.15.111:3300/0 conn(0x7f61bc0af280 0x7f61bc060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 -- 172.21.15.61:0/1046076300 >> v1:172.21.15.111:6789/0 conn(0x7f61bc0b3f60 legacy=0x7f61bc0b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 -- 172.21.15.61:0/1046076300 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc143860 con 0x7f61bc09bce0 2022-07-12T08:50:44.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c0af3700 1 --2- 172.21.15.61:0/1046076300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc09bce0 0x7f61bc0af660 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f61b80032f0 tx=0x7f61b8009290).ready entity=mon.0 client_cookie=a72ac74bb65bf6fc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61a7fff700 1 -- 172.21.15.61:0/1046076300 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b800b870 con 0x7f61bc09bce0 2022-07-12T08:50:44.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61a7fff700 1 -- 172.21.15.61:0/1046076300 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b800b9d0 con 0x7f61bc09bce0 2022-07-12T08:50:44.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61a7fff700 1 -- 172.21.15.61:0/1046076300 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b800bb40 con 0x7f61bc09bce0 2022-07-12T08:50:44.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 -- 172.21.15.61:0/1046076300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc09bce0 msgr2=0x7f61bc0af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.116+0000 7f61c22f6700 1 --2- 172.21.15.61:0/1046076300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc09bce0 0x7f61bc0af660 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7f61b80032f0 tx=0x7f61b8009290).stop 2022-07-12T08:50:44.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- 172.21.15.61:0/1046076300 shutdown_connections 2022-07-12T08:50:44.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 --2- 172.21.15.61:0/1046076300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc09bce0 0x7f61bc0af660 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 --2- 172.21.15.61:0/1046076300 >> v2:172.21.15.111:3300/0 conn(0x7f61bc0af280 0x7f61bc060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- 172.21.15.61:0/1046076300 >> 172.21.15.61:0/1046076300 conn(0x7f61bc0b92c0 msgr2=0x7f61bc0b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:44.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- 172.21.15.61:0/1046076300 shutdown_connections 2022-07-12T08:50:44.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- 172.21.15.61:0/1046076300 wait complete. 2022-07-12T08:50:44.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 Processor -- start 2022-07-12T08:50:44.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- start start 2022-07-12T08:50:44.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61bc09bce0 0x7f61bc09d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61bc0af280 0x7f61bc09dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 0x7f61bc152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0b26f0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc0b2990 con 0x7f61bc09bce0 2022-07-12T08:50:44.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc152010 con 0x7f61bc0af280 2022-07-12T08:50:44.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 0x7f61bc152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c12f4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61bc09bce0 0x7f61bc09d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c0af3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61bc0af280 0x7f61bc09dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 0x7f61bc152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33480/0 (socket says 172.21.15.61:33480) 2022-07-12T08:50:44.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 -- 172.21.15.61:0/313607820 learned_addr learned my addr 172.21.15.61:0/313607820 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:44.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 -- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61bc0af280 msgr2=0x7f61bc09dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61bc0af280 0x7f61bc09dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 -- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61bc09bce0 msgr2=0x7f61bc09d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61bc09bce0 0x7f61bc09d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 -- 172.21.15.61:0/313607820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b8009530 con 0x7f61bc0b3f60 2022-07-12T08:50:44.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c0af3700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61bc0af280 0x7f61bc09dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:44.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c1af5700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 0x7f61bc152750 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f61bc0b70b0 tx=0x7f61b4007ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b400ccd0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b400ce30 con 0x7f61bc0b3f60 2022-07-12T08:50:44.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.120+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61bc1522b0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61b40125f0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc09e0c0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc152440 con 0x7f61bc0b3f60 2022-07-12T08:50:44.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f61b4019030 con 0x7f61bc0b3f60 2022-07-12T08:50:44.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61aeffd700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6198053a40 0x7f6198055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f61b406b060 con 0x7f61bc0b3f60 2022-07-12T08:50:44.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.124+0000 7f61c12f4700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6198053a40 0x7f6198055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.128+0000 7f61c12f4700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6198053a40 0x7f6198055f00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f61b000c000 tx=0x7f61b0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.132+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f61b4041fc0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.296+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61bc057460 con 0x7f61bc0b3f60 2022-07-12T08:50:44.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.296+0000 7f61aeffd700 1 -- 172.21.15.61:0/313607820 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f61b40455e0 con 0x7f61bc0b3f60 2022-07-12T08:50:44.308 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:44.308 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:42.959468+0000","modified":"2022-07-12T08:50:43.963135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/676021650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":676021650},{"type":"v1","addr":"172.21.15.61:6833","nonce":676021650}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-12T08:50:44.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6198053a40 msgr2=0x7f6198055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6198053a40 0x7f6198055f00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f61b000c000 tx=0x7f61b0008040).stop 2022-07-12T08:50:44.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 msgr2=0x7f61bc152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 0x7f61bc152750 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 rx=0x7f61bc0b70b0 tx=0x7f61b4007ba0).stop 2022-07-12T08:50:44.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 shutdown_connections 2022-07-12T08:50:44.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f61bc0b3f60 0x7f61bc152750 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f61bc0af280 0x7f61bc09dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6198053a40 0x7f6198055f00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 --2- 172.21.15.61:0/313607820 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f61bc09bce0 0x7f61bc09d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 >> 172.21.15.61:0/313607820 conn(0x7f61bc0b92c0 msgr2=0x7f61bc0b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:44.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 shutdown_connections 2022-07-12T08:50:44.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.300+0000 7f61c22f6700 1 -- 172.21.15.61:0/313607820 wait complete. 2022-07-12T08:50:44.313 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-12T08:50:44.318 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.420+0000 7f40fd5b3700 1 Processor -- start 2022-07-12T08:50:44.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.420+0000 7f40fd5b3700 1 -- start start 2022-07-12T08:50:44.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40fd5b3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40f8144320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40fd5b3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f80acb60 con 0x7f40f80adff0 2022-07-12T08:50:44.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40fd5b3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f80b45f0 con 0x7f40f80b2490 2022-07-12T08:50:44.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40fd5b3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80acd90 con 0x7f40f80b4210 2022-07-12T08:50:44.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f40f80adff0 0x7f40f813dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49938/0 (socket says 172.21.15.61:49938) 2022-07-12T08:50:44.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f77fe700 1 -- 172.21.15.61:0/2204846333 learned_addr learned my addr 172.21.15.61:0/2204846333 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:44.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f6ffd700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40f8144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116495185 0 0) 0x7f40f80acb60 con 0x7f40f80adff0 2022-07-12T08:50:44.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d8003260 con 0x7f40f80adff0 2022-07-12T08:50:44.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080922930 0 0) 0x7f40f80b45f0 con 0x7f40f80b2490 2022-07-12T08:50:44.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f80acb60 con 0x7f40f80b2490 2022-07-12T08:50:44.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687570721 0 0) 0x7f40d8003260 con 0x7f40f80adff0 2022-07-12T08:50:44.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 msgr2=0x7f40f8144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:44.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40f8144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 >> v1:172.21.15.111:6789/0 conn(0x7f40f80b2490 legacy=0x7f40f80ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f80ad0b0 con 0x7f40f80adff0 2022-07-12T08:50:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f40f0001660 con 0x7f40f80adff0 2022-07-12T08:50:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 >> v1:172.21.15.61:6789/0 conn(0x7f40f80adff0 legacy=0x7f40f813dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40d8003910 0x7f40d8003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40d8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40d80049f0 0x7f40d8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f80acd90 con 0x7f40d80049f0 2022-07-12T08:50:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d800b800 con 0x7f40d8003910 2022-07-12T08:50:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d800b970 con 0x7f40f80b4210 2022-07-12T08:50:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40d8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f77fe700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40d80049f0 0x7f40d8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f6ffd700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40d8003910 0x7f40d8003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 -- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40d8003910 msgr2=0x7f40d8003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40d8003910 0x7f40d8003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 -- 172.21.15.61:0/2204846333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40d80049f0 msgr2=0x7f40d8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:44.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40d80049f0 0x7f40d8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 -- 172.21.15.61:0/2204846333 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d8003260 con 0x7f40f80b4210 2022-07-12T08:50:44.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f67fc700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40d8004500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f40e0001490 tx=0x7f40e0007d80).ready entity=mon.2 client_cookie=c54241236c7f5997 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e0001780 con 0x7f40f80b4210 2022-07-12T08:50:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e0006890 con 0x7f40f80b4210 2022-07-12T08:50:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.424+0000 7f40f5ffb700 1 -- 172.21.15.61:0/2204846333 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e0006bf0 con 0x7f40f80b4210 2022-07-12T08:50:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 -- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 msgr2=0x7f40d8004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/2204846333 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80b4210 0x7f40d8004500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f40e0001490 tx=0x7f40e0007d80).stop 2022-07-12T08:50:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40f77fe700 1 -- 172.21.15.61:0/2204846333 reap_dead start 2022-07-12T08:50:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 -- 172.21.15.61:0/2204846333 shutdown_connections 2022-07-12T08:50:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 -- 172.21.15.61:0/2204846333 >> 172.21.15.61:0/2204846333 conn(0x7f40f8064e10 msgr2=0x7f40f80651f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 -- 172.21.15.61:0/2204846333 shutdown_connections 2022-07-12T08:50:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 -- 172.21.15.61:0/2204846333 wait complete. 2022-07-12T08:50:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 Processor -- start 2022-07-12T08:50:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.428+0000 7f40fd5b3700 1 -- start start 2022-07-12T08:50:44.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f80b4210 0x7f40f8143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 0x7f40f81553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80600c0 0x7f40f813e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8060b80 con 0x7f40f80ae8e0 2022-07-12T08:50:44.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8060db0 con 0x7f40f80b4210 2022-07-12T08:50:44.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f8061050 con 0x7f40f80600c0 2022-07-12T08:50:44.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 0x7f40f81553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 0x7f40f81553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33498/0 (socket says 172.21.15.61:33498) 2022-07-12T08:50:44.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 -- 172.21.15.61:0/3323800602 learned_addr learned my addr 172.21.15.61:0/3323800602 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f6ffd700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f80b4210 0x7f40f8143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f77fe700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80600c0 0x7f40f813e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 -- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80600c0 msgr2=0x7f40f813e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80600c0 0x7f40f813e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 -- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f80b4210 msgr2=0x7f40f8143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f80b4210 0x7f40f8143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 -- 172.21.15.61:0/3323800602 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f805fd20 con 0x7f40f80ae8e0 2022-07-12T08:50:44.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f6ffd700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f80b4210 0x7f40f8143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:44.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40f67fc700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 0x7f40f81553b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f40e0001490 tx=0x7f40e0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e0001780 con 0x7f40f80ae8e0 2022-07-12T08:50:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e000c9e0 con 0x7f40f80ae8e0 2022-07-12T08:50:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f813e5c0 con 0x7f40f80ae8e0 2022-07-12T08:50:44.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e000cd40 con 0x7f40f80ae8e0 2022-07-12T08:50:44.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.432+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f80b0d40 con 0x7f40f80ae8e0 2022-07-12T08:50:44.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.436+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f40e0025030 con 0x7f40f80ae8e0 2022-07-12T08:50:44.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.436+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f813ded0 con 0x7f40f80ae8e0 2022-07-12T08:50:44.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.436+0000 7f40effff700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40dc053a40 0x7f40dc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.436+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f40e006cfe0 con 0x7f40f80ae8e0 2022-07-12T08:50:44.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.436+0000 7f40f6ffd700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40dc053a40 0x7f40dc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.436+0000 7f40f6ffd700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40dc053a40 0x7f40dc055f00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f40e800b000 tx=0x7f40e8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.440+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f40e0043fd0 con 0x7f40f80ae8e0 2022-07-12T08:50:44.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.568+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f805feb0 con 0x7f40f80ae8e0 2022-07-12T08:50:44.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.568+0000 7f40effff700 1 -- 172.21.15.61:0/3323800602 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f40e0030f00 con 0x7f40f80ae8e0 2022-07-12T08:50:44.572 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:44.573 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":138,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:43.963239+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:41.024516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:41.962174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40dc053a40 msgr2=0x7f40dc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40dc053a40 0x7f40dc055f00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f40e800b000 tx=0x7f40e8006df0).stop 2022-07-12T08:50:44.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 msgr2=0x7f40f81553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 0x7f40f81553b0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f40e0001490 tx=0x7f40e0016000).stop 2022-07-12T08:50:44.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 shutdown_connections 2022-07-12T08:50:44.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f40f80600c0 0x7f40f813e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40f80ae8e0 0x7f40f81553b0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f40dc053a40 0x7f40dc055f00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 --2- 172.21.15.61:0/3323800602 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40f80b4210 0x7f40f8143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 >> 172.21.15.61:0/3323800602 conn(0x7f40f8064e10 msgr2=0x7f40f8144900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:44.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 shutdown_connections 2022-07-12T08:50:44.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.572+0000 7f40fd5b3700 1 -- 172.21.15.61:0/3323800602 wait complete. 2022-07-12T08:50:44.586 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:50:44.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 Processor -- start 2022-07-12T08:50:44.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- start start 2022-07-12T08:50:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05d100 0x7fe0cc060e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0cc1465a0 0x7fe0cc1469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc14a560 con 0x7fe0cc05cd20 2022-07-12T08:50:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0cc097d40 con 0x7fe0cc1465a0 2022-07-12T08:50:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc14b340 con 0x7fe0cc05d100 2022-07-12T08:50:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0cc1465a0 0x7fe0cc1469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0cc1465a0 0x7fe0cc1469c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33508/0 (socket says 172.21.15.61:33508) 2022-07-12T08:50:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 -- 172.21.15.61:0/1304509497 learned_addr learned my addr 172.21.15.61:0/1304509497 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d0937700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05d100 0x7fe0cc060e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 -- 172.21.15.61:0/1304509497 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05d100 msgr2=0x7fe0cc060e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:44.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05d100 0x7fe0cc060e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 -- 172.21.15.61:0/1304509497 >> v1:172.21.15.111:6789/0 conn(0x7fe0cc05cd20 legacy=0x7fe0cc14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 -- 172.21.15.61:0/1304509497 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0aafa0 con 0x7fe0cc1465a0 2022-07-12T08:50:44.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d0937700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05d100 0x7fe0cc060e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:44.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d1939700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc1469c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fe0c40032f0 tx=0x7fe0c4009290).ready entity=mon.0 client_cookie=e7cea1343e3bf4f1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0c3fff700 1 -- 172.21.15.61:0/1304509497 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c400b8a0 con 0x7fe0cc1465a0 2022-07-12T08:50:44.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0c3fff700 1 -- 172.21.15.61:0/1304509497 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c400ba00 con 0x7fe0cc1465a0 2022-07-12T08:50:44.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0c3fff700 1 -- 172.21.15.61:0/1304509497 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c400bb70 con 0x7fe0cc1465a0 2022-07-12T08:50:44.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- 172.21.15.61:0/1304509497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc1465a0 msgr2=0x7fe0cc1469c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc1469c0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fe0c40032f0 tx=0x7fe0c4009290).stop 2022-07-12T08:50:44.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- 172.21.15.61:0/1304509497 shutdown_connections 2022-07-12T08:50:44.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc1469c0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 --2- 172.21.15.61:0/1304509497 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05d100 0x7fe0cc060e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- 172.21.15.61:0/1304509497 >> 172.21.15.61:0/1304509497 conn(0x7fe0cc0b1a50 msgr2=0x7fe0cc0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:44.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- 172.21.15.61:0/1304509497 shutdown_connections 2022-07-12T08:50:44.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.704+0000 7fe0d213a700 1 -- 172.21.15.61:0/1304509497 wait complete. 2022-07-12T08:50:44.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 Processor -- start 2022-07-12T08:50:44.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 -- start start 2022-07-12T08:50:44.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05cd20 0x7fe0cc1442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc05d100 0x7fe0cc142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc14d450 con 0x7fe0cc05d100 2022-07-12T08:50:44.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0aed90 con 0x7fe0cc1465a0 2022-07-12T08:50:44.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc14bb30 con 0x7fe0cc05cd20 2022-07-12T08:50:44.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d1138700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05cd20 0x7fe0cc1442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc05d100 0x7fe0cc142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d1939700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d1138700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05cd20 0x7fe0cc1442d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:43938/0 (socket says 172.21.15.61:43938) 2022-07-12T08:50:44.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d1138700 1 -- 172.21.15.61:0/2643793984 learned_addr learned my addr 172.21.15.61:0/2643793984 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:44.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 -- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05cd20 msgr2=0x7fe0cc1442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05cd20 0x7fe0cc1442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 -- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc1465a0 msgr2=0x7fe0cc159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 -- 172.21.15.61:0/2643793984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4009530 con 0x7fe0cc05d100 2022-07-12T08:50:44.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d0937700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc05d100 0x7fe0cc142560 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fe0bc00a000 tx=0x7fe0bc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0bc0106e0 con 0x7fe0cc05d100 2022-07-12T08:50:44.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc010840 con 0x7fe0cc05d100 2022-07-12T08:50:44.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc14bdd0 con 0x7fe0cc05d100 2022-07-12T08:50:44.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0bc010ba0 con 0x7fe0cc05d100 2022-07-12T08:50:44.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.708+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc14c520 con 0x7fe0cc05d100 2022-07-12T08:50:44.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.712+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc057460 con 0x7fe0cc05d100 2022-07-12T08:50:44.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.712+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe0bc01a030 con 0x7fe0cc05d100 2022-07-12T08:50:44.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.712+0000 7fe0c27fc700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe0b4053af0 0x7fe0b4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:44.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.712+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe0bc06bb60 con 0x7fe0cc05d100 2022-07-12T08:50:44.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.712+0000 7fe0d1138700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe0b4053af0 0x7fe0b4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:44.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.712+0000 7fe0d1138700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe0b4053af0 0x7fe0b4055fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fe0c800d000 tx=0x7fe0c8007c50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:44.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.720+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe0bc0429b0 con 0x7fe0cc05d100 2022-07-12T08:50:44.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.888+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe0cc06b2b0 con 0x7fe0cc05d100 2022-07-12T08:50:44.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.896+0000 7fe0c27fc700 1 -- 172.21.15.61:0/2643793984 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v42) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe0bc045fd0 con 0x7fe0cc05d100 2022-07-12T08:50:44.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe0b4053af0 msgr2=0x7fe0b4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe0b4053af0 0x7fe0b4055fb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fe0c800d000 tx=0x7fe0c8007c50).stop 2022-07-12T08:50:44.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc05d100 msgr2=0x7fe0cc142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:44.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc05d100 0x7fe0cc142560 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fe0bc00a000 tx=0x7fe0bc008040).stop 2022-07-12T08:50:44.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 shutdown_connections 2022-07-12T08:50:44.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe0b4053af0 0x7fe0b4055fb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0cc1465a0 0x7fe0cc159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0cc05d100 0x7fe0cc142560 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 --2- 172.21.15.61:0/2643793984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe0cc05cd20 0x7fe0cc1442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:44.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 >> 172.21.15.61:0/2643793984 conn(0x7fe0cc0b1a50 msgr2=0x7fe0cc0afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:44.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 shutdown_connections 2022-07-12T08:50:44.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:44.900+0000 7fe0d213a700 1 -- 172.21.15.61:0/2643793984 wait complete. 2022-07-12T08:50:44.907 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:50:44.916 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:50:45.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.020+0000 7f7bb8969700 1 Processor -- start 2022-07-12T08:50:45.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.020+0000 7f7bb8969700 1 -- start start 2022-07-12T08:50:45.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7bb4140840 0x7f7bb4060520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb4143910 con 0x7f7bb41431e0 2022-07-12T08:50:45.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb2d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7bb41431e0 0x7f7bb40b1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49968/0 (socket says 172.21.15.61:49968) 2022-07-12T08:50:45.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb40b08b0 con 0x7f7bb4064890 2022-07-12T08:50:45.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb2d9d700 1 -- 172.21.15.61:0/2821643181 learned_addr learned my addr 172.21.15.61:0/2821643181 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:45.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- 172.21.15.61:0/2821643181 --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb40b1730 con 0x7f7bb4140840 2022-07-12T08:50:45.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb259c700 1 --2- 172.21.15.61:0/2821643181 >> v2:172.21.15.111:3300/0 conn(0x7f7bb4140840 0x7f7bb4060520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb1d9b700 1 -- 172.21.15.61:0/2821643181 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059133035 0 0) 0x7f7bb4143910 con 0x7f7bb41431e0 2022-07-12T08:50:45.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb1d9b700 1 -- 172.21.15.61:0/2821643181 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98003260 con 0x7f7bb41431e0 2022-07-12T08:50:45.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb259c700 1 -- 172.21.15.61:0/2821643181 >> v1:172.21.15.111:6789/0 conn(0x7f7bb4064890 legacy=0x7f7bb4061130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb1d9b700 1 -- 172.21.15.61:0/2821643181 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514436231 0 0) 0x7f7bb40b08b0 con 0x7f7bb4064890 2022-07-12T08:50:45.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb259c700 1 -- 172.21.15.61:0/2821643181 >> v1:172.21.15.61:6789/0 conn(0x7f7bb41431e0 legacy=0x7f7bb40b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb259c700 1 -- 172.21.15.61:0/2821643181 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb4064470 con 0x7f7bb4140840 2022-07-12T08:50:45.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb259c700 1 --2- 172.21.15.61:0/2821643181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb4140840 0x7f7bb4060520 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f7ba40032f0 tx=0x7f7ba40091d0).ready entity=mon.1 client_cookie=60b903162210d1d1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb1d9b700 1 -- 172.21.15.61:0/2821643181 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba400b6c0 con 0x7f7bb4140840 2022-07-12T08:50:45.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb1d9b700 1 -- 172.21.15.61:0/2821643181 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba400b820 con 0x7f7bb4140840 2022-07-12T08:50:45.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb1d9b700 1 -- 172.21.15.61:0/2821643181 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba400b990 con 0x7f7bb4140840 2022-07-12T08:50:45.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- 172.21.15.61:0/2821643181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb4140840 msgr2=0x7f7bb4060520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 --2- 172.21.15.61:0/2821643181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb4140840 0x7f7bb4060520 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f7ba40032f0 tx=0x7f7ba40091d0).stop 2022-07-12T08:50:45.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- 172.21.15.61:0/2821643181 shutdown_connections 2022-07-12T08:50:45.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 --2- 172.21.15.61:0/2821643181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb4140840 0x7f7bb4060520 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- 172.21.15.61:0/2821643181 >> 172.21.15.61:0/2821643181 conn(0x7f7bb40b5860 msgr2=0x7f7bb40b5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:45.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- 172.21.15.61:0/2821643181 shutdown_connections 2022-07-12T08:50:45.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.024+0000 7f7bb8969700 1 -- 172.21.15.61:0/2821643181 wait complete. 2022-07-12T08:50:45.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 Processor -- start 2022-07-12T08:50:45.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 -- start start 2022-07-12T08:50:45.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 0x7f7bb4151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb4140840 0x7f7bb4140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb41431e0 0x7f7bb41526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb4152200 con 0x7f7bb4064890 2022-07-12T08:50:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb4147650 con 0x7f7bb41431e0 2022-07-12T08:50:45.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb41478f0 con 0x7f7bb4140840 2022-07-12T08:50:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 0x7f7bb4151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 0x7f7bb4151d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33524/0 (socket says 172.21.15.61:33524) 2022-07-12T08:50:45.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 -- 172.21.15.61:0/688158882 learned_addr learned my addr 172.21.15.61:0/688158882 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb259c700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb4140840 0x7f7bb4140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb359e700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb41431e0 0x7f7bb41526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 -- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb4140840 msgr2=0x7f7bb4140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb4140840 0x7f7bb4140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 -- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb41431e0 msgr2=0x7f7bb41526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb41431e0 0x7f7bb41526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 -- 172.21.15.61:0/688158882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba4009470 con 0x7f7bb4064890 2022-07-12T08:50:45.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb359e700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb41431e0 0x7f7bb41526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb2d9d700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 0x7f7bb4151d10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f7bac001110 tx=0x7f7bac007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bac00ce80 con 0x7f7bb4064890 2022-07-12T08:50:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac0070d0 con 0x7f7bb4064890 2022-07-12T08:50:45.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.028+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb413dbd0 con 0x7f7bb4064890 2022-07-12T08:50:45.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bac007430 con 0x7f7bb4064890 2022-07-12T08:50:45.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb4147220 con 0x7f7bb4064890 2022-07-12T08:50:45.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb4057460 con 0x7f7bb4064890 2022-07-12T08:50:45.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7bac010070 con 0x7f7bb4064890 2022-07-12T08:50:45.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7ba3fff700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053a60 0x7f7b90055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7bac06b4a0 con 0x7f7bb4064890 2022-07-12T08:50:45.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.032+0000 7f7bb259c700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053a60 0x7f7b90055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.036+0000 7f7bb259c700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053a60 0x7f7b90055f20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f7ba4019000 tx=0x7f7ba4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.040+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7bac042400 con 0x7f7bb4064890 2022-07-12T08:50:45.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.208+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7bb409bbb0 con 0x7f7bb4064890 2022-07-12T08:50:45.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.220+0000 7f7ba3fff700 1 -- 172.21.15.61:0/688158882 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v43) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7bac045a20 con 0x7f7bb4064890 2022-07-12T08:50:45.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053a60 msgr2=0x7f7b90055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053a60 0x7f7b90055f20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f7ba4019000 tx=0x7f7ba4016000).stop 2022-07-12T08:50:45.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 msgr2=0x7f7bb4151d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 0x7f7bb4151d10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 rx=0x7f7bac001110 tx=0x7f7bac007aa0).stop 2022-07-12T08:50:45.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 shutdown_connections 2022-07-12T08:50:45.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7b90053a60 0x7f7b90055f20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bb41431e0 0x7f7bb41526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7bb4140840 0x7f7bb4140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 --2- 172.21.15.61:0/688158882 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7bb4064890 0x7f7bb4151d10 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 >> 172.21.15.61:0/688158882 conn(0x7f7bb40b5860 msgr2=0x7f7bb4065aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:45.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 shutdown_connections 2022-07-12T08:50:45.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.224+0000 7f7bb8969700 1 -- 172.21.15.61:0/688158882 wait complete. 2022-07-12T08:50:45.232 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:50:45.243 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:50:45.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.360+0000 7f9419c59700 1 Processor -- start 2022-07-12T08:50:45.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.360+0000 7f9419c59700 1 -- start start 2022-07-12T08:50:45.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9414066410 0x7f94140b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9414143af0 0x7f941413e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94140b6780 con 0x7f94140b63a0 2022-07-12T08:50:45.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94140b1f30 con 0x7f9414066410 2022-07-12T08:50:45.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9414143110 con 0x7f9414143af0 2022-07-12T08:50:45.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9414066410 0x7f94140b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9414066410 0x7f94140b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33534/0 (socket says 172.21.15.61:33534) 2022-07-12T08:50:45.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 -- 172.21.15.61:0/571417472 learned_addr learned my addr 172.21.15.61:0/571417472 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:45.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419458700 1 --2- 172.21.15.61:0/571417472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9414143af0 0x7f941413e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 -- 172.21.15.61:0/571417472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9414143af0 msgr2=0x7f941413e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:45.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 --2- 172.21.15.61:0/571417472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9414143af0 0x7f941413e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 -- 172.21.15.61:0/571417472 >> v1:172.21.15.111:6789/0 conn(0x7f94140b63a0 legacy=0x7f94140b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:45.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 -- 172.21.15.61:0/571417472 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94141433b0 con 0x7f9414066410 2022-07-12T08:50:45.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9413fff700 1 --2- 172.21.15.61:0/571417472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9414066410 0x7f94140b1a40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7f9400002f70 tx=0x7f9400006f40).ready entity=mon.0 client_cookie=23a617754c044eaf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f94137fe700 1 -- 172.21.15.61:0/571417472 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940000dc70 con 0x7f9414066410 2022-07-12T08:50:45.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f94137fe700 1 -- 172.21.15.61:0/571417472 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9400000e90 con 0x7f9414066410 2022-07-12T08:50:45.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f94137fe700 1 -- 172.21.15.61:0/571417472 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940000be90 con 0x7f9414066410 2022-07-12T08:50:45.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- 172.21.15.61:0/571417472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9414066410 msgr2=0x7f94140b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 --2- 172.21.15.61:0/571417472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9414066410 0x7f94140b1a40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7f9400002f70 tx=0x7f9400006f40).stop 2022-07-12T08:50:45.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- 172.21.15.61:0/571417472 shutdown_connections 2022-07-12T08:50:45.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 --2- 172.21.15.61:0/571417472 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9414143af0 0x7f941413e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 --2- 172.21.15.61:0/571417472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9414066410 0x7f94140b1a40 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- 172.21.15.61:0/571417472 >> 172.21.15.61:0/571417472 conn(0x7f9414063ad0 msgr2=0x7f9414063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:45.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- 172.21.15.61:0/571417472 shutdown_connections 2022-07-12T08:50:45.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.364+0000 7f9419c59700 1 -- 172.21.15.61:0/571417472 wait complete. 2022-07-12T08:50:45.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 Processor -- start 2022-07-12T08:50:45.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 -- start start 2022-07-12T08:50:45.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 0x7f94140b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9414143af0 0x7f9414066950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94140ad890 0x7f94141510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414141d50 con 0x7f94140b63a0 2022-07-12T08:50:45.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9414141ff0 con 0x7f9414143af0 2022-07-12T08:50:45.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419c59700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9414142290 con 0x7f94140ad890 2022-07-12T08:50:45.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 0x7f94140b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 0x7f94140b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33538/0 (socket says 172.21.15.61:33538) 2022-07-12T08:50:45.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 -- 172.21.15.61:0/2706698639 learned_addr learned my addr 172.21.15.61:0/2706698639 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:45.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9413fff700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9414143af0 0x7f9414066950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419458700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94140ad890 0x7f94141510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 -- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94140ad890 msgr2=0x7f94141510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94140ad890 0x7f94141510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 -- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9414143af0 msgr2=0x7f9414066950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9414143af0 0x7f9414066950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9418c57700 1 -- 172.21.15.61:0/2706698639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940000b460 con 0x7f94140b63a0 2022-07-12T08:50:45.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9413fff700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9414143af0 0x7f9414066950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:45.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.368+0000 7f9419458700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94140ad890 0x7f94141510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:45.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9418c57700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 0x7f94140b00f0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f941406a9d0 tx=0x7f9408007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940800ce30 con 0x7f94140b63a0 2022-07-12T08:50:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940800f470 con 0x7f94140b63a0 2022-07-12T08:50:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94140ae850 con 0x7f94140b63a0 2022-07-12T08:50:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f940800f7d0 con 0x7f94140b63a0 2022-07-12T08:50:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94141515d0 con 0x7f94140b63a0 2022-07-12T08:50:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9408019030 con 0x7f94140b63a0 2022-07-12T08:50:45.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9414151880 con 0x7f94140b63a0 2022-07-12T08:50:45.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9411ffb700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93f4053920 0x7f93f4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f940806b3f0 con 0x7f94140b63a0 2022-07-12T08:50:45.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9413fff700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93f4053920 0x7f93f4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.372+0000 7f9413fff700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93f4053920 0x7f93f4055de0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f9400018000 tx=0x7f94140b2660).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.380+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f94080421e0 con 0x7f94140b63a0 2022-07-12T08:50:45.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.548+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9414057460 con 0x7f94140b63a0 2022-07-12T08:50:45.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.556+0000 7f9411ffb700 1 -- 172.21.15.61:0/2706698639 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v44) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9408045800 con 0x7f94140b63a0 2022-07-12T08:50:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93f4053920 msgr2=0x7f93f4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93f4053920 0x7f93f4055de0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f9400018000 tx=0x7f94140b2660).stop 2022-07-12T08:50:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 msgr2=0x7f94140b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 0x7f94140b00f0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7f941406a9d0 tx=0x7f9408007c50).stop 2022-07-12T08:50:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 shutdown_connections 2022-07-12T08:50:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f94140ad890 0x7f94141510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f93f4053920 0x7f93f4055de0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9414143af0 0x7f9414066950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 --2- 172.21.15.61:0/2706698639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f94140b63a0 0x7f94140b00f0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 >> 172.21.15.61:0/2706698639 conn(0x7f9414063ad0 msgr2=0x7f941413bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:45.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 shutdown_connections 2022-07-12T08:50:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.560+0000 7f9419c59700 1 -- 172.21.15.61:0/2706698639 wait complete. 2022-07-12T08:50:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:50:45.578 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:45.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.684+0000 7fabdd7b5700 1 Processor -- start 2022-07-12T08:50:45.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.684+0000 7fabdd7b5700 1 -- start start 2022-07-12T08:50:45.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdd7b5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd80af280 0x7fabd8060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdd7b5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd80aecb0 con 0x7fabd80b3f60 2022-07-12T08:50:45.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdd7b5700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd81435c0 con 0x7fabd809bce0 2022-07-12T08:50:45.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdd7b5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd80afd70 con 0x7fabd80af280 2022-07-12T08:50:45.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fabd80b3f60 0x7fabd80b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50000/0 (socket says 172.21.15.61:50000) 2022-07-12T08:50:45.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2822881752 learned_addr learned my addr 172.21.15.61:0/2822881752 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:45.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd7fff700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd80af280 0x7fabd8060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091973264 0 0) 0x7fabd80aecb0 con 0x7fabd80b3f60 2022-07-12T08:50:45.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8003260 con 0x7fabd80b3f60 2022-07-12T08:50:45.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853390853 0 0) 0x7fabd81435c0 con 0x7fabd809bce0 2022-07-12T08:50:45.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd80aecb0 con 0x7fabd809bce0 2022-07-12T08:50:45.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 438322039 0 0) 0x7fabb8003260 con 0x7fabd80b3f60 2022-07-12T08:50:45.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd80af280 msgr2=0x7fabd8060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:45.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd80af280 0x7fabd8060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 >> v1:172.21.15.111:6789/0 conn(0x7fabd809bce0 legacy=0x7fabd80af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd8143860 con 0x7fabd80b3f60 2022-07-12T08:50:45.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fabd00019e0 con 0x7fabd80b3f60 2022-07-12T08:50:45.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 >> v1:172.21.15.61:6789/0 conn(0x7fabd80b3f60 legacy=0x7fabd80b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8003910 0x7fabb8003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabb8004500 0x7fabb8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb8004e70 0x7fabb800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd80afd70 con 0x7fabb8004e70 2022-07-12T08:50:45.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800bc40 con 0x7fabb8004500 2022-07-12T08:50:45.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb800be00 con 0x7fabb8003910 2022-07-12T08:50:45.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd7fff700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabd80af280 0x7fabd8060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:45.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb8004e70 0x7fabb800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd77fe700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabb8004500 0x7fabb8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd7fff700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8003910 0x7fabb8003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8003910 msgr2=0x7fabb8003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:45.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8003910 0x7fabb8003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabb8004500 msgr2=0x7fabb8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:45.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabb8004500 0x7fabb8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2822881752 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8003260 con 0x7fabb8004e70 2022-07-12T08:50:45.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb8004e70 0x7fabb800b750 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fabd00017b0 tx=0x7fabd0002150).ready entity=mon.0 client_cookie=29944ed6e9377d19 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabd000cc40 con 0x7fabb8004e70 2022-07-12T08:50:45.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd000cda0 con 0x7fabb8004e70 2022-07-12T08:50:45.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.688+0000 7fabd6ffd700 1 -- 172.21.15.61:0/2822881752 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabd0005b80 con 0x7fabb8004e70 2022-07-12T08:50:45.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2822881752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb8004e70 msgr2=0x7fabb800b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2822881752 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb8004e70 0x7fabb800b750 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7fabd00017b0 tx=0x7fabd0002150).stop 2022-07-12T08:50:45.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2822881752 reap_dead start 2022-07-12T08:50:45.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2822881752 shutdown_connections 2022-07-12T08:50:45.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2822881752 >> 172.21.15.61:0/2822881752 conn(0x7fabd80b92c0 msgr2=0x7fabd80b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:45.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2822881752 shutdown_connections 2022-07-12T08:50:45.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2822881752 wait complete. 2022-07-12T08:50:45.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 Processor -- start 2022-07-12T08:50:45.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- start start 2022-07-12T08:50:45.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8004e70 0x7fabd805c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd81460b0 0x7fabd8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 0x7fabd8155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd809e5b0 con 0x7fabd8145530 2022-07-12T08:50:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd8146630 con 0x7fabd81460b0 2022-07-12T08:50:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdd7b5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd81468d0 con 0x7fabb8004e70 2022-07-12T08:50:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdcfb4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 0x7fabd8155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdcfb4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 0x7fabd8155cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33562/0 (socket says 172.21.15.61:33562) 2022-07-12T08:50:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2818365893 learned_addr learned my addr 172.21.15.61:0/2818365893 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.692+0000 7fabd77fe700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd81460b0 0x7fabd8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabd7fff700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8004e70 0x7fabd805c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8004e70 msgr2=0x7fabd805c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8004e70 0x7fabd805c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd81460b0 msgr2=0x7fabd8155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd81460b0 0x7fabd8155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdcfb4700 1 -- 172.21.15.61:0/2818365893 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd8146b70 con 0x7fabd8145530 2022-07-12T08:50:45.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdcfb4700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 0x7fabd8155cf0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7fabc80011d0 tx=0x7fabd0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabd0007ba0 con 0x7fabd8145530 2022-07-12T08:50:45.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd00074d0 con 0x7fabd8145530 2022-07-12T08:50:45.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd81561e0 con 0x7fabd8145530 2022-07-12T08:50:45.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabd0001e90 con 0x7fabd8145530 2022-07-12T08:50:45.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd8143f70 con 0x7fabd8145530 2022-07-12T08:50:45.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd8146490 con 0x7fabd8145530 2022-07-12T08:50:45.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.696+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fabd001d070 con 0x7fabd8145530 2022-07-12T08:50:45.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.700+0000 7fabd57fa700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabbc0539b0 0x7fabbc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:45.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.700+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fabd006cad0 con 0x7fabd8145530 2022-07-12T08:50:45.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.700+0000 7fabd7fff700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabbc0539b0 0x7fabbc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:45.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.700+0000 7fabd7fff700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabbc0539b0 0x7fabbc055e70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fabc0008000 tx=0x7fabc000b040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:45.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.704+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabd8146490 con 0x7fabd8145530 2022-07-12T08:50:45.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd8146490 con 0x7fabd8145530 2022-07-12T08:50:45.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabd57fa700 1 -- 172.21.15.61:0/2818365893 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fabd8146490 con 0x7fabd8145530 2022-07-12T08:50:45.871 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:45.872 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:42.959468+0000","modified":"2022-07-12T08:50:43.963135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/676021650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":676021650},{"type":"v1","addr":"172.21.15.61:6833","nonce":676021650}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-12T08:50:45.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabbc0539b0 msgr2=0x7fabbc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabbc0539b0 0x7fabbc055e70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7fabc0008000 tx=0x7fabc000b040).stop 2022-07-12T08:50:45.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 msgr2=0x7fabd8155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:45.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 0x7fabd8155cf0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7fabc80011d0 tx=0x7fabd0015000).stop 2022-07-12T08:50:45.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 shutdown_connections 2022-07-12T08:50:45.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd8145530 0x7fabd8155cf0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fabbc0539b0 0x7fabbc055e70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fabd81460b0 0x7fabd8155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 --2- 172.21.15.61:0/2818365893 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fabb8004e70 0x7fabd805c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:45.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 >> 172.21.15.61:0/2818365893 conn(0x7fabd80b92c0 msgr2=0x7fabd80604e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:45.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 shutdown_connections 2022-07-12T08:50:45.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:45.868+0000 7fabdd7b5700 1 -- 172.21.15.61:0/2818365893 wait complete. 2022-07-12T08:50:45.877 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-12T08:50:46.086 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:42.959468+0000', 'modified': '2022-07-12T08:50:43.963135+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8442': {'gid': 8442, 'name': 'a', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/676021650', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 676021650}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 676021650}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:46.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:46.088 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:46.215 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8442,"want_state":"up:active","state":"up:active","rank_uptime":3.2344767700000001,"mdsmap_epoch":77,"osdmap_epoch":138,"osdmap_epoch_barrier":137,"uptime":8.1827137590000003} 2022-07-12T08:50:46.216 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8442, "mdsmap_epoch": 77, "osdmap_epoch": 138, "osdmap_epoch_barrier": 137, "rank_uptime": 3.23447677, "state": "up:active", "uptime": 8.182713759, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:46.227 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:46.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 Processor -- start 2022-07-12T08:50:46.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- start start 2022-07-12T08:50:46.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbb140b1730 0x7fbb140b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb140b1f50 con 0x7fbb14061690 2022-07-12T08:50:46.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb14061a70 con 0x7fbb140988a0 2022-07-12T08:50:46.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb140b2180 con 0x7fbb140b1730 2022-07-12T08:50:46.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb137fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbb14061690 0x7fbb14098190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50016/0 (socket says 172.21.15.61:50016) 2022-07-12T08:50:46.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb137fe700 1 -- 172.21.15.61:0/3466317256 learned_addr learned my addr 172.21.15.61:0/3466317256 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:46.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb12ffd700 1 --2- 172.21.15.61:0/3466317256 >> v2:172.21.15.111:3300/0 conn(0x7fbb140b1730 0x7fbb140b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187063337 0 0) 0x7fbb140b1f50 con 0x7fbb14061690 2022-07-12T08:50:46.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf4003260 con 0x7fbb14061690 2022-07-12T08:50:46.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052357793 0 0) 0x7fbb14061a70 con 0x7fbb140988a0 2022-07-12T08:50:46.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb140b1f50 con 0x7fbb140988a0 2022-07-12T08:50:46.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb12ffd700 1 -- 172.21.15.61:0/3466317256 >> v1:172.21.15.111:6789/0 conn(0x7fbb140988a0 legacy=0x7fbb14144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb12ffd700 1 -- 172.21.15.61:0/3466317256 >> v1:172.21.15.61:6789/0 conn(0x7fbb14061690 legacy=0x7fbb14098190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb12ffd700 1 -- 172.21.15.61:0/3466317256 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb140b2420 con 0x7fbb140b1730 2022-07-12T08:50:46.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb12ffd700 1 --2- 172.21.15.61:0/3466317256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140b1730 0x7fbb140b28c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fbb04002f70 tx=0x7fbb04006df0).ready entity=mon.1 client_cookie=e2108090ca92d54a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb0400d540 con 0x7fbb140b1730 2022-07-12T08:50:46.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb04000e90 con 0x7fbb140b1730 2022-07-12T08:50:46.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb127fc700 1 -- 172.21.15.61:0/3466317256 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb0400d890 con 0x7fbb140b1730 2022-07-12T08:50:46.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- 172.21.15.61:0/3466317256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140b1730 msgr2=0x7fbb140b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 --2- 172.21.15.61:0/3466317256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140b1730 0x7fbb140b28c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7fbb04002f70 tx=0x7fbb04006df0).stop 2022-07-12T08:50:46.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- 172.21.15.61:0/3466317256 shutdown_connections 2022-07-12T08:50:46.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 --2- 172.21.15.61:0/3466317256 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140b1730 0x7fbb140b28c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- 172.21.15.61:0/3466317256 >> 172.21.15.61:0/3466317256 conn(0x7fbb14064720 msgr2=0x7fbb140b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:46.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- 172.21.15.61:0/3466317256 shutdown_connections 2022-07-12T08:50:46.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.344+0000 7fbb19e51700 1 -- 172.21.15.61:0/3466317256 wait complete. 2022-07-12T08:50:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 Processor -- start 2022-07-12T08:50:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 -- start start 2022-07-12T08:50:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 0x7fbb14067480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140988a0 0x7fbb14153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb140b1730 0x7fbb14153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1409a5b0 con 0x7fbb14061690 2022-07-12T08:50:46.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb140ad860 con 0x7fbb140988a0 2022-07-12T08:50:46.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb140adb00 con 0x7fbb140b1730 2022-07-12T08:50:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 0x7fbb14067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 0x7fbb14067480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33572/0 (socket says 172.21.15.61:33572) 2022-07-12T08:50:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 -- 172.21.15.61:0/1682054171 learned_addr learned my addr 172.21.15.61:0/1682054171 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb12ffd700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140988a0 0x7fbb14153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb13fff700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb140b1730 0x7fbb14153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 -- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb140b1730 msgr2=0x7fbb14153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb140b1730 0x7fbb14153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 -- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140988a0 msgr2=0x7fbb14153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140988a0 0x7fbb14153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 -- 172.21.15.61:0/1682054171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0400b180 con 0x7fbb14061690 2022-07-12T08:50:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb13fff700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb140b1730 0x7fbb14153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb137fe700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 0x7fbb14067480 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7fbafc0011d0 tx=0x7fbafc007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbafc0070d0 con 0x7fbb14061690 2022-07-12T08:50:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc007230 con 0x7fbb14061690 2022-07-12T08:50:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb141535e0 con 0x7fbb14061690 2022-07-12T08:50:46.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.348+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbafc005610 con 0x7fbb14061690 2022-07-12T08:50:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb14067970 con 0x7fbb14061690 2022-07-12T08:50:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbafc01a060 con 0x7fbb14061690 2022-07-12T08:50:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb14153770 con 0x7fbb14061690 2022-07-12T08:50:46.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb10ff9700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbaf8053ad0 0x7fbaf8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbafc06b6a0 con 0x7fbb14061690 2022-07-12T08:50:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb12ffd700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbaf8053ad0 0x7fbaf8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.352+0000 7fbb12ffd700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbaf8053ad0 0x7fbaf8055f90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fbb04018000 tx=0x7fbb04014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.360+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbafc042600 con 0x7fbb14061690 2022-07-12T08:50:46.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.484+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb14057460 con 0x7fbb14061690 2022-07-12T08:50:46.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.484+0000 7fbb10ff9700 1 -- 172.21.15.61:0/1682054171 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fbafc018020 con 0x7fbb14061690 2022-07-12T08:50:46.488 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:46.488 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":138,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:43.963239+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:41.024516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:41.962174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:46.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.484+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbaf8053ad0 msgr2=0x7fbaf8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.484+0000 7fbb19e51700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbaf8053ad0 0x7fbaf8055f90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fbb04018000 tx=0x7fbb04014000).stop 2022-07-12T08:50:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.484+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 msgr2=0x7fbb14067480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.484+0000 7fbb19e51700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 0x7fbb14067480 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7fbafc0011d0 tx=0x7fbafc007b50).stop 2022-07-12T08:50:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 shutdown_connections 2022-07-12T08:50:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb140b1730 0x7fbb14153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbaf8053ad0 0x7fbaf8055f90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb140988a0 0x7fbb14153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 --2- 172.21.15.61:0/1682054171 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb14061690 0x7fbb14067480 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 >> 172.21.15.61:0/1682054171 conn(0x7fbb14064720 msgr2=0x7fbb1405cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:46.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 shutdown_connections 2022-07-12T08:50:46.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.488+0000 7fbb19e51700 1 -- 172.21.15.61:0/1682054171 wait complete. 2022-07-12T08:50:46.501 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:50:41.024516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:50:41.962174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:50:46.503 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 Processor -- start 2022-07-12T08:50:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 -- start start 2022-07-12T08:50:46.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f79140b2530 0x7f7914060360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f791413f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79141435e0 con 0x7f7914143110 2022-07-12T08:50:46.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79140b2910 con 0x7f79140b2530 2022-07-12T08:50:46.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f791935c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79140618c0 con 0x7f7914060f60 2022-07-12T08:50:46.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7913fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f79140b2530 0x7f7914060360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7918b5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f791413f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f79137fe700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7914143110 0x7f7914060850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:35404/0 (socket says 172.21.15.61:35404) 2022-07-12T08:50:46.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f79137fe700 1 -- 172.21.15.61:0/1224354877 learned_addr learned my addr 172.21.15.61:0/1224354877 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:46.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7912ffd700 1 -- 172.21.15.61:0/1224354877 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245143957 0 0) 0x7f79141435e0 con 0x7f7914143110 2022-07-12T08:50:46.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7912ffd700 1 -- 172.21.15.61:0/1224354877 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f4003260 con 0x7f7914143110 2022-07-12T08:50:46.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7918b5b700 1 -- 172.21.15.61:0/1224354877 >> v2:172.21.15.111:3300/0 conn(0x7f79140b2530 msgr2=0x7f7914060360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:46.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7918b5b700 1 --2- 172.21.15.61:0/1224354877 >> v2:172.21.15.111:3300/0 conn(0x7f79140b2530 0x7f7914060360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7918b5b700 1 -- 172.21.15.61:0/1224354877 >> v1:172.21.15.111:6789/0 conn(0x7f7914143110 legacy=0x7f7914060850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.608+0000 7f7918b5b700 1 -- 172.21.15.61:0/1224354877 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7914061b60 con 0x7f7914060f60 2022-07-12T08:50:46.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f7918b5b700 1 --2- 172.21.15.61:0/1224354877 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f791413f2b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f790c0032f0 tx=0x7f790c0091d0).ready entity=mon.2 client_cookie=de42fcd706c28f9f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f7912ffd700 1 -- 172.21.15.61:0/1224354877 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f790c00b600 con 0x7f7914060f60 2022-07-12T08:50:46.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f7912ffd700 1 -- 172.21.15.61:0/1224354877 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c00b760 con 0x7f7914060f60 2022-07-12T08:50:46.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f7912ffd700 1 -- 172.21.15.61:0/1224354877 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f790c00b8d0 con 0x7f7914060f60 2022-07-12T08:50:46.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 -- 172.21.15.61:0/1224354877 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 msgr2=0x7f791413f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 --2- 172.21.15.61:0/1224354877 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f791413f2b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f790c0032f0 tx=0x7f790c0091d0).stop 2022-07-12T08:50:46.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 -- 172.21.15.61:0/1224354877 shutdown_connections 2022-07-12T08:50:46.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 --2- 172.21.15.61:0/1224354877 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f791413f2b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 --2- 172.21.15.61:0/1224354877 >> v2:172.21.15.111:3300/0 conn(0x7f79140b2530 0x7f7914060360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 -- 172.21.15.61:0/1224354877 >> 172.21.15.61:0/1224354877 conn(0x7f791405f140 msgr2=0x7f791405f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:46.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 -- 172.21.15.61:0/1224354877 shutdown_connections 2022-07-12T08:50:46.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 -- 172.21.15.61:0/1224354877 wait complete. 2022-07-12T08:50:46.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 Processor -- start 2022-07-12T08:50:46.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.612+0000 7f791935c700 1 -- start start 2022-07-12T08:50:46.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f79140648c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79140b2530 0x7f791414e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7914143110 0x7f79140add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914067270 con 0x7f7914143110 2022-07-12T08:50:46.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791414eb40 con 0x7f79140b2530 2022-07-12T08:50:46.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791414dc70 con 0x7f7914060f60 2022-07-12T08:50:46.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7913fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f79140648c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7914143110 0x7f79140add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7913fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f79140648c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:44014/0 (socket says 172.21.15.61:44014) 2022-07-12T08:50:46.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f79137fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79140b2530 0x7f791414e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7913fff700 1 -- 172.21.15.61:0/2587095315 learned_addr learned my addr 172.21.15.61:0/2587095315 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:46.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 -- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 msgr2=0x7f79140648c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f79140648c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 -- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79140b2530 msgr2=0x7f791414e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79140b2530 0x7f791414e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 -- 172.21.15.61:0/2587095315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c009470 con 0x7f7914143110 2022-07-12T08:50:46.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f7918b5b700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7914143110 0x7f79140add70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7f790c0032f0 tx=0x7f790c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f790c00b8d0 con 0x7f7914143110 2022-07-12T08:50:46.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790c011e60 con 0x7f7914143110 2022-07-12T08:50:46.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791414df10 con 0x7f7914143110 2022-07-12T08:50:46.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f790c00b600 con 0x7f7914143110 2022-07-12T08:50:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7914066930 con 0x7f7914143110 2022-07-12T08:50:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.616+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7914057460 con 0x7f7914143110 2022-07-12T08:50:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.620+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f790c026030 con 0x7f7914143110 2022-07-12T08:50:46.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.620+0000 7f79117fa700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78f80539d0 0x7f78f8055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.620+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f790c06c790 con 0x7f7914143110 2022-07-12T08:50:46.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.620+0000 7f7913fff700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78f80539d0 0x7f78f8055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.620+0000 7f7913fff700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78f80539d0 0x7f78f8055e90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f790400b000 tx=0x7f7904006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.624+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f790c0436f0 con 0x7f7914143110 2022-07-12T08:50:46.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f7914067e20 con 0x7f7914143110 2022-07-12T08:50:46.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f79117fa700 1 -- 172.21.15.61:0/2587095315 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v138) v1 ==== 110+0+10 (secure 0 0 0) 0x7f790c024020 con 0x7f7914143110 2022-07-12T08:50:46.759 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:46.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78f80539d0 msgr2=0x7f78f8055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78f80539d0 0x7f78f8055e90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f790400b000 tx=0x7f7904006df0).stop 2022-07-12T08:50:46.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7914143110 msgr2=0x7f79140add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7914143110 0x7f79140add70 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7f790c0032f0 tx=0x7f790c016000).stop 2022-07-12T08:50:46.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 shutdown_connections 2022-07-12T08:50:46.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7914143110 0x7f79140add70 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f78f80539d0 0x7f78f8055e90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79140b2530 0x7f791414e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 --2- 172.21.15.61:0/2587095315 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7914060f60 0x7f79140648c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 >> 172.21.15.61:0/2587095315 conn(0x7f791405f140 msgr2=0x7f7914139730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:46.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 shutdown_connections 2022-07-12T08:50:46.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.756+0000 7f791935c700 1 -- 172.21.15.61:0/2587095315 wait complete. 2022-07-12T08:50:46.772 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:46.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a40c51700 1 Processor -- start 2022-07-12T08:50:46.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a40c51700 1 -- start start 2022-07-12T08:50:46.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a40c51700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3a3c0b5920 0x7f3a3c14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a40c51700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c05c3e0 con 0x7f3a3c0b0280 2022-07-12T08:50:46.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a40c51700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c0b5d00 con 0x7f3a3c0aa830 2022-07-12T08:50:46.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a40c51700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c05c610 con 0x7f3a3c0b5920 2022-07-12T08:50:46.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3a3c0b5920 0x7f3a3c14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3a3c0b0280 0x7f3a3c14aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50044/0 (socket says 172.21.15.61:50044) 2022-07-12T08:50:46.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3affd700 1 -- 172.21.15.61:0/1651214715 learned_addr learned my addr 172.21.15.61:0/1651214715 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:46.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3a7fc700 1 -- 172.21.15.61:0/1651214715 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615352647 0 0) 0x7f3a3c05c3e0 con 0x7f3a3c0b0280 2022-07-12T08:50:46.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3a7fc700 1 -- 172.21.15.61:0/1651214715 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a28003260 con 0x7f3a3c0b0280 2022-07-12T08:50:46.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3a7fc700 1 -- 172.21.15.61:0/1651214715 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236974222 0 0) 0x7f3a3c0b5d00 con 0x7f3a3c0aa830 2022-07-12T08:50:46.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3b7fe700 1 -- 172.21.15.61:0/1651214715 >> v1:172.21.15.111:6789/0 conn(0x7f3a3c0aa830 legacy=0x7f3a3c0aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3b7fe700 1 -- 172.21.15.61:0/1651214715 >> v1:172.21.15.61:6789/0 conn(0x7f3a3c0b0280 legacy=0x7f3a3c14aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3b7fe700 1 -- 172.21.15.61:0/1651214715 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c05c8b0 con 0x7f3a3c0b5920 2022-07-12T08:50:46.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3b7fe700 1 --2- 172.21.15.61:0/1651214715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c14a9e0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f3a2c002f70 tx=0x7f3a2c006df0).ready entity=mon.0 client_cookie=4cad0f4438d78ae7 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3a7fc700 1 -- 172.21.15.61:0/1651214715 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a2c00d610 con 0x7f3a3c0b5920 2022-07-12T08:50:46.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3a7fc700 1 -- 172.21.15.61:0/1651214715 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c000e90 con 0x7f3a3c0b5920 2022-07-12T08:50:46.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.884+0000 7f3a3a7fc700 1 -- 172.21.15.61:0/1651214715 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a2c00d960 con 0x7f3a3c0b5920 2022-07-12T08:50:46.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- 172.21.15.61:0/1651214715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b5920 msgr2=0x7f3a3c14a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 --2- 172.21.15.61:0/1651214715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c14a9e0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 rx=0x7f3a2c002f70 tx=0x7f3a2c006df0).stop 2022-07-12T08:50:46.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- 172.21.15.61:0/1651214715 shutdown_connections 2022-07-12T08:50:46.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 --2- 172.21.15.61:0/1651214715 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c14a9e0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- 172.21.15.61:0/1651214715 >> 172.21.15.61:0/1651214715 conn(0x7f3a3c063340 msgr2=0x7f3a3c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:46.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- 172.21.15.61:0/1651214715 shutdown_connections 2022-07-12T08:50:46.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- 172.21.15.61:0/1651214715 wait complete. 2022-07-12T08:50:46.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 Processor -- start 2022-07-12T08:50:46.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- start start 2022-07-12T08:50:46.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a3c0aa830 0x7f3a3c0989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 0x7f3a3c1499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0646d0 con 0x7f3a3c0b0280 2022-07-12T08:50:46.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0974b0 con 0x7f3a3c0b5920 2022-07-12T08:50:46.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a40c51700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c097210 con 0x7f3a3c0aa830 2022-07-12T08:50:46.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a3affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 0x7f3a3c1499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a3affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 0x7f3a3c1499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33602/0 (socket says 172.21.15.61:33602) 2022-07-12T08:50:46.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a3affd700 1 -- 172.21.15.61:0/2640426336 learned_addr learned my addr 172.21.15.61:0/2640426336 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:46.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a3bfff700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.888+0000 7f3a3b7fe700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a3c0aa830 0x7f3a3c0989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a3affd700 1 -- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a3c0aa830 msgr2=0x7f3a3c0989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a3affd700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a3c0aa830 0x7f3a3c0989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a3affd700 1 -- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a3c0b5920 msgr2=0x7f3a3c149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:46.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a3affd700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:46.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a3affd700 1 -- 172.21.15.61:0/2640426336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c00b180 con 0x7f3a3c0b0280 2022-07-12T08:50:46.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a3affd700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 0x7f3a3c1499d0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f3a30001110 tx=0x7f3a30007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a3000ce80 con 0x7f3a3c0b0280 2022-07-12T08:50:46.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a300070d0 con 0x7f3a3c0b0280 2022-07-12T08:50:46.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c064b30 con 0x7f3a3c0b0280 2022-07-12T08:50:46.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a30007430 con 0x7f3a3c0b0280 2022-07-12T08:50:46.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c14a3b0 con 0x7f3a3c0b0280 2022-07-12T08:50:46.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3a30010070 con 0x7f3a3c0b0280 2022-07-12T08:50:46.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c057460 con 0x7f3a3c0b0280 2022-07-12T08:50:46.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.892+0000 7f3a38ff9700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a1c053a40 0x7f3a1c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:46.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.896+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3a3006b390 con 0x7f3a3c0b0280 2022-07-12T08:50:46.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.896+0000 7f3a3b7fe700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a1c053a40 0x7f3a1c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:46.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.896+0000 7f3a3b7fe700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a1c053a40 0x7f3a1c055f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f3a2c01a000 tx=0x7f3a2c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:46.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:46.900+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a30042290 con 0x7f3a3c0b0280 2022-07-12T08:50:47.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.028+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3a3c06b2b0 con 0x7f3a3c0b0280 2022-07-12T08:50:47.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.028+0000 7f3a38ff9700 1 -- 172.21.15.61:0/2640426336 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v138) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3a300458b0 con 0x7f3a3c0b0280 2022-07-12T08:50:47.032 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:47.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a1c053a40 msgr2=0x7f3a1c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a1c053a40 0x7f3a1c055f00 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f3a2c01a000 tx=0x7f3a2c014000).stop 2022-07-12T08:50:47.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 msgr2=0x7f3a3c1499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 0x7f3a3c1499d0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f3a30001110 tx=0x7f3a30007aa0).stop 2022-07-12T08:50:47.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 shutdown_connections 2022-07-12T08:50:47.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3a1c053a40 0x7f3a1c055f00 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3a3c0b5920 0x7f3a3c149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a3c0b0280 0x7f3a3c1499d0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 --2- 172.21.15.61:0/2640426336 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3a3c0aa830 0x7f3a3c0989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 >> 172.21.15.61:0/2640426336 conn(0x7f3a3c063340 msgr2=0x7f3a3c0b4340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 shutdown_connections 2022-07-12T08:50:47.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.032+0000 7f3a40c51700 1 -- 172.21.15.61:0/2640426336 wait complete. 2022-07-12T08:50:47.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:50:47.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.160+0000 7fd960edf700 1 Processor -- start 2022-07-12T08:50:47.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.160+0000 7fd960edf700 1 -- start start 2022-07-12T08:50:47.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd95c0af3d0 0x7fd95c144200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd95c0afb90 0x7fd95c05f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c0b2220 con 0x7fd95c05bd10 2022-07-12T08:50:47.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c140b90 con 0x7fd95c0afb90 2022-07-12T08:50:47.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0af7b0 con 0x7fd95c0af3d0 2022-07-12T08:50:47.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd95c0afb90 0x7fd95c05f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd95c0afb90 0x7fd95c05f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33610/0 (socket says 172.21.15.61:33610) 2022-07-12T08:50:47.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 -- 172.21.15.61:0/1174480460 learned_addr learned my addr 172.21.15.61:0/1174480460 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:47.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd95a59c700 1 --2- 172.21.15.61:0/1174480460 >> v2:172.21.15.111:3300/0 conn(0x7fd95c0af3d0 0x7fd95c144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 -- 172.21.15.61:0/1174480460 >> v2:172.21.15.111:3300/0 conn(0x7fd95c0af3d0 msgr2=0x7fd95c144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:47.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 --2- 172.21.15.61:0/1174480460 >> v2:172.21.15.111:3300/0 conn(0x7fd95c0af3d0 0x7fd95c144200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 -- 172.21.15.61:0/1174480460 >> v1:172.21.15.111:6789/0 conn(0x7fd95c05bd10 legacy=0x7fd95c05f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 -- 172.21.15.61:0/1174480460 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c0ac7c0 con 0x7fd95c0afb90 2022-07-12T08:50:47.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd953fff700 1 --2- 172.21.15.61:0/1174480460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0afb90 0x7fd95c05f220 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7fd944002f70 tx=0x7fd944006eb0).ready entity=mon.0 client_cookie=1706f73db8888b32 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd959d9b700 1 -- 172.21.15.61:0/1174480460 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd94400d610 con 0x7fd95c0afb90 2022-07-12T08:50:47.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd959d9b700 1 -- 172.21.15.61:0/1174480460 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd944000e90 con 0x7fd95c0afb90 2022-07-12T08:50:47.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd959d9b700 1 -- 172.21.15.61:0/1174480460 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd94400d9a0 con 0x7fd95c0afb90 2022-07-12T08:50:47.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- 172.21.15.61:0/1174480460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0afb90 msgr2=0x7fd95c05f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 --2- 172.21.15.61:0/1174480460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0afb90 0x7fd95c05f220 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 rx=0x7fd944002f70 tx=0x7fd944006eb0).stop 2022-07-12T08:50:47.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- 172.21.15.61:0/1174480460 shutdown_connections 2022-07-12T08:50:47.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 --2- 172.21.15.61:0/1174480460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0afb90 0x7fd95c05f220 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 --2- 172.21.15.61:0/1174480460 >> v2:172.21.15.111:3300/0 conn(0x7fd95c0af3d0 0x7fd95c144200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- 172.21.15.61:0/1174480460 >> 172.21.15.61:0/1174480460 conn(0x7fd95c0b5a40 msgr2=0x7fd95c0b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- 172.21.15.61:0/1174480460 shutdown_connections 2022-07-12T08:50:47.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.164+0000 7fd960edf700 1 -- 172.21.15.61:0/1174480460 wait complete. 2022-07-12T08:50:47.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 Processor -- start 2022-07-12T08:50:47.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- start start 2022-07-12T08:50:47.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd95c05bd10 0x7fd95c065730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 0x7fd95c0a3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd95c0afb90 0x7fd95c0a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c152ec0 con 0x7fd95c0af3d0 2022-07-12T08:50:47.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0a3e40 con 0x7fd95c0afb90 2022-07-12T08:50:47.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c064b10 con 0x7fd95c05bd10 2022-07-12T08:50:47.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 0x7fd95c0a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 0x7fd95c0a3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33616/0 (socket says 172.21.15.61:33616) 2022-07-12T08:50:47.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 -- 172.21.15.61:0/6956218 learned_addr learned my addr 172.21.15.61:0/6956218 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:47.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd95ad9d700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd95c0afb90 0x7fd95c0a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd95a59c700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd95c05bd10 0x7fd95c065730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 -- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd95c05bd10 msgr2=0x7fd95c065730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd95c05bd10 0x7fd95c065730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 -- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd95c0afb90 msgr2=0x7fd95c0a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd95c0afb90 0x7fd95c0a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 -- 172.21.15.61:0/6956218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94400b1e0 con 0x7fd95c0af3d0 2022-07-12T08:50:47.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd95ad9d700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd95c0afb90 0x7fd95c0a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:47.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd953fff700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 0x7fd95c0a3460 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7fd95c0a3030 tx=0x7fd944015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd944011910 con 0x7fd95c0af3d0 2022-07-12T08:50:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd944011a70 con 0x7fd95c0af3d0 2022-07-12T08:50:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c064d40 con 0x7fd95c0af3d0 2022-07-12T08:50:47.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd944011dd0 con 0x7fd95c0af3d0 2022-07-12T08:50:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c153050 con 0x7fd95c0af3d0 2022-07-12T08:50:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.168+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c066ca0 con 0x7fd95c0af3d0 2022-07-12T08:50:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.172+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd944025030 con 0x7fd95c0af3d0 2022-07-12T08:50:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.172+0000 7fd9537fe700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd940053a40 0x7fd940055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.172+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd94406d130 con 0x7fd95c0af3d0 2022-07-12T08:50:47.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.172+0000 7fd95a59c700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd940053a40 0x7fd940055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.172+0000 7fd95a59c700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd940053a40 0x7fd940055f00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7fd94c00c000 tx=0x7fd94c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.176+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd944044090 con 0x7fd95c0af3d0 2022-07-12T08:50:47.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd95c057460 con 0x7fd95c0af3d0 2022-07-12T08:50:47.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd9537fe700 1 -- 172.21.15.61:0/6956218 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v138) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd944023020 con 0x7fd95c0af3d0 2022-07-12T08:50:47.309 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:47.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd940053a40 msgr2=0x7fd940055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd940053a40 0x7fd940055f00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7fd94c00c000 tx=0x7fd94c008040).stop 2022-07-12T08:50:47.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 msgr2=0x7fd95c0a3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 0x7fd95c0a3460 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7fd95c0a3030 tx=0x7fd944015000).stop 2022-07-12T08:50:47.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 shutdown_connections 2022-07-12T08:50:47.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd940053a40 0x7fd940055f00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd95c0afb90 0x7fd95c0a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd95c0af3d0 0x7fd95c0a3460 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 --2- 172.21.15.61:0/6956218 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd95c05bd10 0x7fd95c065730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 >> 172.21.15.61:0/6956218 conn(0x7fd95c0b5a40 msgr2=0x7fd95c0aba70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 shutdown_connections 2022-07-12T08:50:47.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.304+0000 7fd960edf700 1 -- 172.21.15.61:0/6956218 wait complete. 2022-07-12T08:50:47.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:50:47.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.428+0000 7f7dcc4f0700 1 Processor -- start 2022-07-12T08:50:47.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.428+0000 7f7dcc4f0700 1 -- start start 2022-07-12T08:50:47.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b5480 0x7f7dc40b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7dc40b3730 0x7f7dc40afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40b3560 con 0x7f7dc40b03d0 2022-07-12T08:50:47.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40b5b80 con 0x7f7dc40b3730 2022-07-12T08:50:47.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc40b5e20 con 0x7f7dc40b5480 2022-07-12T08:50:47.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc9a8b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7dc40b3730 0x7f7dc40afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dca28c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b5480 0x7f7dc40b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcaa8d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f7dc40b03d0 0x7f7dc4142f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:35444/0 (socket says 172.21.15.61:35444) 2022-07-12T08:50:47.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcaa8d700 1 -- 172.21.15.61:0/2796638856 learned_addr learned my addr 172.21.15.61:0/2796638856 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:47.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc9a8b700 1 -- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b5480 msgr2=0x7f7dc40b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:47.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc9a8b700 1 --2- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b5480 0x7f7dc40b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc9a8b700 1 -- 172.21.15.61:0/2796638856 >> v1:172.21.15.111:6789/0 conn(0x7f7dc40b03d0 legacy=0x7f7dc4142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc9a8b700 1 -- 172.21.15.61:0/2796638856 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40b0820 con 0x7f7dc40b3730 2022-07-12T08:50:47.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dca28c700 1 --2- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b5480 0x7f7dc40b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:47.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc9a8b700 1 --2- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b3730 0x7f7dc40afee0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f7dc0002f70 tx=0x7f7dc0006df0).ready entity=mon.1 client_cookie=d414841acd8b6ad6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc928a700 1 -- 172.21.15.61:0/2796638856 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dc000d570 con 0x7f7dc40b3730 2022-07-12T08:50:47.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc928a700 1 -- 172.21.15.61:0/2796638856 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc0000e90 con 0x7f7dc40b3730 2022-07-12T08:50:47.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dc928a700 1 -- 172.21.15.61:0/2796638856 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dc000d8c0 con 0x7f7dc40b3730 2022-07-12T08:50:47.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b3730 msgr2=0x7f7dc40afee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b3730 0x7f7dc40afee0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7f7dc0002f70 tx=0x7f7dc0006df0).stop 2022-07-12T08:50:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/2796638856 shutdown_connections 2022-07-12T08:50:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b3730 0x7f7dc40afee0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/2796638856 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b5480 0x7f7dc40b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/2796638856 >> 172.21.15.61:0/2796638856 conn(0x7f7dc40b96b0 msgr2=0x7f7dc40b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/2796638856 shutdown_connections 2022-07-12T08:50:47.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.432+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/2796638856 wait complete. 2022-07-12T08:50:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 Processor -- start 2022-07-12T08:50:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 -- start start 2022-07-12T08:50:47.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 0x7f7dc41453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b3730 0x7f7dc4152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b5480 0x7f7dc4144550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4152560 con 0x7f7dc40b03d0 2022-07-12T08:50:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4148f70 con 0x7f7dc40b5480 2022-07-12T08:50:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcc4f0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4148cd0 con 0x7f7dc40b3730 2022-07-12T08:50:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 0x7f7dc41453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dc9a8b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b3730 0x7f7dc4152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 0x7f7dc41453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33626/0 (socket says 172.21.15.61:33626) 2022-07-12T08:50:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 -- 172.21.15.61:0/633455952 learned_addr learned my addr 172.21.15.61:0/633455952 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dcaa8d700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b5480 0x7f7dc4144550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 -- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b3730 msgr2=0x7f7dc4152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b3730 0x7f7dc4152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 -- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b5480 msgr2=0x7f7dc4144550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b5480 0x7f7dc4144550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.436+0000 7f7dca28c700 1 -- 172.21.15.61:0/633455952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db400b180 con 0x7f7dc40b03d0 2022-07-12T08:50:47.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7dca28c700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 0x7f7dc41453c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f7db400c000 tx=0x7f7db4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7db4010780 con 0x7f7dc40b03d0 2022-07-12T08:50:47.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db40108e0 con 0x7f7dc40b03d0 2022-07-12T08:50:47.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc000b180 con 0x7f7dc40b03d0 2022-07-12T08:50:47.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7db4010c40 con 0x7f7dc40b03d0 2022-07-12T08:50:47.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc41531c0 con 0x7f7dc40b03d0 2022-07-12T08:50:47.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4144c10 con 0x7f7dc40b03d0 2022-07-12T08:50:47.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.440+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7db401c460 con 0x7f7dc40b03d0 2022-07-12T08:50:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.444+0000 7f7db37fe700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7da4053a40 0x7f7da4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.444+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7db406c3c0 con 0x7f7dc40b03d0 2022-07-12T08:50:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.444+0000 7f7dc9a8b700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7da4053a40 0x7f7da4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.444+0000 7f7dc9a8b700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7da4053a40 0x7f7da4055f00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f7dc001a000 tx=0x7f7dc0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.448+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7db40432c0 con 0x7f7dc40b03d0 2022-07-12T08:50:47.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.572+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7dc4057460 con 0x7f7dc40b03d0 2022-07-12T08:50:47.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.572+0000 7f7db37fe700 1 -- 172.21.15.61:0/633455952 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v138) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7db40468e0 con 0x7f7dc40b03d0 2022-07-12T08:50:47.579 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:47.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7da4053a40 msgr2=0x7f7da4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7da4053a40 0x7f7da4055f00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f7dc001a000 tx=0x7f7dc0014000).stop 2022-07-12T08:50:47.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 msgr2=0x7f7dc41453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 0x7f7dc41453c0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f7db400c000 tx=0x7f7db4006df0).stop 2022-07-12T08:50:47.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 shutdown_connections 2022-07-12T08:50:47.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7da4053a40 0x7f7da4055f00 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7dc40b5480 0x7f7dc4144550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7dc40b3730 0x7f7dc4152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 --2- 172.21.15.61:0/633455952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc40b03d0 0x7f7dc41453c0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 >> 172.21.15.61:0/633455952 conn(0x7f7dc40b96b0 msgr2=0x7f7dc40b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 shutdown_connections 2022-07-12T08:50:47.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.576+0000 7f7dcc4f0700 1 -- 172.21.15.61:0/633455952 wait complete. 2022-07-12T08:50:47.598 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:50:47.599 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:47.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 Processor -- start 2022-07-12T08:50:47.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 -- start start 2022-07-12T08:50:47.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fbc980af280 0x7fbc98060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc9809bce0 0x7fbc980af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc980b3f60 0x7fbc980b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc98143b00 con 0x7fbc9809bce0 2022-07-12T08:50:47.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc980aecb0 con 0x7fbc980af280 2022-07-12T08:50:47.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9d17b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc981435c0 con 0x7fbc980b3f60 2022-07-12T08:50:47.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc977fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc9809bce0 0x7fbc980af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc977fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbc9809bce0 0x7fbc980af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33636/0 (socket says 172.21.15.61:33636) 2022-07-12T08:50:47.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc977fe700 1 -- 172.21.15.61:0/1492073933 learned_addr learned my addr 172.21.15.61:0/1492073933 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:47.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc9c97a700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc980b3f60 0x7fbc980b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.700+0000 7fbc97fff700 1 --2- 172.21.15.61:0/1492073933 >> v2:172.21.15.111:3300/0 conn(0x7fbc980af280 0x7fbc98060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc977fe700 1 -- 172.21.15.61:0/1492073933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc980b3f60 msgr2=0x7fbc980b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:47.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc977fe700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc980b3f60 0x7fbc980b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc977fe700 1 -- 172.21.15.61:0/1492073933 >> v2:172.21.15.111:3300/0 conn(0x7fbc980af280 msgr2=0x7fbc98060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:47.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc977fe700 1 --2- 172.21.15.61:0/1492073933 >> v2:172.21.15.111:3300/0 conn(0x7fbc980af280 0x7fbc98060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc977fe700 1 -- 172.21.15.61:0/1492073933 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc98143860 con 0x7fbc9809bce0 2022-07-12T08:50:47.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9c97a700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc980b3f60 0x7fbc980b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:47.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc977fe700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc9809bce0 0x7fbc980af660 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7fbc80002f70 tx=0x7fbc80006eb0).ready entity=mon.0 client_cookie=14699e3ff282d517 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc96ffd700 1 -- 172.21.15.61:0/1492073933 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc8000d570 con 0x7fbc9809bce0 2022-07-12T08:50:47.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc96ffd700 1 -- 172.21.15.61:0/1492073933 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc80000e90 con 0x7fbc9809bce0 2022-07-12T08:50:47.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc96ffd700 1 -- 172.21.15.61:0/1492073933 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc8000d900 con 0x7fbc9809bce0 2022-07-12T08:50:47.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 -- 172.21.15.61:0/1492073933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc9809bce0 msgr2=0x7fbc980af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc9809bce0 0x7fbc980af660 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7fbc80002f70 tx=0x7fbc80006eb0).stop 2022-07-12T08:50:47.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 -- 172.21.15.61:0/1492073933 shutdown_connections 2022-07-12T08:50:47.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc980b3f60 0x7fbc980b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/1492073933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc9809bce0 0x7fbc980af660 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/1492073933 >> v2:172.21.15.111:3300/0 conn(0x7fbc980af280 0x7fbc98060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 -- 172.21.15.61:0/1492073933 >> 172.21.15.61:0/1492073933 conn(0x7fbc980b92c0 msgr2=0x7fbc980b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 -- 172.21.15.61:0/1492073933 shutdown_connections 2022-07-12T08:50:47.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 -- 172.21.15.61:0/1492073933 wait complete. 2022-07-12T08:50:47.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 Processor -- start 2022-07-12T08:50:47.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.704+0000 7fbc9d17b700 1 -- start start 2022-07-12T08:50:47.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc9809bce0 0x7fbc980b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980af280 0x7fbc980b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 0x7fbc980b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980b1920 con 0x7fbc980b3f60 2022-07-12T08:50:47.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9809d660 con 0x7fbc980af280 2022-07-12T08:50:47.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98148e60 con 0x7fbc9809bce0 2022-07-12T08:50:47.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 0x7fbc980b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 0x7fbc980b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33644/0 (socket says 172.21.15.61:33644) 2022-07-12T08:50:47.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc97fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc9809bce0 0x7fbc980b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 -- 172.21.15.61:0/702704325 learned_addr learned my addr 172.21.15.61:0/702704325 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:47.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc977fe700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980af280 0x7fbc980b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 -- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc9809bce0 msgr2=0x7fbc980b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc9809bce0 0x7fbc980b7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 -- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980af280 msgr2=0x7fbc980b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980af280 0x7fbc980b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 -- 172.21.15.61:0/702704325 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8000b1e0 con 0x7fbc980b3f60 2022-07-12T08:50:47.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9c97a700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 0x7fbc980b2240 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7fbc8800a000 tx=0x7fbc88008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc88011780 con 0x7fbc980b3f60 2022-07-12T08:50:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc880118e0 con 0x7fbc980b3f60 2022-07-12T08:50:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc88011c40 con 0x7fbc980b3f60 2022-07-12T08:50:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc980b2730 con 0x7fbc980b3f60 2022-07-12T08:50:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc981469f0 con 0x7fbc980b3f60 2022-07-12T08:50:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.708+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc980b1a90 con 0x7fbc980b3f60 2022-07-12T08:50:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.712+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbc8801c460 con 0x7fbc980b3f60 2022-07-12T08:50:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.712+0000 7fbc957fa700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc7c053920 0x7fbc7c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.712+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbc8806c040 con 0x7fbc980b3f60 2022-07-12T08:50:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.712+0000 7fbc97fff700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc7c053920 0x7fbc7c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.712+0000 7fbc97fff700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc7c053920 0x7fbc7c055de0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fbc9000b000 tx=0x7fbc900091a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:47.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.716+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc88042f40 con 0x7fbc980b3f60 2022-07-12T08:50:47.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.880+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc98057460 con 0x7fbc980b3f60 2022-07-12T08:50:47.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.880+0000 7fbc957fa700 1 -- 172.21.15.61:0/702704325 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fbc88046560 con 0x7fbc980b3f60 2022-07-12T08:50:47.899 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:47.918 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:42.959468+0000","modified":"2022-07-12T08:50:43.963135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/676021650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":676021650},{"type":"v1","addr":"172.21.15.61:6833","nonce":676021650}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-12T08:50:47.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc7c053920 msgr2=0x7fbc7c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc7c053920 0x7fbc7c055de0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fbc9000b000 tx=0x7fbc900091a0).stop 2022-07-12T08:50:47.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 msgr2=0x7fbc980b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 0x7fbc980b2240 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7fbc8800a000 tx=0x7fbc88008040).stop 2022-07-12T08:50:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 shutdown_connections 2022-07-12T08:50:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc980b3f60 0x7fbc980b2240 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbc7c053920 0x7fbc7c055de0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbc980af280 0x7fbc980b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 --2- 172.21.15.61:0/702704325 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbc9809bce0 0x7fbc980b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:47.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 >> 172.21.15.61:0/702704325 conn(0x7fbc980b92c0 msgr2=0x7fbc9805c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:47.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 shutdown_connections 2022-07-12T08:50:47.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:47.884+0000 7fbc9d17b700 1 -- 172.21.15.61:0/702704325 wait complete. 2022-07-12T08:50:47.963 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-12T08:50:48.170 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:42.959468+0000', 'modified': '2022-07-12T08:50:43.963135+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8442': {'gid': 8442, 'name': 'a', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/676021650', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 676021650}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 676021650}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:48.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:48.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:48.300 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8442,"want_state":"up:active","state":"up:active","rank_uptime":5.3271055509999998,"mdsmap_epoch":77,"osdmap_epoch":138,"osdmap_epoch_barrier":137,"uptime":10.275343886} 2022-07-12T08:50:48.301 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8442, "mdsmap_epoch": 77, "osdmap_epoch": 138, "osdmap_epoch_barrier": 137, "rank_uptime": 5.327105551, "state": "up:active", "uptime": 10.275343886, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:48.302 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:50:48.302 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:50:48.303 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:50:48.304 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:48.305 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:48.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:50:48.338 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:48.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:48.411 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:48.412 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:50:48.425 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:50:48.426 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:48.476 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:50:48.504 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:48.505 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:48.535 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:48.535 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:48.536 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:50:48.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:50:48.542+0000 7f7cc51dc080 -1 init, newargv = 0x55d70464c610 newargc=9 2022-07-12T08:50:48.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22256]: starting ceph client 2022-07-12T08:50:48.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22256]: starting fuse 2022-07-12T08:50:49.660 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:50:49.687 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:49.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:49.710 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:49.710 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:49.711 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:50:49.759 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:50:49.760 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:50:49.761 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:49.762 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:50:49.762 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:50:49.763 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:50:49.763 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:50:49.764 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:50:49.764 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:49.764 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:50:49.767 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:50:49.768 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:50:49.769 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:49.769 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:50:49.770 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:50:49.771 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:50:49.772 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:50:49.772 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:49.773 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:50:49.774 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:50:49.776 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:50:49.777 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:50:49.777 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:50:49.778 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:50:49.780 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:50:49.780 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:50:49.827 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.22256.asok 2022-07-12T08:50:49.831 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22256.asok 2022-07-12T08:50:49.832 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:49.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22256.asok status 2022-07-12T08:50:49.969 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:50:49.970 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:50:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:50:49.971 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:50:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:50:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:50:49.972 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:50:49.973 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "22256", 2022-07-12T08:50:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:50:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:49.977 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:50:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:50:49.978 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 8619, 2022-07-12T08:50:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:50:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:50:49.981 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:50:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8619 2022-07-12T08:50:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:50:49.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:50:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:50:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1614496438 2022-07-12T08:50:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:50:49.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:50:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:50:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:50:49.984 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1614496438 2022-07-12T08:50:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:50:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.8619 172.21.15.121:0/1614496438", 2022-07-12T08:50:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/1614496438", 2022-07-12T08:50:49.985 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:50:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 77, 2022-07-12T08:50:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 138, 2022-07-12T08:50:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:50:49.986 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:50:49.987 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:50:49.987 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:50:49.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:49.989 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:49.989 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:49.997 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:50:49.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:50.000 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:50.000 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:50.061 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:50:50.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 Processor -- start 2022-07-12T08:50:50.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 -- start start 2022-07-12T08:50:50.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f55e009bce0 0x7f55e00af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f55e00b3f60 0x7f55e00b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e00aecb0 con 0x7f55e00af280 2022-07-12T08:50:50.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e0143b00 con 0x7f55e00b3f60 2022-07-12T08:50:50.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55dffff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e01435c0 con 0x7f55e009bce0 2022-07-12T08:50:50.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f55e00b3f60 0x7f55e00b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f55e00b3f60 0x7f55e00b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33652/0 (socket says 172.21.15.61:33652) 2022-07-12T08:50:50.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 -- 172.21.15.61:0/3416882544 learned_addr learned my addr 172.21.15.61:0/3416882544 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:50.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55de7fc700 1 --2- 172.21.15.61:0/3416882544 >> v2:172.21.15.111:3300/0 conn(0x7f55e009bce0 0x7f55e00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 -- 172.21.15.61:0/3416882544 >> v2:172.21.15.111:3300/0 conn(0x7f55e009bce0 msgr2=0x7f55e00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:50.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 --2- 172.21.15.61:0/3416882544 >> v2:172.21.15.111:3300/0 conn(0x7f55e009bce0 0x7f55e00af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 -- 172.21.15.61:0/3416882544 >> v1:172.21.15.111:6789/0 conn(0x7f55e00af280 legacy=0x7f55e0060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 -- 172.21.15.61:0/3416882544 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0143860 con 0x7f55e00b3f60 2022-07-12T08:50:50.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55de7fc700 1 --2- 172.21.15.61:0/3416882544 >> v2:172.21.15.111:3300/0 conn(0x7f55e009bce0 0x7f55e00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:50.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.164+0000 7f55df7fe700 1 --2- 172.21.15.61:0/3416882544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e00b3f60 0x7f55e00b4810 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f55d0002f70 tx=0x7f55d0006eb0).ready entity=mon.0 client_cookie=bf55739c88df93d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55ddffb700 1 -- 172.21.15.61:0/3416882544 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d000d5c0 con 0x7f55e00b3f60 2022-07-12T08:50:50.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55ddffb700 1 -- 172.21.15.61:0/3416882544 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d0000e90 con 0x7f55e00b3f60 2022-07-12T08:50:50.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55ddffb700 1 -- 172.21.15.61:0/3416882544 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d000d950 con 0x7f55e00b3f60 2022-07-12T08:50:50.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 -- 172.21.15.61:0/3416882544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e00b3f60 msgr2=0x7f55e00b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 --2- 172.21.15.61:0/3416882544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e00b3f60 0x7f55e00b4810 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f55d0002f70 tx=0x7f55d0006eb0).stop 2022-07-12T08:50:50.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 -- 172.21.15.61:0/3416882544 shutdown_connections 2022-07-12T08:50:50.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 --2- 172.21.15.61:0/3416882544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e00b3f60 0x7f55e00b4810 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 --2- 172.21.15.61:0/3416882544 >> v2:172.21.15.111:3300/0 conn(0x7f55e009bce0 0x7f55e00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 -- 172.21.15.61:0/3416882544 >> 172.21.15.61:0/3416882544 conn(0x7f55e00b92c0 msgr2=0x7f55e00b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 -- 172.21.15.61:0/3416882544 shutdown_connections 2022-07-12T08:50:50.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 -- 172.21.15.61:0/3416882544 wait complete. 2022-07-12T08:50:50.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 Processor -- start 2022-07-12T08:50:50.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.168+0000 7f55dffff700 1 -- start start 2022-07-12T08:50:50.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 0x7f55e00b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55e00af280 0x7f55e013d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55e00b3f60 0x7f55e013d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e00b1f90 con 0x7f55e009bce0 2022-07-12T08:50:50.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e00b12b0 con 0x7f55e00b3f60 2022-07-12T08:50:50.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e00b2670 con 0x7f55e00af280 2022-07-12T08:50:50.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 0x7f55e00b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 0x7f55e00b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33654/0 (socket says 172.21.15.61:33654) 2022-07-12T08:50:50.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 -- 172.21.15.61:0/1109953172 learned_addr learned my addr 172.21.15.61:0/1109953172 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:50.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55df7fe700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55e00b3f60 0x7f55e013d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55de7fc700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55e00af280 0x7f55e013d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 -- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55e00af280 msgr2=0x7f55e013d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55e00af280 0x7f55e013d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 -- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55e00b3f60 msgr2=0x7f55e013d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55e00b3f60 0x7f55e013d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 -- 172.21.15.61:0/1109953172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c8009470 con 0x7f55e009bce0 2022-07-12T08:50:50.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55deffd700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 0x7f55e00b2e00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7f55e00b2800 tx=0x7f55d8007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d8007020 con 0x7f55e009bce0 2022-07-12T08:50:50.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d8007180 con 0x7f55e009bce0 2022-07-12T08:50:50.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d000b1e0 con 0x7f55e009bce0 2022-07-12T08:50:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55d8006560 con 0x7f55e009bce0 2022-07-12T08:50:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e00b1710 con 0x7f55e009bce0 2022-07-12T08:50:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.172+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e013dfb0 con 0x7f55e009bce0 2022-07-12T08:50:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.176+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f55d8010070 con 0x7f55e009bce0 2022-07-12T08:50:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.176+0000 7f55d7fff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55c4053af0 0x7f55c4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.176+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f55d806b040 con 0x7f55e009bce0 2022-07-12T08:50:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.176+0000 7f55de7fc700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55c4053af0 0x7f55c4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.176+0000 7f55de7fc700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55c4053af0 0x7f55c4055fb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f55c800b000 tx=0x7f55c80091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.180+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f55e013dfb0 con 0x7f55e009bce0 2022-07-12T08:50:50.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.308+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55e013dfb0 con 0x7f55e009bce0 2022-07-12T08:50:50.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55d7fff700 1 -- 172.21.15.61:0/1109953172 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f55d8023f00 con 0x7f55e009bce0 2022-07-12T08:50:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:50.315 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":138,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:50:43.963239+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:50:41.024516+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"137","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-12T08:50:41.962174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:50:50.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55c4053af0 msgr2=0x7f55c4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55c4053af0 0x7f55c4055fb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f55c800b000 tx=0x7f55c80091d0).stop 2022-07-12T08:50:50.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 msgr2=0x7f55e00b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 0x7f55e00b2e00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7f55e00b2800 tx=0x7f55d8007c40).stop 2022-07-12T08:50:50.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 shutdown_connections 2022-07-12T08:50:50.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f55c4053af0 0x7f55c4055fb0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55e00b3f60 0x7f55e013d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f55e00af280 0x7f55e013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 --2- 172.21.15.61:0/1109953172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f55e009bce0 0x7f55e00b2e00 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 >> 172.21.15.61:0/1109953172 conn(0x7f55e00b92c0 msgr2=0x7f55e00602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 shutdown_connections 2022-07-12T08:50:50.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.312+0000 7f55dffff700 1 -- 172.21.15.61:0/1109953172 wait complete. 2022-07-12T08:50:50.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:50:41.024516+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '137', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:50:41.962174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:50:50.329 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:50:50.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a1ccda700 1 Processor -- start 2022-07-12T08:50:50.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a1ccda700 1 -- start start 2022-07-12T08:50:50.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a1ccda700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a18060490 0x7f1a180acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a1ccda700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a18060870 con 0x7f1a180b0a90 2022-07-12T08:50:50.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a1ccda700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a18098930 con 0x7f1a18146b40 2022-07-12T08:50:50.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a1ccda700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a18146fd0 con 0x7f1a18060490 2022-07-12T08:50:50.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a177fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a18060490 0x7f1a180acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a16ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1a180b0a90 0x7f1a18146430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50114/0 (socket says 172.21.15.61:50114) 2022-07-12T08:50:50.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a177fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1a18060490 0x7f1a180acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33662/0 (socket says 172.21.15.61:33662) 2022-07-12T08:50:50.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a16ffd700 1 -- 172.21.15.61:0/1049373348 learned_addr learned my addr 172.21.15.61:0/1049373348 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:50.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a177fe700 1 -- 172.21.15.61:0/1049373348 >> v1:172.21.15.111:6789/0 conn(0x7f1a18146b40 legacy=0x7f1a18142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a177fe700 1 -- 172.21.15.61:0/1049373348 >> v1:172.21.15.61:6789/0 conn(0x7f1a180b0a90 legacy=0x7f1a18146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.440+0000 7f1a177fe700 1 -- 172.21.15.61:0/1049373348 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a18098bd0 con 0x7f1a18060490 2022-07-12T08:50:50.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 --2- 172.21.15.61:0/1049373348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a180acd00 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f1a180b0f00 tx=0x7f1a08006df0).ready entity=mon.0 client_cookie=b08ca8d58310d080 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a167fc700 1 -- 172.21.15.61:0/1049373348 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a0800d610 con 0x7f1a18060490 2022-07-12T08:50:50.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a167fc700 1 -- 172.21.15.61:0/1049373348 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08000e90 con 0x7f1a18060490 2022-07-12T08:50:50.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a167fc700 1 -- 172.21.15.61:0/1049373348 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a0800d960 con 0x7f1a18060490 2022-07-12T08:50:50.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- 172.21.15.61:0/1049373348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 msgr2=0x7f1a180acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/1049373348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a180acd00 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f1a180b0f00 tx=0x7f1a08006df0).stop 2022-07-12T08:50:50.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- 172.21.15.61:0/1049373348 shutdown_connections 2022-07-12T08:50:50.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/1049373348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a180acd00 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- 172.21.15.61:0/1049373348 >> 172.21.15.61:0/1049373348 conn(0x7f1a180b6b80 msgr2=0x7f1a18060d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- 172.21.15.61:0/1049373348 shutdown_connections 2022-07-12T08:50:50.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- 172.21.15.61:0/1049373348 wait complete. 2022-07-12T08:50:50.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 Processor -- start 2022-07-12T08:50:50.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- start start 2022-07-12T08:50:50.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a18145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180b0a90 0x7f1a18099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a18146b40 0x7f1a18099730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180adcd0 con 0x7f1a18060490 2022-07-12T08:50:50.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a18148b30 con 0x7f1a18146b40 2022-07-12T08:50:50.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a1ccda700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a18148dd0 con 0x7f1a180b0a90 2022-07-12T08:50:50.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a18145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a18145280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33668/0 (socket says 172.21.15.61:33668) 2022-07-12T08:50:50.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 -- 172.21.15.61:0/2592463921 learned_addr learned my addr 172.21.15.61:0/2592463921 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:50.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a16ffd700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180b0a90 0x7f1a18099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 -- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180b0a90 msgr2=0x7f1a18099240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180b0a90 0x7f1a18099240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 -- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a18146b40 msgr2=0x7f1a18099730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a17fff700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a18146b40 0x7f1a18099730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a18146b40 0x7f1a18099730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a177fe700 1 -- 172.21.15.61:0/2592463921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0800b180 con 0x7f1a18060490 2022-07-12T08:50:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a16ffd700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180b0a90 0x7f1a18099240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.444+0000 7f1a17fff700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a18146b40 0x7f1a18099730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a177fe700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a18145280 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f1a180b0f00 tx=0x7f1a08015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a0800d990 con 0x7f1a18060490 2022-07-12T08:50:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a180ae100 con 0x7f1a18060490 2022-07-12T08:50:50.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08011ce0 con 0x7f1a18060490 2022-07-12T08:50:50.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a0801e650 con 0x7f1a18060490 2022-07-12T08:50:50.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1809c1a0 con 0x7f1a18060490 2022-07-12T08:50:50.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a18057460 con 0x7f1a18060490 2022-07-12T08:50:50.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1a08025030 con 0x7f1a18060490 2022-07-12T08:50:50.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.448+0000 7f1a14ff9700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19f8053a40 0x7f19f8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.452+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1a0806d360 con 0x7f1a18060490 2022-07-12T08:50:50.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.452+0000 7f1a16ffd700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19f8053a40 0x7f19f8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.452+0000 7f1a16ffd700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19f8053a40 0x7f19f8055f00 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f1a0c008000 tx=0x7f1a0c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.456+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a080442c0 con 0x7f1a18060490 2022-07-12T08:50:50.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.580+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f1a180ade60 con 0x7f1a18060490 2022-07-12T08:50:50.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.580+0000 7f1a14ff9700 1 -- 172.21.15.61:0/2592463921 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v138) v1 ==== 110+0+10 (secure 0 0 0) 0x7f1a180ade60 con 0x7f1a18060490 2022-07-12T08:50:50.584 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:50:50.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19f8053a40 msgr2=0x7f19f8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19f8053a40 0x7f19f8055f00 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7f1a0c008000 tx=0x7f1a0c009040).stop 2022-07-12T08:50:50.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 msgr2=0x7f1a18145280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a18145280 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f1a180b0f00 tx=0x7f1a08015000).stop 2022-07-12T08:50:50.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 shutdown_connections 2022-07-12T08:50:50.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f19f8053a40 0x7f19f8055f00 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a18146b40 0x7f1a18099730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180b0a90 0x7f1a18099240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 --2- 172.21.15.61:0/2592463921 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a18060490 0x7f1a18145280 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 >> 172.21.15.61:0/2592463921 conn(0x7f1a180b6b80 msgr2=0x7f1a1805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 shutdown_connections 2022-07-12T08:50:50.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.584+0000 7f1a1ccda700 1 -- 172.21.15.61:0/2592463921 wait complete. 2022-07-12T08:50:50.601 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:50:50.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.708+0000 7f898bfff700 1 Processor -- start 2022-07-12T08:50:50.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.708+0000 7f898bfff700 1 -- start start 2022-07-12T08:50:50.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b6330 0x7f898c0b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f898c065990 0x7f898c0b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898c0b6710 con 0x7f898c0b1e00 2022-07-12T08:50:50.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898c0655a0 con 0x7f898c065990 2022-07-12T08:50:50.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c1431d0 con 0x7f898c0b6330 2022-07-12T08:50:50.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f898c0b1e00 0x7f898c13eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50130/0 (socket says 172.21.15.61:50130) 2022-07-12T08:50:50.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898b7fe700 1 -- 172.21.15.61:0/1889997298 learned_addr learned my addr 172.21.15.61:0/1889997298 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:50.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898affd700 1 --2- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b6330 0x7f898c0b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898a7fc700 1 --2- 172.21.15.61:0/1889997298 >> v2:172.21.15.111:3300/0 conn(0x7f898c065990 0x7f898c0b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f8989ffb700 1 -- 172.21.15.61:0/1889997298 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261070444 0 0) 0x7f898c0b6710 con 0x7f898c0b1e00 2022-07-12T08:50:50.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f8989ffb700 1 -- 172.21.15.61:0/1889997298 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8970003260 con 0x7f898c0b1e00 2022-07-12T08:50:50.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898a7fc700 1 -- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b6330 msgr2=0x7f898c0b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:50.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898a7fc700 1 --2- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b6330 0x7f898c0b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898a7fc700 1 -- 172.21.15.61:0/1889997298 >> v1:172.21.15.61:6789/0 conn(0x7f898c0b1e00 legacy=0x7f898c13eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898a7fc700 1 -- 172.21.15.61:0/1889997298 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c143470 con 0x7f898c065990 2022-07-12T08:50:50.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898a7fc700 1 --2- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b1910 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f8980002f70 tx=0x7f8980006df0).ready entity=mon.1 client_cookie=84d920ca028d2c34 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f8989ffb700 1 -- 172.21.15.61:0/1889997298 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f898000d4a0 con 0x7f898c065990 2022-07-12T08:50:50.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f8989ffb700 1 -- 172.21.15.61:0/1889997298 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8980000e90 con 0x7f898c065990 2022-07-12T08:50:50.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f8989ffb700 1 -- 172.21.15.61:0/1889997298 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f898000d7f0 con 0x7f898c065990 2022-07-12T08:50:50.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 -- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 msgr2=0x7f898c0b1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 --2- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b1910 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f8980002f70 tx=0x7f8980006df0).stop 2022-07-12T08:50:50.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 -- 172.21.15.61:0/1889997298 shutdown_connections 2022-07-12T08:50:50.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 --2- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b1910 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 --2- 172.21.15.61:0/1889997298 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b6330 0x7f898c0b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.712+0000 7f898bfff700 1 -- 172.21.15.61:0/1889997298 >> 172.21.15.61:0/1889997298 conn(0x7f898c063ad0 msgr2=0x7f898c063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 -- 172.21.15.61:0/1889997298 shutdown_connections 2022-07-12T08:50:50.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 -- 172.21.15.61:0/1889997298 wait complete. 2022-07-12T08:50:50.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 Processor -- start 2022-07-12T08:50:50.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 -- start start 2022-07-12T08:50:50.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b1e00 0x7f898c0ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 0x7f898c0aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0670f0 con 0x7f898c0b6330 2022-07-12T08:50:50.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c0afdb0 con 0x7f898c065990 2022-07-12T08:50:50.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898c13bb30 con 0x7f898c0b1e00 2022-07-12T08:50:50.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 0x7f898c0aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 0x7f898c0aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33686/0 (socket says 172.21.15.61:33686) 2022-07-12T08:50:50.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898b7fe700 1 -- 172.21.15.61:0/668908370 learned_addr learned my addr 172.21.15.61:0/668908370 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:50.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898a7fc700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b1e00 0x7f898c0ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.716+0000 7f898affd700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898b7fe700 1 -- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b1e00 msgr2=0x7f898c0ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898b7fe700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b1e00 0x7f898c0ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898b7fe700 1 -- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 msgr2=0x7f898c0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898b7fe700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898b7fe700 1 -- 172.21.15.61:0/668908370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898000b180 con 0x7f898c0b6330 2022-07-12T08:50:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898b7fe700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 0x7f898c0aec10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f8984001490 tx=0x7f8984007e70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8984005ba0 con 0x7f898c0b6330 2022-07-12T08:50:50.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8984005d00 con 0x7f898c0b6330 2022-07-12T08:50:50.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898c142a70 con 0x7f898c0b6330 2022-07-12T08:50:50.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89840072c0 con 0x7f898c0b6330 2022-07-12T08:50:50.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c1419f0 con 0x7f898c0b6330 2022-07-12T08:50:50.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f898401a070 con 0x7f898c0b6330 2022-07-12T08:50:50.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f898c0afc10 con 0x7f898c0b6330 2022-07-12T08:50:50.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f897bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8968053ad0 0x7f8968055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f898406b4e0 con 0x7f898c0b6330 2022-07-12T08:50:50.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.720+0000 7f898affd700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8968053ad0 0x7f8968055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.724+0000 7f898affd700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8968053ad0 0x7f8968055f90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f897c00b000 tx=0x7f897c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.724+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f898c0afc10 con 0x7f898c0b6330 2022-07-12T08:50:50.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f898c0afc10 con 0x7f898c0b6330 2022-07-12T08:50:50.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f897bfff700 1 -- 172.21.15.61:0/668908370 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v138) v1 ==== 92+0+10 (secure 0 0 0) 0x7f898c0afc10 con 0x7f898c0b6330 2022-07-12T08:50:50.856 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:50.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8968053ad0 msgr2=0x7f8968055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8968053ad0 0x7f8968055f90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f897c00b000 tx=0x7f897c006df0).stop 2022-07-12T08:50:50.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 msgr2=0x7f898c0aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 0x7f898c0aec10 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f8984001490 tx=0x7f8984007e70).stop 2022-07-12T08:50:50.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 shutdown_connections 2022-07-12T08:50:50.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f898c0b6330 0x7f898c0aec10 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f898c0b1e00 0x7f898c0ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8968053ad0 0x7f8968055f90 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 --2- 172.21.15.61:0/668908370 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f898c065990 0x7f898c0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 >> 172.21.15.61:0/668908370 conn(0x7f898c063ad0 msgr2=0x7f898c144b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.852+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 shutdown_connections 2022-07-12T08:50:50.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.856+0000 7f898bfff700 1 -- 172.21.15.61:0/668908370 wait complete. 2022-07-12T08:50:50.872 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:50:50.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 Processor -- start 2022-07-12T08:50:50.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 -- start start 2022-07-12T08:50:50.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ccc0b5480 0x7f5ccc0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0b3560 con 0x7f5ccc0b3730 2022-07-12T08:50:50.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b5b80 con 0x7f5ccc0b5480 2022-07-12T08:50:50.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b5e20 con 0x7f5ccc0b05f0 2022-07-12T08:50:50.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ccc0b5480 0x7f5ccc0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ccc0b5480 0x7f5ccc0b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33690/0 (socket says 172.21.15.61:33690) 2022-07-12T08:50:50.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 -- 172.21.15.61:0/3246181273 learned_addr learned my addr 172.21.15.61:0/3246181273 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:50.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd0d00700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 -- 172.21.15.61:0/3246181273 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 msgr2=0x7f5ccc142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 -- 172.21.15.61:0/3246181273 >> v1:172.21.15.111:6789/0 conn(0x7f5ccc0b3730 legacy=0x7f5ccc0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 -- 172.21.15.61:0/3246181273 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0b5860 con 0x7f5ccc0b5480 2022-07-12T08:50:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd0d00700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccbfff700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc0b45d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f5cbc002f70 tx=0x7f5cbc006eb0).ready entity=mon.0 client_cookie=af994bce341d14db server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3246181273 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cbc00d5c0 con 0x7f5ccc0b5480 2022-07-12T08:50:50.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3246181273 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc000e90 con 0x7f5ccc0b5480 2022-07-12T08:50:50.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3246181273 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cbc00d950 con 0x7f5ccc0b5480 2022-07-12T08:50:50.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 -- 172.21.15.61:0/3246181273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 msgr2=0x7f5ccc0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:50.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.976+0000 7f5cd2763700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc0b45d0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 rx=0x7f5cbc002f70 tx=0x7f5cbc006eb0).stop 2022-07-12T08:50:50.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- 172.21.15.61:0/3246181273 shutdown_connections 2022-07-12T08:50:50.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 --2- 172.21.15.61:0/3246181273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc0b45d0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:50.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- 172.21.15.61:0/3246181273 >> 172.21.15.61:0/3246181273 conn(0x7f5ccc0b96b0 msgr2=0x7f5ccc0b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:50.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- 172.21.15.61:0/3246181273 shutdown_connections 2022-07-12T08:50:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- 172.21.15.61:0/3246181273 wait complete. 2022-07-12T08:50:50.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 Processor -- start 2022-07-12T08:50:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- start start 2022-07-12T08:50:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ccc0b3730 0x7f5ccc0af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:50.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b02f0 con 0x7f5ccc0b5480 2022-07-12T08:50:50.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b0050 con 0x7f5ccc0b3730 2022-07-12T08:50:50.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc113030 con 0x7f5ccc0b05f0 2022-07-12T08:50:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd0d00700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:50.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc115270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:44124/0 (socket says 172.21.15.61:44124) 2022-07-12T08:50:50.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 -- 172.21.15.61:0/2957062594 learned_addr learned my addr 172.21.15.61:0/2957062594 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:50.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccb7fe700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ccc0b3730 0x7f5ccc0af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 -- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ccc0b3730 msgr2=0x7f5ccc0af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ccc0b3730 0x7f5ccc0af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 -- 172.21.15.61:0/2957062594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 msgr2=0x7f5ccc155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 -- 172.21.15.61:0/2957062594 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc00b1e0 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5ccbfff700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc115270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5cbc002f70 tx=0x7f5cbc015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cbc0117e0 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cbc011940 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc155d10 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cbc011ca0 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.980+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc112ac0 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.984+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc057460 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.984+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5cbc025030 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.984+0000 7f5cc97fa700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cac053af0 0x7f5cac055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.984+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5cbc06d0c0 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.984+0000 7f5ccb7fe700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cac053af0 0x7f5cac055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.988+0000 7f5ccb7fe700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cac053af0 0x7f5cac055fb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f5cc000d000 tx=0x7f5cc0007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:50.996+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5cbc044020 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.124+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ccc0b1270 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.124+0000 7f5cc97fa700 1 -- 172.21.15.61:0/2957062594 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v138) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5cbc0230a0 con 0x7f5ccc0b05f0 2022-07-12T08:50:51.128 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:51.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cac053af0 msgr2=0x7f5cac055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cac053af0 0x7f5cac055fb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f5cc000d000 tx=0x7f5cc0007b50).stop 2022-07-12T08:50:51.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 msgr2=0x7f5ccc115270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc115270 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5cbc002f70 tx=0x7f5cbc015000).stop 2022-07-12T08:50:51.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 shutdown_connections 2022-07-12T08:50:51.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0b5480 0x7f5ccc155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5cac053af0 0x7f5cac055fb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ccc0b3730 0x7f5ccc0af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 --2- 172.21.15.61:0/2957062594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ccc0b05f0 0x7f5ccc115270 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 >> 172.21.15.61:0/2957062594 conn(0x7f5ccc0b96b0 msgr2=0x7f5ccc05c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:51.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 shutdown_connections 2022-07-12T08:50:51.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.128+0000 7f5cd2763700 1 -- 172.21.15.61:0/2957062594 wait complete. 2022-07-12T08:50:51.144 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:50:51.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.248+0000 7fc642e22700 1 Processor -- start 2022-07-12T08:50:51.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.248+0000 7fc642e22700 1 -- start start 2022-07-12T08:50:51.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b5920 0x7fc63c14a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc63c0b0280 0x7fc63c14aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fc63c05faf0 0x7fc63c05ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c05c3e0 con 0x7fc63c0b0280 2022-07-12T08:50:51.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc63c0b5d00 con 0x7fc63c05faf0 2022-07-12T08:50:51.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c05c610 con 0x7fc63c0b5920 2022-07-12T08:50:51.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc63c0b0280 0x7fc63c14aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc63c0b0280 0x7fc63c14aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33708/0 (socket says 172.21.15.61:33708) 2022-07-12T08:50:51.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 -- 172.21.15.61:0/1121577170 learned_addr learned my addr 172.21.15.61:0/1121577170 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:51.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc641e20700 1 --2- 172.21.15.61:0/1121577170 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b5920 0x7fc63c14a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642621700 1 --2- 172.21.15.61:0/1121577170 >> v2:172.21.15.111:3300/0 conn(0x7fc63c05faf0 0x7fc63c05ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 -- 172.21.15.61:0/1121577170 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b5920 msgr2=0x7fc63c14a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:51.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 --2- 172.21.15.61:0/1121577170 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b5920 0x7fc63c14a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 -- 172.21.15.61:0/1121577170 >> v2:172.21.15.111:3300/0 conn(0x7fc63c05faf0 msgr2=0x7fc63c05ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 --2- 172.21.15.61:0/1121577170 >> v2:172.21.15.111:3300/0 conn(0x7fc63c05faf0 0x7fc63c05ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 -- 172.21.15.61:0/1121577170 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c05c8b0 con 0x7fc63c0b0280 2022-07-12T08:50:51.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc64161f700 1 --2- 172.21.15.61:0/1121577170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c0b0280 0x7fc63c14aed0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fc62c002f70 tx=0x7fc62c006eb0).ready entity=mon.0 client_cookie=77635a1bd9d622cd server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc640e1e700 1 -- 172.21.15.61:0/1121577170 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc62c00d570 con 0x7fc63c0b0280 2022-07-12T08:50:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc640e1e700 1 -- 172.21.15.61:0/1121577170 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62c000e90 con 0x7fc63c0b0280 2022-07-12T08:50:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc640e1e700 1 -- 172.21.15.61:0/1121577170 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc62c00d900 con 0x7fc63c0b0280 2022-07-12T08:50:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- 172.21.15.61:0/1121577170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c0b0280 msgr2=0x7fc63c14aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- 172.21.15.61:0/1121577170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c0b0280 0x7fc63c14aed0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 rx=0x7fc62c002f70 tx=0x7fc62c006eb0).stop 2022-07-12T08:50:51.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- 172.21.15.61:0/1121577170 shutdown_connections 2022-07-12T08:50:51.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- 172.21.15.61:0/1121577170 >> v2:172.21.15.111:3300/0 conn(0x7fc63c05faf0 0x7fc63c05ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- 172.21.15.61:0/1121577170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c0b0280 0x7fc63c14aed0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 --2- 172.21.15.61:0/1121577170 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b5920 0x7fc63c14a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- 172.21.15.61:0/1121577170 >> 172.21.15.61:0/1121577170 conn(0x7fc63c063340 msgr2=0x7fc63c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:51.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- 172.21.15.61:0/1121577170 shutdown_connections 2022-07-12T08:50:51.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.252+0000 7fc642e22700 1 -- 172.21.15.61:0/1121577170 wait complete. 2022-07-12T08:50:51.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 Processor -- start 2022-07-12T08:50:51.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 -- start start 2022-07-12T08:50:51.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 0x7fc63c098830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b0280 0x7fc63c0ad460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc63c0b5920 0x7fc63c0ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c14c230 con 0x7fc63c05faf0 2022-07-12T08:50:51.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c14bcf0 con 0x7fc63c0b5920 2022-07-12T08:50:51.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642e22700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c14bf90 con 0x7fc63c0b0280 2022-07-12T08:50:51.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 0x7fc63c098830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 0x7fc63c098830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33710/0 (socket says 172.21.15.61:33710) 2022-07-12T08:50:51.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 -- 172.21.15.61:0/2847496850 learned_addr learned my addr 172.21.15.61:0/2847496850 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:51.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc64161f700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b0280 0x7fc63c0ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642621700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc63c0b5920 0x7fc63c0ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 -- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b0280 msgr2=0x7fc63c0ad460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b0280 0x7fc63c0ad460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 -- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc63c0b5920 msgr2=0x7fc63c0ad950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc63c0b5920 0x7fc63c0ad950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 -- 172.21.15.61:0/2847496850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62c00b1e0 con 0x7fc63c05faf0 2022-07-12T08:50:51.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc642621700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc63c0b5920 0x7fc63c0ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:51.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.256+0000 7fc641e20700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 0x7fc63c098830 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7fc63800c000 tx=0x7fc638006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc638010780 con 0x7fc63c05faf0 2022-07-12T08:50:51.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6380108e0 con 0x7fc63c05faf0 2022-07-12T08:50:51.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63c0ade40 con 0x7fc63c05faf0 2022-07-12T08:50:51.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc638010c40 con 0x7fc63c05faf0 2022-07-12T08:50:51.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63c14b8c0 con 0x7fc63c05faf0 2022-07-12T08:50:51.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63c0adfd0 con 0x7fc63c05faf0 2022-07-12T08:50:51.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc63801c460 con 0x7fc63c05faf0 2022-07-12T08:50:51.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.260+0000 7fc632ffd700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc624053a40 0x7fc624055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.264+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc63806c2e0 con 0x7fc63c05faf0 2022-07-12T08:50:51.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.264+0000 7fc64161f700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc624053a40 0x7fc624055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.264+0000 7fc64161f700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc624053a40 0x7fc624055f00 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7fc62c01c000 tx=0x7fc62c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.272+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6380431e0 con 0x7fc63c05faf0 2022-07-12T08:50:51.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.400+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc63c0ad150 con 0x7fc63c05faf0 2022-07-12T08:50:51.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.400+0000 7fc632ffd700 1 -- 172.21.15.61:0/2847496850 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v138) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc638046800 con 0x7fc63c05faf0 2022-07-12T08:50:51.416 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:50:51.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc624053a40 msgr2=0x7fc624055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc624053a40 0x7fc624055f00 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7fc62c01c000 tx=0x7fc62c015000).stop 2022-07-12T08:50:51.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 msgr2=0x7fc63c098830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 0x7fc63c098830 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7fc63800c000 tx=0x7fc638006df0).stop 2022-07-12T08:50:51.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 shutdown_connections 2022-07-12T08:50:51.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc624053a40 0x7fc624055f00 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc63c0b5920 0x7fc63c0ad950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc63c0b0280 0x7fc63c0ad460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 --2- 172.21.15.61:0/2847496850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc63c05faf0 0x7fc63c098830 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 >> 172.21.15.61:0/2847496850 conn(0x7fc63c063340 msgr2=0x7fc63c098020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:51.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 shutdown_connections 2022-07-12T08:50:51.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.404+0000 7fc642e22700 1 -- 172.21.15.61:0/2847496850 wait complete. 2022-07-12T08:50:51.427 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:50:51.428 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:51.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.528+0000 7f7243f25700 1 Processor -- start 2022-07-12T08:50:51.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.528+0000 7f7243f25700 1 -- start start 2022-07-12T08:50:51.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f723c0b01f0 0x7f723c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c1443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723c0ad2f0 con 0x7f723c05c050 2022-07-12T08:50:51.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723c0b0a40 con 0x7f723c0b01f0 2022-07-12T08:50:51.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c05c430 con 0x7f723c14a8f0 2022-07-12T08:50:51.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7241cc1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f723c05c050 0x7f723c05cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50170/0 (socket says 172.21.15.61:50170) 2022-07-12T08:50:51.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7241cc1700 1 -- 172.21.15.61:0/1237515843 learned_addr learned my addr 172.21.15.61:0/1237515843 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:51.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72414c0700 1 --2- 172.21.15.61:0/1237515843 >> v2:172.21.15.111:3300/0 conn(0x7f723c0b01f0 0x7f723c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72424c2700 1 --2- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c1443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7240cbf700 1 -- 172.21.15.61:0/1237515843 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771304502 0 0) 0x7f723c0ad2f0 con 0x7f723c05c050 2022-07-12T08:50:51.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7240cbf700 1 -- 172.21.15.61:0/1237515843 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7224003260 con 0x7f723c05c050 2022-07-12T08:50:51.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72414c0700 1 -- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 msgr2=0x7f723c1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:51.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72414c0700 1 --2- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c1443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72414c0700 1 -- 172.21.15.61:0/1237515843 >> v1:172.21.15.61:6789/0 conn(0x7f723c05c050 legacy=0x7f723c05cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72414c0700 1 -- 172.21.15.61:0/1237515843 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c0b2420 con 0x7f723c0b01f0 2022-07-12T08:50:51.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f72414c0700 1 --2- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c0b01f0 0x7f723c0ac710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f7238002f70 tx=0x7f7238006df0).ready entity=mon.1 client_cookie=69f2ec33ef1897f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7240cbf700 1 -- 172.21.15.61:0/1237515843 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723800d4a0 con 0x7f723c0b01f0 2022-07-12T08:50:51.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7240cbf700 1 -- 172.21.15.61:0/1237515843 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7238000e90 con 0x7f723c0b01f0 2022-07-12T08:50:51.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7240cbf700 1 -- 172.21.15.61:0/1237515843 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723800d7f0 con 0x7f723c0b01f0 2022-07-12T08:50:51.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c0b01f0 msgr2=0x7f723c0ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 --2- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c0b01f0 0x7f723c0ac710 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f7238002f70 tx=0x7f7238006df0).stop 2022-07-12T08:50:51.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- 172.21.15.61:0/1237515843 shutdown_connections 2022-07-12T08:50:51.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 --2- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c1443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 --2- 172.21.15.61:0/1237515843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c0b01f0 0x7f723c0ac710 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- 172.21.15.61:0/1237515843 >> 172.21.15.61:0/1237515843 conn(0x7f723c062c00 msgr2=0x7f723c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:51.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- 172.21.15.61:0/1237515843 shutdown_connections 2022-07-12T08:50:51.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.532+0000 7f7243f25700 1 -- 172.21.15.61:0/1237515843 wait complete. 2022-07-12T08:50:51.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 Processor -- start 2022-07-12T08:50:51.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- start start 2022-07-12T08:50:51.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c05c050 0x7f723c14b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 0x7f723c144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c14b910 con 0x7f723c0b01f0 2022-07-12T08:50:51.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c14c140 con 0x7f723c05c050 2022-07-12T08:50:51.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c0a1670 con 0x7f723c14a8f0 2022-07-12T08:50:51.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 0x7f723c144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 0x7f723c144190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33724/0 (socket says 172.21.15.61:33724) 2022-07-12T08:50:51.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 -- 172.21.15.61:0/3207152045 learned_addr learned my addr 172.21.15.61:0/3207152045 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:51.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72424c2700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7241cc1700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c05c050 0x7f723c14b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 -- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 msgr2=0x7f723c144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 -- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c05c050 msgr2=0x7f723c14b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c05c050 0x7f723c14b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 -- 172.21.15.61:0/3207152045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723800b180 con 0x7f723c0b01f0 2022-07-12T08:50:51.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f72414c0700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 0x7f723c144190 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f7238002f70 tx=0x7f7238014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7238013ac0 con 0x7f723c0b01f0 2022-07-12T08:50:51.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7238013c20 con 0x7f723c0b01f0 2022-07-12T08:50:51.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723c14d620 con 0x7f723c0b01f0 2022-07-12T08:50:51.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723801e400 con 0x7f723c0b01f0 2022-07-12T08:50:51.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723c144b70 con 0x7f723c0b01f0 2022-07-12T08:50:51.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.536+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723c057460 con 0x7f723c0b01f0 2022-07-12T08:50:51.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.540+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7238025030 con 0x7f723c0b01f0 2022-07-12T08:50:51.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.540+0000 7f722affd700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f721c053a60 0x7f721c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:51.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.540+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f723806cc00 con 0x7f723c0b01f0 2022-07-12T08:50:51.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.540+0000 7f7241cc1700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f721c053a60 0x7f721c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:51.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.540+0000 7f7241cc1700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f721c053a60 0x7f721c055f20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f722c00b000 tx=0x7f722c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:51.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.544+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7238043ad0 con 0x7f723c0b01f0 2022-07-12T08:50:51.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.708+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f723c0a06f0 con 0x7f723c0b01f0 2022-07-12T08:50:51.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.708+0000 7f722affd700 1 -- 172.21.15.61:0/3207152045 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f7238023070 con 0x7f723c0b01f0 2022-07-12T08:50:51.713 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:51.713 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:42.959468+0000","modified":"2022-07-12T08:50:43.963135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/676021650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":676021650},{"type":"v1","addr":"172.21.15.61:6833","nonce":676021650}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-12T08:50:51.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f721c053a60 msgr2=0x7f721c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f721c053a60 0x7f721c055f20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f722c00b000 tx=0x7f722c009040).stop 2022-07-12T08:50:51.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 msgr2=0x7f723c144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:51.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 0x7f723c144190 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 rx=0x7f7238002f70 tx=0x7f7238014000).stop 2022-07-12T08:50:51.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 shutdown_connections 2022-07-12T08:50:51.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f723c14a8f0 0x7f723c144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f723c0b01f0 0x7f723c144190 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f721c053a60 0x7f721c055f20 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 --2- 172.21.15.61:0/3207152045 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f723c05c050 0x7f723c14b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:51.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 >> 172.21.15.61:0/3207152045 conn(0x7f723c062c00 msgr2=0x7f723c0b3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:51.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 shutdown_connections 2022-07-12T08:50:51.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:51.712+0000 7f7243f25700 1 -- 172.21.15.61:0/3207152045 wait complete. 2022-07-12T08:50:51.724 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-12T08:50:51.928 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:50:42.959468+0000', 'modified': '2022-07-12T08:50:43.963135+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8442': {'gid': 8442, 'name': 'a', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/676021650', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 676021650}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 676021650}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:50:51.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:50:51.930 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:50:52.054 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8442,"want_state":"up:active","state":"up:active","rank_uptime":9.0781151710000003,"mdsmap_epoch":77,"osdmap_epoch":138,"osdmap_epoch_barrier":137,"uptime":14.026354433} 2022-07-12T08:50:52.055 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8442, "mdsmap_epoch": 77, "osdmap_epoch": 138, "osdmap_epoch_barrier": 137, "rank_uptime": 9.078115171, "state": "up:active", "uptime": 14.026354433, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:50:52.055 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:50:52.056 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:50:52.056 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:50:52.057 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:52.057 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:52.073 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:52.134 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:52.135 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:52.151 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:52.153 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:52.154 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:52.160 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:50:52.161 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:52.206 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:52.224 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:52.225 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:52.237 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:52.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:52.239 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:52.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:50:52.276+0000 7f22370b9080 -1 init, newargv = 0x55b29e2517d0 newargc=9ceph-fuse[21746]: starting ceph client 2022-07-12T08:50:52.280 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr: 2022-07-12T08:50:52.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21746]: starting fuse 2022-07-12T08:50:53.387 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:50:53.411 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:50:53.412 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:50:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:50:53.472 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:50:53.473 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.474 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:50:53.483 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:50:53.483 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:50:53.484 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.484 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:50:53.485 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:50:53.485 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:50:53.486 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:50:53.486 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:50:53.487 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:53.487 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:50:53.488 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:50:53.488 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:50:53.489 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:53.489 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:50:53.489 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:50:53.490 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:50:53.490 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:50:53.491 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:53.491 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:50:53.492 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:50:53.493 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:50:53.493 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:50:53.494 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:50:53.494 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:50:53.494 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:50:53.495 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:50:53.572 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21746.asok 2022-07-12T08:50:53.578 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21746.asok 2022-07-12T08:50:53.579 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.579 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21746.asok status 2022-07-12T08:50:53.672 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:50:53.672 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:50:53.673 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:50:53.673 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:50:53.673 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:50:53.673 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:50:53.674 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:50:53.674 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21746", 2022-07-12T08:50:53.674 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:50:53.674 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:53.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:50:53.675 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:50:53.675 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 8461, 2022-07-12T08:50:53.675 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:50:53.676 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:50:53.676 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:50:53.676 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 8461 2022-07-12T08:50:53.676 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:53.676 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:50:53.677 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:50:53.677 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:50:53.677 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 14823537 2022-07-12T08:50:53.677 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:50:53.678 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:53.678 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:50:53.678 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:50:53.678 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:50:53.678 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 14823537 2022-07-12T08:50:53.679 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:50:53.679 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.8461 172.21.15.130:0/14823537", 2022-07-12T08:50:53.679 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/14823537", 2022-07-12T08:50:53.679 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:50:53.680 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 77, 2022-07-12T08:50:53.680 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 138, 2022-07-12T08:50:53.680 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:50:53.680 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:50:53.681 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:50:53.681 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:50:53.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.683 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:53.683 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:53.696 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:53.697 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:53.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.699 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:53.726 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-12T08:50:53.727 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.rctime . 2022-07-12T08:50:53.777 INFO:teuthology.orchestra.run.smithi121.stdout:1657615853.722485285 2022-07-12T08:50:53.778 INFO:tasks.cephfs.test_misc:rctime = 1657615853.722485285 2022-07-12T08:50:53.778 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:50:53.779 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:53.801 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:53.829 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:53.855 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:53.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:50:53.902 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:50:53.903 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.904 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:53.904 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:53.916 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:50:53.917 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:50:53.918 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2022-07-12T08:50:53.919 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:50:53.920 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:53.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22256]: fuse finished with error 0 and tester_r 0 2022-07-12T08:50:53.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:53.978 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:53.978 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:54.000 INFO:teuthology.orchestra.run.smithi121.stdout:ext2/ext3 2022-07-12T08:50:54.001 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:50:54.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:54.119 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:50:54.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:54.129 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:54.130 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:54.148 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:50:54.149 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:50:54.150 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:50:54.150 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:50:54.151 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:54.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21746]: fuse finished with error 0 and tester_r 0 2022-07-12T08:50:54.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:54.220 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:54.221 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:54.232 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:50:54.233 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:50:54.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:54.279 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:50:54.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:50:54.290 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:50:54.291 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:50:54.299 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:50:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:54.305 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:50:54.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:50:54.307 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:50:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:50:54.365 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:50:54.374 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:50:54.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.484+0000 7fa5652d1700 1 Processor -- start 2022-07-12T08:50:54.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.484+0000 7fa5652d1700 1 -- start start 2022-07-12T08:50:54.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa560146c50 0x7fa56005faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:54.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5600ab3f0 con 0x7fa5600b5780 2022-07-12T08:50:54.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5600ab950 con 0x7fa5600abdf0 2022-07-12T08:50:54.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5600ab690 con 0x7fa560146c50 2022-07-12T08:50:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa557fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa560146c50 0x7fa56005faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa55ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa5600b5780 0x7fa560146540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50182/0 (socket says 172.21.15.61:50182) 2022-07-12T08:50:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa55ffff700 1 -- 172.21.15.61:0/1872782353 learned_addr learned my addr 172.21.15.61:0/1872782353 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:54.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa557fff700 1 -- 172.21.15.61:0/1872782353 >> v1:172.21.15.111:6789/0 conn(0x7fa5600abdf0 legacy=0x7fa56005ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa557fff700 1 -- 172.21.15.61:0/1872782353 >> v1:172.21.15.61:6789/0 conn(0x7fa5600b5780 legacy=0x7fa560146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:54.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa55f7fe700 1 -- 172.21.15.61:0/1872782353 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088948795 0 0) 0x7fa5600ab3f0 con 0x7fa5600b5780 2022-07-12T08:50:54.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa557fff700 1 -- 172.21.15.61:0/1872782353 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5600b5b60 con 0x7fa560146c50 2022-07-12T08:50:54.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa557fff700 1 --2- 172.21.15.61:0/1872782353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa560146c50 0x7fa56005faf0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7fa5600aa5b0 tx=0x7fa548006df0).ready entity=mon.0 client_cookie=6295d50f2b5b463 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa55f7fe700 1 -- 172.21.15.61:0/1872782353 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54800d610 con 0x7fa560146c50 2022-07-12T08:50:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa55f7fe700 1 -- 172.21.15.61:0/1872782353 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa548000e90 con 0x7fa560146c50 2022-07-12T08:50:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa55f7fe700 1 -- 172.21.15.61:0/1872782353 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54800d960 con 0x7fa560146c50 2022-07-12T08:50:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- 172.21.15.61:0/1872782353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa560146c50 msgr2=0x7fa56005faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 --2- 172.21.15.61:0/1872782353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa560146c50 0x7fa56005faf0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7fa5600aa5b0 tx=0x7fa548006df0).stop 2022-07-12T08:50:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- 172.21.15.61:0/1872782353 shutdown_connections 2022-07-12T08:50:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 --2- 172.21.15.61:0/1872782353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa560146c50 0x7fa56005faf0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:54.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- 172.21.15.61:0/1872782353 >> 172.21.15.61:0/1872782353 conn(0x7fa560063380 msgr2=0x7fa5600b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:54.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- 172.21.15.61:0/1872782353 shutdown_connections 2022-07-12T08:50:54.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- 172.21.15.61:0/1872782353 wait complete. 2022-07-12T08:50:54.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 Processor -- start 2022-07-12T08:50:54.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.488+0000 7fa5652d1700 1 -- start start 2022-07-12T08:50:54.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5600abdf0 0x7fa560159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:54.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 0x7fa56015a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:54.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa560146c50 0x7fa560066ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:54.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa560146700 con 0x7fa5600b5780 2022-07-12T08:50:54.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56014ada0 con 0x7fa560146c50 2022-07-12T08:50:54.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56014a860 con 0x7fa5600abdf0 2022-07-12T08:50:54.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 0x7fa56015a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:54.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 0x7fa56015a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33740/0 (socket says 172.21.15.61:33740) 2022-07-12T08:50:54.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 -- 172.21.15.61:0/4188667292 learned_addr learned my addr 172.21.15.61:0/4188667292 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:54.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa564ad0700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa560146c50 0x7fa560066ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:54.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa55ffff700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5600abdf0 0x7fa560159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:54.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 -- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5600abdf0 msgr2=0x7fa560159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:54.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5600abdf0 0x7fa560159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:54.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 -- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa560146c50 msgr2=0x7fa560066ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:54.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa560146c50 0x7fa560066ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:54.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 -- 172.21.15.61:0/4188667292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54800b180 con 0x7fa5600b5780 2022-07-12T08:50:54.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa564ad0700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa560146c50 0x7fa560066ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:54.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa557fff700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 0x7fa56015a020 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7fa5600aa5b0 tx=0x7fa548015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:54.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa54800de90 con 0x7fa5600b5780 2022-07-12T08:50:54.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54800bec0 con 0x7fa5600b5780 2022-07-12T08:50:54.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56014ab00 con 0x7fa5600b5780 2022-07-12T08:50:54.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa548011e90 con 0x7fa5600b5780 2022-07-12T08:50:54.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.492+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5600af050 con 0x7fa5600b5780 2022-07-12T08:50:54.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.496+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa560057460 con 0x7fa5600b5780 2022-07-12T08:50:54.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.496+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa548025030 con 0x7fa5600b5780 2022-07-12T08:50:54.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.496+0000 7fa55dffb700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa544053af0 0x7fa544055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:54.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.496+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa54806d170 con 0x7fa5600b5780 2022-07-12T08:50:54.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.496+0000 7fa55ffff700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa544053af0 0x7fa544055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:54.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.496+0000 7fa55ffff700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa544053af0 0x7fa544055fb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fa55000b000 tx=0x7fa550009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:54.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.500+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa5480440d0 con 0x7fa5600b5780 2022-07-12T08:50:54.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.628+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa56005be90 con 0x7fa5600b5780 2022-07-12T08:50:54.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa55dffb700 1 -- 172.21.15.61:0/4188667292 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v138) v1 ==== 81+0+33707 (secure 0 0 0) 0x7fa548023090 con 0x7fa5600b5780 2022-07-12T08:50:54.637 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:54.650 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:50:54.651 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 138, 2022-07-12T08:50:54.651 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:50:54.652 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:50:54.652 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:50:43.963239+0000", 2022-07-12T08:50:54.653 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:54.653 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:50:54.655 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:50:54.655 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:50:54.655 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:50:54.656 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:50:54.656 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:50:54.657 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:50:54.657 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:50:54.662 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:54.663 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:50:54.663 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:50:54.664 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:50:54.664 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:50:54.665 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:50:54.665 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 28, 2022-07-12T08:50:54.665 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:50:54.667 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:50:54.668 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:50:54.668 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:50:54.668 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:50:54.669 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.669 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:50:54.670 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:50:54.670 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:50:54.670 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:54.671 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:54.671 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:54.672 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:54.672 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:54.673 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:54.673 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:54.674 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:54.674 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:50:54.675 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:50:54.675 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:50:54.675 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:50:54.676 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:50:54.676 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:54.677 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:54.677 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:54.678 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:54.678 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:54.678 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:54.679 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:54.679 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.680 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:50:54.680 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:54.680 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:54.681 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:54.681 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:54.682 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:54.683 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:54.683 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:54.684 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:54.684 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:54.685 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:54.685 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:54.686 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:54.686 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:54.687 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:54.687 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:54.688 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:54.688 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:54.689 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:54.689 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:54.690 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:54.690 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:54.690 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:54.691 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:54.691 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:54.692 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:54.692 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:54.692 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.693 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:54.693 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:54.693 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:54.694 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:54.694 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:54.695 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:54.695 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:54.696 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:54.696 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:54.696 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:54.697 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:54.697 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:50:54.698 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:50:54.698 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.700 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:54.700 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:50:54.701 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.702 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.702 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:50:54.703 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:50:54.703 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:50:54.704 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:54.704 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:54.705 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:54.705 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:54.705 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:54.706 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:54.706 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:54.707 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:54.707 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:54.708 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:54.708 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:54.708 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:54.709 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:54.709 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:54.710 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:54.711 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:54.711 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:54.712 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:54.712 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:54.713 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:54.713 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.713 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:50:54.714 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:54.714 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:54.715 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:54.715 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:54.715 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:54.716 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:54.716 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:54.717 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:54.717 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:54.718 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:54.718 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:54.719 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:54.719 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:54.720 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:54.720 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:54.720 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:54.721 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:54.721 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:54.731 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:54.731 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:54.732 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:54.732 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:54.733 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:54.733 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:54.734 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:54.734 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:54.735 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.735 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:54.736 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:54.736 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:54.737 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:54.737 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:54.738 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:54.738 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:54.739 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:54.739 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:54.740 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:54.740 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:54.741 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:50:54.741 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:54.741 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:50:54.742 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.743 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.743 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.743 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 27, 2022-07-12T08:50:54.745 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:50:54.745 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:50:41.024516+0000", 2022-07-12T08:50:54.745 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:54.746 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:54.746 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:54.747 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:54.747 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:54.748 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:54.748 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:54.749 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:54.749 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:54.749 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:54.750 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:54.750 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:54.751 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:54.751 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:54.752 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:54.752 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:54.753 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:54.753 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:54.754 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:54.754 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:54.755 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.755 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "137", 2022-07-12T08:50:54.756 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:54.756 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:54.757 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:54.757 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:54.767 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:54.767 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:54.769 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:54.774 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:54.776 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:54.777 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:54.777 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:54.778 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:54.779 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:54.781 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:54.781 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:54.782 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:54.782 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:54.783 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:54.783 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:54.783 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:54.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:54.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:54.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:54.785 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:54.785 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:54.786 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:54.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.786 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:54.787 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:54.787 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:54.787 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:54.787 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:54.788 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:54.788 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:54.788 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:54.789 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:54.789 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:54.790 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:54.791 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:50:54.791 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:50:54.791 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:50:54.792 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:50:54.792 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.792 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:54.793 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:50:54.794 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:50:54.802 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.803 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.803 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.804 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.804 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 28, 2022-07-12T08:50:54.805 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:50:54.805 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:50:41.962174+0000", 2022-07-12T08:50:54.806 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:50:54.806 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:50:54.807 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:50:54.807 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:50:54.808 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:50:54.808 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:50:54.809 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:50:54.809 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:50:54.809 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:50:54.810 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:50:54.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:50:54.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:50:54.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:50:54.813 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:50:54.813 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:50:54.814 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:50:54.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:50:54.815 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:50:54.815 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:50:54.815 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:50:54.815 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "138", 2022-07-12T08:50:54.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:50:54.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:50:54.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:50:54.817 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:50:54.817 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:50:54.817 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:50:54.817 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:50:54.818 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:50:54.818 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:50:54.818 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:50:54.818 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:50:54.819 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:50:54.819 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:50:54.819 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:50:54.819 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:50:54.820 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:50:54.820 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:50:54.820 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:50:54.820 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:50:54.821 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:50:54.821 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:50:54.821 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:50:54.821 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:50:54.821 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:50:54.822 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:50:54.822 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:50:54.822 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.822 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:50:54.823 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:50:54.823 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:50:54.825 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:50:54.825 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:50:54.825 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:50:54.826 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:50:54.826 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:50:54.826 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:50:54.827 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:50:54.827 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:50:54.827 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:50:54.827 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:50:54.828 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:50:54.828 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:50:54.828 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.828 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.829 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.829 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:54.829 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:50:54.829 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.830 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:54.830 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:50:54.830 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:54.830 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:54.831 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:54.831 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:54.831 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:54.832 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:54.832 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:54.832 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:50:54.832 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:54.833 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:54.833 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:54.833 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.834 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.834 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.835 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:50:54.838 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.838 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.839 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.839 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.839 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:50:54.840 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.840 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.840 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.841 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.841 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:54.841 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.842 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.842 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.843 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:50:54.843 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.843 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.844 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.844 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.845 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:50:54.845 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.845 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.846 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.846 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.847 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:54.847 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.848 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:50:54.849 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.850 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.850 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.851 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:50:54.851 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.852 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.852 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.852 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.853 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:54.853 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.854 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.855 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.855 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:50:54.856 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.856 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.857 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.857 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.857 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:50:54.858 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:50:54.858 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.859 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.859 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.859 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:50:54.860 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:50:54.860 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:50:54.860 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:50:54.861 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:54.861 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:54.862 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:54.862 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.862 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.863 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.863 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:54.864 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:50:54.864 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:54.864 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:54.865 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:54.865 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:54.866 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:54.866 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:54.867 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:54.867 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:50:54.867 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:54.868 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:54.868 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:54.869 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.869 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.869 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.870 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:50:54.871 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.871 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.872 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.872 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.872 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:50:54.873 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.873 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.874 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.874 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.875 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:54.875 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.875 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.876 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.876 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:50:54.876 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.877 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.878 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:50:54.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.879 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.879 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.880 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:54.880 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.880 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.880 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:50:54.881 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.881 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.881 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.882 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.882 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:50:54.882 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.883 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.883 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.883 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.883 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:54.883 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.884 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.884 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:50:54.884 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.885 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.885 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.885 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.885 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:50:54.885 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:50:54.886 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.886 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.887 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.887 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:50:54.887 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:50:54.887 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:50:54.888 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:50:54.888 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:54.888 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:54.888 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:54.889 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.889 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.889 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.889 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:54.890 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:50:54.890 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:54.890 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:54.890 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:54.891 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:54.891 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:54.891 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:54.892 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:54.892 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 135, 2022-07-12T08:50:54.892 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:54.893 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:54.893 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:54.894 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.894 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.894 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.895 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:50:54.895 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.895 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.896 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.897 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:50:54.897 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.898 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.898 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.899 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.899 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:54.900 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.900 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.901 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:50:54.902 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.903 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.903 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.904 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.904 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:50:54.905 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.905 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.905 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.907 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.907 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:54.908 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.908 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.909 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:50:54.909 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.910 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.910 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.911 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.911 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:50:54.912 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.912 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.913 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.913 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.914 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:54.914 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.915 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.915 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.916 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:50:54.916 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.917 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:50:54.918 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:50:54.919 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.919 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.920 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.920 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:50:54.921 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:50:54.921 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:50:54.922 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:50:54.922 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:54.923 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:54.923 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:54.923 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.924 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.924 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.925 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:54.925 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:50:54.926 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:54.926 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:54.926 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:54.927 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:54.927 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:54.927 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:54.928 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:54.928 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 125, 2022-07-12T08:50:54.929 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:54.929 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:54.929 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:54.931 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.931 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.931 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.931 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:50:54.931 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.932 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.932 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.932 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.932 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:50:54.933 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.933 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.933 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.934 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.934 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:54.935 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.935 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.936 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.936 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:50:54.936 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.937 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.937 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.937 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.938 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:50:54.938 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.939 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.940 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.940 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.940 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:54.941 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.941 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.942 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.942 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:50:54.942 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.943 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.943 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.944 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.944 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:50:54.944 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.945 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.945 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.946 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.946 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:54.947 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.947 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.948 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:50:54.949 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.949 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.949 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.950 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.950 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:50:54.951 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:50:54.951 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.951 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.952 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.952 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:50:54.953 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:50:54.953 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:50:54.953 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:50:54.954 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:54.954 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:54.955 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:54.955 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.956 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.956 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.957 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:54.957 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:50:54.958 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:54.958 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:54.959 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:54.960 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:54.960 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:54.961 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:54.961 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:54.962 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 135, 2022-07-12T08:50:54.962 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:54.962 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:54.963 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:54.963 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.964 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.964 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.965 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:50:54.965 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.965 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.966 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.966 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.967 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:50:54.967 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.967 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.968 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:54.969 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.969 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.970 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.970 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:50:54.971 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.971 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.972 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:50:54.973 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.973 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.973 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.974 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.974 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:54.974 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.975 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.976 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:50:54.976 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.976 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.977 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.977 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.978 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:50:54.978 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.979 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.979 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.979 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.980 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:54.980 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.981 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.981 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.982 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:50:54.982 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.982 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.983 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.983 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.984 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:50:54.984 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:50:54.984 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.985 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.985 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.986 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:50:54.986 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:50:54.986 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:50:54.987 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:50:54.987 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:54.988 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:54.988 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:54.988 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.989 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.989 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.989 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:54.990 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:50:54.990 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:54.990 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:54.991 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:54.991 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:54.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:54.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:54.992 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:54.992 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:50:54.992 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:54.992 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:54.992 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:54.993 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:54.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.994 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:54.995 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:50:54.995 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:54.996 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:54.996 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:54.996 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:54.997 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:50:54.998 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:54.999 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:54.999 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:54.999 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:55.000 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.001 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.001 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.001 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:50:55.002 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:55.002 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.002 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.003 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:50:55.004 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:55.005 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.005 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.006 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.006 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:55.007 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.007 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.008 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.009 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:50:55.009 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:55.010 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.010 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.011 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.011 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:50:55.012 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:55.012 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.013 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.013 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.014 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:55.014 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.015 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.015 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.015 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:50:55.016 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:55.016 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.017 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.017 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:50:55.018 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:50:55.019 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.019 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.019 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.020 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:50:55.020 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:50:55.021 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:50:55.021 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:50:55.022 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:55.022 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:55.022 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:55.023 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.024 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.024 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:55.024 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:50:55.025 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:55.025 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:55.026 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:55.026 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:55.026 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:55.027 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:55.027 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:55.028 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:50:55.028 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:55.028 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:55.029 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:55.029 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.030 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.030 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:50:55.031 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.032 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.032 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:50:55.032 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.033 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.033 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.033 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.033 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:55.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:50:55.034 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.035 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.035 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:50:55.036 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.036 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.036 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.036 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.037 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:55.037 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.038 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:50:55.039 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.039 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:50:55.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.041 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.041 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.042 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.042 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:55.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:50:55.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:50:55.046 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:50:55.046 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.047 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.047 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.047 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:50:55.048 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:50:55.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:50:55.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:50:55.049 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:55.049 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:55.050 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:55.050 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.051 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.051 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:55.052 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:50:55.052 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:50:55.053 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:50:55.053 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:50:55.053 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:50:55.054 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:50:55.054 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:50:55.055 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:50:55.055 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:50:55.056 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:50:55.056 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:50:55.056 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:50:55.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:50:55.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:50:55.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.061 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.062 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:50:55.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.062 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:50:55.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.064 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:50:55.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.067 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.068 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.069 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:50:55.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.070 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.070 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.070 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:50:55.071 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.071 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.072 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:50:55.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.073 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.074 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.074 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:50:55.075 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:50:55.075 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.076 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:50:55.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:50:55.076 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.077 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.078 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:50:55.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:50:55.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:50:55.079 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.079 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.080 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.080 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:50:55.081 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:50:55.081 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:50:55.081 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:50:55.081 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:50:55.082 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:50:55.082 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:50:55.082 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:50:55.082 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.083 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:55.083 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:50:55.084 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.084 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:50:55.085 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.085 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.085 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.085 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.086 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.086 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:50:55.086 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.087 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:50:55.087 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.088 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.088 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.089 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.090 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.090 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:50:55.090 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.091 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.092 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:50:55.092 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.093 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.093 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.094 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.094 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.094 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:50:55.095 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.095 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.096 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:50:55.097 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.097 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.097 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.098 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.099 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.099 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:50:55.100 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.100 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.101 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:50:55.101 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.102 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.103 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.103 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.104 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.104 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:50:55.105 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.106 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:50:55.107 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.107 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.107 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.114 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.115 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:50:55.116 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.116 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.118 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:50:55.118 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.119 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.119 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.119 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.120 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.120 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:50:55.121 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:50:55.131 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:50:55.131 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:50:55.131 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:50:55.132 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:50:55.132 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:50:55.133 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:50:55.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:50:55.134 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:50:55.134 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.135 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:50:55.135 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:50:55.135 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:50:55.136 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:50:55.136 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:50:55.137 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:50:55.137 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:50:55.138 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:50:55.139 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:50:55.140 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:50:55.140 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:50:55.140 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:50:55.141 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:50:55.141 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:50:55.142 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:50:55.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:50:55.143 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:50:55.143 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:50:55.143 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:50:55.144 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:50:55.144 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:50:55.145 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:50:55.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa544053af0 msgr2=0x7fa544055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa544053af0 0x7fa544055fb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7fa55000b000 tx=0x7fa550009040).stop 2022-07-12T08:50:55.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 msgr2=0x7fa56015a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 0x7fa56015a020 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 rx=0x7fa5600aa5b0 tx=0x7fa548015000).stop 2022-07-12T08:50:55.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 shutdown_connections 2022-07-12T08:50:55.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa544053af0 0x7fa544055fb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa560146c50 0x7fa560066ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa5600b5780 0x7fa56015a020 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 --2- 172.21.15.61:0/4188667292 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa5600abdf0 0x7fa560159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.632+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 >> 172.21.15.61:0/4188667292 conn(0x7fa560063380 msgr2=0x7fa5600b4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:55.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.636+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 shutdown_connections 2022-07-12T08:50:55.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:54.636+0000 7fa5652d1700 1 -- 172.21.15.61:0/4188667292 wait complete. 2022-07-12T08:50:55.165 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:55.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 Processor -- start 2022-07-12T08:50:55.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 -- start start 2022-07-12T08:50:55.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f747c0ab670 0x7f747c05f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c0abc70 con 0x7f747c0ab290 2022-07-12T08:50:55.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c05c720 con 0x7f747c0afa60 2022-07-12T08:50:55.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c0aba90 con 0x7f747c0ab670 2022-07-12T08:50:55.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f747c0ab670 0x7f747c05f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f747c0ab670 0x7f747c05f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33748/0 (socket says 172.21.15.61:33748) 2022-07-12T08:50:55.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 -- 172.21.15.61:0/587387951 learned_addr learned my addr 172.21.15.61:0/587387951 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:55.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 -- 172.21.15.61:0/587387951 >> v1:172.21.15.111:6789/0 conn(0x7f747c0afa60 legacy=0x7f747c147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:55.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 -- 172.21.15.61:0/587387951 >> v1:172.21.15.61:6789/0 conn(0x7f747c0ab290 legacy=0x7f747c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 -- 172.21.15.61:0/587387951 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c05c9c0 con 0x7f747c0ab670 2022-07-12T08:50:55.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747bfff700 1 --2- 172.21.15.61:0/587387951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab670 0x7f747c05f140 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7f747c0af510 tx=0x7f746c009290).ready entity=mon.0 client_cookie=4620b27680c78849 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:55.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747b7fe700 1 -- 172.21.15.61:0/587387951 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f746c00b860 con 0x7f747c0ab670 2022-07-12T08:50:55.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747b7fe700 1 -- 172.21.15.61:0/587387951 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c00b9c0 con 0x7f747c0ab670 2022-07-12T08:50:55.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f747b7fe700 1 -- 172.21.15.61:0/587387951 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f746c00bb30 con 0x7f747c0ab670 2022-07-12T08:50:55.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 -- 172.21.15.61:0/587387951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab670 msgr2=0x7f747c05f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.284+0000 7f7481d9a700 1 --2- 172.21.15.61:0/587387951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab670 0x7f747c05f140 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7f747c0af510 tx=0x7f746c009290).stop 2022-07-12T08:50:55.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- 172.21.15.61:0/587387951 shutdown_connections 2022-07-12T08:50:55.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 --2- 172.21.15.61:0/587387951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab670 0x7f747c05f140 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- 172.21.15.61:0/587387951 >> 172.21.15.61:0/587387951 conn(0x7f747c0b1950 msgr2=0x7f747c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:55.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- 172.21.15.61:0/587387951 shutdown_connections 2022-07-12T08:50:55.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- 172.21.15.61:0/587387951 wait complete. 2022-07-12T08:50:55.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 Processor -- start 2022-07-12T08:50:55.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- start start 2022-07-12T08:50:55.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 0x7f747c063860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f747c0ab670 0x7f747c063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c0afa60 0x7f747c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0aeda0 con 0x7f747c0ab290 2022-07-12T08:50:55.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c156e40 con 0x7f747c0afa60 2022-07-12T08:50:55.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.288+0000 7f7481d9a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0aeb00 con 0x7f747c0ab670 2022-07-12T08:50:55.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 0x7f747c063860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 0x7f747c063860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33752/0 (socket says 172.21.15.61:33752) 2022-07-12T08:50:55.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 -- 172.21.15.61:0/3606909534 learned_addr learned my addr 172.21.15.61:0/3606909534 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:55.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7481599700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c0afa60 0x7f747c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f747bfff700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f747c0ab670 0x7f747c063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 -- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f747c0ab670 msgr2=0x7f747c063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f747c0ab670 0x7f747c063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 -- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c0afa60 msgr2=0x7f747c159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c0afa60 0x7f747c159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 -- 172.21.15.61:0/3606909534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c009530 con 0x7f747c0ab290 2022-07-12T08:50:55.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7480d98700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 0x7f747c063860 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f747c0615d0 tx=0x7f7464007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:55.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f746400ce30 con 0x7f747c0ab290 2022-07-12T08:50:55.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7464007240 con 0x7f747c0ab290 2022-07-12T08:50:55.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7464006700 con 0x7f747c0ab290 2022-07-12T08:50:55.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c0af110 con 0x7f747c0ab290 2022-07-12T08:50:55.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c159c80 con 0x7f747c0ab290 2022-07-12T08:50:55.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c057460 con 0x7f747c0ab290 2022-07-12T08:50:55.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7464010030 con 0x7f747c0ab290 2022-07-12T08:50:55.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7479ffb700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7454053a60 0x7f7454055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.292+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f746406bae0 con 0x7f747c0ab290 2022-07-12T08:50:55.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.296+0000 7f747bfff700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7454053a60 0x7f7454055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.296+0000 7f747bfff700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7454053a60 0x7f7454055f20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f746c019000 tx=0x7f746c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:55.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.300+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7464042a40 con 0x7f747c0ab290 2022-07-12T08:50:55.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.464+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f747c05f670 con 0x7f747c0ab290 2022-07-12T08:50:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.464+0000 7f7479ffb700 1 -- 172.21.15.61:0/3606909534 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f7464046060 con 0x7f747c0ab290 2022-07-12T08:50:55.470 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:55.470 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:42.959468+0000","modified":"2022-07-12T08:50:43.963135+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/676021650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":676021650},{"type":"v1","addr":"172.21.15.61:6833","nonce":676021650}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-12T08:50:55.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7454053a60 msgr2=0x7f7454055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7454053a60 0x7f7454055f20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f746c019000 tx=0x7f746c016000).stop 2022-07-12T08:50:55.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 msgr2=0x7f747c063860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 0x7f747c063860 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 rx=0x7f747c0615d0 tx=0x7f7464007c10).stop 2022-07-12T08:50:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 shutdown_connections 2022-07-12T08:50:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7454053a60 0x7f7454055f20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f747c0afa60 0x7f747c159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f747c0ab670 0x7f747c063d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 --2- 172.21.15.61:0/3606909534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f747c0ab290 0x7f747c063860 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 >> 172.21.15.61:0/3606909534 conn(0x7f747c0b1950 msgr2=0x7f747c0b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 shutdown_connections 2022-07-12T08:50:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.468+0000 7f7481d9a700 1 -- 172.21.15.61:0/3606909534 wait complete. 2022-07-12T08:50:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-12T08:50:55.490 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:50:55.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.596+0000 7f7b1a079700 1 Processor -- start 2022-07-12T08:50:55.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.596+0000 7f7b1a079700 1 -- start start 2022-07-12T08:50:55.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b140b2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7b14064490 0x7f7b1405f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b14142d60 con 0x7f7b141394b0 2022-07-12T08:50:55.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b14142f90 con 0x7f7b14064490 2022-07-12T08:50:55.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1413fa60 con 0x7f7b140b2a50 2022-07-12T08:50:55.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b19077700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b140b2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7b14064490 0x7f7b1405f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7b14064490 0x7f7b1405f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:40566/0 (socket says 172.21.15.61:40566) 2022-07-12T08:50:55.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 -- 172.21.15.61:0/2439150165 learned_addr learned my addr 172.21.15.61:0/2439150165 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:55.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 -- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 msgr2=0x7f7b140b2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:55.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 --2- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b140b2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 -- 172.21.15.61:0/2439150165 >> v1:172.21.15.111:6789/0 conn(0x7f7b141394b0 legacy=0x7f7b1413ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 -- 172.21.15.61:0/2439150165 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1405f630 con 0x7f7b14064490 2022-07-12T08:50:55.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b18876700 1 --2- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b1405f140 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f7b10002f70 tx=0x7f7b10006eb0).ready entity=mon.1 client_cookie=799d7f753fc1bd75 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:55.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b07fff700 1 -- 172.21.15.61:0/2439150165 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b1000d570 con 0x7f7b14064490 2022-07-12T08:50:55.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b07fff700 1 -- 172.21.15.61:0/2439150165 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b10000e90 con 0x7f7b14064490 2022-07-12T08:50:55.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b07fff700 1 -- 172.21.15.61:0/2439150165 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b1000d900 con 0x7f7b14064490 2022-07-12T08:50:55.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 msgr2=0x7f7b1405f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 --2- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b1405f140 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f7b10002f70 tx=0x7f7b10006eb0).stop 2022-07-12T08:50:55.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- 172.21.15.61:0/2439150165 shutdown_connections 2022-07-12T08:50:55.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 --2- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b1405f140 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 --2- 172.21.15.61:0/2439150165 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b140b2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- 172.21.15.61:0/2439150165 >> 172.21.15.61:0/2439150165 conn(0x7f7b14061e70 msgr2=0x7f7b14063a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:55.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- 172.21.15.61:0/2439150165 shutdown_connections 2022-07-12T08:50:55.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.600+0000 7f7b1a079700 1 -- 172.21.15.61:0/2439150165 wait complete. 2022-07-12T08:50:55.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 Processor -- start 2022-07-12T08:50:55.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 -- start start 2022-07-12T08:50:55.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b14064c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b14066e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 0x7f7b14150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14141750 con 0x7f7b141394b0 2022-07-12T08:50:55.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14066700 con 0x7f7b14064490 2022-07-12T08:50:55.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b140669a0 con 0x7f7b140b2a50 2022-07-12T08:50:55.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 0x7f7b14150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 0x7f7b14150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33770/0 (socket says 172.21.15.61:33770) 2022-07-12T08:50:55.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 -- 172.21.15.61:0/1564267789 learned_addr learned my addr 172.21.15.61:0/1564267789 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:55.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b18876700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b14066e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19077700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b14064c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 -- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 msgr2=0x7f7b14066e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b14066e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 -- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 msgr2=0x7f7b14064c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:55.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b14064c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:55.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 -- 172.21.15.61:0/1564267789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1000b1e0 con 0x7f7b141394b0 2022-07-12T08:50:55.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b19878700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 0x7f7b14150d80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f7b14143230 tx=0x7f7b0c007f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:55.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b0c00f470 con 0x7f7b141394b0 2022-07-12T08:50:55.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c00f5d0 con 0x7f7b141394b0 2022-07-12T08:50:55.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b14142b50 con 0x7f7b141394b0 2022-07-12T08:50:55.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b0c00f930 con 0x7f7b141394b0 2022-07-12T08:50:55.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.604+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b14136a00 con 0x7f7b141394b0 2022-07-12T08:50:55.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.608+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b14066b60 con 0x7f7b141394b0 2022-07-12T08:50:55.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.608+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7b0c019030 con 0x7f7b141394b0 2022-07-12T08:50:55.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.608+0000 7f7b067fc700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7afc053af0 0x7f7afc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:55.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.608+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7b0c06b450 con 0x7f7b141394b0 2022-07-12T08:50:55.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.608+0000 7f7b19077700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7afc053af0 0x7f7afc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:55.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.608+0000 7f7b19077700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7afc053af0 0x7f7afc055fb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f7b0800f000 tx=0x7f7b0800f040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:55.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.616+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7b0c0423b0 con 0x7f7b141394b0 2022-07-12T08:50:55.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:55.780+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b14141570 con 0x7f7b141394b0 2022-07-12T08:50:56.331 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:50:56.328+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:50:56.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.336+0000 7f7b067fc700 1 -- 172.21.15.61:0/1564267789 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b0c017020 con 0x7f7b141394b0 2022-07-12T08:50:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7afc053af0 msgr2=0x7f7afc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7afc053af0 0x7f7afc055fb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f7b0800f000 tx=0x7f7b0800f040).stop 2022-07-12T08:50:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 msgr2=0x7f7b14150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 0x7f7b14150d80 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7f7b14143230 tx=0x7f7b0c007f00).stop 2022-07-12T08:50:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 shutdown_connections 2022-07-12T08:50:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b141394b0 0x7f7b14150d80 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7b140b2a50 0x7f7b14066e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7afc053af0 0x7f7afc055fb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 --2- 172.21.15.61:0/1564267789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b14064490 0x7f7b14064c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 >> 172.21.15.61:0/1564267789 conn(0x7f7b14061e70 msgr2=0x7f7b140b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:56.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 shutdown_connections 2022-07-12T08:50:56.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.340+0000 7f7b1a079700 1 -- 172.21.15.61:0/1564267789 wait complete. 2022-07-12T08:50:56.353 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:50:56.361 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:50:56.365 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:50:56.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 Processor -- start 2022-07-12T08:50:56.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 -- start start 2022-07-12T08:50:56.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16b80b3f60 0x7f16b80b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80aecb0 con 0x7f16b809bce0 2022-07-12T08:50:56.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b80afd70 con 0x7f16b80b3f60 2022-07-12T08:50:56.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8143b00 con 0x7f16b80af280 2022-07-12T08:50:56.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b77fe700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f16b80b3f60 0x7f16b80b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b67fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f16b809bce0 0x7f16b80af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50240/0 (socket says 172.21.15.61:50240) 2022-07-12T08:50:56.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8060680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:44214/0 (socket says 172.21.15.61:44214) 2022-07-12T08:50:56.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b67fc700 1 -- 172.21.15.61:0/3674991065 learned_addr learned my addr 172.21.15.61:0/3674991065 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:56.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b5ffb700 1 -- 172.21.15.61:0/3674991065 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691825296 0 0) 0x7f16b80aecb0 con 0x7f16b809bce0 2022-07-12T08:50:56.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3674991065 >> v2:172.21.15.111:3300/0 conn(0x7f16b80b3f60 msgr2=0x7f16b80b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:56.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3674991065 >> v2:172.21.15.111:3300/0 conn(0x7f16b80b3f60 0x7f16b80b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3674991065 >> v1:172.21.15.61:6789/0 conn(0x7f16b809bce0 legacy=0x7f16b80af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3674991065 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b81435c0 con 0x7f16b80af280 2022-07-12T08:50:56.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3674991065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8060680 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f16a0002f70 tx=0x7f16a0006d90).ready entity=mon.2 client_cookie=41e451236185a74d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:56.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.464+0000 7f16b5ffb700 1 -- 172.21.15.61:0/3674991065 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a000d4a0 con 0x7f16b80af280 2022-07-12T08:50:56.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b5ffb700 1 -- 172.21.15.61:0/3674991065 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0000e90 con 0x7f16b80af280 2022-07-12T08:50:56.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b5ffb700 1 -- 172.21.15.61:0/3674991065 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a000d7f0 con 0x7f16b80af280 2022-07-12T08:50:56.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 -- 172.21.15.61:0/3674991065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 msgr2=0x7f16b8060680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3674991065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8060680 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f16a0002f70 tx=0x7f16a0006d90).stop 2022-07-12T08:50:56.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 -- 172.21.15.61:0/3674991065 shutdown_connections 2022-07-12T08:50:56.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3674991065 >> v2:172.21.15.111:3300/0 conn(0x7f16b80b3f60 0x7f16b80b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3674991065 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8060680 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 -- 172.21.15.61:0/3674991065 >> 172.21.15.61:0/3674991065 conn(0x7f16b80b92c0 msgr2=0x7f16b80b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:56.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 -- 172.21.15.61:0/3674991065 shutdown_connections 2022-07-12T08:50:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 -- 172.21.15.61:0/3674991065 wait complete. 2022-07-12T08:50:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 Processor -- start 2022-07-12T08:50:56.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.468+0000 7f16b7fff700 1 -- start start 2022-07-12T08:50:56.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 0x7f16b809e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b80b3f60 0x7f16b8141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8155270 con 0x7f16b809bce0 2022-07-12T08:50:56.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b809e860 con 0x7f16b80b3f60 2022-07-12T08:50:56.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b809eb00 con 0x7f16b80af280 2022-07-12T08:50:56.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 0x7f16b809e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 0x7f16b809e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33792/0 (socket says 172.21.15.61:33792) 2022-07-12T08:50:56.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3911101943 learned_addr learned my addr 172.21.15.61:0/3911101943 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:56.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b67fc700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b77fe700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b80b3f60 0x7f16b8141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 msgr2=0x7f16b8151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b80b3f60 msgr2=0x7f16b8141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b80b3f60 0x7f16b8141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3911101943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a000b120 con 0x7f16b809bce0 2022-07-12T08:50:56.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 0x7f16b809e320 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f16a0002f70 tx=0x7f16a0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:56.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a0011840 con 0x7f16b809bce0 2022-07-12T08:50:56.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a00119a0 con 0x7f16b809bce0 2022-07-12T08:50:56.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b809eda0 con 0x7f16b809bce0 2022-07-12T08:50:56.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a0011d00 con 0x7f16b809bce0 2022-07-12T08:50:56.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8141d10 con 0x7f16b809bce0 2022-07-12T08:50:56.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.472+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b81553e0 con 0x7f16b809bce0 2022-07-12T08:50:56.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.476+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f16a000f070 con 0x7f16b809bce0 2022-07-12T08:50:56.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.476+0000 7f16affff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f169c053a60 0x7f169c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.476+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f16a006d310 con 0x7f16b809bce0 2022-07-12T08:50:56.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.476+0000 7f16b67fc700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f169c053a60 0x7f169c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.476+0000 7f16b67fc700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f169c053a60 0x7f169c055f20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f16a800d000 tx=0x7f16a8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:56.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.484+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16b81553e0 con 0x7f16b809bce0 2022-07-12T08:50:56.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.644+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16b81553b0 con 0x7f16b809bce0 2022-07-12T08:50:56.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.644+0000 7f16affff700 1 -- 172.21.15.61:0/3911101943 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f16a0044220 con 0x7f16b809bce0 2022-07-12T08:50:56.649 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:50:56.650 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":78,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":78,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:50:42.959468+0000","modified":"2022-07-12T08:50:56.330874+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":139,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-12T08:50:56.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f169c053a60 msgr2=0x7f169c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f169c053a60 0x7f169c055f20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f16a800d000 tx=0x7f16a8009040).stop 2022-07-12T08:50:56.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 msgr2=0x7f16b809e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 0x7f16b809e320 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7f16a0002f70 tx=0x7f16a0014000).stop 2022-07-12T08:50:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 shutdown_connections 2022-07-12T08:50:56.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f169c053a60 0x7f169c055f20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16b80b3f60 0x7f16b8141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f16b80af280 0x7f16b8151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3911101943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b809bce0 0x7f16b809e320 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 >> 172.21.15.61:0/3911101943 conn(0x7f16b80b92c0 msgr2=0x7f16b80b8000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 shutdown_connections 2022-07-12T08:50:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.648+0000 7f16b7fff700 1 -- 172.21.15.61:0/3911101943 wait complete. 2022-07-12T08:50:56.661 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 78 2022-07-12T08:50:56.663 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:50:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.768+0000 7f62fc6d2700 1 Processor -- start 2022-07-12T08:50:56.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.768+0000 7f62fc6d2700 1 -- start start 2022-07-12T08:50:56.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f62f40b4050 0x7f62f40b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f62f4141e50 0x7f62f41422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f40b56c0 con 0x7f62f40b52e0 2022-07-12T08:50:56.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f4142f00 con 0x7f62f4141e50 2022-07-12T08:50:56.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f40b1320 con 0x7f62f40b4050 2022-07-12T08:50:56.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fac6f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f62f4141e50 0x7f62f41422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fa46e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f62f40b52e0 0x7f62f40b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50250/0 (socket says 172.21.15.61:50250) 2022-07-12T08:50:56.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fa46e700 1 -- 172.21.15.61:0/2090512704 learned_addr learned my addr 172.21.15.61:0/2090512704 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:56.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62f9c6d700 1 --2- 172.21.15.61:0/2090512704 >> v2:172.21.15.111:3300/0 conn(0x7f62f40b4050 0x7f62f40b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fac6f700 1 -- 172.21.15.61:0/2090512704 >> v2:172.21.15.111:3300/0 conn(0x7f62f40b4050 msgr2=0x7f62f40b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:56.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fac6f700 1 --2- 172.21.15.61:0/2090512704 >> v2:172.21.15.111:3300/0 conn(0x7f62f40b4050 0x7f62f40b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fac6f700 1 -- 172.21.15.61:0/2090512704 >> v1:172.21.15.61:6789/0 conn(0x7f62f40b52e0 legacy=0x7f62f40b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fac6f700 1 -- 172.21.15.61:0/2090512704 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40b4bd0 con 0x7f62f4141e50 2022-07-12T08:50:56.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62f9c6d700 1 --2- 172.21.15.61:0/2090512704 >> v2:172.21.15.111:3300/0 conn(0x7f62f40b4050 0x7f62f40b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:56.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62f946c700 1 -- 172.21.15.61:0/2090512704 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922114817 0 0) 0x7f62f40b56c0 con 0x7f62f40b52e0 2022-07-12T08:50:56.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fac6f700 1 --2- 172.21.15.61:0/2090512704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f4141e50 0x7f62f41422c0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f62f40b1460 tx=0x7f62ec0091d0).ready entity=mon.0 client_cookie=1f5b34dfbc6c4a91 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62f946c700 1 -- 172.21.15.61:0/2090512704 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62ec00b680 con 0x7f62f4141e50 2022-07-12T08:50:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62f946c700 1 -- 172.21.15.61:0/2090512704 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec00b7e0 con 0x7f62f4141e50 2022-07-12T08:50:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62f946c700 1 -- 172.21.15.61:0/2090512704 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62ec00b950 con 0x7f62f4141e50 2022-07-12T08:50:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 -- 172.21.15.61:0/2090512704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f4141e50 msgr2=0x7f62f41422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/2090512704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f4141e50 0x7f62f41422c0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f62f40b1460 tx=0x7f62ec0091d0).stop 2022-07-12T08:50:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.772+0000 7f62fc6d2700 1 -- 172.21.15.61:0/2090512704 shutdown_connections 2022-07-12T08:50:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/2090512704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f4141e50 0x7f62f41422c0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/2090512704 >> v2:172.21.15.111:3300/0 conn(0x7f62f40b4050 0x7f62f40b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- 172.21.15.61:0/2090512704 >> 172.21.15.61:0/2090512704 conn(0x7f62f4067020 msgr2=0x7f62f4067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- 172.21.15.61:0/2090512704 shutdown_connections 2022-07-12T08:50:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- 172.21.15.61:0/2090512704 wait complete. 2022-07-12T08:50:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 Processor -- start 2022-07-12T08:50:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- start start 2022-07-12T08:50:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 0x7f62f41464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 0x7f62f4155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62f4141e50 0x7f62f4156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4144640 con 0x7f62f40b52e0 2022-07-12T08:50:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4144870 con 0x7f62f40b4050 2022-07-12T08:50:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fc6d2700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f4144b10 con 0x7f62f4141e50 2022-07-12T08:50:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62f9c6d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 0x7f62f4155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fa46e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 0x7f62f41464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fac6f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62f4141e50 0x7f62f4156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fa46e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 0x7f62f41464e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:40612/0 (socket says 172.21.15.61:40612) 2022-07-12T08:50:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62f9c6d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 0x7f62f4155d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33808/0 (socket says 172.21.15.61:33808) 2022-07-12T08:50:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.776+0000 7f62fa46e700 1 -- 172.21.15.61:0/1985671395 learned_addr learned my addr 172.21.15.61:0/1985671395 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:56.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62f9c6d700 1 -- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62f4141e50 msgr2=0x7f62f4156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62f9c6d700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62f4141e50 0x7f62f4156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62f9c6d700 1 -- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 msgr2=0x7f62f41464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62f9c6d700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 0x7f62f41464e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62f9c6d700 1 -- 172.21.15.61:0/1985671395 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e800b180 con 0x7f62f40b52e0 2022-07-12T08:50:56.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62fa46e700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 0x7f62f41464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62f9c6d700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 0x7f62f4155d60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f62e800c000 tx=0x7f62e8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62e8010780 con 0x7f62f40b52e0 2022-07-12T08:50:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e80108e0 con 0x7f62f40b52e0 2022-07-12T08:50:56.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62ec009470 con 0x7f62f40b52e0 2022-07-12T08:50:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62e8010c40 con 0x7f62f40b52e0 2022-07-12T08:50:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f4155620 con 0x7f62f40b52e0 2022-07-12T08:50:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4144cd0 con 0x7f62f40b52e0 2022-07-12T08:50:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.780+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f62e801c460 con 0x7f62f40b52e0 2022-07-12T08:50:56.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.784+0000 7f62e77fe700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62d4053ad0 0x7f62d4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.784+0000 7f62fa46e700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62d4053ad0 0x7f62d4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.784+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f62e806c480 con 0x7f62f40b52e0 2022-07-12T08:50:56.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.784+0000 7f62fa46e700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62d4053ad0 0x7f62d4055f90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f62f000d000 tx=0x7f62f0007ac0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:56.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.792+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62e8043220 con 0x7f62f40b52e0 2022-07-12T08:50:56.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:56.956+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f62f4057460 con 0x7f62f40b52e0 2022-07-12T08:50:57.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.352+0000 7f62e77fe700 1 -- 172.21.15.61:0/1985671395 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7f62e8046840 con 0x7f62f40b52e0 2022-07-12T08:50:57.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62d4053ad0 msgr2=0x7f62d4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:57.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62d4053ad0 0x7f62d4055f90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7f62f000d000 tx=0x7f62f0007ac0).stop 2022-07-12T08:50:57.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 msgr2=0x7f62f4155d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:57.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 0x7f62f4155d60 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f62e800c000 tx=0x7f62e8006df0).stop 2022-07-12T08:50:57.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 shutdown_connections 2022-07-12T08:50:57.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62f4141e50 0x7f62f4156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f40b52e0 0x7f62f4155d60 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62d4053ad0 0x7f62d4055f90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 --2- 172.21.15.61:0/1985671395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62f40b4050 0x7f62f41464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 >> 172.21.15.61:0/1985671395 conn(0x7f62f4067020 msgr2=0x7f62f4067400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:57.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 shutdown_connections 2022-07-12T08:50:57.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.360+0000 7f62fc6d2700 1 -- 172.21.15.61:0/1985671395 wait complete. 2022-07-12T08:50:57.378 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:50:57.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.496+0000 7f8dfe202700 1 Processor -- start 2022-07-12T08:50:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.496+0000 7f8dfe202700 1 -- start start 2022-07-12T08:50:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df80b6780 0x7f8df80b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8df80b25c0 0x7f8df813f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:57.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df80668f0 con 0x7f8df81436b0 2022-07-12T08:50:57.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df8142f70 con 0x7f8df80b25c0 2022-07-12T08:50:57.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8143210 con 0x7f8df80b6780 2022-07-12T08:50:57.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfc9ff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8df80b25c0 0x7f8df813f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:57.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfda01700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8df81436b0 0x7f8df8065a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50270/0 (socket says 172.21.15.61:50270) 2022-07-12T08:50:57.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfda01700 1 -- 172.21.15.61:0/3972456304 learned_addr learned my addr 172.21.15.61:0/3972456304 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:57.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfd200700 1 --2- 172.21.15.61:0/3972456304 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df80b6780 0x7f8df80b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:57.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfc9ff700 1 -- 172.21.15.61:0/3972456304 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df80b6780 msgr2=0x7f8df80b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfc9ff700 1 --2- 172.21.15.61:0/3972456304 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df80b6780 0x7f8df80b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfc9ff700 1 -- 172.21.15.61:0/3972456304 >> v1:172.21.15.61:6789/0 conn(0x7f8df81436b0 legacy=0x7f8df8065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfc9ff700 1 -- 172.21.15.61:0/3972456304 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80607a0 con 0x7f8df80b25c0 2022-07-12T08:50:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8de3fff700 1 -- 172.21.15.61:0/3972456304 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803372632 0 0) 0x7f8df80668f0 con 0x7f8df81436b0 2022-07-12T08:50:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfc9ff700 1 --2- 172.21.15.61:0/3972456304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813f900 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f8df8064f80 tx=0x7f8dec006df0).ready entity=mon.0 client_cookie=960ac935e417310f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8de3fff700 1 -- 172.21.15.61:0/3972456304 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dec00d570 con 0x7f8df80b25c0 2022-07-12T08:50:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8de3fff700 1 -- 172.21.15.61:0/3972456304 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dec000e90 con 0x7f8df80b25c0 2022-07-12T08:50:57.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8de3fff700 1 -- 172.21.15.61:0/3972456304 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8dec00d8c0 con 0x7f8df80b25c0 2022-07-12T08:50:57.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- 172.21.15.61:0/3972456304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 msgr2=0x7f8df813f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:57.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 --2- 172.21.15.61:0/3972456304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813f900 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f8df8064f80 tx=0x7f8dec006df0).stop 2022-07-12T08:50:57.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- 172.21.15.61:0/3972456304 shutdown_connections 2022-07-12T08:50:57.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 --2- 172.21.15.61:0/3972456304 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813f900 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 --2- 172.21.15.61:0/3972456304 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df80b6780 0x7f8df80b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- 172.21.15.61:0/3972456304 >> 172.21.15.61:0/3972456304 conn(0x7f8df8064570 msgr2=0x7f8df80b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:57.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- 172.21.15.61:0/3972456304 shutdown_connections 2022-07-12T08:50:57.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.500+0000 7f8dfe202700 1 -- 172.21.15.61:0/3972456304 wait complete. 2022-07-12T08:50:57.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 Processor -- start 2022-07-12T08:50:57.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 -- start start 2022-07-12T08:50:57.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:57.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8df80b6780 0x7f8df81299f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:57.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df81436b0 0x7f8df80aeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:57.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80ad730 con 0x7f8df80b25c0 2022-07-12T08:50:57.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80ad9d0 con 0x7f8df80b6780 2022-07-12T08:50:57.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80ad420 con 0x7f8df81436b0 2022-07-12T08:50:57.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:57.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfda01700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df81436b0 0x7f8df80aeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:57.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33824/0 (socket says 172.21.15.61:33824) 2022-07-12T08:50:57.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 -- 172.21.15.61:0/2529633347 learned_addr learned my addr 172.21.15.61:0/2529633347 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:57.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfc9ff700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8df80b6780 0x7f8df81299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:57.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 -- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df81436b0 msgr2=0x7f8df80aeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:57.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df81436b0 0x7f8df80aeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 -- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8df80b6780 msgr2=0x7f8df81299f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:57.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8df80b6780 0x7f8df81299f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:57.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 -- 172.21.15.61:0/2529633347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec00b180 con 0x7f8df80b25c0 2022-07-12T08:50:57.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfc9ff700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8df80b6780 0x7f8df81299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:57.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfd200700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813e760 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f8df40030b0 tx=0x7f8df4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:57.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8df4011780 con 0x7f8df80b25c0 2022-07-12T08:50:57.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df40118e0 con 0x7f8df80b25c0 2022-07-12T08:50:57.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df8129ee0 con 0x7f8df80b25c0 2022-07-12T08:50:57.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8df4011c40 con 0x7f8df80b25c0 2022-07-12T08:50:57.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.504+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df81392a0 con 0x7f8df80b25c0 2022-07-12T08:50:57.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.508+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df80adb90 con 0x7f8df80b25c0 2022-07-12T08:50:57.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.508+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8df401c460 con 0x7f8df80b25c0 2022-07-12T08:50:57.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.508+0000 7f8deaffd700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dd40539d0 0x7f8dd4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:57.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.508+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f8df406c0c0 con 0x7f8df80b25c0 2022-07-12T08:50:57.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.508+0000 7f8dfc9ff700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dd40539d0 0x7f8dd4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:57.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.508+0000 7f8dfc9ff700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dd40539d0 0x7f8dd4055e90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f8dec01c000 tx=0x7f8dec015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:57.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.516+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8df4042e60 con 0x7f8df80b25c0 2022-07-12T08:50:57.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:57.644+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8df8057460 con 0x7f8df80b25c0 2022-07-12T08:50:58.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.400+0000 7f8deaffd700 1 -- 172.21.15.61:0/2529633347 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v140) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8df4046480 con 0x7f8df80b25c0 2022-07-12T08:50:58.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dd40539d0 msgr2=0x7f8dd4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:58.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dd40539d0 0x7f8dd4055e90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f8dec01c000 tx=0x7f8dec015000).stop 2022-07-12T08:50:58.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 msgr2=0x7f8df813e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:58.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813e760 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f8df40030b0 tx=0x7f8df4008040).stop 2022-07-12T08:50:58.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 shutdown_connections 2022-07-12T08:50:58.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8df81436b0 0x7f8df80aeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8dd40539d0 0x7f8dd4055e90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8df80b6780 0x7f8df81299f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 --2- 172.21.15.61:0/2529633347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8df80b25c0 0x7f8df813e760 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 >> 172.21.15.61:0/2529633347 conn(0x7f8df8064570 msgr2=0x7f8df805ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:58.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 shutdown_connections 2022-07-12T08:50:58.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.404+0000 7f8dfe202700 1 -- 172.21.15.61:0/2529633347 wait complete. 2022-07-12T08:50:58.421 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:50:58.423 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:50:58.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.536+0000 7f9c7c8f5700 1 Processor -- start 2022-07-12T08:50:58.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.536+0000 7f9c7c8f5700 1 -- start start 2022-07-12T08:50:58.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c780b6ac0 0x7f9c78146220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c78060430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f9c781420b0 0x7f9c78142540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c78146710 con 0x7f9c780b6ac0 2022-07-12T08:50:58.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c780b6ea0 con 0x7f9c781420b0 2022-07-12T08:50:58.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c78060200 con 0x7f9c7805f550 2022-07-12T08:50:58.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c780b6ac0 0x7f9c78146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c780b6ac0 0x7f9c78146220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33832/0 (socket says 172.21.15.61:33832) 2022-07-12T08:50:58.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 -- 172.21.15.61:0/2439211584 learned_addr learned my addr 172.21.15.61:0/2439211584 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:58.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c77fff700 1 --2- 172.21.15.61:0/2439211584 >> v2:172.21.15.111:3300/0 conn(0x7f9c781420b0 0x7f9c78142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c6effd700 1 --2- 172.21.15.61:0/2439211584 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c78060430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 -- 172.21.15.61:0/2439211584 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 msgr2=0x7f9c78060430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:58.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 --2- 172.21.15.61:0/2439211584 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c78060430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 -- 172.21.15.61:0/2439211584 >> v2:172.21.15.111:3300/0 conn(0x7f9c781420b0 msgr2=0x7f9c78142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:50:58.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 --2- 172.21.15.61:0/2439211584 >> v2:172.21.15.111:3300/0 conn(0x7f9c781420b0 0x7f9c78142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 -- 172.21.15.61:0/2439211584 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780b1360 con 0x7f9c780b6ac0 2022-07-12T08:50:58.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c77fff700 1 --2- 172.21.15.61:0/2439211584 >> v2:172.21.15.111:3300/0 conn(0x7f9c781420b0 0x7f9c78142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:50:58.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c777fe700 1 --2- 172.21.15.61:0/2439211584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c78146220 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f9c68002f70 tx=0x7f9c68006eb0).ready entity=mon.0 client_cookie=8afde6379a354ca8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:58.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c76ffd700 1 -- 172.21.15.61:0/2439211584 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c6800d570 con 0x7f9c780b6ac0 2022-07-12T08:50:58.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c76ffd700 1 -- 172.21.15.61:0/2439211584 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c68000e90 con 0x7f9c780b6ac0 2022-07-12T08:50:58.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c76ffd700 1 -- 172.21.15.61:0/2439211584 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c6800d900 con 0x7f9c780b6ac0 2022-07-12T08:50:58.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/2439211584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 msgr2=0x7f9c78146220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:58.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/2439211584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c78146220 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f9c68002f70 tx=0x7f9c68006eb0).stop 2022-07-12T08:50:58.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/2439211584 shutdown_connections 2022-07-12T08:50:58.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/2439211584 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c78060430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/2439211584 >> v2:172.21.15.111:3300/0 conn(0x7f9c781420b0 0x7f9c78142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/2439211584 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c78146220 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/2439211584 >> 172.21.15.61:0/2439211584 conn(0x7f9c78064570 msgr2=0x7f9c780b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:58.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/2439211584 shutdown_connections 2022-07-12T08:50:58.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.540+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/2439211584 wait complete. 2022-07-12T08:50:58.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 Processor -- start 2022-07-12T08:50:58.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 -- start start 2022-07-12T08:50:58.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c780b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c780af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781420b0 0x7f9c780af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780673f0 con 0x7f9c780b6ac0 2022-07-12T08:50:58.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780657a0 con 0x7f9c781420b0 2022-07-12T08:50:58.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c7c8f5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c78066520 con 0x7f9c7805f550 2022-07-12T08:50:58.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c6effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c780af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c6effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c780af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33838/0 (socket says 172.21.15.61:33838) 2022-07-12T08:50:58.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c6effd700 1 -- 172.21.15.61:0/3212844969 learned_addr learned my addr 172.21.15.61:0/3212844969 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:58.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c777fe700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c780b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.544+0000 7f9c77fff700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781420b0 0x7f9c780af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c6effd700 1 -- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 msgr2=0x7f9c780b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:58.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c6effd700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c780b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c6effd700 1 -- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781420b0 msgr2=0x7f9c780af930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:58.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c6effd700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781420b0 0x7f9c780af930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:58.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c6effd700 1 -- 172.21.15.61:0/3212844969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c70009470 con 0x7f9c780b6ac0 2022-07-12T08:50:58.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c777fe700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c780b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:50:58.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c6effd700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c780af440 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f9c6000c000 tx=0x7f9c60006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:58.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c60010780 con 0x7f9c780b6ac0 2022-07-12T08:50:58.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c600108e0 con 0x7f9c780b6ac0 2022-07-12T08:50:58.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6800b1e0 con 0x7f9c780b6ac0 2022-07-12T08:50:58.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c60010c40 con 0x7f9c780b6ac0 2022-07-12T08:50:58.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c780afe20 con 0x7f9c780b6ac0 2022-07-12T08:50:58.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c780b05d0 con 0x7f9c780b6ac0 2022-07-12T08:50:58.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9c6001c460 con 0x7f9c780b6ac0 2022-07-12T08:50:58.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c757fa700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9c5c053ad0 0x7f9c5c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:58.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f9c6006c280 con 0x7f9c780b6ac0 2022-07-12T08:50:58.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.548+0000 7f9c777fe700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9c5c053ad0 0x7f9c5c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:58.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.552+0000 7f9c777fe700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9c5c053ad0 0x7f9c5c055f90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f9c6801a000 tx=0x7f9c68015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:58.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.556+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c780b05d0 con 0x7f9c780b6ac0 2022-07-12T08:50:58.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:58.688+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9c780b05d0 con 0x7f9c780b6ac0 2022-07-12T08:50:59.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.408+0000 7f9c757fa700 1 -- 172.21.15.61:0/3212844969 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v141) v1 ==== 173+0+0 (secure 0 0 0) 0x7f9c780b05d0 con 0x7f9c780b6ac0 2022-07-12T08:50:59.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9c5c053ad0 msgr2=0x7f9c5c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9c5c053ad0 0x7f9c5c055f90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f9c6801a000 tx=0x7f9c68015000).stop 2022-07-12T08:50:59.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 msgr2=0x7f9c780af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c780af440 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 rx=0x7f9c6000c000 tx=0x7f9c60006df0).stop 2022-07-12T08:50:59.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 shutdown_connections 2022-07-12T08:50:59.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9c5c053ad0 0x7f9c5c055f90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9c781420b0 0x7f9c780af930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c780b6ac0 0x7f9c780af440 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 --2- 172.21.15.61:0/3212844969 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9c7805f550 0x7f9c780b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 >> 172.21.15.61:0/3212844969 conn(0x7f9c78064570 msgr2=0x7f9c781400d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:59.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 shutdown_connections 2022-07-12T08:50:59.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.412+0000 7f9c7c8f5700 1 -- 172.21.15.61:0/3212844969 wait complete. 2022-07-12T08:50:59.418 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:50:59.432 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-12T08:50:59.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45bfff700 1 Processor -- start 2022-07-12T08:50:59.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45bfff700 1 -- start start 2022-07-12T08:50:59.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7ff45c146be0 0x7ff45c05d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c05ff60 con 0x7ff45c05cd20 2022-07-12T08:50:59.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45bfff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c146200 con 0x7ff45c0af490 2022-07-12T08:50:59.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45bfff700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c1464a0 con 0x7ff45c146be0 2022-07-12T08:50:59.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff45c05cd20 0x7ff45c05f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50294/0 (socket says 172.21.15.61:50294) 2022-07-12T08:50:59.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.540+0000 7ff45affd700 1 -- 172.21.15.61:0/2454846868 learned_addr learned my addr 172.21.15.61:0/2454846868 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:59.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45b7fe700 1 --2- 172.21.15.61:0/2454846868 >> v2:172.21.15.111:3300/0 conn(0x7ff45c146be0 0x7ff45c05d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073260311 0 0) 0x7ff45c05ff60 con 0x7ff45c05cd20 2022-07-12T08:50:59.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c003260 con 0x7ff45c05cd20 2022-07-12T08:50:59.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707627743 0 0) 0x7ff45c146200 con 0x7ff45c0af490 2022-07-12T08:50:59.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff45c05ff60 con 0x7ff45c0af490 2022-07-12T08:50:59.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45b7fe700 1 -- 172.21.15.61:0/2454846868 >> v1:172.21.15.111:6789/0 conn(0x7ff45c0af490 legacy=0x7ff45c05f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45b7fe700 1 -- 172.21.15.61:0/2454846868 >> v1:172.21.15.61:6789/0 conn(0x7ff45c05cd20 legacy=0x7ff45c05f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45b7fe700 1 -- 172.21.15.61:0/2454846868 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c146740 con 0x7ff45c146be0 2022-07-12T08:50:59.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45b7fe700 1 --2- 172.21.15.61:0/2454846868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c146be0 0x7ff45c05d100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff4540032f0 tx=0x7ff4540091d0).ready entity=mon.1 client_cookie=164a097e5442d295 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:59.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff45400b600 con 0x7ff45c146be0 2022-07-12T08:50:59.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45400b760 con 0x7ff45c146be0 2022-07-12T08:50:59.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff459ffb700 1 -- 172.21.15.61:0/2454846868 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff45400b8d0 con 0x7ff45c146be0 2022-07-12T08:50:59.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 -- 172.21.15.61:0/2454846868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c146be0 msgr2=0x7ff45c05d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2454846868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c146be0 0x7ff45c05d100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff4540032f0 tx=0x7ff4540091d0).stop 2022-07-12T08:50:59.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 -- 172.21.15.61:0/2454846868 shutdown_connections 2022-07-12T08:50:59.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2454846868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c146be0 0x7ff45c05d100 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 -- 172.21.15.61:0/2454846868 >> 172.21.15.61:0/2454846868 conn(0x7ff45c0b9e20 msgr2=0x7ff45c0b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:59.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 -- 172.21.15.61:0/2454846868 shutdown_connections 2022-07-12T08:50:59.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.544+0000 7ff45bfff700 1 -- 172.21.15.61:0/2454846868 wait complete. 2022-07-12T08:50:59.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 Processor -- start 2022-07-12T08:50:59.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 -- start start 2022-07-12T08:50:59.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 0x7ff45c09ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c0af490 0x7ff45c09af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff45c146be0 0x7ff45c09b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b0ca0 con 0x7ff45c05cd20 2022-07-12T08:50:59.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b0f40 con 0x7ff45c0af490 2022-07-12T08:50:59.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0b11e0 con 0x7ff45c146be0 2022-07-12T08:50:59.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 0x7ff45c09ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 0x7ff45c09ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33850/0 (socket says 172.21.15.61:33850) 2022-07-12T08:50:59.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 -- 172.21.15.61:0/2364328546 learned_addr learned my addr 172.21.15.61:0/2364328546 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:59.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45b7fe700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff45c146be0 0x7ff45c09b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45a7fc700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c0af490 0x7ff45c09af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 -- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff45c146be0 msgr2=0x7ff45c09b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff45c146be0 0x7ff45c09b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 -- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c0af490 msgr2=0x7ff45c09af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c0af490 0x7ff45c09af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 -- 172.21.15.61:0/2364328546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff454009470 con 0x7ff45c05cd20 2022-07-12T08:50:59.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45affd700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 0x7ff45c09ef10 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7ff4440011d0 tx=0x7ff444007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:59.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4440070d0 con 0x7ff45c05cd20 2022-07-12T08:50:59.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff444007230 con 0x7ff45c05cd20 2022-07-12T08:50:59.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c0b3880 con 0x7ff45c05cd20 2022-07-12T08:50:59.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff444005610 con 0x7ff45c05cd20 2022-07-12T08:50:59.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.548+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c0b2120 con 0x7ff45c05cd20 2022-07-12T08:50:59.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.552+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c145fa0 con 0x7ff45c05cd20 2022-07-12T08:50:59.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.552+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff444010070 con 0x7ff45c05cd20 2022-07-12T08:50:59.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.552+0000 7ff453fff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4400539b0 0x7ff440055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.552+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff44406b250 con 0x7ff45c05cd20 2022-07-12T08:50:59.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.552+0000 7ff45a7fc700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4400539b0 0x7ff440055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.552+0000 7ff45a7fc700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4400539b0 0x7ff440055e70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7ff44c00d000 tx=0x7ff44c007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:59.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.556+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff4440423f0 con 0x7ff45c05cd20 2022-07-12T08:50:59.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.688+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff45c057460 con 0x7ff45c05cd20 2022-07-12T08:50:59.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff453fff700 1 -- 172.21.15.61:0/2364328546 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff444045a10 con 0x7ff45c05cd20 2022-07-12T08:50:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4400539b0 msgr2=0x7ff440055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4400539b0 0x7ff440055e70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7ff44c00d000 tx=0x7ff44c007ad0).stop 2022-07-12T08:50:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 msgr2=0x7ff45c09ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 0x7ff45c09ef10 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7ff4440011d0 tx=0x7ff444007b50).stop 2022-07-12T08:50:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 shutdown_connections 2022-07-12T08:50:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff45c146be0 0x7ff45c09b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff4400539b0 0x7ff440055e70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff45c0af490 0x7ff45c09af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 --2- 172.21.15.61:0/2364328546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff45c05cd20 0x7ff45c09ef10 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 >> 172.21.15.61:0/2364328546 conn(0x7ff45c0b9e20 msgr2=0x7ff45c05f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.696+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 shutdown_connections 2022-07-12T08:50:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.700+0000 7ff45bfff700 1 -- 172.21.15.61:0/2364328546 wait complete. 2022-07-12T08:50:59.715 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-12T08:50:59.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddc7d0700 1 Processor -- start 2022-07-12T08:50:59.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddc7d0700 1 -- start start 2022-07-12T08:50:59.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddc7d0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3dd40b1e40 0x7f3dd405d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddc7d0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd413fd50 con 0x7f3dd40ad590 2022-07-12T08:50:59.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddc7d0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd4067900 con 0x7f3dd40ac710 2022-07-12T08:50:59.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddc7d0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd4067ba0 con 0x7f3dd40b1e40 2022-07-12T08:50:59.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dda56c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3dd40b1e40 0x7f3dd405d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddad6d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3dd40ad590 0x7f3dd41401b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50310/0 (socket says 172.21.15.61:50310) 2022-07-12T08:50:59.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3ddad6d700 1 -- 172.21.15.61:0/1699478010 learned_addr learned my addr 172.21.15.61:0/1699478010 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:50:59.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dd956a700 1 -- 172.21.15.61:0/1699478010 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085078327 0 0) 0x7f3dd413fd50 con 0x7f3dd40ad590 2022-07-12T08:50:59.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dd956a700 1 -- 172.21.15.61:0/1699478010 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dbc003260 con 0x7f3dd40ad590 2022-07-12T08:50:59.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dda56c700 1 -- 172.21.15.61:0/1699478010 >> v1:172.21.15.111:6789/0 conn(0x7f3dd40ac710 legacy=0x7f3dd40ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dda56c700 1 -- 172.21.15.61:0/1699478010 >> v1:172.21.15.61:6789/0 conn(0x7f3dd40ad590 legacy=0x7f3dd41401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dda56c700 1 -- 172.21.15.61:0/1699478010 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40675f0 con 0x7f3dd40b1e40 2022-07-12T08:50:59.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dda56c700 1 --2- 172.21.15.61:0/1699478010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b1e40 0x7f3dd405d0d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f3dc4002f70 tx=0x7f3dc4006df0).ready entity=mon.0 client_cookie=f5470ca7c2076e4b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:59.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dd956a700 1 -- 172.21.15.61:0/1699478010 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc400d610 con 0x7f3dd40b1e40 2022-07-12T08:50:59.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dd956a700 1 -- 172.21.15.61:0/1699478010 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc4000e90 con 0x7f3dd40b1e40 2022-07-12T08:50:59.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.820+0000 7f3dd956a700 1 -- 172.21.15.61:0/1699478010 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc400d960 con 0x7f3dd40b1e40 2022-07-12T08:50:59.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1699478010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b1e40 msgr2=0x7f3dd405d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1699478010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b1e40 0x7f3dd405d0d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f3dc4002f70 tx=0x7f3dc4006df0).stop 2022-07-12T08:50:59.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1699478010 shutdown_connections 2022-07-12T08:50:59.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1699478010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b1e40 0x7f3dd405d0d0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1699478010 >> 172.21.15.61:0/1699478010 conn(0x7f3dd4062c00 msgr2=0x7f3dd4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:50:59.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1699478010 shutdown_connections 2022-07-12T08:50:59.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1699478010 wait complete. 2022-07-12T08:50:59.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 Processor -- start 2022-07-12T08:50:59.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- start start 2022-07-12T08:50:59.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 0x7f3dd405c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dd40ad590 0x7f3dd413eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3dd40b1e40 0x7f3dd413b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd409e480 con 0x7f3dd40ac710 2022-07-12T08:50:59.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd409e720 con 0x7f3dd40ad590 2022-07-12T08:50:59.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddc7d0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd413b520 con 0x7f3dd40b1e40 2022-07-12T08:50:59.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 0x7f3dd405c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 0x7f3dd405c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33864/0 (socket says 172.21.15.61:33864) 2022-07-12T08:50:59.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 -- 172.21.15.61:0/1044868301 learned_addr learned my addr 172.21.15.61:0/1044868301 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:50:59.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3ddad6d700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3dd40b1e40 0x7f3dd413b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dd9d6b700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dd40ad590 0x7f3dd413eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 -- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3dd40b1e40 msgr2=0x7f3dd413b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3dd40b1e40 0x7f3dd413b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 -- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dd40ad590 msgr2=0x7f3dd413eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:50:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dd40ad590 0x7f3dd413eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:50:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dda56c700 1 -- 172.21.15.61:0/1044868301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc400b180 con 0x7f3dd40ac710 2022-07-12T08:50:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.824+0000 7f3dd9d6b700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dd40ad590 0x7f3dd413eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:50:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dda56c700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 0x7f3dd405c050 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f3dc4002f70 tx=0x7f3dc4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:59.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc400d290 con 0x7f3dd40ac710 2022-07-12T08:50:59.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc401e400 con 0x7f3dd40ac710 2022-07-12T08:50:59.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd413f3e0 con 0x7f3dd40ac710 2022-07-12T08:50:59.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc401e760 con 0x7f3dd40ac710 2022-07-12T08:50:59.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd413beb0 con 0x7f3dd40ac710 2022-07-12T08:50:59.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd413b6e0 con 0x7f3dd40ac710 2022-07-12T08:50:59.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3dc4025030 con 0x7f3dd40ac710 2022-07-12T08:50:59.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dc37fe700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3db4053a60 0x7f3db4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:50:59.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f3dc406c810 con 0x7f3dd40ac710 2022-07-12T08:50:59.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.828+0000 7f3dd9d6b700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3db4053a60 0x7f3db4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:50:59.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.832+0000 7f3dd9d6b700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3db4053a60 0x7f3db4055f20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f3dd0008000 tx=0x7f3dd0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:50:59.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:50:59.836+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3dc4043a50 con 0x7f3dd40ac710 2022-07-12T08:51:00.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.008+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7f3dd4057460 con 0x7f3dd40ac710 2022-07-12T08:51:00.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.416+0000 7f3dc37fe700 1 -- 172.21.15.61:0/1044868301 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v460) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3dc4023020 con 0x7f3dd40ac710 2022-07-12T08:51:00.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.420+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3db4053a60 msgr2=0x7f3db4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:00.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.420+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3db4053a60 0x7f3db4055f20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f3dd0008000 tx=0x7f3dd0009040).stop 2022-07-12T08:51:00.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.420+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 msgr2=0x7f3dd405c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:00.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.420+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 0x7f3dd405c050 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f3dc4002f70 tx=0x7f3dc4014000).stop 2022-07-12T08:51:00.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 shutdown_connections 2022-07-12T08:51:00.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3dd40b1e40 0x7f3dd413b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3db4053a60 0x7f3db4055f20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3dd40ad590 0x7f3dd413eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 --2- 172.21.15.61:0/1044868301 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40ac710 0x7f3dd405c050 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 >> 172.21.15.61:0/1044868301 conn(0x7f3dd4062c00 msgr2=0x7f3dd40acc50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:00.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 shutdown_connections 2022-07-12T08:51:00.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.424+0000 7f3ddc7d0700 1 -- 172.21.15.61:0/1044868301 wait complete. 2022-07-12T08:51:00.443 INFO:tasks.cephfs_test_runner:test_root_rctime (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-12T08:51:00.444 INFO:tasks.cephfs_test_runner:Starting test: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T08:51:00.445 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs' 2022-07-12T08:51:00.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 Processor -- start 2022-07-12T08:51:00.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- start start 2022-07-12T08:51:00.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f731413ce60 0x7f73140b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f7314142d50 0x7f73140b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f73140b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73140b24c0 con 0x7f731413ce60 2022-07-12T08:51:00.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731413d240 con 0x7f7314142d50 2022-07-12T08:51:00.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73141431f0 con 0x7f73140b60a0 2022-07-12T08:51:00.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f731413ce60 0x7f73140b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f731413ce60 0x7f73140b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33872/0 (socket says 172.21.15.61:33872) 2022-07-12T08:51:00.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 -- 172.21.15.61:0/2480948754 learned_addr learned my addr 172.21.15.61:0/2480948754 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:00.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731918e700 1 --2- 172.21.15.61:0/2480948754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f73140b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f7313fff700 1 --2- 172.21.15.61:0/2480948754 >> v2:172.21.15.111:3300/0 conn(0x7f7314142d50 0x7f73140b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 -- 172.21.15.61:0/2480948754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 msgr2=0x7f73140b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:00.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 --2- 172.21.15.61:0/2480948754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f73140b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 -- 172.21.15.61:0/2480948754 >> v2:172.21.15.111:3300/0 conn(0x7f7314142d50 msgr2=0x7f73140b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:00.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 --2- 172.21.15.61:0/2480948754 >> v2:172.21.15.111:3300/0 conn(0x7f7314142d50 0x7f73140b5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 -- 172.21.15.61:0/2480948754 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314143490 con 0x7f731413ce60 2022-07-12T08:51:00.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731898d700 1 --2- 172.21.15.61:0/2480948754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f73140b15a0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f72fc002f70 tx=0x7f72fc006eb0).ready entity=mon.0 client_cookie=d91d3f0239b5ed6b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:00.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f73137fe700 1 -- 172.21.15.61:0/2480948754 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72fc00d570 con 0x7f731413ce60 2022-07-12T08:51:00.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f73137fe700 1 -- 172.21.15.61:0/2480948754 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc000e90 con 0x7f731413ce60 2022-07-12T08:51:00.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f73137fe700 1 -- 172.21.15.61:0/2480948754 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72fc00d900 con 0x7f731413ce60 2022-07-12T08:51:00.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- 172.21.15.61:0/2480948754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 msgr2=0x7f73140b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:00.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- 172.21.15.61:0/2480948754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f73140b15a0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 rx=0x7f72fc002f70 tx=0x7f72fc006eb0).stop 2022-07-12T08:51:00.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- 172.21.15.61:0/2480948754 shutdown_connections 2022-07-12T08:51:00.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- 172.21.15.61:0/2480948754 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f73140b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- 172.21.15.61:0/2480948754 >> v2:172.21.15.111:3300/0 conn(0x7f7314142d50 0x7f73140b5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 --2- 172.21.15.61:0/2480948754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f73140b15a0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- 172.21.15.61:0/2480948754 >> 172.21.15.61:0/2480948754 conn(0x7f731405f9f0 msgr2=0x7f7314066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:00.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- 172.21.15.61:0/2480948754 shutdown_connections 2022-07-12T08:51:00.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.552+0000 7f731abf1700 1 -- 172.21.15.61:0/2480948754 wait complete. 2022-07-12T08:51:00.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 Processor -- start 2022-07-12T08:51:00.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 -- start start 2022-07-12T08:51:00.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f7314146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f731413b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7314142d50 0x7f73141552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7314146bd0 con 0x7f731413ce60 2022-07-12T08:51:00.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7314145940 con 0x7f7314142d50 2022-07-12T08:51:00.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73140ad340 con 0x7f73140b60a0 2022-07-12T08:51:00.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f731413b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731898d700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f7314146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f731413b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33882/0 (socket says 172.21.15.61:33882) 2022-07-12T08:51:00.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731918e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7314142d50 0x7f73141552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 -- 172.21.15.61:0/604423012 learned_addr learned my addr 172.21.15.61:0/604423012 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:00.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 -- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 msgr2=0x7f7314146d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:00.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f7314146d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 -- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7314142d50 msgr2=0x7f73141552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:00.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7314142d50 0x7f73141552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:00.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 -- 172.21.15.61:0/604423012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc00b1e0 con 0x7f731413ce60 2022-07-12T08:51:00.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7313fff700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f731413b820 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f7304003430 tx=0x7f7304008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:00.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f730400f710 con 0x7f731413ce60 2022-07-12T08:51:00.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730400f870 con 0x7f731413ce60 2022-07-12T08:51:00.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73140ad5e0 con 0x7f731413ce60 2022-07-12T08:51:00.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f730400fbd0 con 0x7f731413ce60 2022-07-12T08:51:00.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.556+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73141453c0 con 0x7f731413ce60 2022-07-12T08:51:00.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.560+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73140ad770 con 0x7f731413ce60 2022-07-12T08:51:00.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.560+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f730401a030 con 0x7f731413ce60 2022-07-12T08:51:00.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.560+0000 7f7311ffb700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72f8053a40 0x7f72f8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:00.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.560+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f730406bb20 con 0x7f731413ce60 2022-07-12T08:51:00.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.560+0000 7f731898d700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72f8053a40 0x7f72f8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:00.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.560+0000 7f731898d700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72f8053a40 0x7f72f8055f00 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f72fc01c000 tx=0x7f72fc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:00.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.564+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7304042d60 con 0x7f731413ce60 2022-07-12T08:51:00.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:00.732+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]} v 0) v1 -- 0x7f7314057460 con 0x7f731413ce60 2022-07-12T08:51:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.424+0000 7f7311ffb700 1 -- 172.21.15.61:0/604423012 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]}]=0 v461) v1 ==== 139+0+0 (secure 0 0 0) 0x7f7304046380 con 0x7f731413ce60 2022-07-12T08:51:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72f8053a40 msgr2=0x7f72f8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72f8053a40 0x7f72f8055f00 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7f72fc01c000 tx=0x7f72fc015000).stop 2022-07-12T08:51:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 msgr2=0x7f731413b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f731413b820 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f7304003430 tx=0x7f7304008040).stop 2022-07-12T08:51:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 shutdown_connections 2022-07-12T08:51:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f72f8053a40 0x7f72f8055f00 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7314142d50 0x7f73141552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f731413ce60 0x7f731413b820 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 --2- 172.21.15.61:0/604423012 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f73140b60a0 0x7f7314146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 >> 172.21.15.61:0/604423012 conn(0x7f731405f9f0 msgr2=0x7f7314142550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 shutdown_connections 2022-07-12T08:51:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.428+0000 7f731abf1700 1 -- 172.21.15.61:0/604423012 wait complete. 2022-07-12T08:51:01.451 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-12T08:51:01.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 Processor -- start 2022-07-12T08:51:01.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 -- start start 2022-07-12T08:51:01.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f661c05cec0 0x7f661c1467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f661c067620 0x7f661c146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c05f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c0aef80 con 0x7f661c067620 2022-07-12T08:51:01.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c1460b0 con 0x7f661c05cec0 2022-07-12T08:51:01.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6623194700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0af1b0 con 0x7f661c0aff70 2022-07-12T08:51:01.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f661c067620 0x7f661c146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f661c067620 0x7f661c146ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33886/0 (socket says 172.21.15.61:33886) 2022-07-12T08:51:01.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 -- 172.21.15.61:0/3285118077 learned_addr learned my addr 172.21.15.61:0/3285118077 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:01.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6622993700 1 --2- 172.21.15.61:0/3285118077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c05f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6622192700 1 --2- 172.21.15.61:0/3285118077 >> v2:172.21.15.111:3300/0 conn(0x7f661c05cec0 0x7f661c1467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 -- 172.21.15.61:0/3285118077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 msgr2=0x7f661c05f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:01.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 --2- 172.21.15.61:0/3285118077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c05f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 -- 172.21.15.61:0/3285118077 >> v2:172.21.15.111:3300/0 conn(0x7f661c05cec0 msgr2=0x7f661c1467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:01.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 --2- 172.21.15.61:0/3285118077 >> v2:172.21.15.111:3300/0 conn(0x7f661c05cec0 0x7f661c1467b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 -- 172.21.15.61:0/3285118077 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c146310 con 0x7f661c067620 2022-07-12T08:51:01.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.572+0000 7f6621991700 1 --2- 172.21.15.61:0/3285118077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c146ca0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f660c002f70 tx=0x7f660c006eb0).ready entity=mon.0 client_cookie=4d29f422460f4d93 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:01.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6621190700 1 -- 172.21.15.61:0/3285118077 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c00d570 con 0x7f661c067620 2022-07-12T08:51:01.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6621190700 1 -- 172.21.15.61:0/3285118077 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c000e90 con 0x7f661c067620 2022-07-12T08:51:01.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6621190700 1 -- 172.21.15.61:0/3285118077 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c00d900 con 0x7f661c067620 2022-07-12T08:51:01.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 -- 172.21.15.61:0/3285118077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 msgr2=0x7f661c146ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 --2- 172.21.15.61:0/3285118077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c146ca0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 rx=0x7f660c002f70 tx=0x7f660c006eb0).stop 2022-07-12T08:51:01.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 -- 172.21.15.61:0/3285118077 shutdown_connections 2022-07-12T08:51:01.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 --2- 172.21.15.61:0/3285118077 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c05f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 --2- 172.21.15.61:0/3285118077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c146ca0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 --2- 172.21.15.61:0/3285118077 >> v2:172.21.15.111:3300/0 conn(0x7f661c05cec0 0x7f661c1467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 -- 172.21.15.61:0/3285118077 >> 172.21.15.61:0/3285118077 conn(0x7f661c0b6880 msgr2=0x7f661c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 -- 172.21.15.61:0/3285118077 shutdown_connections 2022-07-12T08:51:01.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 -- 172.21.15.61:0/3285118077 wait complete. 2022-07-12T08:51:01.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 Processor -- start 2022-07-12T08:51:01.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.576+0000 7f6623194700 1 -- start start 2022-07-12T08:51:01.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f661c05cec0 0x7f661c151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c13fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c13ff10 con 0x7f661c067620 2022-07-12T08:51:01.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c148240 con 0x7f661c05cec0 2022-07-12T08:51:01.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c147370 con 0x7f661c0aff70 2022-07-12T08:51:01.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c13fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c13fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33894/0 (socket says 172.21.15.61:33894) 2022-07-12T08:51:01.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 -- 172.21.15.61:0/2142956091 learned_addr learned my addr 172.21.15.61:0/2142956091 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:01.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6622192700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f661c05cec0 0x7f661c151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6622993700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 -- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 msgr2=0x7f661c147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 -- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f661c05cec0 msgr2=0x7f661c151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f661c05cec0 0x7f661c151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 -- 172.21.15.61:0/2142956091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c00b1e0 con 0x7f661c067620 2022-07-12T08:51:01.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6621991700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c13fa20 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f661c1483d0 tx=0x7f660c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:01.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c00dea0 con 0x7f661c067620 2022-07-12T08:51:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c006b40 con 0x7f661c067620 2022-07-12T08:51:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c147610 con 0x7f661c067620 2022-07-12T08:51:01.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f660c011de0 con 0x7f661c067620 2022-07-12T08:51:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c1488b0 con 0x7f661c067620 2022-07-12T08:51:01.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.580+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c057460 con 0x7f661c067620 2022-07-12T08:51:01.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.584+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f660c025030 con 0x7f661c067620 2022-07-12T08:51:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.584+0000 7f661b7fe700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65fc053a60 0x7f65fc055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:01.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.584+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f660c06cbf0 con 0x7f661c067620 2022-07-12T08:51:01.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.584+0000 7f6622192700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65fc053a60 0x7f65fc055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:01.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.584+0000 7f6622192700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65fc053a60 0x7f65fc055f20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f661000d000 tx=0x7f6610008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:01.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.588+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f660c043e30 con 0x7f661c067620 2022-07-12T08:51:01.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.716+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f661c09bfc0 con 0x7f661c067620 2022-07-12T08:51:01.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.724+0000 7f661b7fe700 1 -- 172.21.15.61:0/2142956091 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f660c023020 con 0x7f661c067620 2022-07-12T08:51:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65fc053a60 msgr2=0x7f65fc055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65fc053a60 0x7f65fc055f20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7f661000d000 tx=0x7f6610008040).stop 2022-07-12T08:51:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 msgr2=0x7f661c13fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c13fa20 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 rx=0x7f661c1483d0 tx=0x7f660c015000).stop 2022-07-12T08:51:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 shutdown_connections 2022-07-12T08:51:01.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f661c0aff70 0x7f661c147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f661c067620 0x7f661c13fa20 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f65fc053a60 0x7f65fc055f20 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 --2- 172.21.15.61:0/2142956091 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f661c05cec0 0x7f661c151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:01.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 >> 172.21.15.61:0/2142956091 conn(0x7f661c0b6880 msgr2=0x7f661c0b8710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:01.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 shutdown_connections 2022-07-12T08:51:01.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:01.728+0000 7f6623194700 1 -- 172.21.15.61:0/2142956091 wait complete. 2022-07-12T08:51:01.747 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T08:51:01.748 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:51:01.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:51:01.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:51:01.812 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:51:01.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T08:51:01.857 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T08:51:01.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:01.858 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:01.859 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:51:01.866 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-12T08:51:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:51:01.868 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-12T08:51:01.869 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:01.870 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:01.871 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:51:01.882 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-12T08:51:01.882 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-12T08:51:01.883 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:01.885 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:01.885 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-12T08:51:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T08:51:01.910 INFO:teuthology.orchestra.run.smithi137.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-12T08:51:01.910 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-12T08:51:01.911 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:51:02.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.020+0000 7f8f7359e700 1 Processor -- start 2022-07-12T08:51:02.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.020+0000 7f8f7359e700 1 -- start start 2022-07-12T08:51:02.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f7405fb90 0x7f8f740ad000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f740b0980 con 0x7f8f740b0b80 2022-07-12T08:51:02.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f740af0d0 con 0x7f8f74142690 2022-07-12T08:51:02.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7405ff70 con 0x7f8f7405fb90 2022-07-12T08:51:02.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f6bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8f740b0b80 0x7f8f740b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50350/0 (socket says 172.21.15.61:50350) 2022-07-12T08:51:02.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f6bfff700 1 -- 172.21.15.61:0/337329088 learned_addr learned my addr 172.21.15.61:0/337329088 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:02.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7259c700 1 --2- 172.21.15.61:0/337329088 >> v2:172.21.15.61:3300/0 conn(0x7f8f7405fb90 0x7f8f740ad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731797510 0 0) 0x7f8f740b0980 con 0x7f8f740b0b80 2022-07-12T08:51:02.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f54003260 con 0x7f8f740b0b80 2022-07-12T08:51:02.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171612242 0 0) 0x7f8f740af0d0 con 0x7f8f74142690 2022-07-12T08:51:02.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f740b0980 con 0x7f8f74142690 2022-07-12T08:51:02.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7259c700 1 -- 172.21.15.61:0/337329088 >> v1:172.21.15.111:6789/0 conn(0x7f8f74142690 legacy=0x7f8f74142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7259c700 1 -- 172.21.15.61:0/337329088 >> v1:172.21.15.61:6789/0 conn(0x7f8f740b0b80 legacy=0x7f8f740b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7259c700 1 -- 172.21.15.61:0/337329088 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f740af3b0 con 0x7f8f7405fb90 2022-07-12T08:51:02.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7259c700 1 --2- 172.21.15.61:0/337329088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f7405fb90 0x7f8f740ad000 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f8f64002f70 tx=0x7f8f64007f00).ready entity=mon.0 client_cookie=53c60e9b10bae655 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6400e4a0 con 0x7f8f7405fb90 2022-07-12T08:51:02.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f64000e90 con 0x7f8f7405fb90 2022-07-12T08:51:02.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f71d9b700 1 -- 172.21.15.61:0/337329088 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6400e7f0 con 0x7f8f7405fb90 2022-07-12T08:51:02.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- 172.21.15.61:0/337329088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f7405fb90 msgr2=0x7f8f740ad000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 --2- 172.21.15.61:0/337329088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f7405fb90 0x7f8f740ad000 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 rx=0x7f8f64002f70 tx=0x7f8f64007f00).stop 2022-07-12T08:51:02.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- 172.21.15.61:0/337329088 shutdown_connections 2022-07-12T08:51:02.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 --2- 172.21.15.61:0/337329088 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f7405fb90 0x7f8f740ad000 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- 172.21.15.61:0/337329088 >> 172.21.15.61:0/337329088 conn(0x7f8f74063cd0 msgr2=0x7f8f7405f140 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:02.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- 172.21.15.61:0/337329088 shutdown_connections 2022-07-12T08:51:02.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.024+0000 7f8f7359e700 1 -- 172.21.15.61:0/337329088 wait complete. 2022-07-12T08:51:02.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 Processor -- start 2022-07-12T08:51:02.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 -- start start 2022-07-12T08:51:02.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f7405fb90 0x7f8f74065980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f740b0b80 0x7f8f7413a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 0x7f8f7413ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f741404b0 con 0x7f8f74142690 2022-07-12T08:51:02.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f74142440 con 0x7f8f740b0b80 2022-07-12T08:51:02.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f741406e0 con 0x7f8f7405fb90 2022-07-12T08:51:02.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 0x7f8f7413ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 0x7f8f7413ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33910/0 (socket says 172.21.15.61:33910) 2022-07-12T08:51:02.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 -- 172.21.15.61:0/4106714955 learned_addr learned my addr 172.21.15.61:0/4106714955 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:02.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7259c700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f7405fb90 0x7f8f74065980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f6bfff700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f740b0b80 0x7f8f7413a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 -- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f7405fb90 msgr2=0x7f8f74065980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f7405fb90 0x7f8f74065980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 -- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f740b0b80 msgr2=0x7f8f7413a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f740b0b80 0x7f8f7413a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 -- 172.21.15.61:0/4106714955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6400c200 con 0x7f8f74142690 2022-07-12T08:51:02.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f72d9d700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 0x7f8f7413ab70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f8f6c001490 tx=0x7f8f6c007e60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6c002310 con 0x7f8f74142690 2022-07-12T08:51:02.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c002470 con 0x7f8f74142690 2022-07-12T08:51:02.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f74141310 con 0x7f8f74142690 2022-07-12T08:51:02.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6c0017a0 con 0x7f8f74142690 2022-07-12T08:51:02.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.028+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7413e490 con 0x7f8f74142690 2022-07-12T08:51:02.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.032+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f741414a0 con 0x7f8f74142690 2022-07-12T08:51:02.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.032+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f6c019020 con 0x7f8f74142690 2022-07-12T08:51:02.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.032+0000 7f8f6b7fe700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f58053a40 0x7f8f58055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.032+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8f6c06bac0 con 0x7f8f74142690 2022-07-12T08:51:02.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.032+0000 7f8f7259c700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f58053a40 0x7f8f58055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.032+0000 7f8f7259c700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f58053a40 0x7f8f58055f00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f8f6401c000 tx=0x7f8f64016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.040+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f6c042b70 con 0x7f8f74142690 2022-07-12T08:51:02.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.164+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8f74057460 con 0x7f8f74142690 2022-07-12T08:51:02.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f6b7fe700 1 -- 172.21.15.61:0/4106714955 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v141) v1 ==== 81+0+28883 (secure 0 0 0) 0x7f8f6c046190 con 0x7f8f74142690 2022-07-12T08:51:02.176 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:02.176 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:51:02.176 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 141, 2022-07-12T08:51:02.177 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:51:02.177 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:51:02.177 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:50:59.396517+0000", 2022-07-12T08:51:02.177 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:51:02.177 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:51:02.178 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:51:02.178 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:51:02.178 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:51:02.178 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:51:02.179 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:51:02.179 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:51:02.179 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:51:02.179 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:02.179 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:51:02.180 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:51:02.180 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:51:02.180 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:51:02.181 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:51:02.182 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 28, 2022-07-12T08:51:02.182 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:51:02.182 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:51:02.183 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:51:02.183 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:51:02.183 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:51:02.183 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.184 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:51:02.184 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:51:02.184 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:51:02.184 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:51:02.184 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:51:02.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:51:02.185 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:51:02.185 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:51:02.185 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:51:02.186 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:51:02.186 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:51:02.186 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:51:02.186 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:51:02.187 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:51:02.187 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:51:02.187 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:51:02.187 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:51:02.187 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:51:02.188 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:51:02.188 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:51:02.188 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:51:02.188 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:51:02.189 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:51:02.189 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.189 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:51:02.189 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:51:02.190 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:51:02.190 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:51:02.190 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:51:02.190 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:51:02.190 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:51:02.191 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:51:02.191 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:51:02.191 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:51:02.191 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:51:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:51:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:51:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:51:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:51:02.193 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:51:02.193 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:51:02.193 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:51:02.193 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:51:02.194 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:51:02.194 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:51:02.194 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:51:02.194 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:51:02.195 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:51:02.195 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:51:02.195 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:51:02.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:51:02.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.196 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:51:02.196 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:51:02.196 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:51:02.196 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:51:02.197 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:51:02.197 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:51:02.197 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:51:02.197 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:51:02.197 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:51:02.198 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:51:02.198 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:51:02.198 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:51:02.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:51:02.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.199 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:51:02.199 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:51:02.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:51:02.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:51:02.201 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:51:02.201 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:51:02.201 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:51:02.201 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:51:02.202 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:51:02.202 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:51:02.202 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:51:02.202 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:51:02.203 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:51:02.213 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:51:02.213 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:51:02.214 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:51:02.214 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:51:02.215 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:51:02.215 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:51:02.215 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:51:02.215 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:51:02.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:51:02.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:51:02.216 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:51:02.217 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:51:02.217 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.217 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:51:02.218 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:51:02.218 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:51:02.218 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:51:02.219 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:51:02.219 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:51:02.219 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:51:02.220 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:51:02.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:51:02.220 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:51:02.221 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:51:02.221 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:51:02.221 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:51:02.221 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:51:02.222 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:51:02.222 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:51:02.223 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:51:02.223 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:51:02.223 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:51:02.223 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:51:02.224 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:51:02.224 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:51:02.224 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:51:02.225 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:51:02.225 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:51:02.225 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:51:02.226 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:51:02.226 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.226 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:51:02.227 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:51:02.227 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:51:02.227 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:51:02.228 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:51:02.228 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:51:02.228 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:51:02.230 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:51:02.231 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:51:02.231 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:51:02.231 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:51:02.232 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:51:02.232 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:51:02.233 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:51:02.233 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.233 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.234 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:02.236 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:51:02.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.237 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:51:02.237 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:51:02.237 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.238 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.238 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.239 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.240 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.240 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.241 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.241 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:51:02.242 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.243 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.244 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:51:02.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:51:02.251 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.251 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.252 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.252 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:51:02.254 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.255 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.255 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.256 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:51:02.256 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.257 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.258 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.258 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.260 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:51:02.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.261 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.261 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.262 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:51:02.262 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.263 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.263 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.264 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.265 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:51:02.266 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.267 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:51:02.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:02.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.268 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.269 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:51:02.269 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:51:02.270 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:51:02.270 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:51:02.271 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.271 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.271 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.272 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.273 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:51:02.273 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:51:02.274 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.274 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.275 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.275 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.276 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.276 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.277 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.277 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:51:02.278 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.278 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.279 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.280 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.280 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:51:02.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.281 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:51:02.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.284 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.284 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.286 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.286 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:51:02.286 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.287 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.288 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:51:02.288 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.289 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.289 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.290 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.291 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.291 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.292 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.292 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:51:02.293 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.293 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.293 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.294 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:51:02.295 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.296 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.296 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.297 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.298 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:51:02.299 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.300 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:51:02.301 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:02.302 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.302 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.303 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:51:02.304 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:51:02.304 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:51:02.304 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:51:02.305 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.305 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.306 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.306 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.307 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.308 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:51:02.308 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:51:02.308 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.309 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.309 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.310 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.311 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.311 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.311 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 135, 2022-07-12T08:51:02.312 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.312 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.313 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.314 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.314 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:51:02.315 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.316 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.317 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:51:02.317 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.317 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.318 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.319 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.319 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.320 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.320 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.320 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:51:02.321 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.322 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.322 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.322 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:51:02.322 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.323 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.323 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.323 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.323 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.323 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.324 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.324 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.324 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:51:02.324 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.325 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:51:02.325 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.326 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.326 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.326 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.326 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.327 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.327 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.328 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:51:02.328 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:51:02.330 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:02.330 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.330 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.331 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.331 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:51:02.332 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:51:02.332 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:51:02.332 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:51:02.332 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.333 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.333 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.333 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.334 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.334 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:51:02.334 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:51:02.335 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.335 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.335 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.335 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.336 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.336 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.336 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.336 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 125, 2022-07-12T08:51:02.336 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.337 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.337 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.338 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:51:02.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.339 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.340 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.341 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:51:02.341 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.341 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.342 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.343 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.343 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.344 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.344 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:51:02.345 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.345 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.346 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.346 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.347 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:51:02.347 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.347 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.348 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.348 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.348 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.349 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:51:02.350 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.350 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.351 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.351 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.352 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:51:02.352 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.353 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.353 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.354 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.355 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:51:02.356 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.358 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:51:02.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:02.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.360 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.360 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:51:02.361 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:51:02.361 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:51:02.362 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:51:02.362 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.363 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.363 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.363 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.364 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.364 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.365 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:51:02.365 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:51:02.366 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.366 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.367 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.367 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.368 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.369 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.369 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.369 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 135, 2022-07-12T08:51:02.370 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.370 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.371 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.372 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.372 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.372 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:51:02.373 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.373 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:51:02.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.375 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.376 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.376 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.376 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.377 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.378 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:51:02.378 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.379 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.379 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:51:02.380 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.380 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.381 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.381 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.382 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.382 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.383 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.383 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:51:02.384 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.384 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.384 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.385 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.385 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:51:02.386 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.386 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.386 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.387 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.387 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.388 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.388 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:51:02.389 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.390 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.390 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:51:02.390 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:02.390 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.391 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.391 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.391 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:51:02.391 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:51:02.392 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:51:02.392 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:51:02.392 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.392 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.393 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.393 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.393 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.394 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:51:02.394 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:51:02.395 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.395 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.396 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.396 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.396 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.396 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.396 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.397 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:51:02.397 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.397 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.397 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.398 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.398 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:51:02.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.399 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:51:02.400 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.400 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.401 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.401 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.401 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.402 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.402 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:51:02.402 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.403 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.403 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.403 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:51:02.403 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.403 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.404 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.404 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.404 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.405 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.405 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:51:02.405 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.406 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.406 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.406 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.406 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:51:02.406 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.407 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.407 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.407 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.407 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.407 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.408 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.408 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:51:02.409 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.409 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.409 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.409 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.409 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:51:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.410 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:51:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:51:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:51:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:51:02.411 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.412 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.412 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.412 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.412 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.413 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:51:02.413 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:51:02.413 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.413 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.414 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.414 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.415 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.415 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.415 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.416 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:51:02.416 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.416 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.417 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:51:02.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.418 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.418 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:51:02.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.419 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.419 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.420 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.420 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.420 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.420 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.420 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.421 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:51:02.421 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.421 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.422 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.422 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:51:02.422 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.422 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.422 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.423 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.423 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.423 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.423 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.424 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.424 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:51:02.424 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.424 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.424 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.425 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:51:02.425 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.425 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.426 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.426 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.426 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.427 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.427 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:51:02.427 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.427 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.428 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.428 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:51:02.428 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:02.429 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.429 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.429 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.429 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:51:02.429 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:51:02.430 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:51:02.430 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:51:02.430 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.430 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.431 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.431 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.431 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.431 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.431 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:51:02.432 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:51:02.432 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:02.432 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:02.433 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:02.433 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:02.433 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:02.433 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:02.434 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:02.434 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 136, 2022-07-12T08:51:02.434 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:02.434 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:02.434 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:02.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.435 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:51:02.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.436 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.436 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:51:02.437 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.437 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.437 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.438 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:02.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.439 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.439 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.439 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:51:02.439 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.439 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.440 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.440 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.440 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:51:02.440 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.441 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.441 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.441 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.442 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:02.442 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.442 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.442 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.443 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:51:02.443 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.443 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.443 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:51:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.445 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.445 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.445 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:02.445 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:02.446 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.446 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:02.446 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:51:02.446 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.447 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.447 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:02.448 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:51:02.448 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:02.448 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.448 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.448 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.449 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:51:02.449 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:51:02.449 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:51:02.450 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:51:02.450 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:02.450 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:02.450 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:02.451 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:02.451 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.451 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:02.451 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:51:02.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.452 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:51:02.452 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.452 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.453 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.453 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.453 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.453 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:51:02.454 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.454 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.454 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:51:02.454 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.455 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.455 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.455 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.455 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.456 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:51:02.456 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.456 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:51:02.457 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.457 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.457 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.457 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.458 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.458 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:51:02.458 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.458 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.459 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.459 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:51:02.459 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.459 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.459 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.460 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.460 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.460 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:51:02.460 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.461 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.461 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:51:02.461 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.462 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.462 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.462 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.462 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.462 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:51:02.463 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.463 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.464 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.464 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:51:02.464 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.465 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.465 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.465 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.466 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.466 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:51:02.467 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.467 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.468 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.468 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:51:02.468 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.469 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.469 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.470 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.470 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.471 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:51:02.471 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.471 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.472 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:02.472 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:51:02.473 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:02.473 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:02.474 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:02.474 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:02.475 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:02.475 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:51:02.476 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:02.476 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.477 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:02.477 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:51:02.478 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:51:02.478 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:51:02.479 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:51:02.479 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-12T08:51:02.480 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/676021650": "2022-07-13T08:50:55.783985+0000", 2022-07-12T08:51:02.480 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/676021650": "2022-07-13T08:50:55.783985+0000" 2022-07-12T08:51:02.481 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.481 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:51:02.481 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:51:02.482 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:51:02.482 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:51:02.483 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:51:02.483 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:51:02.484 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:51:02.484 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:51:02.485 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:02.485 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:02.486 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:51:02.486 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:51:02.487 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:51:02.487 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:51:02.488 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:51:02.488 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:51:02.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f58053a40 msgr2=0x7f8f58055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f58053a40 0x7f8f58055f00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f8f6401c000 tx=0x7f8f64016000).stop 2022-07-12T08:51:02.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 msgr2=0x7f8f7413ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 0x7f8f7413ab70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f8f6c001490 tx=0x7f8f6c007e60).stop 2022-07-12T08:51:02.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 shutdown_connections 2022-07-12T08:51:02.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f74142690 0x7f8f7413ab70 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f58053a40 0x7f8f58055f00 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f740b0b80 0x7f8f7413a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 --2- 172.21.15.61:0/4106714955 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f7405fb90 0x7f8f74065980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 >> 172.21.15.61:0/4106714955 conn(0x7f8f74063cd0 msgr2=0x7f8f7413fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:02.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 shutdown_connections 2022-07-12T08:51:02.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.168+0000 7f8f7359e700 1 -- 172.21.15.61:0/4106714955 wait complete. 2022-07-12T08:51:02.497 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:02.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.608+0000 7f8e3ba92700 1 Processor -- start 2022-07-12T08:51:02.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.608+0000 7f8e3ba92700 1 -- start start 2022-07-12T08:51:02.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e340b06b0 0x7f8e3405c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e34140150 0x7f8e341405e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e34141f90 con 0x7f8e34064850 2022-07-12T08:51:02.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e34064150 con 0x7f8e34140150 2022-07-12T08:51:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3405c580 con 0x7f8e340b06b0 2022-07-12T08:51:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3982e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e340b06b0 0x7f8e3405c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3902d700 1 --1- >> v1:172.21.15.111:6789/0 conn(0x7f8e34064850 0x7f8e34144320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:35736/0 (socket says 172.21.15.61:35736) 2022-07-12T08:51:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3902d700 1 -- 172.21.15.61:0/1422356953 learned_addr learned my addr 172.21.15.61:0/1422356953 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:02.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3a02f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8e34140150 0x7f8e341405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3882c700 1 -- 172.21.15.61:0/1422356953 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635234746 0 0) 0x7f8e34141f90 con 0x7f8e34064850 2022-07-12T08:51:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3882c700 1 -- 172.21.15.61:0/1422356953 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e20003260 con 0x7f8e34064850 2022-07-12T08:51:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3982e700 1 -- 172.21.15.61:0/1422356953 >> v2:172.21.15.111:3300/0 conn(0x7f8e34140150 msgr2=0x7f8e341405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3982e700 1 --2- 172.21.15.61:0/1422356953 >> v2:172.21.15.111:3300/0 conn(0x7f8e34140150 0x7f8e341405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3982e700 1 -- 172.21.15.61:0/1422356953 >> v1:172.21.15.111:6789/0 conn(0x7f8e34064850 legacy=0x7f8e34144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3982e700 1 -- 172.21.15.61:0/1422356953 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e340643b0 con 0x7f8e340b06b0 2022-07-12T08:51:02.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3982e700 1 --2- 172.21.15.61:0/1422356953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e340b06b0 0x7f8e3405c940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8e240032f0 tx=0x7f8e240091d0).ready entity=mon.2 client_cookie=dacc0befa91ab44c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3882c700 1 -- 172.21.15.61:0/1422356953 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e2400b600 con 0x7f8e340b06b0 2022-07-12T08:51:02.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3882c700 1 -- 172.21.15.61:0/1422356953 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e2400b760 con 0x7f8e340b06b0 2022-07-12T08:51:02.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3882c700 1 -- 172.21.15.61:0/1422356953 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e2400b8d0 con 0x7f8e340b06b0 2022-07-12T08:51:02.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- 172.21.15.61:0/1422356953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e340b06b0 msgr2=0x7f8e3405c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/1422356953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e340b06b0 0x7f8e3405c940 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f8e240032f0 tx=0x7f8e240091d0).stop 2022-07-12T08:51:02.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- 172.21.15.61:0/1422356953 shutdown_connections 2022-07-12T08:51:02.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/1422356953 >> v2:172.21.15.111:3300/0 conn(0x7f8e34140150 0x7f8e341405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/1422356953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e340b06b0 0x7f8e3405c940 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- 172.21.15.61:0/1422356953 >> 172.21.15.61:0/1422356953 conn(0x7f8e34062c00 msgr2=0x7f8e34062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:02.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- 172.21.15.61:0/1422356953 shutdown_connections 2022-07-12T08:51:02.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.612+0000 7f8e3ba92700 1 -- 172.21.15.61:0/1422356953 wait complete. 2022-07-12T08:51:02.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 Processor -- start 2022-07-12T08:51:02.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 -- start start 2022-07-12T08:51:02.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e34064850 0x7f8e341434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 0x7f8e340a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e34140150 0x7f8e3414fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e340a1240 con 0x7f8e340b06b0 2022-07-12T08:51:02.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e340a2740 con 0x7f8e34064850 2022-07-12T08:51:02.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e340a29e0 con 0x7f8e34140150 2022-07-12T08:51:02.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 0x7f8e340a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 0x7f8e340a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33922/0 (socket says 172.21.15.61:33922) 2022-07-12T08:51:02.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 -- 172.21.15.61:0/4182144594 learned_addr learned my addr 172.21.15.61:0/4182144594 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:02.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3982e700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e34064850 0x7f8e341434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3a02f700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e34140150 0x7f8e3414fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 -- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e34140150 msgr2=0x7f8e3414fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e34140150 0x7f8e3414fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 -- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e34064850 msgr2=0x7f8e341434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e34064850 0x7f8e341434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 -- 172.21.15.61:0/4182144594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e24009470 con 0x7f8e340b06b0 2022-07-12T08:51:02.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3902d700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 0x7f8e340a2e80 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f8e28001110 tx=0x7f8e28007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e2800ce20 con 0x7f8e340b06b0 2022-07-12T08:51:02.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e280072e0 con 0x7f8e340b06b0 2022-07-12T08:51:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e340a3370 con 0x7f8e340b06b0 2022-07-12T08:51:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8e28005800 con 0x7f8e340b06b0 2022-07-12T08:51:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.616+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e340a0e10 con 0x7f8e340b06b0 2022-07-12T08:51:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.620+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e340a01d0 con 0x7f8e340b06b0 2022-07-12T08:51:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.620+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8e28010070 con 0x7f8e340b06b0 2022-07-12T08:51:02.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.620+0000 7f8e32ffd700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e14053940 0x7f8e14055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.620+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8e2806b2b0 con 0x7f8e340b06b0 2022-07-12T08:51:02.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.620+0000 7f8e3982e700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e14053940 0x7f8e14055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.620+0000 7f8e3982e700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e14053940 0x7f8e14055e00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f8e2401b000 tx=0x7f8e24016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.624+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8e340a01d0 con 0x7f8e340b06b0 2022-07-12T08:51:02.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e340a01d0 con 0x7f8e340b06b0 2022-07-12T08:51:02.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e32ffd700 1 -- 172.21.15.61:0/4182144594 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f8e340a01d0 con 0x7f8e340b06b0 2022-07-12T08:51:02.805 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:02.805 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8688,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3171104982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3171104982},{"type":"v1","addr":"172.21.15.61:6833","nonce":3171104982}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[]} 2022-07-12T08:51:02.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e14053940 msgr2=0x7f8e14055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e14053940 0x7f8e14055e00 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f8e2401b000 tx=0x7f8e24016000).stop 2022-07-12T08:51:02.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 msgr2=0x7f8e340a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 0x7f8e340a2e80 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f8e28001110 tx=0x7f8e28007b20).stop 2022-07-12T08:51:02.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 shutdown_connections 2022-07-12T08:51:02.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8e34140150 0x7f8e3414fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8e340b06b0 0x7f8e340a2e80 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8e14053940 0x7f8e14055e00 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 --2- 172.21.15.61:0/4182144594 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8e34064850 0x7f8e341434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 >> 172.21.15.61:0/4182144594 conn(0x7f8e34062c00 msgr2=0x7f8e340b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:02.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 shutdown_connections 2022-07-12T08:51:02.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.800+0000 7f8e3ba92700 1 -- 172.21.15.61:0/4182144594 wait complete. 2022-07-12T08:51:02.815 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 79 2022-07-12T08:51:02.817 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:02.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 Processor -- start 2022-07-12T08:51:02.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- start start 2022-07-12T08:51:02.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ab290 0x7fb3f40ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f405f140 0x7fb3f405ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb3f40afa60 0x7fb3f40afef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f405f580 con 0x7fb3f40ab290 2022-07-12T08:51:02.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f40abc70 con 0x7fb3f40afa60 2022-07-12T08:51:02.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f405c720 con 0x7fb3f405f140 2022-07-12T08:51:02.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f8e31700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ab290 0x7fb3f40ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f8e31700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ab290 0x7fb3f40ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33930/0 (socket says 172.21.15.61:33930) 2022-07-12T08:51:02.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f8e31700 1 -- 172.21.15.61:0/12600398 learned_addr learned my addr 172.21.15.61:0/12600398 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:02.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/12600398 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f405f140 0x7fb3f405ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9632700 1 --2- 172.21.15.61:0/12600398 >> v2:172.21.15.111:3300/0 conn(0x7fb3f40afa60 0x7fb3f40afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 -- 172.21.15.61:0/12600398 >> v2:172.21.15.111:3300/0 conn(0x7fb3f40afa60 msgr2=0x7fb3f40afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:02.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/12600398 >> v2:172.21.15.111:3300/0 conn(0x7fb3f40afa60 0x7fb3f40afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 -- 172.21.15.61:0/12600398 >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ab290 msgr2=0x7fb3f40ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:02.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/12600398 >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ab290 0x7fb3f40ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 -- 172.21.15.61:0/12600398 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f405c9c0 con 0x7fb3f405f140 2022-07-12T08:51:02.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/12600398 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f405f140 0x7fb3f405ce60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb3dc002f70 tx=0x7fb3dc006df0).ready entity=mon.2 client_cookie=4df9807036d9c6f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f37fe700 1 -- 172.21.15.61:0/12600398 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc00d4a0 con 0x7fb3f405f140 2022-07-12T08:51:02.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f37fe700 1 -- 172.21.15.61:0/12600398 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc000e90 con 0x7fb3f405f140 2022-07-12T08:51:02.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f37fe700 1 -- 172.21.15.61:0/12600398 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc00d7f0 con 0x7fb3f405f140 2022-07-12T08:51:02.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- 172.21.15.61:0/12600398 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f405f140 msgr2=0x7fb3f405ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/12600398 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f405f140 0x7fb3f405ce60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb3dc002f70 tx=0x7fb3dc006df0).stop 2022-07-12T08:51:02.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- 172.21.15.61:0/12600398 shutdown_connections 2022-07-12T08:51:02.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/12600398 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f405f140 0x7fb3f405ce60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/12600398 >> v2:172.21.15.111:3300/0 conn(0x7fb3f40afa60 0x7fb3f40afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/12600398 >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ab290 0x7fb3f40ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- 172.21.15.61:0/12600398 >> 172.21.15.61:0/12600398 conn(0x7fb3f40b1950 msgr2=0x7fb3f40b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:02.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- 172.21.15.61:0/12600398 shutdown_connections 2022-07-12T08:51:02.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.932+0000 7fb3f9e33700 1 -- 172.21.15.61:0/12600398 wait complete. 2022-07-12T08:51:02.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 Processor -- start 2022-07-12T08:51:02.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 -- start start 2022-07-12T08:51:02.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 0x7fb3f4066710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f40ab290 0x7fb3f40b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3f40afa60 0x7fb3f40aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40af290 con 0x7fb3f405f140 2022-07-12T08:51:02.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4065d30 con 0x7fb3f40afa60 2022-07-12T08:51:02.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4065fd0 con 0x7fb3f40ab290 2022-07-12T08:51:02.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 0x7fb3f4066710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f3fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f40ab290 0x7fb3f40b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 0x7fb3f4066710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33936/0 (socket says 172.21.15.61:33936) 2022-07-12T08:51:02.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9632700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3f40afa60 0x7fb3f40aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 -- 172.21.15.61:0/1503209757 learned_addr learned my addr 172.21.15.61:0/1503209757 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:02.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 -- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f40ab290 msgr2=0x7fb3f40b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f40ab290 0x7fb3f40b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 -- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3f40afa60 msgr2=0x7fb3f40aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:02.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3f40afa60 0x7fb3f40aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:02.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 -- 172.21.15.61:0/1503209757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc00b180 con 0x7fb3f405f140 2022-07-12T08:51:02.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f8e31700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 0x7fb3f4066710 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fb3ec006ec0 tx=0x7fb3ec009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3ec009680 con 0x7fb3f405f140 2022-07-12T08:51:02.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec00c850 con 0x7fb3f405f140 2022-07-12T08:51:02.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f4066270 con 0x7fb3f405f140 2022-07-12T08:51:02.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3ec00cbb0 con 0x7fb3f405f140 2022-07-12T08:51:02.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.936+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f40ae3e0 con 0x7fb3f405f140 2022-07-12T08:51:02.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.940+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4057460 con 0x7fb3f405f140 2022-07-12T08:51:02.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.940+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb3ec022070 con 0x7fb3f405f140 2022-07-12T08:51:02.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.940+0000 7fb3f1ffb700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3d80539b0 0x7fb3d8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:02.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.940+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb3ec06c380 con 0x7fb3f405f140 2022-07-12T08:51:02.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.940+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3d80539b0 0x7fb3d8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:02.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.940+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3d80539b0 0x7fb3d8055e70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fb3dc01a000 tx=0x7fb3dc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:02.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:02.948+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3ec043450 con 0x7fb3f405f140 2022-07-12T08:51:03.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f406b2b0 con 0x7fb3f405f140 2022-07-12T08:51:03.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f1ffb700 1 -- 172.21.15.61:0/1503209757 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+1212 (secure 0 0 0) 0x7fb3ec046a70 con 0x7fb3f405f140 2022-07-12T08:51:03.115 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:03.116 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8260,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/2838716448","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":2838716448},{"type":"v1","addr":"172.21.15.111:6835","nonce":2838716448}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8688,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3171104982","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3171104982},{"type":"v1","addr":"172.21.15.61:6833","nonce":3171104982}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":79}],"filesystems":[]} 2022-07-12T08:51:03.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3d80539b0 msgr2=0x7fb3d8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:03.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3d80539b0 0x7fb3d8055e70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7fb3dc01a000 tx=0x7fb3dc014000).stop 2022-07-12T08:51:03.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 msgr2=0x7fb3f4066710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:03.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 0x7fb3f4066710 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7fb3ec006ec0 tx=0x7fb3ec009290).stop 2022-07-12T08:51:03.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 shutdown_connections 2022-07-12T08:51:03.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb3d80539b0 0x7fb3d8055e70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3f40afa60 0x7fb3f40aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb3f40ab290 0x7fb3f40b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 --2- 172.21.15.61:0/1503209757 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f405f140 0x7fb3f4066710 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 >> 172.21.15.61:0/1503209757 conn(0x7fb3f40b1950 msgr2=0x7fb3f4099b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:03.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 shutdown_connections 2022-07-12T08:51:03.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.112+0000 7fb3f9e33700 1 -- 172.21.15.61:0/1503209757 wait complete. 2022-07-12T08:51:03.124 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 79 2022-07-12T08:51:03.128 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-12T08:51:03.129 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-12T08:51:03.129 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T08:51:03.130 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:51:03.130 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-12T08:51:03.131 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-12T08:51:03.132 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T08:51:03.132 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T08:51:03.144 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T08:51:03.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-12T08:51:03.152 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T08:51:03.152 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-12T08:51:03.155 INFO:tasks.ceph.mds.b:Started 2022-07-12T08:51:03.156 DEBUG:teuthology.parallel:result is None 2022-07-12T08:51:03.191 INFO:tasks.ceph.mds.a:Started 2022-07-12T08:51:03.192 DEBUG:teuthology.parallel:result is None 2022-07-12T08:51:03.193 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-12T08:51:03.236 INFO:tasks.ceph.mds.b.smithi111.stdout:starting mds.b at 2022-07-12T08:51:03.265 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:51:03.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 Processor -- start 2022-07-12T08:51:03.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 -- start start 2022-07-12T08:51:03.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fca480b0120 0x7fca480aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:03.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca4813f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:03.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480acee0 con 0x7fca480b1320 2022-07-12T08:51:03.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca480ad180 con 0x7fca480b0120 2022-07-12T08:51:03.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca4cc7b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca480acb60 con 0x7fca4813ec70 2022-07-12T08:51:03.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca477fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fca480b1320 0x7fca480ae500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50410/0 (socket says 172.21.15.61:50410) 2022-07-12T08:51:03.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca477fe700 1 -- 172.21.15.61:0/2775553499 learned_addr learned my addr 172.21.15.61:0/2775553499 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:03.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca47fff700 1 --2- 172.21.15.61:0/2775553499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca4813f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:03.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca46ffd700 1 --2- 172.21.15.61:0/2775553499 >> v2:172.21.15.111:3300/0 conn(0x7fca480b0120 0x7fca480aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:03.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca467fc700 1 -- 172.21.15.61:0/2775553499 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769949775 0 0) 0x7fca480acee0 con 0x7fca480b1320 2022-07-12T08:51:03.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca467fc700 1 -- 172.21.15.61:0/2775553499 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca34003260 con 0x7fca480b1320 2022-07-12T08:51:03.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca467fc700 1 -- 172.21.15.61:0/2775553499 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327391591 0 0) 0x7fca34003260 con 0x7fca480b1320 2022-07-12T08:51:03.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca47fff700 1 -- 172.21.15.61:0/2775553499 >> v2:172.21.15.111:3300/0 conn(0x7fca480b0120 msgr2=0x7fca480aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:03.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca47fff700 1 --2- 172.21.15.61:0/2775553499 >> v2:172.21.15.111:3300/0 conn(0x7fca480b0120 0x7fca480aec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca47fff700 1 -- 172.21.15.61:0/2775553499 >> v1:172.21.15.61:6789/0 conn(0x7fca480b1320 legacy=0x7fca480ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:03.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca47fff700 1 -- 172.21.15.61:0/2775553499 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca480b0630 con 0x7fca4813ec70 2022-07-12T08:51:03.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca47fff700 1 --2- 172.21.15.61:0/2775553499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca4813f120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fca38001800 tx=0x7fca400091d0).ready entity=mon.2 client_cookie=26bc6a1d9cc4700f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:03.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.300+0000 7fca467fc700 1 -- 172.21.15.61:0/2775553499 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca38001660 con 0x7fca4813ec70 2022-07-12T08:51:03.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca467fc700 1 -- 172.21.15.61:0/2775553499 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4000f510 con 0x7fca4813ec70 2022-07-12T08:51:03.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca467fc700 1 -- 172.21.15.61:0/2775553499 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca4000f680 con 0x7fca4813ec70 2022-07-12T08:51:03.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 -- 172.21.15.61:0/2775553499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 msgr2=0x7fca4813f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:03.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/2775553499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca4813f120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fca38001800 tx=0x7fca400091d0).stop 2022-07-12T08:51:03.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 -- 172.21.15.61:0/2775553499 shutdown_connections 2022-07-12T08:51:03.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/2775553499 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca4813f120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/2775553499 >> v2:172.21.15.111:3300/0 conn(0x7fca480b0120 0x7fca480aec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 -- 172.21.15.61:0/2775553499 >> 172.21.15.61:0/2775553499 conn(0x7fca480b5780 msgr2=0x7fca480b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:03.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 -- 172.21.15.61:0/2775553499 shutdown_connections 2022-07-12T08:51:03.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 -- 172.21.15.61:0/2775553499 wait complete. 2022-07-12T08:51:03.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 Processor -- start 2022-07-12T08:51:03.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.304+0000 7fca4cc7b700 1 -- start start 2022-07-12T08:51:03.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 0x7fca48142090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:03.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca480b1320 0x7fca480664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:03.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca48064510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:03.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca48141e60 con 0x7fca480b0120 2022-07-12T08:51:03.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca480671d0 con 0x7fca480b1320 2022-07-12T08:51:03.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca48141a60 con 0x7fca4813ec70 2022-07-12T08:51:03.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 0x7fca48142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:03.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 0x7fca48142090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33966/0 (socket says 172.21.15.61:33966) 2022-07-12T08:51:03.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 -- 172.21.15.61:0/746926821 learned_addr learned my addr 172.21.15.61:0/746926821 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:03.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca47fff700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca48064510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:03.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca46ffd700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca480b1320 0x7fca480664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:03.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 -- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 msgr2=0x7fca48064510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:03.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca48064510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 -- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca480b1320 msgr2=0x7fca480664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:03.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca480b1320 0x7fca480664e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:03.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 -- 172.21.15.61:0/746926821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca40009470 con 0x7fca480b0120 2022-07-12T08:51:03.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca477fe700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 0x7fca48142090 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7fca380018c0 tx=0x7fca38001ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:03.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca38007170 con 0x7fca480b0120 2022-07-12T08:51:03.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca380072d0 con 0x7fca480b0120 2022-07-12T08:51:03.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca48064a00 con 0x7fca480b0120 2022-07-12T08:51:03.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca38004920 con 0x7fca480b0120 2022-07-12T08:51:03.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca48142a00 con 0x7fca480b0120 2022-07-12T08:51:03.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.308+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48067390 con 0x7fca480b0120 2022-07-12T08:51:03.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.312+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fca3801b070 con 0x7fca480b0120 2022-07-12T08:51:03.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.312+0000 7fca44ff9700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca280539b0 0x7fca28055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:03.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.312+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fca3806b360 con 0x7fca480b0120 2022-07-12T08:51:03.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.312+0000 7fca46ffd700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca280539b0 0x7fca28055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:03.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.312+0000 7fca46ffd700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca280539b0 0x7fca28055e70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fca3c00b000 tx=0x7fca3c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:03.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.316+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca38042540 con 0x7fca480b0120 2022-07-12T08:51:03.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:03.451+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fca48057460 con 0x7fca480b0120 2022-07-12T08:51:03.745 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:03.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:03.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:03.760 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:03.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:04.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.451+0000 7fca44ff9700 1 -- 172.21.15.61:0/746926821 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v143) v1 ==== 97+0+0 (secure 0 0 0) 0x7fca38045b60 con 0x7fca480b0120 2022-07-12T08:51:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca280539b0 msgr2=0x7fca28055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca280539b0 0x7fca28055e70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fca3c00b000 tx=0x7fca3c006df0).stop 2022-07-12T08:51:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 msgr2=0x7fca48142090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 0x7fca48142090 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7fca380018c0 tx=0x7fca38001ac0).stop 2022-07-12T08:51:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 shutdown_connections 2022-07-12T08:51:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fca4813ec70 0x7fca48064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fca280539b0 0x7fca28055e70 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca480b1320 0x7fca480664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 --2- 172.21.15.61:0/746926821 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca480b0120 0x7fca48142090 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 >> 172.21.15.61:0/746926821 conn(0x7fca480b5780 msgr2=0x7fca48144660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 shutdown_connections 2022-07-12T08:51:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.455+0000 7fca4cc7b700 1 -- 172.21.15.61:0/746926821 wait complete. 2022-07-12T08:51:04.464 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-12T08:51:04.479 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-12T08:51:04.480 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-12T08:51:04.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.587+0000 7f2264fc8700 1 Processor -- start 2022-07-12T08:51:04.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.587+0000 7f2264fc8700 1 -- start start 2022-07-12T08:51:04.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f22600b0380 0x7f2260144140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226005d3d0 con 0x7f226005bfc0 2022-07-12T08:51:04.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600b5e80 con 0x7f22600b5aa0 2022-07-12T08:51:04.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226014b590 con 0x7f22600b0380 2022-07-12T08:51:04.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225dd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f226005bfc0 0x7f2260147000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50426/0 (socket says 172.21.15.61:50426) 2022-07-12T08:51:04.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225dd9b700 1 -- 172.21.15.61:0/3848520187 learned_addr learned my addr 172.21.15.61:0/3848520187 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:04.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225ed9d700 1 --2- 172.21.15.61:0/3848520187 >> v2:172.21.15.111:3300/0 conn(0x7f22600b0380 0x7f2260144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225d59a700 1 -- 172.21.15.61:0/3848520187 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240344699 0 0) 0x7f226005d3d0 con 0x7f226005bfc0 2022-07-12T08:51:04.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225d59a700 1 -- 172.21.15.61:0/3848520187 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2244003260 con 0x7f226005bfc0 2022-07-12T08:51:04.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225ed9d700 1 -- 172.21.15.61:0/3848520187 >> v1:172.21.15.111:6789/0 conn(0x7f22600b5aa0 legacy=0x7f226014ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225ed9d700 1 -- 172.21.15.61:0/3848520187 >> v1:172.21.15.61:6789/0 conn(0x7f226005bfc0 legacy=0x7f2260147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225ed9d700 1 -- 172.21.15.61:0/3848520187 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226014a740 con 0x7f22600b0380 2022-07-12T08:51:04.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225ed9d700 1 --2- 172.21.15.61:0/3848520187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f2260144140 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f2250002f70 tx=0x7f2250006df0).ready entity=mon.1 client_cookie=f15729a4dd854263 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:04.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225d59a700 1 -- 172.21.15.61:0/3848520187 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f225000d610 con 0x7f22600b0380 2022-07-12T08:51:04.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225d59a700 1 -- 172.21.15.61:0/3848520187 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2250000e90 con 0x7f22600b0380 2022-07-12T08:51:04.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f225d59a700 1 -- 172.21.15.61:0/3848520187 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f225000d960 con 0x7f22600b0380 2022-07-12T08:51:04.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- 172.21.15.61:0/3848520187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 msgr2=0x7f2260144140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 --2- 172.21.15.61:0/3848520187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f2260144140 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f2250002f70 tx=0x7f2250006df0).stop 2022-07-12T08:51:04.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- 172.21.15.61:0/3848520187 shutdown_connections 2022-07-12T08:51:04.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 --2- 172.21.15.61:0/3848520187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f2260144140 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- 172.21.15.61:0/3848520187 >> 172.21.15.61:0/3848520187 conn(0x7f2260063520 msgr2=0x7f22600b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:04.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- 172.21.15.61:0/3848520187 shutdown_connections 2022-07-12T08:51:04.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.591+0000 7f2264fc8700 1 -- 172.21.15.61:0/3848520187 wait complete. 2022-07-12T08:51:04.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 Processor -- start 2022-07-12T08:51:04.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 -- start start 2022-07-12T08:51:04.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f226005bfc0 0x7f22601563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f226014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 0x7f2260063980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260065c00 con 0x7f22600b5aa0 2022-07-12T08:51:04.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260065ea0 con 0x7f22600b0380 2022-07-12T08:51:04.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f2264fc8700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2260066140 con 0x7f226005bfc0 2022-07-12T08:51:04.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 0x7f2260063980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 0x7f2260063980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33984/0 (socket says 172.21.15.61:33984) 2022-07-12T08:51:04.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225dd9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f226014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225e59c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f226005bfc0 0x7f22601563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 -- 172.21.15.61:0/1680490505 learned_addr learned my addr 172.21.15.61:0/1680490505 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 -- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f226005bfc0 msgr2=0x7f22601563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f226005bfc0 0x7f22601563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 -- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 msgr2=0x7f226014c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f226014c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 -- 172.21.15.61:0/1680490505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225000b180 con 0x7f22600b5aa0 2022-07-12T08:51:04.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f225ed9d700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 0x7f2260063980 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f2250002f70 tx=0x7f2250014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:04.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.595+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2250016070 con 0x7f22600b5aa0 2022-07-12T08:51:04.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225000d290 con 0x7f22600b5aa0 2022-07-12T08:51:04.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2260065590 con 0x7f22600b5aa0 2022-07-12T08:51:04.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2250013bf0 con 0x7f22600b5aa0 2022-07-12T08:51:04.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2260157520 con 0x7f22600b5aa0 2022-07-12T08:51:04.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2260066300 con 0x7f22600b5aa0 2022-07-12T08:51:04.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2250025030 con 0x7f22600b5aa0 2022-07-12T08:51:04.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f224f7fe700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f223c053ad0 0x7f223c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f225006cb90 con 0x7f22600b5aa0 2022-07-12T08:51:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.599+0000 7f225e59c700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f223c053ad0 0x7f223c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.603+0000 7f225e59c700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f223c053ad0 0x7f223c055f90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f225800d000 tx=0x7f2258007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:04.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.611+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2250043d80 con 0x7f22600b5aa0 2022-07-12T08:51:04.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.783+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2260057460 con 0x7f22600b5aa0 2022-07-12T08:51:04.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.783+0000 7f224f7fe700 1 -- 172.21.15.61:0/1680490505 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f2250023020 con 0x7f22600b5aa0 2022-07-12T08:51:04.796 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:04.797 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:51:04.797 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-12T08:51:04.797 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.798 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-12T08:51:04.798 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAYQyEBAARGKo1P3Ch/pggCgNXwqzZA==", 2022-07-12T08:51:04.799 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.799 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:51:04.799 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.800 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.800 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.800 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.801 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.801 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-12T08:51:04.801 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81igbHDFhAAjLFLSbdKlXe1KsUZnItTSQ==", 2022-07-12T08:51:04.802 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.802 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:51:04.802 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.802 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.803 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.803 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.803 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.803 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.804 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-12T08:51:04.804 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCEM81iMHnSIhAAGJJrlXvar1GJZqqgpvBTIw==", 2022-07-12T08:51:04.804 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.804 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.805 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.805 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.805 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.805 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.806 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.806 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-12T08:51:04.806 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i/HgtARAARc69sulFOHvg7NUyCPAaLA==", 2022-07-12T08:51:04.806 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.807 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.807 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.807 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.807 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.807 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.808 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.808 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-12T08:51:04.808 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81iEZtfFxAA51xjXxiBrfplFFSCYZRVHQ==", 2022-07-12T08:51:04.808 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-12T08:51:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCFM81i3NxQJxAA/J4DSnFYgED43ACNEZHdnQ==", 2022-07-12T08:51:04.811 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.812 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.812 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.812 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.813 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.813 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.813 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-12T08:51:04.813 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCIM81i0s2WMRAAPq1tZXBiSTc7Zi51AN44Fg==", 2022-07-12T08:51:04.814 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.814 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.814 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.815 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.815 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.815 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.815 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.816 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-12T08:51:04.816 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iQr/IERAAc442pIAiI296lSsKrt1GYA==", 2022-07-12T08:51:04.816 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.816 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.817 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.817 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.817 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.817 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.818 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.818 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-12T08:51:04.818 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCJM81iqIdmLBAAGFFXc2AdDCzxyR3nzSOiUQ==", 2022-07-12T08:51:04.818 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.819 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.819 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.819 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.819 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.819 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.820 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.820 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-12T08:51:04.820 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCKM81iFD1bBRAAkcSOdJMZO7nFS4Ai+gjp3w==", 2022-07-12T08:51:04.820 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.821 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.821 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.821 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.821 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.822 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.822 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.822 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-12T08:51:04.823 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iAfW/HBAAduXrmHLRKMHO9DvQGQiZ7g==", 2022-07-12T08:51:04.823 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.823 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:51:04.823 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:51:04.824 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:51:04.824 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.824 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.825 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-12T08:51:04.825 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81i6rHVIRAA8Yksf608IciS6xOr66O8pw==", 2022-07-12T08:51:04.825 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.825 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:51:04.826 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:51:04.826 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:51:04.826 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.827 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.827 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-12T08:51:04.827 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iNMYQJxAAAPKduSWTUh2g8EDl4BQY7A==", 2022-07-12T08:51:04.827 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.827 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-12T08:51:04.828 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-12T08:51:04.828 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-12T08:51:04.828 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.829 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.829 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-12T08:51:04.829 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCAM81i0gHdHhAAgab/9YBKVqpfoaxOc4vpIQ==", 2022-07-12T08:51:04.829 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.829 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:51:04.830 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-12T08:51:04.830 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-12T08:51:04.830 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.830 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.831 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.831 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.831 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-12T08:51:04.831 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iJV+gCxAAoMvnAkD51YqrqUanZ9lWxA==", 2022-07-12T08:51:04.831 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.832 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:51:04.832 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:51:04.832 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.832 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.833 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:04.833 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:04.833 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-12T08:51:04.833 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQCBM81iwp3pBxAAtusaBPljAXsNj21JXpoKbg==", 2022-07-12T08:51:04.833 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-12T08:51:04.834 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-12T08:51:04.834 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-12T08:51:04.834 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-12T08:51:04.834 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.835 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:04.835 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:04.835 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:51:04.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f223c053ad0 msgr2=0x7f223c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f223c053ad0 0x7f223c055f90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f225800d000 tx=0x7f2258007f10).stop 2022-07-12T08:51:04.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 msgr2=0x7f2260063980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 0x7f2260063980 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f2250002f70 tx=0x7f2250014000).stop 2022-07-12T08:51:04.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 shutdown_connections 2022-07-12T08:51:04.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22600b5aa0 0x7f2260063980 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f223c053ad0 0x7f223c055f90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f22600b0380 0x7f226014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 --2- 172.21.15.61:0/1680490505 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f226005bfc0 0x7f22601563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 >> 172.21.15.61:0/1680490505 conn(0x7f2260063520 msgr2=0x7f2260147150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:04.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 shutdown_connections 2022-07-12T08:51:04.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.787+0000 7f2264fc8700 1 -- 172.21.15.61:0/1680490505 wait complete. 2022-07-12T08:51:04.840 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:51:04.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 Processor -- start 2022-07-12T08:51:04.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 -- start start 2022-07-12T08:51:04.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9ec0140160 0x7f9ec013ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec005f860 0x7f9ec0141410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec0140680 con 0x7f9ec005f480 2022-07-12T08:51:04.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec0140cd0 con 0x7f9ec0140160 2022-07-12T08:51:04.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0140f70 con 0x7f9ec005f860 2022-07-12T08:51:04.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebe7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9ec0140160 0x7f9ec013ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebeffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9ec005f480 0x7f9ec013c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50438/0 (socket says 172.21.15.61:50438) 2022-07-12T08:51:04.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebeffd700 1 -- 172.21.15.61:0/657606068 learned_addr learned my addr 172.21.15.61:0/657606068 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:04.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebf7fe700 1 --2- 172.21.15.61:0/657606068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec005f860 0x7f9ec0141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebe7fc700 1 -- 172.21.15.61:0/657606068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec005f860 msgr2=0x7f9ec0141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:04.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebe7fc700 1 --2- 172.21.15.61:0/657606068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec005f860 0x7f9ec0141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebe7fc700 1 -- 172.21.15.61:0/657606068 >> v1:172.21.15.61:6789/0 conn(0x7f9ec005f480 legacy=0x7f9ec013c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebe7fc700 1 -- 172.21.15.61:0/657606068 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec005fca0 con 0x7f9ec0140160 2022-07-12T08:51:04.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebdffb700 1 -- 172.21.15.61:0/657606068 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633097283 0 0) 0x7f9ec0140680 con 0x7f9ec005f480 2022-07-12T08:51:04.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebe7fc700 1 --2- 172.21.15.61:0/657606068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec0140160 0x7f9ec013ce20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f9eb00032f0 tx=0x7f9eb000b470).ready entity=mon.0 client_cookie=81b3d5098401c1d5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:04.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebdffb700 1 -- 172.21.15.61:0/657606068 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb000de20 con 0x7f9ec0140160 2022-07-12T08:51:04.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebdffb700 1 -- 172.21.15.61:0/657606068 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0010470 con 0x7f9ec0140160 2022-07-12T08:51:04.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebdffb700 1 -- 172.21.15.61:0/657606068 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb00107d0 con 0x7f9ec0140160 2022-07-12T08:51:04.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 -- 172.21.15.61:0/657606068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec0140160 msgr2=0x7f9ec013ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.943+0000 7f9ebffff700 1 --2- 172.21.15.61:0/657606068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec0140160 0x7f9ec013ce20 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7f9eb00032f0 tx=0x7f9eb000b470).stop 2022-07-12T08:51:04.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- 172.21.15.61:0/657606068 shutdown_connections 2022-07-12T08:51:04.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 --2- 172.21.15.61:0/657606068 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec005f860 0x7f9ec0141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 --2- 172.21.15.61:0/657606068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec0140160 0x7f9ec013ce20 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- 172.21.15.61:0/657606068 >> 172.21.15.61:0/657606068 conn(0x7f9ec0063a60 msgr2=0x7f9ec0063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:04.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- 172.21.15.61:0/657606068 shutdown_connections 2022-07-12T08:51:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- 172.21.15.61:0/657606068 wait complete. 2022-07-12T08:51:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 Processor -- start 2022-07-12T08:51:04.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- start start 2022-07-12T08:51:04.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 0x7f9ec0065490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec005f860 0x7f9ec0065980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec0140160 0x7f9ec0141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01424d0 con 0x7f9ec005f480 2022-07-12T08:51:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0152660 con 0x7f9ec005f860 2022-07-12T08:51:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebffff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec0152900 con 0x7f9ec0140160 2022-07-12T08:51:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 0x7f9ec0065490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 0x7f9ec0065490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:33996/0 (socket says 172.21.15.61:33996) 2022-07-12T08:51:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 -- 172.21.15.61:0/189632732 learned_addr learned my addr 172.21.15.61:0/189632732 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:04.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebf7fe700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec0140160 0x7f9ec0141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebe7fc700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec005f860 0x7f9ec0065980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 -- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec0140160 msgr2=0x7f9ec0141330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec0140160 0x7f9ec0141330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 -- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec005f860 msgr2=0x7f9ec0065980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:04.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec005f860 0x7f9ec0065980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:04.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.947+0000 7f9ebeffd700 1 -- 172.21.15.61:0/189632732 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb000b710 con 0x7f9ec005f480 2022-07-12T08:51:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9ebeffd700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 0x7f9ec0065490 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7f9eb4001000 tx=0x7f9eb4007b40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb40070d0 con 0x7f9ec005f480 2022-07-12T08:51:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4007230 con 0x7f9ec005f480 2022-07-12T08:51:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec00677e0 con 0x7f9ec005f480 2022-07-12T08:51:04.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9eb40055d0 con 0x7f9ec005f480 2022-07-12T08:51:04.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec0142f80 con 0x7f9ec005f480 2022-07-12T08:51:04.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec0142660 con 0x7f9ec005f480 2022-07-12T08:51:04.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9eb4010070 con 0x7f9ec005f480 2022-07-12T08:51:04.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9eaffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e9c053ad0 0x7f9e9c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:04.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f9eb406b170 con 0x7f9ec005f480 2022-07-12T08:51:04.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.951+0000 7f9ebe7fc700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e9c053ad0 0x7f9e9c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:04.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.955+0000 7f9ebe7fc700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e9c053ad0 0x7f9e9c055f90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f9eb001b000 tx=0x7f9ec013f760).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:04.959+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ec0142660 con 0x7f9ec005f480 2022-07-12T08:51:05.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.087+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ec0142660 con 0x7f9ec005f480 2022-07-12T08:51:05.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.087+0000 7f9eaffff700 1 -- 172.21.15.61:0/189632732 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f9ec0142660 con 0x7f9ec005f480 2022-07-12T08:51:05.093 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:05.094 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":143,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:04.439595+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:51:05.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e9c053ad0 msgr2=0x7f9e9c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e9c053ad0 0x7f9e9c055f90 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f9eb001b000 tx=0x7f9ec013f760).stop 2022-07-12T08:51:05.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 msgr2=0x7f9ec0065490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 0x7f9ec0065490 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 rx=0x7f9eb4001000 tx=0x7f9eb4007b40).stop 2022-07-12T08:51:05.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 shutdown_connections 2022-07-12T08:51:05.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9ec0140160 0x7f9ec0141330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9e9c053ad0 0x7f9e9c055f90 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ec005f860 0x7f9ec0065980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 --2- 172.21.15.61:0/189632732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ec005f480 0x7f9ec0065490 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 >> 172.21.15.61:0/189632732 conn(0x7f9ec0063a60 msgr2=0x7f9ec013d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 shutdown_connections 2022-07-12T08:51:05.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.091+0000 7f9ebffff700 1 -- 172.21.15.61:0/189632732 wait complete. 2022-07-12T08:51:05.114 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-12T08:51:05.115 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:51:05.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 Processor -- start 2022-07-12T08:51:05.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 -- start start 2022-07-12T08:51:05.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4764060a70 0x7f476409a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f476409ae20 0x7f4764142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47640ab010 con 0x7f47640abe90 2022-07-12T08:51:05.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4764097aa0 con 0x7f476409ae20 2022-07-12T08:51:05.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f476ab89700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4764143ea0 con 0x7f4764060a70 2022-07-12T08:51:05.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f4769386700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f476409ae20 0x7f4764142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f4769386700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f476409ae20 0x7f4764142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34002/0 (socket says 172.21.15.61:34002) 2022-07-12T08:51:05.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.227+0000 7f4769386700 1 -- 172.21.15.61:0/1990687502 learned_addr learned my addr 172.21.15.61:0/1990687502 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:05.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4769b87700 1 --2- 172.21.15.61:0/1990687502 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4764060a70 0x7f476409a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4769386700 1 -- 172.21.15.61:0/1990687502 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4764060a70 msgr2=0x7f476409a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:05.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4769386700 1 --2- 172.21.15.61:0/1990687502 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4764060a70 0x7f476409a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4769386700 1 -- 172.21.15.61:0/1990687502 >> v1:172.21.15.111:6789/0 conn(0x7f47640abe90 legacy=0x7f47641431c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4769386700 1 -- 172.21.15.61:0/1990687502 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47641438d0 con 0x7f476409ae20 2022-07-12T08:51:05.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4769386700 1 --2- 172.21.15.61:0/1990687502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f476409ae20 0x7f4764142cd0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f4754002f70 tx=0x7f4754006eb0).ready entity=mon.0 client_cookie=f2636c8ef14e591a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4768b85700 1 -- 172.21.15.61:0/1990687502 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f475400d5c0 con 0x7f476409ae20 2022-07-12T08:51:05.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4768b85700 1 -- 172.21.15.61:0/1990687502 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4754000e90 con 0x7f476409ae20 2022-07-12T08:51:05.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f4768b85700 1 -- 172.21.15.61:0/1990687502 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f475400d950 con 0x7f476409ae20 2022-07-12T08:51:05.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 -- 172.21.15.61:0/1990687502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f476409ae20 msgr2=0x7f4764142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 --2- 172.21.15.61:0/1990687502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f476409ae20 0x7f4764142cd0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f4754002f70 tx=0x7f4754006eb0).stop 2022-07-12T08:51:05.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 -- 172.21.15.61:0/1990687502 shutdown_connections 2022-07-12T08:51:05.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 --2- 172.21.15.61:0/1990687502 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f476409ae20 0x7f4764142cd0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 --2- 172.21.15.61:0/1990687502 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f4764060a70 0x7f476409a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 -- 172.21.15.61:0/1990687502 >> 172.21.15.61:0/1990687502 conn(0x7f476405bd10 msgr2=0x7f476405c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 -- 172.21.15.61:0/1990687502 shutdown_connections 2022-07-12T08:51:05.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.231+0000 7f476ab89700 1 -- 172.21.15.61:0/1990687502 wait complete. 2022-07-12T08:51:05.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 Processor -- start 2022-07-12T08:51:05.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 -- start start 2022-07-12T08:51:05.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4764060a70 0x7f4764155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f476409ae20 0x7f476413f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 0x7f476413f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47640adec0 con 0x7f47640abe90 2022-07-12T08:51:05.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4764155e20 con 0x7f4764060a70 2022-07-12T08:51:05.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4764146b40 con 0x7f476409ae20 2022-07-12T08:51:05.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 0x7f476413f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 0x7f476413f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34012/0 (socket says 172.21.15.61:34012) 2022-07-12T08:51:05.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 -- 172.21.15.61:0/2991810606 learned_addr learned my addr 172.21.15.61:0/2991810606 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:05.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f4769386700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f476409ae20 0x7f476413f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f4769b87700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4764060a70 0x7f4764155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 -- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f476409ae20 msgr2=0x7f476413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f476409ae20 0x7f476413f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 -- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4764060a70 msgr2=0x7f4764155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4764060a70 0x7f4764155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 -- 172.21.15.61:0/2991810606 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475400b1e0 con 0x7f47640abe90 2022-07-12T08:51:05.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476a388700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 0x7f476413f8f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f47640af090 tx=0x7f475c0013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f475c006aa0 con 0x7f47640abe90 2022-07-12T08:51:05.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475c006c00 con 0x7f47640abe90 2022-07-12T08:51:05.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4764156060 con 0x7f47640abe90 2022-07-12T08:51:05.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f475c007030 con 0x7f47640abe90 2022-07-12T08:51:05.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.235+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4764145c50 con 0x7f47640abe90 2022-07-12T08:51:05.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.239+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4764146d00 con 0x7f47640abe90 2022-07-12T08:51:05.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.239+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f475c010030 con 0x7f47640abe90 2022-07-12T08:51:05.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.239+0000 7f4752ffd700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4744053a40 0x7f4744055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.239+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f475c06bb70 con 0x7f47640abe90 2022-07-12T08:51:05.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.239+0000 7f4769b87700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4744053a40 0x7f4744055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.239+0000 7f4769b87700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4744053a40 0x7f4744055f00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f476000c000 tx=0x7f4760008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.243+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f475c042cf0 con 0x7f47640abe90 2022-07-12T08:51:05.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.367+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f47641561f0 con 0x7f47640abe90 2022-07-12T08:51:05.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.367+0000 7f4752ffd700 1 -- 172.21.15.61:0/2991810606 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v143) v1 ==== 110+0+10 (secure 0 0 0) 0x7f475c046310 con 0x7f47640abe90 2022-07-12T08:51:05.372 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:51:05.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4744053a40 msgr2=0x7f4744055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4744053a40 0x7f4744055f00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f476000c000 tx=0x7f4760008040).stop 2022-07-12T08:51:05.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 msgr2=0x7f476413f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 0x7f476413f8f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f47640af090 tx=0x7f475c0013a0).stop 2022-07-12T08:51:05.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 shutdown_connections 2022-07-12T08:51:05.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47640abe90 0x7f476413f8f0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f476409ae20 0x7f476413f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f4744053a40 0x7f4744055f00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 --2- 172.21.15.61:0/2991810606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4764060a70 0x7f4764155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 >> 172.21.15.61:0/2991810606 conn(0x7f476405bd10 msgr2=0x7f476413c610 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 shutdown_connections 2022-07-12T08:51:05.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.371+0000 7f476ab89700 1 -- 172.21.15.61:0/2991810606 wait complete. 2022-07-12T08:51:05.385 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:51:05.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 Processor -- start 2022-07-12T08:51:05.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- start start 2022-07-12T08:51:05.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f031c0b0120 0x7f031c144150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f031c0ab730 0x7f031c144640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c14aad0 0x7f031c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031c0b1040 con 0x7f031c0b0120 2022-07-12T08:51:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031c0b0500 con 0x7f031c0ab730 2022-07-12T08:51:05.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031c05d0b0 con 0x7f031c14aad0 2022-07-12T08:51:05.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f031c0b0120 0x7f031c144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f031c0b0120 0x7f031c144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34016/0 (socket says 172.21.15.61:34016) 2022-07-12T08:51:05.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 -- 172.21.15.61:0/2759761852 learned_addr learned my addr 172.21.15.61:0/2759761852 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:05.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03127fc700 1 --2- 172.21.15.61:0/2759761852 >> v2:172.21.15.111:3300/0 conn(0x7f031c0ab730 0x7f031c144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031b7fe700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c14aad0 0x7f031c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 -- 172.21.15.61:0/2759761852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c14aad0 msgr2=0x7f031c14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:05.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c14aad0 0x7f031c14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 -- 172.21.15.61:0/2759761852 >> v2:172.21.15.111:3300/0 conn(0x7f031c0ab730 msgr2=0x7f031c144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:05.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 --2- 172.21.15.61:0/2759761852 >> v2:172.21.15.111:3300/0 conn(0x7f031c0ab730 0x7f031c144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 -- 172.21.15.61:0/2759761852 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031c05d350 con 0x7f031c0b0120 2022-07-12T08:51:05.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031b7fe700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c14aad0 0x7f031c14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:05.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03127fc700 1 --2- 172.21.15.61:0/2759761852 >> v2:172.21.15.111:3300/0 conn(0x7f031c0ab730 0x7f031c144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:05.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031affd700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c0b0120 0x7f031c144150 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f030c002f70 tx=0x7f030c006eb0).ready entity=mon.0 client_cookie=c1ee65490cc0f984 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031a7fc700 1 -- 172.21.15.61:0/2759761852 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f030c00d570 con 0x7f031c0b0120 2022-07-12T08:51:05.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031a7fc700 1 -- 172.21.15.61:0/2759761852 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c000e90 con 0x7f031c0b0120 2022-07-12T08:51:05.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f031a7fc700 1 -- 172.21.15.61:0/2759761852 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f030c00d900 con 0x7f031c0b0120 2022-07-12T08:51:05.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- 172.21.15.61:0/2759761852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c0b0120 msgr2=0x7f031c144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c0b0120 0x7f031c144150 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f030c002f70 tx=0x7f030c006eb0).stop 2022-07-12T08:51:05.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- 172.21.15.61:0/2759761852 shutdown_connections 2022-07-12T08:51:05.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c14aad0 0x7f031c14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- 172.21.15.61:0/2759761852 >> v2:172.21.15.111:3300/0 conn(0x7f031c0ab730 0x7f031c144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 --2- 172.21.15.61:0/2759761852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c0b0120 0x7f031c144150 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- 172.21.15.61:0/2759761852 >> 172.21.15.61:0/2759761852 conn(0x7f031c0b5290 msgr2=0x7f031c0b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- 172.21.15.61:0/2759761852 shutdown_connections 2022-07-12T08:51:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.483+0000 7f03218a7700 1 -- 172.21.15.61:0/2759761852 wait complete. 2022-07-12T08:51:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 Processor -- start 2022-07-12T08:51:05.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 -- start start 2022-07-12T08:51:05.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c0ab730 0x7f031c0aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031c0b0120 0x7f031c063b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 0x7f031c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c064ab0 con 0x7f031c14aad0 2022-07-12T08:51:05.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031c065510 con 0x7f031c0b0120 2022-07-12T08:51:05.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031c064ce0 con 0x7f031c0ab730 2022-07-12T08:51:05.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 0x7f031c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031affd700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c0ab730 0x7f031c0aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 0x7f031c0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34026/0 (socket says 172.21.15.61:34026) 2022-07-12T08:51:05.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 -- 172.21.15.61:0/1748101628 learned_addr learned my addr 172.21.15.61:0/1748101628 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:05.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03127fc700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031c0b0120 0x7f031c063b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 -- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c0ab730 msgr2=0x7f031c0aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c0ab730 0x7f031c0aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 -- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031c0b0120 msgr2=0x7f031c063b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031c0b0120 0x7f031c063b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 -- 172.21.15.61:0/1748101628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c00b1e0 con 0x7f031c14aad0 2022-07-12T08:51:05.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f031b7fe700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 0x7f031c0ad8d0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f0314007000 tx=0x7f0314008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f031400e510 con 0x7f031c14aad0 2022-07-12T08:51:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031c067a50 con 0x7f031c14aad0 2022-07-12T08:51:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031400e670 con 0x7f031c14aad0 2022-07-12T08:51:05.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031c065970 con 0x7f031c14aad0 2022-07-12T08:51:05.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.487+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f031400e9d0 con 0x7f031c14aad0 2022-07-12T08:51:05.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.491+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031c065680 con 0x7f031c14aad0 2022-07-12T08:51:05.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.491+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0314019030 con 0x7f031c14aad0 2022-07-12T08:51:05.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.491+0000 7f0318ff9700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03000539b0 0x7f0300055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.491+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f031406b970 con 0x7f031c14aad0 2022-07-12T08:51:05.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.491+0000 7f031affd700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03000539b0 0x7f0300055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.491+0000 7f031affd700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03000539b0 0x7f0300055e70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f030c01c000 tx=0x7f030c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.499+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0314042c10 con 0x7f031c14aad0 2022-07-12T08:51:05.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.623+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f031c065270 con 0x7f031c14aad0 2022-07-12T08:51:05.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.623+0000 7f0318ff9700 1 -- 172.21.15.61:0/1748101628 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v143) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0314046230 con 0x7f031c14aad0 2022-07-12T08:51:05.630 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03000539b0 msgr2=0x7f0300055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03000539b0 0x7f0300055e70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f030c01c000 tx=0x7f030c015000).stop 2022-07-12T08:51:05.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 msgr2=0x7f031c0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 0x7f031c0ad8d0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7f0314007000 tx=0x7f0314008040).stop 2022-07-12T08:51:05.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 shutdown_connections 2022-07-12T08:51:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f031c14aad0 0x7f031c0ad8d0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f03000539b0 0x7f0300055e70 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f031c0b0120 0x7f031c063b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 --2- 172.21.15.61:0/1748101628 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f031c0ab730 0x7f031c0aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 >> 172.21.15.61:0/1748101628 conn(0x7f031c0b5290 msgr2=0x7f031c1478a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 shutdown_connections 2022-07-12T08:51:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.627+0000 7f03218a7700 1 -- 172.21.15.61:0/1748101628 wait complete. 2022-07-12T08:51:05.644 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-12T08:51:05.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa1afe0700 1 Processor -- start 2022-07-12T08:51:05.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa1afe0700 1 -- start start 2022-07-12T08:51:05.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa1afe0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7faa14060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa1afe0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa140aecb0 con 0x7faa1409bce0 2022-07-12T08:51:05.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa1afe0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa141435c0 con 0x7faa140b3f60 2022-07-12T08:51:05.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa1afe0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa140afd70 con 0x7faa140af280 2022-07-12T08:51:05.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa197dd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7faa1409bce0 0x7faa140af660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50480/0 (socket says 172.21.15.61:50480) 2022-07-12T08:51:05.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa197dd700 1 -- 172.21.15.61:0/2386336639 learned_addr learned my addr 172.21.15.61:0/2386336639 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:05.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa19fde700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7faa14060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741161158 0 0) 0x7faa140aecb0 con 0x7faa1409bce0 2022-07-12T08:51:05.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc003260 con 0x7faa1409bce0 2022-07-12T08:51:05.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001804487 0 0) 0x7faa141435c0 con 0x7faa140b3f60 2022-07-12T08:51:05.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.747+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa140aecb0 con 0x7faa140b3f60 2022-07-12T08:51:05.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 460124783 0 0) 0x7fa9fc003260 con 0x7faa1409bce0 2022-07-12T08:51:05.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 msgr2=0x7faa14060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:05.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7faa14060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 >> v1:172.21.15.111:6789/0 conn(0x7faa140b3f60 legacy=0x7faa140b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14143860 con 0x7faa1409bce0 2022-07-12T08:51:05.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7faa04001660 con 0x7faa1409bce0 2022-07-12T08:51:05.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 >> v1:172.21.15.61:6789/0 conn(0x7faa1409bce0 legacy=0x7faa140af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9fc003910 0x7fa9fc003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7fa9fc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9fc0049f0 0x7fa9fc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa140afd70 con 0x7fa9fc003910 2022-07-12T08:51:05.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc00b780 con 0x7fa9fc0049f0 2022-07-12T08:51:05.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc00b940 con 0x7faa140af280 2022-07-12T08:51:05.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9fc003910 0x7fa9fc003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa197dd700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7fa9fc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1a7df700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9fc0049f0 0x7fa9fc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 -- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 msgr2=0x7fa9fc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:05.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7fa9fc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 -- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9fc0049f0 msgr2=0x7fa9fc004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:05.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9fc0049f0 0x7fa9fc004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 -- 172.21.15.61:0/2386336639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc003260 con 0x7fa9fc003910 2022-07-12T08:51:05.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1a7df700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9fc0049f0 0x7fa9fc004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:05.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa197dd700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faa140af280 0x7fa9fc004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:05.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa19fde700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9fc003910 0x7fa9fc003d10 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7faa1000c000 tx=0x7faa10006d90).ready entity=mon.0 client_cookie=f05069409d1e36c8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa10013450 con 0x7fa9fc003910 2022-07-12T08:51:05.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa100135b0 con 0x7fa9fc003910 2022-07-12T08:51:05.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa18fdc700 1 -- 172.21.15.61:0/2386336639 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa10013910 con 0x7fa9fc003910 2022-07-12T08:51:05.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1afe0700 1 -- 172.21.15.61:0/2386336639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9fc003910 msgr2=0x7fa9fc003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1afe0700 1 --2- 172.21.15.61:0/2386336639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9fc003910 0x7fa9fc003d10 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7faa1000c000 tx=0x7faa10006d90).stop 2022-07-12T08:51:05.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1a7df700 1 -- 172.21.15.61:0/2386336639 reap_dead start 2022-07-12T08:51:05.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1afe0700 1 -- 172.21.15.61:0/2386336639 shutdown_connections 2022-07-12T08:51:05.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1afe0700 1 -- 172.21.15.61:0/2386336639 >> 172.21.15.61:0/2386336639 conn(0x7faa140b92c0 msgr2=0x7faa140b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1afe0700 1 -- 172.21.15.61:0/2386336639 shutdown_connections 2022-07-12T08:51:05.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.751+0000 7faa1afe0700 1 -- 172.21.15.61:0/2386336639 wait complete. 2022-07-12T08:51:05.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 Processor -- start 2022-07-12T08:51:05.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 -- start start 2022-07-12T08:51:05.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9fc003910 0x7faa141444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa140b0700 0x7faa141463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 0x7faa141468e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa140b2ca0 con 0x7faa1409eb60 2022-07-12T08:51:05.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa140b1e50 con 0x7faa140b0700 2022-07-12T08:51:05.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1409e160 con 0x7fa9fc003910 2022-07-12T08:51:05.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 0x7faa141468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 0x7faa141468e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34044/0 (socket says 172.21.15.61:34044) 2022-07-12T08:51:05.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 -- 172.21.15.61:0/4222168232 learned_addr learned my addr 172.21.15.61:0/4222168232 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:05.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa19fde700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9fc003910 0x7faa141444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa197dd700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa140b0700 0x7faa141463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 -- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9fc003910 msgr2=0x7faa141444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9fc003910 0x7faa141444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 -- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa140b0700 msgr2=0x7faa141463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa140b0700 0x7faa141463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 -- 172.21.15.61:0/4222168232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14144b30 con 0x7faa1409eb60 2022-07-12T08:51:05.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa197dd700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa140b0700 0x7faa141463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:05.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa19fde700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9fc003910 0x7faa141444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:05.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1a7df700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 0x7faa141468e0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7faa1409e920 tx=0x7faa0c007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa0c012850 con 0x7faa1409eb60 2022-07-12T08:51:05.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c0129b0 con 0x7faa1409eb60 2022-07-12T08:51:05.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa14144dd0 con 0x7faa1409eb60 2022-07-12T08:51:05.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa0c012d10 con 0x7faa1409eb60 2022-07-12T08:51:05.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.755+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa140b2230 con 0x7faa1409eb60 2022-07-12T08:51:05.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.759+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faa0c01d460 con 0x7faa1409eb60 2022-07-12T08:51:05.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.759+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa14057460 con 0x7faa1409eb60 2022-07-12T08:51:05.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.759+0000 7faa02ffd700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9f40539b0 0x7fa9f4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:05.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.759+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5253+0+0 (secure 0 0 0) 0x7faa0c06c200 con 0x7faa1409eb60 2022-07-12T08:51:05.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.759+0000 7faa19fde700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9f40539b0 0x7fa9f4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:05.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.759+0000 7faa19fde700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9f40539b0 0x7fa9f4055e70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7faa1001b000 tx=0x7faa10017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:05.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.767+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa0c043320 con 0x7faa1409eb60 2022-07-12T08:51:05.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.895+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7faa1406b2b0 con 0x7faa1409eb60 2022-07-12T08:51:05.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.895+0000 7faa02ffd700 1 -- 172.21.15.61:0/4222168232 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v143) v1 ==== 84+0+143 (secure 0 0 0) 0x7faa0c046940 con 0x7faa1409eb60 2022-07-12T08:51:05.900 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:05.901 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-12T08:51:05.901 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:05.901 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-12T08:51:05.903 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-12T08:51:05.903 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:05.904 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:05.905 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-12T08:51:05.906 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-12T08:51:05.906 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:05.907 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-12T08:51:05.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9f40539b0 msgr2=0x7fa9f4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9f40539b0 0x7fa9f4055e70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7faa1001b000 tx=0x7faa10017000).stop 2022-07-12T08:51:05.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 msgr2=0x7faa141468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:05.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 0x7faa141468e0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 rx=0x7faa1409e920 tx=0x7faa0c007d80).stop 2022-07-12T08:51:05.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 shutdown_connections 2022-07-12T08:51:05.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa1409eb60 0x7faa141468e0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9f40539b0 0x7fa9f4055e70 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa140b0700 0x7faa141463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 --2- 172.21.15.61:0/4222168232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9fc003910 0x7faa141444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:05.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 >> 172.21.15.61:0/4222168232 conn(0x7faa140b92c0 msgr2=0x7faa14148230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:05.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 shutdown_connections 2022-07-12T08:51:05.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:05.899+0000 7faa1afe0700 1 -- 172.21.15.61:0/4222168232 wait complete. 2022-07-12T08:51:05.914 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-12T08:51:05.915 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-12T08:51:06.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 Processor -- start 2022-07-12T08:51:06.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 -- start start 2022-07-12T08:51:06.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f83ac144860 0x7f83ac14b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac0b0500 con 0x7f83ac0ab730 2022-07-12T08:51:06.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83ac05d0b0 con 0x7f83ac0b0120 2022-07-12T08:51:06.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83ac0b1040 con 0x7f83ac144860 2022-07-12T08:51:06.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83a3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f83ac0ab730 0x7f83ac14aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50500/0 (socket says 172.21.15.61:50500) 2022-07-12T08:51:06.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83a3fff700 1 -- 172.21.15.61:0/1903945554 learned_addr learned my addr 172.21.15.61:0/1903945554 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:06.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b0a04700 1 --2- 172.21.15.61:0/1903945554 >> v2:172.21.15.61:3300/0 conn(0x7f83ac144860 0x7f83ac14b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83ab7fe700 1 -- 172.21.15.61:0/1903945554 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267269386 0 0) 0x7f83ac0b0500 con 0x7f83ac0ab730 2022-07-12T08:51:06.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83ab7fe700 1 -- 172.21.15.61:0/1903945554 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f838c003260 con 0x7f83ac0ab730 2022-07-12T08:51:06.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83ab7fe700 1 -- 172.21.15.61:0/1903945554 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751374869 0 0) 0x7f838c003260 con 0x7f83ac0ab730 2022-07-12T08:51:06.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b0a04700 1 -- 172.21.15.61:0/1903945554 >> v1:172.21.15.111:6789/0 conn(0x7f83ac0b0120 legacy=0x7f83ac144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b0a04700 1 -- 172.21.15.61:0/1903945554 >> v1:172.21.15.61:6789/0 conn(0x7f83ac0ab730 legacy=0x7f83ac14aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b0a04700 1 -- 172.21.15.61:0/1903945554 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac05d350 con 0x7f83ac144860 2022-07-12T08:51:06.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b0a04700 1 --2- 172.21.15.61:0/1903945554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac144860 0x7f83ac14b1e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7f8394001800 tx=0x7f83a40091d0).ready entity=mon.0 client_cookie=2f26010313818787 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83ab7fe700 1 -- 172.21.15.61:0/1903945554 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8394001660 con 0x7f83ac144860 2022-07-12T08:51:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83ab7fe700 1 -- 172.21.15.61:0/1903945554 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a400b580 con 0x7f83ac144860 2022-07-12T08:51:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83ab7fe700 1 -- 172.21.15.61:0/1903945554 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83a400b6f0 con 0x7f83ac144860 2022-07-12T08:51:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 -- 172.21.15.61:0/1903945554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac144860 msgr2=0x7f83ac14b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.023+0000 7f83b2467700 1 --2- 172.21.15.61:0/1903945554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac144860 0x7f83ac14b1e0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7f8394001800 tx=0x7f83a40091d0).stop 2022-07-12T08:51:06.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 -- 172.21.15.61:0/1903945554 shutdown_connections 2022-07-12T08:51:06.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 --2- 172.21.15.61:0/1903945554 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac144860 0x7f83ac14b1e0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 -- 172.21.15.61:0/1903945554 >> 172.21.15.61:0/1903945554 conn(0x7f83ac0b5290 msgr2=0x7f83ac0b5670 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:06.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 -- 172.21.15.61:0/1903945554 shutdown_connections 2022-07-12T08:51:06.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 -- 172.21.15.61:0/1903945554 wait complete. 2022-07-12T08:51:06.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 Processor -- start 2022-07-12T08:51:06.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.027+0000 7f83b2467700 1 -- start start 2022-07-12T08:51:06.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 0x7f83ac0aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83ac0b0120 0x7f83ac159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83ac144860 0x7f83ac15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0ad300 con 0x7f83ac0ab730 2022-07-12T08:51:06.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac0ad600 con 0x7f83ac144860 2022-07-12T08:51:06.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac067570 con 0x7f83ac0b0120 2022-07-12T08:51:06.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 0x7f83ac0aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 0x7f83ac0aed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34056/0 (socket says 172.21.15.61:34056) 2022-07-12T08:51:06.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 -- 172.21.15.61:0/511829327 learned_addr learned my addr 172.21.15.61:0/511829327 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:06.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b0a04700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83ac144860 0x7f83ac15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83a3fff700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83ac0b0120 0x7f83ac159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 -- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83ac0b0120 msgr2=0x7f83ac159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83ac0b0120 0x7f83ac159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 -- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83ac144860 msgr2=0x7f83ac15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83ac144860 0x7f83ac15a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 -- 172.21.15.61:0/511829327 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4009470 con 0x7f83ac0ab730 2022-07-12T08:51:06.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83abfff700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 0x7f83ac0aed90 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f83ac061b40 tx=0x7f839c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:06.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f839c00ce30 con 0x7f83ac0ab730 2022-07-12T08:51:06.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839c00f470 con 0x7f83ac0ab730 2022-07-12T08:51:06.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac159b30 con 0x7f83ac0ab730 2022-07-12T08:51:06.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f839c00f7d0 con 0x7f83ac0ab730 2022-07-12T08:51:06.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac063b60 con 0x7f83ac0ab730 2022-07-12T08:51:06.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.031+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac067730 con 0x7f83ac0ab730 2022-07-12T08:51:06.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.035+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f839c019030 con 0x7f83ac0ab730 2022-07-12T08:51:06.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.035+0000 7f83a9ffb700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8390053ad0 0x7f8390055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.035+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f839c06afb0 con 0x7f83ac0ab730 2022-07-12T08:51:06.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.035+0000 7f83a3fff700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8390053ad0 0x7f8390055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.035+0000 7f83a3fff700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8390053ad0 0x7f8390055f90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f839400b000 tx=0x7f8394000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:06.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.039+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f839c0421e0 con 0x7f83ac0ab730 2022-07-12T08:51:06.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.167+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f83ac057460 con 0x7f83ac0ab730 2022-07-12T08:51:06.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.471+0000 7f83a9ffb700 1 -- 172.21.15.61:0/511829327 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v144) v1 ==== 133+0+0 (secure 0 0 0) 0x7f839c045800 con 0x7f83ac0ab730 2022-07-12T08:51:06.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8390053ad0 msgr2=0x7f8390055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8390053ad0 0x7f8390055f90 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f839400b000 tx=0x7f8394000bc0).stop 2022-07-12T08:51:06.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 msgr2=0x7f83ac0aed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 0x7f83ac0aed90 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7f83ac061b40 tx=0x7f839c007c50).stop 2022-07-12T08:51:06.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 shutdown_connections 2022-07-12T08:51:06.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8390053ad0 0x7f8390055f90 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83ac144860 0x7f83ac15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f83ac0b0120 0x7f83ac159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 --2- 172.21.15.61:0/511829327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f83ac0ab730 0x7f83ac0aed90 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 >> 172.21.15.61:0/511829327 conn(0x7f83ac0b5290 msgr2=0x7f83ac148b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:06.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 shutdown_connections 2022-07-12T08:51:06.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.475+0000 7f83b2467700 1 -- 172.21.15.61:0/511829327 wait complete. 2022-07-12T08:51:06.495 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-12T08:51:06.498 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-12T08:51:06.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 Processor -- start 2022-07-12T08:51:06.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 -- start start 2022-07-12T08:51:06.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f94700af4b0 0x7f94700af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f947005bdf0 0x7f947005c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9470143610 con 0x7f9470060b50 2022-07-12T08:51:06.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9470143840 con 0x7f94700af4b0 2022-07-12T08:51:06.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94700afef0 con 0x7f947005bdf0 2022-07-12T08:51:06.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9475149700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9470060b50 0x7f9470142120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50516/0 (socket says 172.21.15.61:50516) 2022-07-12T08:51:06.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9475149700 1 -- 172.21.15.61:0/4176786507 learned_addr learned my addr 172.21.15.61:0/4176786507 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:06.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947614b700 1 --2- 172.21.15.61:0/4176786507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f947005bdf0 0x7f947005c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947594a700 1 --2- 172.21.15.61:0/4176786507 >> v2:172.21.15.111:3300/0 conn(0x7f94700af4b0 0x7f94700af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9474948700 1 -- 172.21.15.61:0/4176786507 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156242796 0 0) 0x7f9470143610 con 0x7f9470060b50 2022-07-12T08:51:06.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9474948700 1 -- 172.21.15.61:0/4176786507 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f945c003260 con 0x7f9470060b50 2022-07-12T08:51:06.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947614b700 1 -- 172.21.15.61:0/4176786507 >> v2:172.21.15.111:3300/0 conn(0x7f94700af4b0 msgr2=0x7f94700af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:06.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947614b700 1 --2- 172.21.15.61:0/4176786507 >> v2:172.21.15.111:3300/0 conn(0x7f94700af4b0 0x7f94700af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947614b700 1 -- 172.21.15.61:0/4176786507 >> v1:172.21.15.61:6789/0 conn(0x7f9470060b50 legacy=0x7f9470142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947614b700 1 -- 172.21.15.61:0/4176786507 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9470142830 con 0x7f947005bdf0 2022-07-12T08:51:06.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947614b700 1 --2- 172.21.15.61:0/4176786507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f947005bdf0 0x7f947005c230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9468002f70 tx=0x7f9468006df0).ready entity=mon.2 client_cookie=4c8e797e095bed8c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:06.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9474948700 1 -- 172.21.15.61:0/4176786507 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946800d4a0 con 0x7f947005bdf0 2022-07-12T08:51:06.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9474948700 1 -- 172.21.15.61:0/4176786507 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9468000e90 con 0x7f947005bdf0 2022-07-12T08:51:06.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f9474948700 1 -- 172.21.15.61:0/4176786507 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946800d7f0 con 0x7f947005bdf0 2022-07-12T08:51:06.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 -- 172.21.15.61:0/4176786507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f947005bdf0 msgr2=0x7f947005c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.607+0000 7f947694c700 1 --2- 172.21.15.61:0/4176786507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f947005bdf0 0x7f947005c230 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9468002f70 tx=0x7f9468006df0).stop 2022-07-12T08:51:06.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- 172.21.15.61:0/4176786507 shutdown_connections 2022-07-12T08:51:06.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 --2- 172.21.15.61:0/4176786507 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f947005bdf0 0x7f947005c230 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 --2- 172.21.15.61:0/4176786507 >> v2:172.21.15.111:3300/0 conn(0x7f94700af4b0 0x7f94700af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- 172.21.15.61:0/4176786507 >> 172.21.15.61:0/4176786507 conn(0x7f94700b92c0 msgr2=0x7f94700b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:06.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- 172.21.15.61:0/4176786507 shutdown_connections 2022-07-12T08:51:06.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- 172.21.15.61:0/4176786507 wait complete. 2022-07-12T08:51:06.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 Processor -- start 2022-07-12T08:51:06.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- start start 2022-07-12T08:51:06.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 0x7f94701529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9470060b50 0x7f947009a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94700af4b0 0x7f947013d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947013c7f0 con 0x7f947005bdf0 2022-07-12T08:51:06.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9470152000 con 0x7f94700af4b0 2022-07-12T08:51:06.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94701522a0 con 0x7f9470060b50 2022-07-12T08:51:06.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 0x7f94701529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 0x7f94701529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34070/0 (socket says 172.21.15.61:34070) 2022-07-12T08:51:06.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 -- 172.21.15.61:0/3973686051 learned_addr learned my addr 172.21.15.61:0/3973686051 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:06.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947614b700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94700af4b0 0x7f947013d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f9475149700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9470060b50 0x7f947009a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 -- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9470060b50 msgr2=0x7f947009a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9470060b50 0x7f947009a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 -- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94700af4b0 msgr2=0x7f947013d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94700af4b0 0x7f947013d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:06.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 -- 172.21.15.61:0/3973686051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946800b180 con 0x7f947005bdf0 2022-07-12T08:51:06.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f9475149700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9470060b50 0x7f947009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:06.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947594a700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 0x7f94701529e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f946000c000 tx=0x7f94600091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:06.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946000c850 con 0x7f947005bdf0 2022-07-12T08:51:06.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946000c9b0 con 0x7f947005bdf0 2022-07-12T08:51:06.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f946000cd10 con 0x7f947005bdf0 2022-07-12T08:51:06.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9470152540 con 0x7f947005bdf0 2022-07-12T08:51:06.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.611+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947009aa50 con 0x7f947005bdf0 2022-07-12T08:51:06.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.615+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9470151e70 con 0x7f947005bdf0 2022-07-12T08:51:06.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.615+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9460022070 con 0x7f947005bdf0 2022-07-12T08:51:06.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.615+0000 7f946effd700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9450053a40 0x7f9450055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:06.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.615+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f946006c3d0 con 0x7f947005bdf0 2022-07-12T08:51:06.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.615+0000 7f9475149700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9450053a40 0x7f9450055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:06.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.615+0000 7f9475149700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9450053a40 0x7f9450055f00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f9464008000 tx=0x7f9464009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:06.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.623+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9460043500 con 0x7f947005bdf0 2022-07-12T08:51:06.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:06.747+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f9470057460 con 0x7f947005bdf0 2022-07-12T08:51:07.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.279+0000 7f946effd700 1 -- 172.21.15.61:0/3973686051 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v145) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9460046b20 con 0x7f947005bdf0 2022-07-12T08:51:07.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9450053a40 msgr2=0x7f9450055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9450053a40 0x7f9450055f00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f9464008000 tx=0x7f9464009040).stop 2022-07-12T08:51:07.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 msgr2=0x7f94701529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 0x7f94701529e0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7f946000c000 tx=0x7f94600091d0).stop 2022-07-12T08:51:07.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 shutdown_connections 2022-07-12T08:51:07.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9450053a40 0x7f9450055f00 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f94700af4b0 0x7f947013d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f9470060b50 0x7f947009a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 --2- 172.21.15.61:0/3973686051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f947005bdf0 0x7f94701529e0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 >> 172.21.15.61:0/3973686051 conn(0x7f94700b92c0 msgr2=0x7f947005c810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:07.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 shutdown_connections 2022-07-12T08:51:07.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.283+0000 7f947694c700 1 -- 172.21.15.61:0/3973686051 wait complete. 2022-07-12T08:51:07.301 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-12T08:51:07.310 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-12T08:51:07.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f956a51e700 1 Processor -- start 2022-07-12T08:51:07.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f956a51e700 1 -- start start 2022-07-12T08:51:07.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f956a51e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f95640b2940 0x7f95640b5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f956a51e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95640b2f20 con 0x7f9564066fc0 2022-07-12T08:51:07.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f956a51e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95640b3170 con 0x7f95640b45d0 2022-07-12T08:51:07.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f956a51e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95640b4170 con 0x7f95640b2940 2022-07-12T08:51:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f9563fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9564066fc0 0x7f95640b4d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50528/0 (socket says 172.21.15.61:50528) 2022-07-12T08:51:07.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f9563fff700 1 -- 172.21.15.61:0/3563157058 learned_addr learned my addr 172.21.15.61:0/3563157058 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:07.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f9568abb700 1 --2- 172.21.15.61:0/3563157058 >> v2:172.21.15.111:3300/0 conn(0x7f95640b2940 0x7f95640b5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f9562ffd700 1 -- 172.21.15.61:0/3563157058 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180210947 0 0) 0x7f95640b2f20 con 0x7f9564066fc0 2022-07-12T08:51:07.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.431+0000 7f9562ffd700 1 -- 172.21.15.61:0/3563157058 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9550003260 con 0x7f9564066fc0 2022-07-12T08:51:07.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9568abb700 1 -- 172.21.15.61:0/3563157058 >> v1:172.21.15.111:6789/0 conn(0x7f95640b45d0 legacy=0x7f956405c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9568abb700 1 -- 172.21.15.61:0/3563157058 >> v1:172.21.15.61:6789/0 conn(0x7f9564066fc0 legacy=0x7f95640b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9568abb700 1 -- 172.21.15.61:0/3563157058 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95640673a0 con 0x7f95640b2940 2022-07-12T08:51:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9568abb700 1 --2- 172.21.15.61:0/3563157058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f95640b5410 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f955c0032f0 tx=0x7f955c0091d0).ready entity=mon.1 client_cookie=721c9e71215b5cf5 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9562ffd700 1 -- 172.21.15.61:0/3563157058 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f955c00b710 con 0x7f95640b2940 2022-07-12T08:51:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9562ffd700 1 -- 172.21.15.61:0/3563157058 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c00b870 con 0x7f95640b2940 2022-07-12T08:51:07.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f9562ffd700 1 -- 172.21.15.61:0/3563157058 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f955c00b9e0 con 0x7f95640b2940 2022-07-12T08:51:07.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 -- 172.21.15.61:0/3563157058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 msgr2=0x7f95640b5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 --2- 172.21.15.61:0/3563157058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f95640b5410 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7f955c0032f0 tx=0x7f955c0091d0).stop 2022-07-12T08:51:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 -- 172.21.15.61:0/3563157058 shutdown_connections 2022-07-12T08:51:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 --2- 172.21.15.61:0/3563157058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f95640b5410 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 -- 172.21.15.61:0/3563157058 >> 172.21.15.61:0/3563157058 conn(0x7f95640b94d0 msgr2=0x7f95640b98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:07.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 -- 172.21.15.61:0/3563157058 shutdown_connections 2022-07-12T08:51:07.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.435+0000 7f956a51e700 1 -- 172.21.15.61:0/3563157058 wait complete. 2022-07-12T08:51:07.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 Processor -- start 2022-07-12T08:51:07.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 -- start start 2022-07-12T08:51:07.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 0x7f95641482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f956413e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95640b45d0 0x7f956413d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95641489c0 con 0x7f9564066fc0 2022-07-12T08:51:07.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9564148c60 con 0x7f95640b2940 2022-07-12T08:51:07.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9564148f00 con 0x7f95640b45d0 2022-07-12T08:51:07.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 0x7f95641482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 0x7f95641482b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34084/0 (socket says 172.21.15.61:34084) 2022-07-12T08:51:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 -- 172.21.15.61:0/2716188545 learned_addr learned my addr 172.21.15.61:0/2716188545 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9568abb700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95640b45d0 0x7f956413d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f95637fe700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f956413e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 -- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95640b45d0 msgr2=0x7f956413d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95640b45d0 0x7f956413d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 -- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 msgr2=0x7f956413e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f956413e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 -- 172.21.15.61:0/2716188545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c009470 con 0x7f9564066fc0 2022-07-12T08:51:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f9563fff700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 0x7f95641482b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f9558001110 tx=0x7f9558007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:07.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95580070d0 con 0x7f9564066fc0 2022-07-12T08:51:07.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9558007230 con 0x7f9564066fc0 2022-07-12T08:51:07.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9558005610 con 0x7f9564066fc0 2022-07-12T08:51:07.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9564151ae0 con 0x7f9564066fc0 2022-07-12T08:51:07.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.439+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9564144380 con 0x7f9564066fc0 2022-07-12T08:51:07.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.443+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95641490c0 con 0x7f9564066fc0 2022-07-12T08:51:07.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.443+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9558010070 con 0x7f9564066fc0 2022-07-12T08:51:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.443+0000 7f95617fa700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95440539b0 0x7f9544055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.443+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f955806b560 con 0x7f9564066fc0 2022-07-12T08:51:07.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.443+0000 7f95637fe700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95440539b0 0x7f9544055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.447+0000 7f95637fe700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95440539b0 0x7f9544055e70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f955400d000 tx=0x7f9554007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:07.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.451+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9558042530 con 0x7f9564066fc0 2022-07-12T08:51:07.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.627+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f9564057460 con 0x7f9564066fc0 2022-07-12T08:51:07.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.639+0000 7f95617fa700 1 -- 172.21.15.61:0/2716188545 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 29 and data pool 30 v82) v1 ==== 189+0+0 (secure 0 0 0) 0x7f9558045b50 con 0x7f9564066fc0 2022-07-12T08:51:07.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95440539b0 msgr2=0x7f9544055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95440539b0 0x7f9544055e70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f955400d000 tx=0x7f9554007b70).stop 2022-07-12T08:51:07.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 msgr2=0x7f95641482b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 0x7f95641482b0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f9558001110 tx=0x7f9558007b80).stop 2022-07-12T08:51:07.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 shutdown_connections 2022-07-12T08:51:07.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f95640b45d0 0x7f956413d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f95440539b0 0x7f9544055e70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95640b2940 0x7f956413e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 --2- 172.21.15.61:0/2716188545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9564066fc0 0x7f95641482b0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 >> 172.21.15.61:0/2716188545 conn(0x7f95640b94d0 msgr2=0x7f95640b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:07.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 shutdown_connections 2022-07-12T08:51:07.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.643+0000 7f956a51e700 1 -- 172.21.15.61:0/2716188545 wait complete. 2022-07-12T08:51:07.659 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 29 and data pool 30 2022-07-12T08:51:07.661 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:51:07.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 Processor -- start 2022-07-12T08:51:07.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- start start 2022-07-12T08:51:07.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdb2c142d50 0x7fdb2c0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c13d240 con 0x7fdb2c13ce60 2022-07-12T08:51:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2c1431f0 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb2c0b24c0 con 0x7fdb2c142d50 2022-07-12T08:51:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb32058700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdb2c13ce60 0x7fdb2c0b5bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50550/0 (socket says 172.21.15.61:50550) 2022-07-12T08:51:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31857700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdb2c142d50 0x7fdb2c0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb32058700 1 -- 172.21.15.61:0/1860252084 learned_addr learned my addr 172.21.15.61:0/1860252084 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31056700 1 -- 172.21.15.61:0/1860252084 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204086926 0 0) 0x7fdb2c13d240 con 0x7fdb2c13ce60 2022-07-12T08:51:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31056700 1 -- 172.21.15.61:0/1860252084 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb14003260 con 0x7fdb2c13ce60 2022-07-12T08:51:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31857700 1 -- 172.21.15.61:0/1860252084 >> v1:172.21.15.111:6789/0 conn(0x7fdb2c0b47f0 legacy=0x7fdb2c0b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31857700 1 -- 172.21.15.61:0/1860252084 >> v1:172.21.15.61:6789/0 conn(0x7fdb2c13ce60 legacy=0x7fdb2c0b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31857700 1 -- 172.21.15.61:0/1860252084 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c143490 con 0x7fdb2c142d50 2022-07-12T08:51:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31857700 1 --2- 172.21.15.61:0/1860252084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c142d50 0x7fdb2c0b15a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fdb280032f0 tx=0x7fdb280091d0).ready entity=mon.0 client_cookie=2df1c29b7e7885e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:07.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31056700 1 -- 172.21.15.61:0/1860252084 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb2800b710 con 0x7fdb2c142d50 2022-07-12T08:51:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31056700 1 -- 172.21.15.61:0/1860252084 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb2800b870 con 0x7fdb2c142d50 2022-07-12T08:51:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb31056700 1 -- 172.21.15.61:0/1860252084 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb2800b9e0 con 0x7fdb2c142d50 2022-07-12T08:51:07.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- 172.21.15.61:0/1860252084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c142d50 msgr2=0x7fdb2c0b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 --2- 172.21.15.61:0/1860252084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c142d50 0x7fdb2c0b15a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 rx=0x7fdb280032f0 tx=0x7fdb280091d0).stop 2022-07-12T08:51:07.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- 172.21.15.61:0/1860252084 shutdown_connections 2022-07-12T08:51:07.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 --2- 172.21.15.61:0/1860252084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c142d50 0x7fdb2c0b15a0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- 172.21.15.61:0/1860252084 >> 172.21.15.61:0/1860252084 conn(0x7fdb2c05f9f0 msgr2=0x7fdb2c066e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:07.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- 172.21.15.61:0/1860252084 shutdown_connections 2022-07-12T08:51:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.755+0000 7fdb342bc700 1 -- 172.21.15.61:0/1860252084 wait complete. 2022-07-12T08:51:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 Processor -- start 2022-07-12T08:51:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 -- start start 2022-07-12T08:51:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 0x7fdb2c05c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb2c13ce60 0x7fdb2c155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb2c142d50 0x7fdb2c0ad340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c0b97a0 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c1554a0 con 0x7fdb2c13ce60 2022-07-12T08:51:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.759+0000 7fdb342bc700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2c155740 con 0x7fdb2c142d50 2022-07-12T08:51:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 0x7fdb2c05c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 0x7fdb2c05c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34106/0 (socket says 172.21.15.61:34106) 2022-07-12T08:51:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 -- 172.21.15.61:0/11950236 learned_addr learned my addr 172.21.15.61:0/11950236 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb31857700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb2c13ce60 0x7fdb2c155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32859700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb2c142d50 0x7fdb2c0ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 -- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb2c142d50 msgr2=0x7fdb2c0ad340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb2c142d50 0x7fdb2c0ad340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 -- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb2c13ce60 msgr2=0x7fdb2c155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb2c13ce60 0x7fdb2c155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 -- 172.21.15.61:0/11950236 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28009470 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb31857700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb2c13ce60 0x7fdb2c155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb32058700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 0x7fdb2c05c560 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7fdb20001110 tx=0x7fdb20007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb200072e0 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb20007440 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb2c1560d0 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb20005800 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2c0afcb0 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2c155900 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdb2001a060 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb1f7fe700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb0c0539b0 0x7fdb0c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdb2006b780 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.763+0000 7fdb31857700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb0c0539b0 0x7fdb0c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.767+0000 7fdb31857700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb0c0539b0 0x7fdb0c055e70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fdb2801a000 tx=0x7fdb28016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.771+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb200426e0 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.895+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb2c0af730 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb1f7fe700 1 -- 172.21.15.61:0/11950236 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fdb20018020 con 0x7fdb2c0b47f0 2022-07-12T08:51:07.902 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:07.902 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":146,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:07.631932+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:51:06.224007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-12T08:51:06.803491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:51:07.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb0c0539b0 msgr2=0x7fdb0c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb0c0539b0 0x7fdb0c055e70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7fdb2801a000 tx=0x7fdb28016000).stop 2022-07-12T08:51:07.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 msgr2=0x7fdb2c05c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:07.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 0x7fdb2c05c560 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7fdb20001110 tx=0x7fdb20007b80).stop 2022-07-12T08:51:07.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 shutdown_connections 2022-07-12T08:51:07.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fdb2c142d50 0x7fdb2c0ad340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fdb0c0539b0 0x7fdb0c055e70 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb2c13ce60 0x7fdb2c155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 --2- 172.21.15.61:0/11950236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb2c0b47f0 0x7fdb2c05c560 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:07.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 >> 172.21.15.61:0/11950236 conn(0x7fdb2c05f9f0 msgr2=0x7fdb2c0b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:07.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 shutdown_connections 2022-07-12T08:51:07.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:07.899+0000 7fdb342bc700 1 -- 172.21.15.61:0/11950236 wait complete. 2022-07-12T08:51:07.922 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:51:08.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ea4a700 1 Processor -- start 2022-07-12T08:51:08.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ea4a700 1 -- start start 2022-07-12T08:51:08.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ea4a700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8f3805f140 0x7f8f38151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ea4a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f380abc70 con 0x7f8f380ab290 2022-07-12T08:51:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ea4a700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3805c720 con 0x7f8f380ab670 2022-07-12T08:51:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ea4a700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f380aba90 con 0x7f8f3805f140 2022-07-12T08:51:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3da48700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8f380ab290 0x7f8f3805ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50566/0 (socket says 172.21.15.61:50566) 2022-07-12T08:51:08.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3da48700 1 -- 172.21.15.61:0/1425265902 learned_addr learned my addr 172.21.15.61:0/1425265902 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:08.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3e249700 1 --2- 172.21.15.61:0/1425265902 >> v2:172.21.15.111:3300/0 conn(0x7f8f3805f140 0x7f8f38151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ca46700 1 -- 172.21.15.61:0/1425265902 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78812884 0 0) 0x7f8f380abc70 con 0x7f8f380ab290 2022-07-12T08:51:08.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ca46700 1 -- 172.21.15.61:0/1425265902 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f20003260 con 0x7f8f380ab290 2022-07-12T08:51:08.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3e249700 1 -- 172.21.15.61:0/1425265902 >> v1:172.21.15.111:6789/0 conn(0x7f8f380ab670 legacy=0x7f8f380afa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3e249700 1 -- 172.21.15.61:0/1425265902 >> v1:172.21.15.61:6789/0 conn(0x7f8f380ab290 legacy=0x7f8f3805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3e249700 1 -- 172.21.15.61:0/1425265902 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3805c9c0 con 0x7f8f3805f140 2022-07-12T08:51:08.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3e249700 1 --2- 172.21.15.61:0/1425265902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3805f140 0x7f8f38151bf0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f8f300032f0 tx=0x7f8f300091d0).ready entity=mon.1 client_cookie=e35fb4b5332696af server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ca46700 1 -- 172.21.15.61:0/1425265902 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f3000b6c0 con 0x7f8f3805f140 2022-07-12T08:51:08.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.027+0000 7f8f3ca46700 1 -- 172.21.15.61:0/1425265902 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3000b820 con 0x7f8f3805f140 2022-07-12T08:51:08.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ca46700 1 -- 172.21.15.61:0/1425265902 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f3000b990 con 0x7f8f3805f140 2022-07-12T08:51:08.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/1425265902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3805f140 msgr2=0x7f8f38151bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/1425265902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3805f140 0x7f8f38151bf0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7f8f300032f0 tx=0x7f8f300091d0).stop 2022-07-12T08:51:08.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/1425265902 shutdown_connections 2022-07-12T08:51:08.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/1425265902 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f3805f140 0x7f8f38151bf0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/1425265902 >> 172.21.15.61:0/1425265902 conn(0x7f8f380b1950 msgr2=0x7f8f380b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:08.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/1425265902 shutdown_connections 2022-07-12T08:51:08.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/1425265902 wait complete. 2022-07-12T08:51:08.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 Processor -- start 2022-07-12T08:51:08.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- start start 2022-07-12T08:51:08.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 0x7f8f38067360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f380ab290 0x7f8f3814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f380ab670 0x7f8f38148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f380653c0 con 0x7f8f3805f140 2022-07-12T08:51:08.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f38066840 con 0x7f8f380ab290 2022-07-12T08:51:08.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3ea4a700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3814b870 con 0x7f8f380ab670 2022-07-12T08:51:08.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3da48700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 0x7f8f38067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3da48700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 0x7f8f38067360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34120/0 (socket says 172.21.15.61:34120) 2022-07-12T08:51:08.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3da48700 1 -- 172.21.15.61:0/3876352725 learned_addr learned my addr 172.21.15.61:0/3876352725 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:08.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3e249700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f380ab670 0x7f8f38148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.031+0000 7f8f3d247700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f380ab290 0x7f8f3814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3da48700 1 -- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f380ab670 msgr2=0x7f8f38148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3da48700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f380ab670 0x7f8f38148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3da48700 1 -- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f380ab290 msgr2=0x7f8f3814ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3da48700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f380ab290 0x7f8f3814ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3da48700 1 -- 172.21.15.61:0/3876352725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f30009470 con 0x7f8f3805f140 2022-07-12T08:51:08.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3d247700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f380ab290 0x7f8f3814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:08.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3e249700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f380ab670 0x7f8f38148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:08.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3da48700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 0x7f8f38067360 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f8f28001110 tx=0x7f8f28007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f280070d0 con 0x7f8f3805f140 2022-07-12T08:51:08.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f28007230 con 0x7f8f3805f140 2022-07-12T08:51:08.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3814bb10 con 0x7f8f3805f140 2022-07-12T08:51:08.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f28005610 con 0x7f8f3805f140 2022-07-12T08:51:08.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3814b490 con 0x7f8f3805f140 2022-07-12T08:51:08.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f38066d30 con 0x7f8f3805f140 2022-07-12T08:51:08.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.035+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f28010070 con 0x7f8f3805f140 2022-07-12T08:51:08.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.039+0000 7f8f26ffd700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f18053ad0 0x7f8f18055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.039+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8f2806b600 con 0x7f8f3805f140 2022-07-12T08:51:08.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.039+0000 7f8f3d247700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f18053ad0 0x7f8f18055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.039+0000 7f8f3d247700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f18053ad0 0x7f8f18055f90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f8f3400d000 tx=0x7f8f34009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.043+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f38066d30 con 0x7f8f3805f140 2022-07-12T08:51:08.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.167+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f38066d30 con 0x7f8f3805f140 2022-07-12T08:51:08.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f26ffd700 1 -- 172.21.15.61:0/3876352725 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f8f38066d30 con 0x7f8f3805f140 2022-07-12T08:51:08.178 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:08.178 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":146,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:07.631932+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:51:06.224007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-12T08:51:06.803491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:51:08.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f18053ad0 msgr2=0x7f8f18055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f18053ad0 0x7f8f18055f90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f8f3400d000 tx=0x7f8f34009040).stop 2022-07-12T08:51:08.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 msgr2=0x7f8f38067360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 0x7f8f38067360 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7f8f28001110 tx=0x7f8f28007b80).stop 2022-07-12T08:51:08.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 shutdown_connections 2022-07-12T08:51:08.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8f380ab670 0x7f8f38148610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8f18053ad0 0x7f8f18055f90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8f380ab290 0x7f8f3814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 --2- 172.21.15.61:0/3876352725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f3805f140 0x7f8f38067360 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 >> 172.21.15.61:0/3876352725 conn(0x7f8f380b1950 msgr2=0x7f8f380b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:08.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 shutdown_connections 2022-07-12T08:51:08.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.171+0000 7f8f3ea4a700 1 -- 172.21.15.61:0/3876352725 wait complete. 2022-07-12T08:51:08.195 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-12T08:51:08.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.311+0000 7f8b86c7c700 1 Processor -- start 2022-07-12T08:51:08.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.311+0000 7f8b86c7c700 1 -- start start 2022-07-12T08:51:08.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b800b3ea0 0x7f8b800b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8b8005c810 0x7f8b800b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b800b3120 con 0x7f8b800b9db0 2022-07-12T08:51:08.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8005bd10 con 0x7f8b8005c810 2022-07-12T08:51:08.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b800b33c0 con 0x7f8b800b3ea0 2022-07-12T08:51:08.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b85219700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8b8005c810 0x7f8b800b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b84a18700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8b800b9db0 0x7f8b8005c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50578/0 (socket says 172.21.15.61:50578) 2022-07-12T08:51:08.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b84a18700 1 -- 172.21.15.61:0/2995963112 learned_addr learned my addr 172.21.15.61:0/2995963112 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:08.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b7ffff700 1 --2- 172.21.15.61:0/2995963112 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b800b3ea0 0x7f8b800b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b85219700 1 -- 172.21.15.61:0/2995963112 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b800b3ea0 msgr2=0x7f8b800b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:08.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b85219700 1 --2- 172.21.15.61:0/2995963112 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b800b3ea0 0x7f8b800b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b7f7fe700 1 -- 172.21.15.61:0/2995963112 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200563923 0 0) 0x7f8b800b3120 con 0x7f8b800b9db0 2022-07-12T08:51:08.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b85219700 1 -- 172.21.15.61:0/2995963112 >> v1:172.21.15.61:6789/0 conn(0x7f8b800b9db0 legacy=0x7f8b8005c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b85219700 1 -- 172.21.15.61:0/2995963112 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8005cc60 con 0x7f8b8005c810 2022-07-12T08:51:08.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b85219700 1 --2- 172.21.15.61:0/2995963112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b8005c810 0x7f8b800b4c90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f8b780032f0 tx=0x7f8b780091d0).ready entity=mon.0 client_cookie=35dce51c1b1848a4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b7f7fe700 1 -- 172.21.15.61:0/2995963112 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b7800b6c0 con 0x7f8b8005c810 2022-07-12T08:51:08.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b7f7fe700 1 -- 172.21.15.61:0/2995963112 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7800b820 con 0x7f8b8005c810 2022-07-12T08:51:08.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b7f7fe700 1 -- 172.21.15.61:0/2995963112 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b7800b990 con 0x7f8b8005c810 2022-07-12T08:51:08.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- 172.21.15.61:0/2995963112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b8005c810 msgr2=0x7f8b800b4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/2995963112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b8005c810 0x7f8b800b4c90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 rx=0x7f8b780032f0 tx=0x7f8b780091d0).stop 2022-07-12T08:51:08.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- 172.21.15.61:0/2995963112 shutdown_connections 2022-07-12T08:51:08.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/2995963112 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b8005c810 0x7f8b800b4c90 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/2995963112 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b800b3ea0 0x7f8b800b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- 172.21.15.61:0/2995963112 >> 172.21.15.61:0/2995963112 conn(0x7f8b80067960 msgr2=0x7f8b800b9500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:08.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- 172.21.15.61:0/2995963112 shutdown_connections 2022-07-12T08:51:08.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.315+0000 7f8b86c7c700 1 -- 172.21.15.61:0/2995963112 wait complete. 2022-07-12T08:51:08.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 Processor -- start 2022-07-12T08:51:08.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 -- start start 2022-07-12T08:51:08.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b8005c810 0x7f8b801159f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b800b3ea0 0x7f8b80112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 0x7f8b80113490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80116040 con 0x7f8b800b9db0 2022-07-12T08:51:08.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80155db0 con 0x7f8b800b3ea0 2022-07-12T08:51:08.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80116470 con 0x7f8b8005c810 2022-07-12T08:51:08.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 0x7f8b80113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 0x7f8b80113490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34136/0 (socket says 172.21.15.61:34136) 2022-07-12T08:51:08.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 -- 172.21.15.61:0/987693352 learned_addr learned my addr 172.21.15.61:0/987693352 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:08.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b7ffff700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b800b3ea0 0x7f8b80112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b84a18700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b8005c810 0x7f8b801159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 -- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b8005c810 msgr2=0x7f8b801159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b8005c810 0x7f8b801159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 -- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b800b3ea0 msgr2=0x7f8b80112fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b800b3ea0 0x7f8b80112fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 -- 172.21.15.61:0/987693352 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78009470 con 0x7f8b800b9db0 2022-07-12T08:51:08.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b84a18700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b8005c810 0x7f8b801159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:08.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b7ffff700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b800b3ea0 0x7f8b80112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:08.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b85219700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 0x7f8b80113490 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f8b780032f0 tx=0x7f8b78016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b7800b6c0 con 0x7f8b800b9db0 2022-07-12T08:51:08.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b780107a0 con 0x7f8b800b9db0 2022-07-12T08:51:08.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b80156050 con 0x7f8b800b9db0 2022-07-12T08:51:08.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b78010b00 con 0x7f8b800b9db0 2022-07-12T08:51:08.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.319+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b801126f0 con 0x7f8b800b9db0 2022-07-12T08:51:08.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.323+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b801161b0 con 0x7f8b800b9db0 2022-07-12T08:51:08.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.323+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8b7800f070 con 0x7f8b800b9db0 2022-07-12T08:51:08.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.323+0000 7f8b7dffb700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b60053a40 0x7f8b60055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.323+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8b7806cc20 con 0x7f8b800b9db0 2022-07-12T08:51:08.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.323+0000 7f8b84a18700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b60053a40 0x7f8b60055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.323+0000 7f8b84a18700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b60053a40 0x7f8b60055f00 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f8b74008000 tx=0x7f8b74009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.331+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b801161b0 con 0x7f8b800b9db0 2022-07-12T08:51:08.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.499+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8b80155c00 con 0x7f8b800b9db0 2022-07-12T08:51:08.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b7dffb700 1 -- 172.21.15.61:0/987693352 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v83) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8b78015070 con 0x7f8b800b9db0 2022-07-12T08:51:08.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b60053a40 msgr2=0x7f8b60055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b60053a40 0x7f8b60055f00 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f8b74008000 tx=0x7f8b74009040).stop 2022-07-12T08:51:08.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 msgr2=0x7f8b80113490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 0x7f8b80113490 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f8b780032f0 tx=0x7f8b78016000).stop 2022-07-12T08:51:08.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 shutdown_connections 2022-07-12T08:51:08.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b800b9db0 0x7f8b80113490 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b60053a40 0x7f8b60055f00 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b800b3ea0 0x7f8b80112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 --2- 172.21.15.61:0/987693352 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b8005c810 0x7f8b801159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 >> 172.21.15.61:0/987693352 conn(0x7f8b80067960 msgr2=0x7f8b800b8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:08.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 shutdown_connections 2022-07-12T08:51:08.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.651+0000 7f8b86c7c700 1 -- 172.21.15.61:0/987693352 wait complete. 2022-07-12T08:51:08.679 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:08.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 Processor -- start 2022-07-12T08:51:08.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 -- start start 2022-07-12T08:51:08.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64780ab670 0x7f647805f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780afa60 0x7f64780afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64780abc70 con 0x7f64780ab290 2022-07-12T08:51:08.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64780aba90 con 0x7f64780ab670 2022-07-12T08:51:08.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647e245700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647805c720 con 0x7f64780afa60 2022-07-12T08:51:08.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64780ab670 0x7f647805f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647da44700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780afa60 0x7f64780afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f64780ab670 0x7f647805f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:40946/0 (socket says 172.21.15.61:40946) 2022-07-12T08:51:08.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 -- 172.21.15.61:0/1814387433 learned_addr learned my addr 172.21.15.61:0/1814387433 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:08.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 -- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780afa60 msgr2=0x7f64780afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:08.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 --2- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780afa60 0x7f64780afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 -- 172.21.15.61:0/1814387433 >> v1:172.21.15.111:6789/0 conn(0x7f64780ab290 legacy=0x7f647805ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 -- 172.21.15.61:0/1814387433 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647805c9c0 con 0x7f64780ab670 2022-07-12T08:51:08.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f647ca42700 1 --2- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780ab670 0x7f647805f140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f6474002f70 tx=0x7f6474006eb0).ready entity=mon.1 client_cookie=13ea8bcec757aceb server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f646ffff700 1 -- 172.21.15.61:0/1814387433 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f647400d570 con 0x7f64780ab670 2022-07-12T08:51:08.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f646ffff700 1 -- 172.21.15.61:0/1814387433 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474000e90 con 0x7f64780ab670 2022-07-12T08:51:08.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.795+0000 7f646ffff700 1 -- 172.21.15.61:0/1814387433 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f647400d900 con 0x7f64780ab670 2022-07-12T08:51:08.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 -- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780ab670 msgr2=0x7f647805f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 --2- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780ab670 0x7f647805f140 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7f6474002f70 tx=0x7f6474006eb0).stop 2022-07-12T08:51:08.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 -- 172.21.15.61:0/1814387433 shutdown_connections 2022-07-12T08:51:08.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 --2- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780afa60 0x7f64780afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 --2- 172.21.15.61:0/1814387433 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780ab670 0x7f647805f140 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 -- 172.21.15.61:0/1814387433 >> 172.21.15.61:0/1814387433 conn(0x7f64780b1950 msgr2=0x7f64780b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:08.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 -- 172.21.15.61:0/1814387433 shutdown_connections 2022-07-12T08:51:08.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 -- 172.21.15.61:0/1814387433 wait complete. 2022-07-12T08:51:08.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 Processor -- start 2022-07-12T08:51:08.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.799+0000 7f647e245700 1 -- start start 2022-07-12T08:51:08.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 0x7f64780672d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780ab670 0x7f64780ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780afa60 0x7f64780ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64781450c0 con 0x7f64780ab290 2022-07-12T08:51:08.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6478145360 con 0x7f64780afa60 2022-07-12T08:51:08.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64780af170 con 0x7f64780ab670 2022-07-12T08:51:08.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 0x7f64780672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 0x7f64780672d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34146/0 (socket says 172.21.15.61:34146) 2022-07-12T08:51:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 -- 172.21.15.61:0/258895599 learned_addr learned my addr 172.21.15.61:0/258895599 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647ca42700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780ab670 0x7f64780ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647da44700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780afa60 0x7f64780ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 -- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780ab670 msgr2=0x7f64780ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780ab670 0x7f64780ae790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 -- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780afa60 msgr2=0x7f64780ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780afa60 0x7f64780ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 -- 172.21.15.61:0/258895599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647400b1e0 con 0x7f64780ab290 2022-07-12T08:51:08.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647d243700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 0x7f64780672d0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f64780615d0 tx=0x7f6468007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6468007030 con 0x7f64780ab290 2022-07-12T08:51:08.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468007190 con 0x7f64780ab290 2022-07-12T08:51:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6478066cb0 con 0x7f64780ab290 2022-07-12T08:51:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6468006510 con 0x7f64780ab290 2022-07-12T08:51:08.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6478066f60 con 0x7f64780ab290 2022-07-12T08:51:08.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.803+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6478145520 con 0x7f64780ab290 2022-07-12T08:51:08.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.807+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6468010030 con 0x7f64780ab290 2022-07-12T08:51:08.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.807+0000 7f646e7fc700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6460053a60 0x7f6460055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:08.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.807+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f646806ba30 con 0x7f64780ab290 2022-07-12T08:51:08.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.807+0000 7f647ca42700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6460053a60 0x7f6460055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:08.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.807+0000 7f647ca42700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6460053a60 0x7f6460055f20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f647401c000 tx=0x7f6474015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:08.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.815+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64680428f0 con 0x7f64780ab290 2022-07-12T08:51:08.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.983+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6478057460 con 0x7f64780ab290 2022-07-12T08:51:08.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f646e7fc700 1 -- 172.21.15.61:0/258895599 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f6468045f10 con 0x7f64780ab290 2022-07-12T08:51:08.991 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:08.991 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1725259132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1725259132},{"type":"v1","addr":"172.21.15.111:6835","nonce":1725259132}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:51:07.632105+0000","modified":"2022-07-12T08:51:08.635747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8775":{"gid":8775,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/502386216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":502386216},{"type":"v1","addr":"172.21.15.61:6833","nonce":502386216}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-12T08:51:08.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6460053a60 msgr2=0x7f6460055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6460053a60 0x7f6460055f20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f647401c000 tx=0x7f6474015000).stop 2022-07-12T08:51:08.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 msgr2=0x7f64780672d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:08.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 0x7f64780672d0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f64780615d0 tx=0x7f6468007c10).stop 2022-07-12T08:51:08.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 shutdown_connections 2022-07-12T08:51:08.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f6460053a60 0x7f6460055f20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64780afa60 0x7f64780ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f64780ab670 0x7f64780ae790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 --2- 172.21.15.61:0/258895599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64780ab290 0x7f64780672d0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:08.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 >> 172.21.15.61:0/258895599 conn(0x7f64780b1950 msgr2=0x7f64780b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:08.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 shutdown_connections 2022-07-12T08:51:09.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:08.987+0000 7f647e245700 1 -- 172.21.15.61:0/258895599 wait complete. 2022-07-12T08:51:09.003 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-12T08:51:09.008 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:51:09.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 Processor -- start 2022-07-12T08:51:09.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 -- start start 2022-07-12T08:51:09.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00680af0b0 0x7f0068060920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0068060e10 0x7f006805bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00680b02e0 con 0x7f0068061270 2022-07-12T08:51:09.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0068067040 con 0x7f0068060e10 2022-07-12T08:51:09.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006eecb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00680af490 con 0x7f00680af0b0 2022-07-12T08:51:09.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006d6c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0068060e10 0x7f006805bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006d6c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0068060e10 0x7f006805bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34156/0 (socket says 172.21.15.61:34156) 2022-07-12T08:51:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006d6c8700 1 -- 172.21.15.61:0/3991052114 learned_addr learned my addr 172.21.15.61:0/3991052114 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.107+0000 7f006dec9700 1 --2- 172.21.15.61:0/3991052114 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00680af0b0 0x7f0068060920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006d6c8700 1 -- 172.21.15.61:0/3991052114 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00680af0b0 msgr2=0x7f0068060920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006d6c8700 1 --2- 172.21.15.61:0/3991052114 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00680af0b0 0x7f0068060920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006d6c8700 1 -- 172.21.15.61:0/3991052114 >> v1:172.21.15.61:6789/0 conn(0x7f0068061270 legacy=0x7f006805c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:09.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006d6c8700 1 -- 172.21.15.61:0/3991052114 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00680641f0 con 0x7f0068060e10 2022-07-12T08:51:09.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006d6c8700 1 --2- 172.21.15.61:0/3991052114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f006805bd10 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7f0058002f70 tx=0x7f0058006eb0).ready entity=mon.0 client_cookie=2a91db94f1886db server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006cec7700 1 -- 172.21.15.61:0/3991052114 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f005800d5c0 con 0x7f0068060e10 2022-07-12T08:51:09.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006cec7700 1 -- 172.21.15.61:0/3991052114 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0058000e90 con 0x7f0068060e10 2022-07-12T08:51:09.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006cec7700 1 -- 172.21.15.61:0/3991052114 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f005800d950 con 0x7f0068060e10 2022-07-12T08:51:09.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 -- 172.21.15.61:0/3991052114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 msgr2=0x7f006805bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 --2- 172.21.15.61:0/3991052114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f006805bd10 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7f0058002f70 tx=0x7f0058006eb0).stop 2022-07-12T08:51:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 -- 172.21.15.61:0/3991052114 shutdown_connections 2022-07-12T08:51:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 --2- 172.21.15.61:0/3991052114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f006805bd10 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 --2- 172.21.15.61:0/3991052114 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f00680af0b0 0x7f0068060920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 -- 172.21.15.61:0/3991052114 >> 172.21.15.61:0/3991052114 conn(0x7f00680b5780 msgr2=0x7f00680b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:09.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 -- 172.21.15.61:0/3991052114 shutdown_connections 2022-07-12T08:51:09.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.111+0000 7f006eecb700 1 -- 172.21.15.61:0/3991052114 wait complete. 2022-07-12T08:51:09.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 Processor -- start 2022-07-12T08:51:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 -- start start 2022-07-12T08:51:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f0068141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0068061270 0x7f00680b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00680af0b0 0x7f0068152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068142540 con 0x7f0068060e10 2022-07-12T08:51:09.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00681453a0 con 0x7f00680af0b0 2022-07-12T08:51:09.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00681422a0 con 0x7f0068061270 2022-07-12T08:51:09.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f0068141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f0068141c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34160/0 (socket says 172.21.15.61:34160) 2022-07-12T08:51:09.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 -- 172.21.15.61:0/396167663 learned_addr learned my addr 172.21.15.61:0/396167663 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:09.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006d6c8700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0068061270 0x7f00680b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006e6ca700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00680af0b0 0x7f0068152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 -- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0068061270 msgr2=0x7f00680b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0068061270 0x7f00680b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 -- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00680af0b0 msgr2=0x7f0068152f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00680af0b0 0x7f0068152f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 -- 172.21.15.61:0/396167663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005800b1e0 con 0x7f0068060e10 2022-07-12T08:51:09.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006dec9700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f0068141c70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f00640030b0 tx=0x7f0064008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0064011780 con 0x7f0068060e10 2022-07-12T08:51:09.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00640118e0 con 0x7f0068060e10 2022-07-12T08:51:09.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006813e760 con 0x7f0068060e10 2022-07-12T08:51:09.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0064011c40 con 0x7f0068060e10 2022-07-12T08:51:09.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.115+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0068146c40 con 0x7f0068060e10 2022-07-12T08:51:09.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.119+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0068145560 con 0x7f0068060e10 2022-07-12T08:51:09.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.119+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0064012420 con 0x7f0068060e10 2022-07-12T08:51:09.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.119+0000 7f0056ffd700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0048053a40 0x7f0048055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.119+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f006406bcd0 con 0x7f0068060e10 2022-07-12T08:51:09.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.119+0000 7f006d6c8700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0048053a40 0x7f0048055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.119+0000 7f006d6c8700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0048053a40 0x7f0048055f00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f005801a000 tx=0x7f0058015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.123+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0068145560 con 0x7f0068060e10 2022-07-12T08:51:09.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.251+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0068145560 con 0x7f0068060e10 2022-07-12T08:51:09.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.255+0000 7f0056ffd700 1 -- 172.21.15.61:0/396167663 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f0068145560 con 0x7f0068060e10 2022-07-12T08:51:09.265 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:09.265 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":147,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:08.634798+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:51:06.224007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-12T08:51:06.803491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:51:09.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0048053a40 msgr2=0x7f0048055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0048053a40 0x7f0048055f00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7f005801a000 tx=0x7f0058015000).stop 2022-07-12T08:51:09.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 msgr2=0x7f0068141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f0068141c70 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 rx=0x7f00640030b0 tx=0x7f0064008040).stop 2022-07-12T08:51:09.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 shutdown_connections 2022-07-12T08:51:09.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0048053a40 0x7f0048055f00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f00680af0b0 0x7f0068152f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0068061270 0x7f00680b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 --2- 172.21.15.61:0/396167663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0068060e10 0x7f0068141c70 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 >> 172.21.15.61:0/396167663 conn(0x7f00680b5780 msgr2=0x7f006813b790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:09.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 shutdown_connections 2022-07-12T08:51:09.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.259+0000 7f006eecb700 1 -- 172.21.15.61:0/396167663 wait complete. 2022-07-12T08:51:09.277 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:51:09.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 Processor -- start 2022-07-12T08:51:09.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 -- start start 2022-07-12T08:51:09.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0146140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f99c0060760 0x7f99c01419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c00655a0 con 0x7f99c005f480 2022-07-12T08:51:09.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c0142a60 con 0x7f99c0060760 2022-07-12T08:51:09.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99c4c5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00b29f0 con 0x7f99c00b0ef0 2022-07-12T08:51:09.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99bf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f99c005f480 0x7f99c0060050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50618/0 (socket says 172.21.15.61:50618) 2022-07-12T08:51:09.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99bf7fe700 1 -- 172.21.15.61:0/3265523926 learned_addr learned my addr 172.21.15.61:0/3265523926 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:09.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99beffd700 1 --2- 172.21.15.61:0/3265523926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0146140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99bffff700 1 --2- 172.21.15.61:0/3265523926 >> v2:172.21.15.111:3300/0 conn(0x7f99c0060760 0x7f99c01419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.391+0000 7f99be7fc700 1 -- 172.21.15.61:0/3265523926 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501725199 0 0) 0x7f99c00655a0 con 0x7f99c005f480 2022-07-12T08:51:09.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99be7fc700 1 -- 172.21.15.61:0/3265523926 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99ac003260 con 0x7f99c005f480 2022-07-12T08:51:09.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99beffd700 1 -- 172.21.15.61:0/3265523926 >> v2:172.21.15.111:3300/0 conn(0x7f99c0060760 msgr2=0x7f99c01419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:09.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99be7fc700 1 -- 172.21.15.61:0/3265523926 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 111906280 0 0) 0x7f99ac003260 con 0x7f99c005f480 2022-07-12T08:51:09.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99beffd700 1 --2- 172.21.15.61:0/3265523926 >> v2:172.21.15.111:3300/0 conn(0x7f99c0060760 0x7f99c01419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99beffd700 1 -- 172.21.15.61:0/3265523926 >> v1:172.21.15.61:6789/0 conn(0x7f99c005f480 legacy=0x7f99c0060050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99beffd700 1 -- 172.21.15.61:0/3265523926 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c005f860 con 0x7f99c00b0ef0 2022-07-12T08:51:09.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99bffff700 1 --2- 172.21.15.61:0/3265523926 >> v2:172.21.15.111:3300/0 conn(0x7f99c0060760 0x7f99c01419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:09.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99beffd700 1 --2- 172.21.15.61:0/3265523926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0146140 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f99b0001800 tx=0x7f99b4006df0).ready entity=mon.2 client_cookie=ae373b4f9f53935f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99be7fc700 1 -- 172.21.15.61:0/3265523926 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b0001660 con 0x7f99c00b0ef0 2022-07-12T08:51:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99be7fc700 1 -- 172.21.15.61:0/3265523926 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b4000e90 con 0x7f99c00b0ef0 2022-07-12T08:51:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99be7fc700 1 -- 172.21.15.61:0/3265523926 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b4012460 con 0x7f99c00b0ef0 2022-07-12T08:51:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 -- 172.21.15.61:0/3265523926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 msgr2=0x7f99c0146140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/3265523926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0146140 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f99b0001800 tx=0x7f99b4006df0).stop 2022-07-12T08:51:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 -- 172.21.15.61:0/3265523926 shutdown_connections 2022-07-12T08:51:09.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/3265523926 >> v2:172.21.15.111:3300/0 conn(0x7f99c0060760 0x7f99c01419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/3265523926 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0146140 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 -- 172.21.15.61:0/3265523926 >> 172.21.15.61:0/3265523926 conn(0x7f99c0063ad0 msgr2=0x7f99c0063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:09.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 -- 172.21.15.61:0/3265523926 shutdown_connections 2022-07-12T08:51:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 -- 172.21.15.61:0/3265523926 wait complete. 2022-07-12T08:51:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 Processor -- start 2022-07-12T08:51:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.395+0000 7f99c4c5b700 1 -- start start 2022-07-12T08:51:09.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c005f480 0x7f99c0116720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 0x7f99c0116c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c01153e0 con 0x7f99c0060760 2022-07-12T08:51:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c00b02e0 con 0x7f99c005f480 2022-07-12T08:51:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c01147f0 con 0x7f99c00b0ef0 2022-07-12T08:51:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 0x7f99c0116c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 0x7f99c0116c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34176/0 (socket says 172.21.15.61:34176) 2022-07-12T08:51:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 -- 172.21.15.61:0/1947209351 learned_addr learned my addr 172.21.15.61:0/1947209351 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99bffff700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99bf7fe700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c005f480 0x7f99c0116720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 -- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 msgr2=0x7f99c0154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 -- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c005f480 msgr2=0x7f99c0116720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c005f480 0x7f99c0116720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 -- 172.21.15.61:0/1947209351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8009470 con 0x7f99c0060760 2022-07-12T08:51:09.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99beffd700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 0x7f99c0116c10 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f99b0001800 tx=0x7f99b4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b4016070 con 0x7f99c0060760 2022-07-12T08:51:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b4013e40 con 0x7f99c0060760 2022-07-12T08:51:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b400b180 con 0x7f99c0060760 2022-07-12T08:51:09.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99b400de60 con 0x7f99c0060760 2022-07-12T08:51:09.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c00afd60 con 0x7f99c0060760 2022-07-12T08:51:09.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.399+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c00b00f0 con 0x7f99c0060760 2022-07-12T08:51:09.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.403+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f99b400f030 con 0x7f99c0060760 2022-07-12T08:51:09.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.403+0000 7f99bcff9700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a0053a40 0x7f99a0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.403+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f99b406cc30 con 0x7f99c0060760 2022-07-12T08:51:09.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.403+0000 7f99bf7fe700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a0053a40 0x7f99a0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.403+0000 7f99bf7fe700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a0053a40 0x7f99a0055f00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f99b000b000 tx=0x7f99b0001d40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.411+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99b4043c40 con 0x7f99c0060760 2022-07-12T08:51:09.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.579+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f99c0057460 con 0x7f99c0060760 2022-07-12T08:51:09.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.587+0000 7f99bcff9700 1 -- 172.21.15.61:0/1947209351 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v45) v1 ==== 184+0+0 (secure 0 0 0) 0x7f99b4047260 con 0x7f99c0060760 2022-07-12T08:51:09.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.587+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a0053a40 msgr2=0x7f99a0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.587+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a0053a40 0x7f99a0055f00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f99b000b000 tx=0x7f99b0001d40).stop 2022-07-12T08:51:09.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.587+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 msgr2=0x7f99c0116c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.587+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 0x7f99c0116c10 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7f99b0001800 tx=0x7f99b4014000).stop 2022-07-12T08:51:09.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 shutdown_connections 2022-07-12T08:51:09.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f99c00b0ef0 0x7f99c0154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99c0060760 0x7f99c0116c10 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f99a0053a40 0x7f99a0055f00 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 --2- 172.21.15.61:0/1947209351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f99c005f480 0x7f99c0116720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 >> 172.21.15.61:0/1947209351 conn(0x7f99c0063ad0 msgr2=0x7f99c00b5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 shutdown_connections 2022-07-12T08:51:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.591+0000 7f99c4c5b700 1 -- 172.21.15.61:0/1947209351 wait complete. 2022-07-12T08:51:09.599 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-12T08:51:09.606 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:51:09.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9fb90700 1 Processor -- start 2022-07-12T08:51:09.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9fb90700 1 -- start start 2022-07-12T08:51:09.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9fb90700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8b980b4050 0x7f8b980b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9fb90700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980b56c0 con 0x7f8b980b52e0 2022-07-12T08:51:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9fb90700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b980b4bd0 con 0x7f8b98141e50 2022-07-12T08:51:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9fb90700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b98142f00 con 0x7f8b980b4050 2022-07-12T08:51:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9d92c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8b980b52e0 0x7f8b980b21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50632/0 (socket says 172.21.15.61:50632) 2022-07-12T08:51:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.719+0000 7f8b9d92c700 1 -- 172.21.15.61:0/2141083550 learned_addr learned my addr 172.21.15.61:0/2141083550 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/2141083550 >> v2:172.21.15.111:3300/0 conn(0x7f8b980b4050 0x7f8b980b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777516550 0 0) 0x7f8b980b56c0 con 0x7f8b980b52e0 2022-07-12T08:51:09.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b80003260 con 0x7f8b980b52e0 2022-07-12T08:51:09.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195025758 0 0) 0x7f8b980b4bd0 con 0x7f8b98141e50 2022-07-12T08:51:09.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b980b56c0 con 0x7f8b98141e50 2022-07-12T08:51:09.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236426069 0 0) 0x7f8b80003260 con 0x7f8b980b52e0 2022-07-12T08:51:09.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 >> v2:172.21.15.111:3300/0 conn(0x7f8b980b4050 msgr2=0x7f8b980b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:09.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 --2- 172.21.15.61:0/2141083550 >> v2:172.21.15.111:3300/0 conn(0x7f8b980b4050 0x7f8b980b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 >> v1:172.21.15.111:6789/0 conn(0x7f8b98141e50 legacy=0x7f8b981422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980b0e80 con 0x7f8b980b52e0 2022-07-12T08:51:09.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7f8b94001660 con 0x7f8b980b52e0 2022-07-12T08:51:09.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 >> v1:172.21.15.61:6789/0 conn(0x7f8b980b52e0 legacy=0x7f8b980b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b80003930 0x7f8b80003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b80004500 0x7f8b80004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004e70 0x7f8b8000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8000bcf0 con 0x7f8b80004500 2022-07-12T08:51:09.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8000be60 con 0x7f8b80004e70 2022-07-12T08:51:09.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8000c020 con 0x7f8b80003930 2022-07-12T08:51:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b80004500 0x7f8b80004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d92c700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b80003930 0x7f8b80003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9e12d700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004e70 0x7f8b8000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 -- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b80003930 msgr2=0x7f8b80003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b80003930 0x7f8b80003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 -- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004e70 msgr2=0x7f8b8000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004e70 0x7f8b8000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9e12d700 1 -- 172.21.15.61:0/2141083550 reap_dead start 2022-07-12T08:51:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 -- 172.21.15.61:0/2141083550 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b80003260 con 0x7f8b80004500 2022-07-12T08:51:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b80004500 0x7f8b80004980 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f8b8800c000 tx=0x7f8b88006df0).ready entity=mon.0 client_cookie=a8f5faecb684485d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b880125e0 con 0x7f8b80004500 2022-07-12T08:51:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b88012740 con 0x7f8b80004500 2022-07-12T08:51:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9c92a700 1 -- 172.21.15.61:0/2141083550 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b88012aa0 con 0x7f8b80004500 2022-07-12T08:51:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 -- 172.21.15.61:0/2141083550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b80004500 msgr2=0x7f8b80004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b80004500 0x7f8b80004980 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7f8b8800c000 tx=0x7f8b88006df0).stop 2022-07-12T08:51:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 -- 172.21.15.61:0/2141083550 shutdown_connections 2022-07-12T08:51:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/2141083550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b80004500 0x7f8b80004980 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 -- 172.21.15.61:0/2141083550 >> 172.21.15.61:0/2141083550 conn(0x7f8b98067020 msgr2=0x7f8b98067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 -- 172.21.15.61:0/2141083550 shutdown_connections 2022-07-12T08:51:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.723+0000 7f8b9fb90700 1 -- 172.21.15.61:0/2141083550 wait complete. 2022-07-12T08:51:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 Processor -- start 2022-07-12T08:51:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 -- start start 2022-07-12T08:51:09.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004500 0x7f8b980afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b98147640 0x7f8b9813db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 0x7f8b981522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b981481b0 con 0x7f8b9813e040 2022-07-12T08:51:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9813e4c0 con 0x7f8b80004500 2022-07-12T08:51:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b98149320 con 0x7f8b98147640 2022-07-12T08:51:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 0x7f8b981522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 0x7f8b981522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34198/0 (socket says 172.21.15.61:34198) 2022-07-12T08:51:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 -- 172.21.15.61:0/714203650 learned_addr learned my addr 172.21.15.61:0/714203650 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9d92c700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004500 0x7f8b980afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b98147640 0x7f8b9813db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 -- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b98147640 msgr2=0x7f8b9813db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b98147640 0x7f8b9813db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 -- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004500 msgr2=0x7f8b980afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004500 0x7f8b980afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 -- 172.21.15.61:0/714203650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98147f10 con 0x7f8b9813e040 2022-07-12T08:51:09.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9d12b700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b98147640 0x7f8b9813db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:09.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9d92c700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004500 0x7f8b980afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:09.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9e12d700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 0x7f8b981522a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f8b98148320 tx=0x7f8b90007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b90006d60 con 0x7f8b9813e040 2022-07-12T08:51:09.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b90007030 con 0x7f8b9813e040 2022-07-12T08:51:09.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b98148e80 con 0x7f8b9813e040 2022-07-12T08:51:09.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b981445a0 con 0x7f8b9813e040 2022-07-12T08:51:09.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.727+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b900073d0 con 0x7f8b9813e040 2022-07-12T08:51:09.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.731+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b74003410 con 0x7f8b9813e040 2022-07-12T08:51:09.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.731+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8b9001b030 con 0x7f8b9813e040 2022-07-12T08:51:09.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.731+0000 7f8b86ffd700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b78053af0 0x7f8b78055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:09.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.731+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8b9006bb70 con 0x7f8b9813e040 2022-07-12T08:51:09.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.731+0000 7f8b9d92c700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b78053af0 0x7f8b78055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:09.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.731+0000 7f8b9d92c700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b78053af0 0x7f8b78055fb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f8b94005000 tx=0x7f8b94009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:09.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.735+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b90042ad0 con 0x7f8b9813e040 2022-07-12T08:51:09.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.907+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8b740036c0 con 0x7f8b9813e040 2022-07-12T08:51:09.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.915+0000 7f8b86ffd700 1 -- 172.21.15.61:0/714203650 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v46) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8b900460f0 con 0x7f8b9813e040 2022-07-12T08:51:09.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b78053af0 msgr2=0x7f8b78055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b78053af0 0x7f8b78055fb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f8b94005000 tx=0x7f8b94009040).stop 2022-07-12T08:51:09.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 msgr2=0x7f8b981522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:09.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 0x7f8b981522a0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 rx=0x7f8b98148320 tx=0x7f8b90007d80).stop 2022-07-12T08:51:09.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 shutdown_connections 2022-07-12T08:51:09.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b9813e040 0x7f8b981522a0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8b98147640 0x7f8b9813db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8b78053af0 0x7f8b78055fb0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 --2- 172.21.15.61:0/714203650 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b80004500 0x7f8b980afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:09.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 >> 172.21.15.61:0/714203650 conn(0x7f8b98067020 msgr2=0x7f8b980b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:09.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 shutdown_connections 2022-07-12T08:51:09.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:09.919+0000 7f8b9fb90700 1 -- 172.21.15.61:0/714203650 wait complete. 2022-07-12T08:51:09.930 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-12T08:51:09.939 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-12T08:51:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 Processor -- start 2022-07-12T08:51:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 -- start start 2022-07-12T08:51:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5c2c0b2de0 0x7f5c2c13f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c13c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c142df0 con 0x7f5c2c0af270 2022-07-12T08:51:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2c0b31c0 con 0x7f5c2c0b2de0 2022-07-12T08:51:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c30cc5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c0b0e00 con 0x7f5c2c0b3640 2022-07-12T08:51:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c2affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5c2c0af270 0x7f5c2c1434c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50652/0 (socket says 172.21.15.61:50652) 2022-07-12T08:51:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c2affd700 1 -- 172.21.15.61:0/718862286 learned_addr learned my addr 172.21.15.61:0/718862286 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c2b7fe700 1 --2- 172.21.15.61:0/718862286 >> v2:172.21.15.61:3300/0 conn(0x7f5c2c0b2de0 0x7f5c2c13f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c2bfff700 1 --2- 172.21.15.61:0/718862286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c13c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.055+0000 7f5c2a7fc700 1 -- 172.21.15.61:0/718862286 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035897497 0 0) 0x7f5c2c142df0 con 0x7f5c2c0af270 2022-07-12T08:51:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2a7fc700 1 -- 172.21.15.61:0/718862286 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c003260 con 0x7f5c2c0af270 2022-07-12T08:51:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2b7fe700 1 -- 172.21.15.61:0/718862286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 msgr2=0x7f5c2c13c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2b7fe700 1 --2- 172.21.15.61:0/718862286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c13c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2b7fe700 1 -- 172.21.15.61:0/718862286 >> v1:172.21.15.61:6789/0 conn(0x7f5c2c0af270 legacy=0x7f5c2c1434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2b7fe700 1 -- 172.21.15.61:0/718862286 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c143020 con 0x7f5c2c0b2de0 2022-07-12T08:51:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2b7fe700 1 --2- 172.21.15.61:0/718862286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c13f820 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f5c240032f0 tx=0x7f5c240091d0).ready entity=mon.0 client_cookie=a1c9bf34812bd5d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2a7fc700 1 -- 172.21.15.61:0/718862286 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c2400b600 con 0x7f5c2c0b2de0 2022-07-12T08:51:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2a7fc700 1 -- 172.21.15.61:0/718862286 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2400b760 con 0x7f5c2c0b2de0 2022-07-12T08:51:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c2a7fc700 1 -- 172.21.15.61:0/718862286 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c2400b8d0 con 0x7f5c2c0b2de0 2022-07-12T08:51:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 -- 172.21.15.61:0/718862286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 msgr2=0x7f5c2c13f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/718862286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c13f820 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 rx=0x7f5c240032f0 tx=0x7f5c240091d0).stop 2022-07-12T08:51:10.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 -- 172.21.15.61:0/718862286 shutdown_connections 2022-07-12T08:51:10.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/718862286 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c13c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/718862286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c13f820 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 -- 172.21.15.61:0/718862286 >> 172.21.15.61:0/718862286 conn(0x7f5c2c063a60 msgr2=0x7f5c2c063e70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:10.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 -- 172.21.15.61:0/718862286 shutdown_connections 2022-07-12T08:51:10.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 -- 172.21.15.61:0/718862286 wait complete. 2022-07-12T08:51:10.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 Processor -- start 2022-07-12T08:51:10.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.059+0000 7f5c30cc5700 1 -- start start 2022-07-12T08:51:10.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c2c0af270 0x7f5c2c146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c146920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c0b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c13b890 con 0x7f5c2c0b2de0 2022-07-12T08:51:10.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c144c60 con 0x7f5c2c0af270 2022-07-12T08:51:10.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2c144f00 con 0x7f5c2c0b3640 2022-07-12T08:51:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c0b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c146920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34210/0 (socket says 172.21.15.61:34210) 2022-07-12T08:51:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2affd700 1 -- 172.21.15.61:0/1392457640 learned_addr learned my addr 172.21.15.61:0/1392457640 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2b7fe700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c2c0af270 0x7f5c2c146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 -- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c2c0af270 msgr2=0x7f5c2c146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c2c0af270 0x7f5c2c146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 -- 172.21.15.61:0/1392457640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 msgr2=0x7f5c2c146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c146920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 -- 172.21.15.61:0/1392457640 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c24009470 con 0x7f5c2c0b3640 2022-07-12T08:51:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c2bfff700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c0b6530 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5c1c00c000 tx=0x7f5c1c006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c1c016070 con 0x7f5c2c0b3640 2022-07-12T08:51:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c011750 con 0x7f5c2c0b3640 2022-07-12T08:51:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c139ad0 con 0x7f5c2c0b3640 2022-07-12T08:51:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c1c011ab0 con 0x7f5c2c0b3640 2022-07-12T08:51:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c0b4040 con 0x7f5c2c0b3640 2022-07-12T08:51:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2c13ba00 con 0x7f5c2c0b3640 2022-07-12T08:51:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.063+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5c1c011c10 con 0x7f5c2c0b3640 2022-07-12T08:51:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.067+0000 7f5c28ff9700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c10053af0 0x7f5c10055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.067+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5c1c06bf80 con 0x7f5c2c0b3640 2022-07-12T08:51:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.067+0000 7f5c2b7fe700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c10053af0 0x7f5c10055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.067+0000 7f5c2b7fe700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c10053af0 0x7f5c10055fb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f5c2401b000 tx=0x7f5c24016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:10.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.075+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5c1c042ee0 con 0x7f5c2c0b3640 2022-07-12T08:51:10.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.239+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5c2c057460 con 0x7f5c2c0b3640 2022-07-12T08:51:10.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.251+0000 7f5c28ff9700 1 -- 172.21.15.61:0/1392457640 <== mon.2 v2:172.21.15.111:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5c1c046500 con 0x7f5c2c0b3640 2022-07-12T08:51:10.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c10053af0 msgr2=0x7f5c10055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c10053af0 0x7f5c10055fb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f5c2401b000 tx=0x7f5c24016000).stop 2022-07-12T08:51:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 msgr2=0x7f5c2c0b6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c0b6530 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f5c1c00c000 tx=0x7f5c1c006df0).stop 2022-07-12T08:51:10.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 shutdown_connections 2022-07-12T08:51:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5c2c0b3640 0x7f5c2c0b6530 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c2c0b2de0 0x7f5c2c146920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c10053af0 0x7f5c10055fb0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 --2- 172.21.15.61:0/1392457640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5c2c0af270 0x7f5c2c146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 >> 172.21.15.61:0/1392457640 conn(0x7f5c2c063a60 msgr2=0x7f5c2c060050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 shutdown_connections 2022-07-12T08:51:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.255+0000 7f5c30cc5700 1 -- 172.21.15.61:0/1392457640 wait complete. 2022-07-12T08:51:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-12T08:51:10.272 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:10.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 Processor -- start 2022-07-12T08:51:10.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 -- start start 2022-07-12T08:51:10.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd3180af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd3180b3f60 0x7fd3180b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3180aecb0 con 0x7fd3180af280 2022-07-12T08:51:10.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd318143b00 con 0x7fd3180b3f60 2022-07-12T08:51:10.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ee96700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3181435c0 con 0x7fd31809bce0 2022-07-12T08:51:10.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31e695700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd3180b3f60 0x7fd3180b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd3180af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd3180af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:44646/0 (socket says 172.21.15.61:44646) 2022-07-12T08:51:10.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 -- 172.21.15.61:0/2543290712 learned_addr learned my addr 172.21.15.61:0/2543290712 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:10.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31de94700 1 --1- 172.21.15.61:0/2543290712 >> v1:172.21.15.111:6789/0 conn(0x7fd3180af280 0x7fd318060680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.111:6789/0 says I am v1:172.21.15.61:36036/0 (socket says 172.21.15.61:36036) 2022-07-12T08:51:10.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 -- 172.21.15.61:0/2543290712 >> v2:172.21.15.61:3300/0 conn(0x7fd3180b3f60 msgr2=0x7fd3180b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:10.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 --2- 172.21.15.61:0/2543290712 >> v2:172.21.15.61:3300/0 conn(0x7fd3180b3f60 0x7fd3180b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ce92700 1 -- 172.21.15.61:0/2543290712 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304647754 0 0) 0x7fd3180aecb0 con 0x7fd3180af280 2022-07-12T08:51:10.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 -- 172.21.15.61:0/2543290712 >> v1:172.21.15.111:6789/0 conn(0x7fd3180af280 legacy=0x7fd318060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 -- 172.21.15.61:0/2543290712 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318143860 con 0x7fd31809bce0 2022-07-12T08:51:10.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31d693700 1 --2- 172.21.15.61:0/2543290712 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd3180af660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd3000032f0 tx=0x7fd300009290).ready entity=mon.2 client_cookie=b8bce4b80fa70fdf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:10.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ce92700 1 -- 172.21.15.61:0/2543290712 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30000b8a0 con 0x7fd31809bce0 2022-07-12T08:51:10.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ce92700 1 -- 172.21.15.61:0/2543290712 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30000ba00 con 0x7fd31809bce0 2022-07-12T08:51:10.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.375+0000 7fd31ce92700 1 -- 172.21.15.61:0/2543290712 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30000bb70 con 0x7fd31809bce0 2022-07-12T08:51:10.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- 172.21.15.61:0/2543290712 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 msgr2=0x7fd3180af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 --2- 172.21.15.61:0/2543290712 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd3180af660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fd3000032f0 tx=0x7fd300009290).stop 2022-07-12T08:51:10.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- 172.21.15.61:0/2543290712 shutdown_connections 2022-07-12T08:51:10.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 --2- 172.21.15.61:0/2543290712 >> v2:172.21.15.61:3300/0 conn(0x7fd3180b3f60 0x7fd3180b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 --2- 172.21.15.61:0/2543290712 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd3180af660 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- 172.21.15.61:0/2543290712 >> 172.21.15.61:0/2543290712 conn(0x7fd3180b92c0 msgr2=0x7fd3180b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:10.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- 172.21.15.61:0/2543290712 shutdown_connections 2022-07-12T08:51:10.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- 172.21.15.61:0/2543290712 wait complete. 2022-07-12T08:51:10.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 Processor -- start 2022-07-12T08:51:10.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- start start 2022-07-12T08:51:10.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd318152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 0x7fd3180b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3180b3f60 0x7fd31809d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b1360 con 0x7fd3180af280 2022-07-12T08:51:10.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318152d20 con 0x7fd3180b3f60 2022-07-12T08:51:10.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31ee96700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3180b24d0 con 0x7fd31809bce0 2022-07-12T08:51:10.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31d693700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 0x7fd3180b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31de94700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd318152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31d693700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 0x7fd3180b1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34224/0 (socket says 172.21.15.61:34224) 2022-07-12T08:51:10.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31de94700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd318152830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:44650/0 (socket says 172.21.15.61:44650) 2022-07-12T08:51:10.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31d693700 1 -- 172.21.15.61:0/639780376 learned_addr learned my addr 172.21.15.61:0/639780376 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:10.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.379+0000 7fd31e695700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3180b3f60 0x7fd31809d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31d693700 1 -- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 msgr2=0x7fd318152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31d693700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd318152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31d693700 1 -- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3180b3f60 msgr2=0x7fd31809d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31d693700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3180b3f60 0x7fd31809d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31d693700 1 -- 172.21.15.61:0/639780376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd300009530 con 0x7fd3180af280 2022-07-12T08:51:10.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31d693700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 0x7fd3180b1d30 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fd3000032f0 tx=0x7fd300016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:10.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30000bb70 con 0x7fd3180af280 2022-07-12T08:51:10.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd300021630 con 0x7fd3180af280 2022-07-12T08:51:10.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3180b1590 con 0x7fd3180af280 2022-07-12T08:51:10.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30000b8a0 con 0x7fd3180af280 2022-07-12T08:51:10.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3180b0700 con 0x7fd3180af280 2022-07-12T08:51:10.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3180b26d0 con 0x7fd3180af280 2022-07-12T08:51:10.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd30000f070 con 0x7fd3180af280 2022-07-12T08:51:10.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd316ffd700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2fc053a40 0x7fd2fc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:10.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd30006d5b0 con 0x7fd3180af280 2022-07-12T08:51:10.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.383+0000 7fd31de94700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2fc053a40 0x7fd2fc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:10.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.387+0000 7fd31de94700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2fc053a40 0x7fd2fc055f00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fd30800d000 tx=0x7fd3080011f0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:10.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.391+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd300044510 con 0x7fd3180af280 2022-07-12T08:51:10.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.555+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd318057460 con 0x7fd3180af280 2022-07-12T08:51:10.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.555+0000 7fd316ffd700 1 -- 172.21.15.61:0/639780376 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fd300015070 con 0x7fd3180af280 2022-07-12T08:51:10.561 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:10.562 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1725259132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1725259132},{"type":"v1","addr":"172.21.15.111:6835","nonce":1725259132}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:51:07.632105+0000","modified":"2022-07-12T08:51:08.635747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8775":{"gid":8775,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/502386216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":502386216},{"type":"v1","addr":"172.21.15.61:6833","nonce":502386216}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-12T08:51:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2fc053a40 msgr2=0x7fd2fc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2fc053a40 0x7fd2fc055f00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7fd30800d000 tx=0x7fd3080011f0).stop 2022-07-12T08:51:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 msgr2=0x7fd3180b1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 0x7fd3180b1d30 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7fd3000032f0 tx=0x7fd300016000).stop 2022-07-12T08:51:10.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 shutdown_connections 2022-07-12T08:51:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd2fc053a40 0x7fd2fc055f00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd3180b3f60 0x7fd31809d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3180af280 0x7fd3180b1d30 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 --2- 172.21.15.61:0/639780376 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd31809bce0 0x7fd318152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:10.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 >> 172.21.15.61:0/639780376 conn(0x7fd3180b92c0 msgr2=0x7fd3180b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:10.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 shutdown_connections 2022-07-12T08:51:10.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:10.559+0000 7fd31ee96700 1 -- 172.21.15.61:0/639780376 wait complete. 2022-07-12T08:51:10.568 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-12T08:51:10.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:51:07.632105+0000', 'modified': '2022-07-12T08:51:08.635747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8775': {'gid': 8775, 'name': 'a', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/502386216', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 502386216}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 502386216}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:51:10.786 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:51:10.787 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:51:10.905 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8775,"want_state":"up:active","state":"up:active","rank_uptime":3.2609320159999999,"mdsmap_epoch":83,"osdmap_epoch":148,"osdmap_epoch_barrier":147,"uptime":7.6300119410000002} 2022-07-12T08:51:10.906 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8775, "mdsmap_epoch": 83, "osdmap_epoch": 148, "osdmap_epoch_barrier": 147, "rank_uptime": 3.260932016, "state": "up:active", "uptime": 7.630011941, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:51:10.908 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:51:11.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 Processor -- start 2022-07-12T08:51:11.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- start start 2022-07-12T08:51:11.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3af40b1f80 0x7f3af40ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3af40abdf0 0x7f3af405bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af405d480 con 0x7f3af40afc70 2022-07-12T08:51:11.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40b0fc0 con 0x7f3af40b1f80 2022-07-12T08:51:11.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40b2360 con 0x7f3af40abdf0 2022-07-12T08:51:11.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3af40b1f80 0x7f3af40ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3af40b1f80 0x7f3af40ab900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34230/0 (socket says 172.21.15.61:34230) 2022-07-12T08:51:11.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 -- 172.21.15.61:0/3765816883 learned_addr learned my addr 172.21.15.61:0/3765816883 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:11.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af1d9b700 1 --2- 172.21.15.61:0/3765816883 >> v2:172.21.15.111:3300/0 conn(0x7f3af40abdf0 0x7f3af405bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 -- 172.21.15.61:0/3765816883 >> v2:172.21.15.111:3300/0 conn(0x7f3af40abdf0 msgr2=0x7f3af405bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:11.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 --2- 172.21.15.61:0/3765816883 >> v2:172.21.15.111:3300/0 conn(0x7f3af40abdf0 0x7f3af405bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 -- 172.21.15.61:0/3765816883 >> v1:172.21.15.111:6789/0 conn(0x7f3af40afc70 legacy=0x7f3af405c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:11.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 -- 172.21.15.61:0/3765816883 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40b16d0 con 0x7f3af40b1f80 2022-07-12T08:51:11.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af1d9b700 1 --2- 172.21.15.61:0/3765816883 >> v2:172.21.15.111:3300/0 conn(0x7f3af40abdf0 0x7f3af405bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:11.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af259c700 1 --2- 172.21.15.61:0/3765816883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af40ab900 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7f3af40b62e0 tx=0x7f3aec006eb0).ready entity=mon.0 client_cookie=2bd2cb79891181 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af159a700 1 -- 172.21.15.61:0/3765816883 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aec00d5c0 con 0x7f3af40b1f80 2022-07-12T08:51:11.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af159a700 1 -- 172.21.15.61:0/3765816883 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec000e90 con 0x7f3af40b1f80 2022-07-12T08:51:11.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af159a700 1 -- 172.21.15.61:0/3765816883 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aec00d910 con 0x7f3af40b1f80 2022-07-12T08:51:11.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- 172.21.15.61:0/3765816883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 msgr2=0x7f3af40ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/3765816883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af40ab900 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7f3af40b62e0 tx=0x7f3aec006eb0).stop 2022-07-12T08:51:11.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- 172.21.15.61:0/3765816883 shutdown_connections 2022-07-12T08:51:11.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/3765816883 >> v2:172.21.15.111:3300/0 conn(0x7f3af40abdf0 0x7f3af405bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/3765816883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af40ab900 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- 172.21.15.61:0/3765816883 >> 172.21.15.61:0/3765816883 conn(0x7f3af4062c00 msgr2=0x7f3af4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- 172.21.15.61:0/3765816883 shutdown_connections 2022-07-12T08:51:11.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.011+0000 7f3af8ffa700 1 -- 172.21.15.61:0/3765816883 wait complete. 2022-07-12T08:51:11.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 Processor -- start 2022-07-12T08:51:11.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 -- start start 2022-07-12T08:51:11.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3af40abdf0 0x7f3af40663f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3af40afc70 0x7f3af4063b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af413f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4148130 con 0x7f3af40b1f80 2022-07-12T08:51:11.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4141fa0 con 0x7f3af40abdf0 2022-07-12T08:51:11.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4142240 con 0x7f3af40afc70 2022-07-12T08:51:11.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af413f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af1d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3af40afc70 0x7f3af4063b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af413f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34240/0 (socket says 172.21.15.61:34240) 2022-07-12T08:51:11.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 -- 172.21.15.61:0/2495092894 learned_addr learned my addr 172.21.15.61:0/2495092894 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:11.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af259c700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3af40abdf0 0x7f3af40663f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 -- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3af40afc70 msgr2=0x7f3af4063b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3af40afc70 0x7f3af4063b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 -- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3af40abdf0 msgr2=0x7f3af40663f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3af40abdf0 0x7f3af40663f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 -- 172.21.15.61:0/2495092894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec00b1e0 con 0x7f3af40b1f80 2022-07-12T08:51:11.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af2d9d700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af413f140 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f3ae4002320 tx=0x7f3ae40023f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ae4006aa0 con 0x7f3af40b1f80 2022-07-12T08:51:11.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4006c00 con 0x7f3af40b1f80 2022-07-12T08:51:11.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af41424e0 con 0x7f3af40b1f80 2022-07-12T08:51:11.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ae4007030 con 0x7f3af40b1f80 2022-07-12T08:51:11.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.015+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4141760 con 0x7f3af40b1f80 2022-07-12T08:51:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.019+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4057460 con 0x7f3af40b1f80 2022-07-12T08:51:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.019+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3ae4010030 con 0x7f3af40b1f80 2022-07-12T08:51:11.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.019+0000 7f3aeb7fe700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ad8053920 0x7f3ad8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.019+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ae406be60 con 0x7f3af40b1f80 2022-07-12T08:51:11.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.019+0000 7f3af259c700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ad8053920 0x7f3ad8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.019+0000 7f3af259c700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ad8053920 0x7f3ad8055de0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f3aec01c000 tx=0x7f3aec015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.023+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ae4042cf0 con 0x7f3af40b1f80 2022-07-12T08:51:11.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.151+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3af40447a0 con 0x7f3af40b1f80 2022-07-12T08:51:11.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.151+0000 7f3aeb7fe700 1 -- 172.21.15.61:0/2495092894 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f3ae4046310 con 0x7f3af40b1f80 2022-07-12T08:51:11.158 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:11.158 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":148,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:09.648732+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:51:06.224007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-12T08:51:06.803491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:51:11.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ad8053920 msgr2=0x7f3ad8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ad8053920 0x7f3ad8055de0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f3aec01c000 tx=0x7f3aec015000).stop 2022-07-12T08:51:11.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 msgr2=0x7f3af413f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af413f140 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 rx=0x7f3ae4002320 tx=0x7f3ae40023f0).stop 2022-07-12T08:51:11.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 shutdown_connections 2022-07-12T08:51:11.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3af40b1f80 0x7f3af413f140 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3af40afc70 0x7f3af4063b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3ad8053920 0x7f3ad8055de0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 --2- 172.21.15.61:0/2495092894 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3af40abdf0 0x7f3af40663f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 >> 172.21.15.61:0/2495092894 conn(0x7f3af4062c00 msgr2=0x7f3af40b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 shutdown_connections 2022-07-12T08:51:11.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.155+0000 7f3af8ffa700 1 -- 172.21.15.61:0/2495092894 wait complete. 2022-07-12T08:51:11.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:51:06.224007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:51:06.803491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:51:11.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:51:11.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 Processor -- start 2022-07-12T08:51:11.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- start start 2022-07-12T08:51:11.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58140b2530 0x7f5814060360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814060f60 0x7f581413f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58141435e0 con 0x7f5814143110 2022-07-12T08:51:11.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58140b2910 con 0x7f58140b2530 2022-07-12T08:51:11.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58140618c0 con 0x7f5814060f60 2022-07-12T08:51:11.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5812d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58140b2530 0x7f5814060360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f581259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5814143110 0x7f5814060850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50694/0 (socket says 172.21.15.61:50694) 2022-07-12T08:51:11.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f581259c700 1 -- 172.21.15.61:0/3278990169 learned_addr learned my addr 172.21.15.61:0/3278990169 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:11.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f581359e700 1 --2- 172.21.15.61:0/3278990169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814060f60 0x7f581413f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5811d9b700 1 -- 172.21.15.61:0/3278990169 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600980191 0 0) 0x7f58141435e0 con 0x7f5814143110 2022-07-12T08:51:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5811d9b700 1 -- 172.21.15.61:0/3278990169 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f8003260 con 0x7f5814143110 2022-07-12T08:51:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5812d9d700 1 -- 172.21.15.61:0/3278990169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814060f60 msgr2=0x7f581413f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:11.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5812d9d700 1 --2- 172.21.15.61:0/3278990169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814060f60 0x7f581413f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5812d9d700 1 -- 172.21.15.61:0/3278990169 >> v1:172.21.15.61:6789/0 conn(0x7f5814143110 legacy=0x7f5814060850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5812d9d700 1 -- 172.21.15.61:0/3278990169 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5814061b60 con 0x7f58140b2530 2022-07-12T08:51:11.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5812d9d700 1 --2- 172.21.15.61:0/3278990169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814060360 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f57fc002f70 tx=0x7f57fc006df0).ready entity=mon.0 client_cookie=747d30ffbc44e9bc server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5811d9b700 1 -- 172.21.15.61:0/3278990169 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc011660 con 0x7f58140b2530 2022-07-12T08:51:11.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5811d9b700 1 -- 172.21.15.61:0/3278990169 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57fc000e90 con 0x7f58140b2530 2022-07-12T08:51:11.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5811d9b700 1 -- 172.21.15.61:0/3278990169 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57fc0119b0 con 0x7f58140b2530 2022-07-12T08:51:11.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- 172.21.15.61:0/3278990169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 msgr2=0x7f5814060360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 --2- 172.21.15.61:0/3278990169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814060360 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f57fc002f70 tx=0x7f57fc006df0).stop 2022-07-12T08:51:11.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- 172.21.15.61:0/3278990169 shutdown_connections 2022-07-12T08:51:11.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 --2- 172.21.15.61:0/3278990169 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814060f60 0x7f581413f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 --2- 172.21.15.61:0/3278990169 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814060360 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.287+0000 7f5818816700 1 -- 172.21.15.61:0/3278990169 >> 172.21.15.61:0/3278990169 conn(0x7f581405f140 msgr2=0x7f581405f520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 -- 172.21.15.61:0/3278990169 shutdown_connections 2022-07-12T08:51:11.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 -- 172.21.15.61:0/3278990169 wait complete. 2022-07-12T08:51:11.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 Processor -- start 2022-07-12T08:51:11.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 -- start start 2022-07-12T08:51:11.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5814060f60 0x7f58141388f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814066360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814143110 0x7f58140641f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5814065d30 con 0x7f58140b2530 2022-07-12T08:51:11.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5814066d20 con 0x7f5814060f60 2022-07-12T08:51:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5818816700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5814065600 con 0x7f5814143110 2022-07-12T08:51:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814066360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814066360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34252/0 (socket says 172.21.15.61:34252) 2022-07-12T08:51:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 -- 172.21.15.61:0/2312300616 learned_addr learned my addr 172.21.15.61:0/2312300616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581359e700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814143110 0x7f58140641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f5812d9d700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5814060f60 0x7f58141388f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 -- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814143110 msgr2=0x7f58140641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814143110 0x7f58140641f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 -- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5814060f60 msgr2=0x7f58141388f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5814060f60 0x7f58141388f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 -- 172.21.15.61:0/2312300616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc00c290 con 0x7f58140b2530 2022-07-12T08:51:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f581259c700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814066360 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f5804001110 tx=0x7f5804007b80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f580400ce20 con 0x7f58140b2530 2022-07-12T08:51:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.291+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58040072e0 con 0x7f58140b2530 2022-07-12T08:51:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581413b8a0 con 0x7f58140b2530 2022-07-12T08:51:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5804005800 con 0x7f58140b2530 2022-07-12T08:51:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58140668d0 con 0x7f58140b2530 2022-07-12T08:51:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f580401a060 con 0x7f58140b2530 2022-07-12T08:51:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581413b590 con 0x7f58140b2530 2022-07-12T08:51:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f580bfff700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec0539b0 0x7f57ec055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f580406b0a0 con 0x7f58140b2530 2022-07-12T08:51:11.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f5812d9d700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec0539b0 0x7f57ec055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.295+0000 7f5812d9d700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec0539b0 0x7f57ec055e70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f57fc019000 tx=0x7f57fc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.299+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5804041e30 con 0x7f58140b2530 2022-07-12T08:51:11.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.427+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f581413b8a0 con 0x7f58140b2530 2022-07-12T08:51:11.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.427+0000 7f580bfff700 1 -- 172.21.15.61:0/2312300616 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v148) v1 ==== 110+0+10 (secure 0 0 0) 0x7f581413b8a0 con 0x7f58140b2530 2022-07-12T08:51:11.442 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:51:11.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec0539b0 msgr2=0x7f57ec055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec0539b0 0x7f57ec055e70 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7f57fc019000 tx=0x7f57fc015000).stop 2022-07-12T08:51:11.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 msgr2=0x7f5814066360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814066360 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 rx=0x7f5804001110 tx=0x7f5804007b80).stop 2022-07-12T08:51:11.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 shutdown_connections 2022-07-12T08:51:11.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5814143110 0x7f58140641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58140b2530 0x7f5814066360 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f57ec0539b0 0x7f57ec055e70 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 --2- 172.21.15.61:0/2312300616 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5814060f60 0x7f58141388f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 >> 172.21.15.61:0/2312300616 conn(0x7f581405f140 msgr2=0x7f58140b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 shutdown_connections 2022-07-12T08:51:11.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.431+0000 7f5818816700 1 -- 172.21.15.61:0/2312300616 wait complete. 2022-07-12T08:51:11.453 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:51:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.563+0000 7f0847fff700 1 Processor -- start 2022-07-12T08:51:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- start start 2022-07-12T08:51:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08480b0280 0x7f084814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084805c3e0 con 0x7f08480aa830 2022-07-12T08:51:11.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480b5d00 con 0x7f08480b5920 2022-07-12T08:51:11.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084805c610 con 0x7f08480b0280 2022-07-12T08:51:11.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08467fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08480b0280 0x7f084814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08477fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f08480aa830 0x7f08480aac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50710/0 (socket says 172.21.15.61:50710) 2022-07-12T08:51:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08477fe700 1 -- 172.21.15.61:0/1853491828 learned_addr learned my addr 172.21.15.61:0/1853491828 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:11.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22383831 0 0) 0x7f084805c3e0 con 0x7f08480aa830 2022-07-12T08:51:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0828003260 con 0x7f08480aa830 2022-07-12T08:51:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119377220 0 0) 0x7f08480b5d00 con 0x7f08480b5920 2022-07-12T08:51:11.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084805c3e0 con 0x7f08480b5920 2022-07-12T08:51:11.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08467fc700 1 -- 172.21.15.61:0/1853491828 >> v1:172.21.15.111:6789/0 conn(0x7f08480b5920 legacy=0x7f084814a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 889959661 0 0) 0x7f0828003260 con 0x7f08480aa830 2022-07-12T08:51:11.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08467fc700 1 -- 172.21.15.61:0/1853491828 >> v1:172.21.15.61:6789/0 conn(0x7f08480aa830 legacy=0x7f08480aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08467fc700 1 -- 172.21.15.61:0/1853491828 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084805c8b0 con 0x7f08480b0280 2022-07-12T08:51:11.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f08467fc700 1 --2- 172.21.15.61:0/1853491828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f084814b0f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f0840001110 tx=0x7f0830006df0).ready entity=mon.0 client_cookie=a28470a4bfef70f1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0840001660 con 0x7f08480b0280 2022-07-12T08:51:11.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0830000e90 con 0x7f08480b0280 2022-07-12T08:51:11.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0845ffb700 1 -- 172.21.15.61:0/1853491828 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830012460 con 0x7f08480b0280 2022-07-12T08:51:11.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- 172.21.15.61:0/1853491828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 msgr2=0x7f084814b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 --2- 172.21.15.61:0/1853491828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f084814b0f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f0840001110 tx=0x7f0830006df0).stop 2022-07-12T08:51:11.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- 172.21.15.61:0/1853491828 shutdown_connections 2022-07-12T08:51:11.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 --2- 172.21.15.61:0/1853491828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f084814b0f0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- 172.21.15.61:0/1853491828 >> 172.21.15.61:0/1853491828 conn(0x7f0848063340 msgr2=0x7f08480b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- 172.21.15.61:0/1853491828 shutdown_connections 2022-07-12T08:51:11.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.567+0000 7f0847fff700 1 -- 172.21.15.61:0/1853491828 wait complete. 2022-07-12T08:51:11.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 Processor -- start 2022-07-12T08:51:11.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 -- start start 2022-07-12T08:51:11.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08480aa830 0x7f0848099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f0848148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08480b5920 0x7f0848148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848063cb0 con 0x7f08480b0280 2022-07-12T08:51:11.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0848063f50 con 0x7f08480aa830 2022-07-12T08:51:11.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f0847fff700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08480641f0 con 0x7f08480b5920 2022-07-12T08:51:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f08467fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f0848148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f08467fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f0848148560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34266/0 (socket says 172.21.15.61:34266) 2022-07-12T08:51:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.571+0000 7f08467fc700 1 -- 172.21.15.61:0/1272521195 learned_addr learned my addr 172.21.15.61:0/1272521195 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:11.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f0846ffd700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08480aa830 0x7f0848099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08477fe700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08480b5920 0x7f0848148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08467fc700 1 -- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08480b5920 msgr2=0x7f0848148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08467fc700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08480b5920 0x7f0848148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08467fc700 1 -- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08480aa830 msgr2=0x7f0848099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08467fc700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08480aa830 0x7f0848099340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08467fc700 1 -- 172.21.15.61:0/1272521195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083000b180 con 0x7f08480b0280 2022-07-12T08:51:11.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08477fe700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08480b5920 0x7f0848148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f0846ffd700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08480aa830 0x7f0848099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f08467fc700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f0848148560 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f0840001110 tx=0x7f0830014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830016070 con 0x7f08480b0280 2022-07-12T08:51:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083000bec0 con 0x7f08480b0280 2022-07-12T08:51:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0848099830 con 0x7f08480b0280 2022-07-12T08:51:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0830012460 con 0x7f08480b0280 2022-07-12T08:51:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08480ae160 con 0x7f08480b0280 2022-07-12T08:51:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0830025030 con 0x7f08480b0280 2022-07-12T08:51:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0848057460 con 0x7f08480b0280 2022-07-12T08:51:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f083ffff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f082c053a40 0x7f082c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f083006cdd0 con 0x7f08480b0280 2022-07-12T08:51:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.575+0000 7f0846ffd700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f082c053a40 0x7f082c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.579+0000 7f0846ffd700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f082c053a40 0x7f082c055f00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f0838008000 tx=0x7f0838009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.583+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0830043d30 con 0x7f08480b0280 2022-07-12T08:51:11.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f08480b4be0 con 0x7f08480b0280 2022-07-12T08:51:11.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f083ffff700 1 -- 172.21.15.61:0/1272521195 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v148) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0830023070 con 0x7f08480b0280 2022-07-12T08:51:11.714 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f082c053a40 msgr2=0x7f082c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f082c053a40 0x7f082c055f00 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f0838008000 tx=0x7f0838009040).stop 2022-07-12T08:51:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 msgr2=0x7f0848148560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f0848148560 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f0840001110 tx=0x7f0830014000).stop 2022-07-12T08:51:11.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 shutdown_connections 2022-07-12T08:51:11.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08480b5920 0x7f0848148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08480b0280 0x7f0848148560 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f082c053a40 0x7f082c055f00 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 --2- 172.21.15.61:0/1272521195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08480aa830 0x7f0848099340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 >> 172.21.15.61:0/1272521195 conn(0x7f0848063340 msgr2=0x7f08480b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 shutdown_connections 2022-07-12T08:51:11.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.711+0000 7f0847fff700 1 -- 172.21.15.61:0/1272521195 wait complete. 2022-07-12T08:51:11.729 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:51:11.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.831+0000 7fa8e9bf6700 1 Processor -- start 2022-07-12T08:51:11.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.831+0000 7fa8e9bf6700 1 -- start start 2022-07-12T08:51:11.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fa8e40acb60 0x7fa8e41468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e414b0b0 con 0x7fa8e40b7eb0 2022-07-12T08:51:11.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e40b8290 con 0x7fa8e40ad600 2022-07-12T08:51:11.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e40b25f0 con 0x7fa8e40acb60 2022-07-12T08:51:11.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa8e40b7eb0 0x7fa8e414a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50722/0 (socket says 172.21.15.61:50722) 2022-07-12T08:51:11.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 -- 172.21.15.61:0/3858234022 learned_addr learned my addr 172.21.15.61:0/3858234022 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:11.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e3fff700 1 --2- 172.21.15.61:0/3858234022 >> v2:172.21.15.111:3300/0 conn(0x7fa8e40acb60 0x7fa8e41468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792184267 0 0) 0x7fa8e414b0b0 con 0x7fa8e40b7eb0 2022-07-12T08:51:11.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c4003260 con 0x7fa8e40b7eb0 2022-07-12T08:51:11.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450492927 0 0) 0x7fa8e40b8290 con 0x7fa8e40ad600 2022-07-12T08:51:11.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8e414b0b0 con 0x7fa8e40ad600 2022-07-12T08:51:11.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750716481 0 0) 0x7fa8c4003260 con 0x7fa8e40b7eb0 2022-07-12T08:51:11.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 >> v2:172.21.15.111:3300/0 conn(0x7fa8e40acb60 msgr2=0x7fa8e41468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:11.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 --2- 172.21.15.61:0/3858234022 >> v2:172.21.15.111:3300/0 conn(0x7fa8e40acb60 0x7fa8e41468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 >> v1:172.21.15.111:6789/0 conn(0x7fa8e40ad600 legacy=0x7fa8e40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e40adbc0 con 0x7fa8e40b7eb0 2022-07-12T08:51:11.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fa8d4001660 con 0x7fa8e40b7eb0 2022-07-12T08:51:11.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 >> v1:172.21.15.61:6789/0 conn(0x7fa8e40b7eb0 legacy=0x7fa8e414a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8c4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8c4004500 0x7fa8c4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8c4004e70 0x7fa8c400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40b25f0 con 0x7fa8c4003930 2022-07-12T08:51:11.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c400bd20 con 0x7fa8c4004e70 2022-07-12T08:51:11.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8c4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c400bee0 con 0x7fa8c4004500 2022-07-12T08:51:11.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8daffd700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8c4004500 0x7fa8c4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e3fff700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8c4004e70 0x7fa8c400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 -- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8c4004500 msgr2=0x7fa8c4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:11.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8c4004500 0x7fa8c4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 -- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8c4004e70 msgr2=0x7fa8c400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:11.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8c4004e70 0x7fa8c400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e3fff700 1 -- 172.21.15.61:0/3858234022 reap_dead start 2022-07-12T08:51:11.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 -- 172.21.15.61:0/3858234022 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4003260 con 0x7fa8c4003930 2022-07-12T08:51:11.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8c4003d30 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7fa8d4001430 tx=0x7fa8d4001d00).ready entity=mon.0 client_cookie=eecdf472eba34955 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d4001d70 con 0x7fa8c4003930 2022-07-12T08:51:11.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d40070d0 con 0x7fa8c4003930 2022-07-12T08:51:11.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e2ffd700 1 -- 172.21.15.61:0/3858234022 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d4007430 con 0x7fa8c4003930 2022-07-12T08:51:11.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/3858234022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 msgr2=0x7fa8c4003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8c4003d30 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7fa8d4001430 tx=0x7fa8d4001d00).stop 2022-07-12T08:51:11.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/3858234022 shutdown_connections 2022-07-12T08:51:11.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/3858234022 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8c4003d30 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/3858234022 >> 172.21.15.61:0/3858234022 conn(0x7fa8e4065850 msgr2=0x7fa8e40b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/3858234022 shutdown_connections 2022-07-12T08:51:11.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.835+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/3858234022 wait complete. 2022-07-12T08:51:11.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 Processor -- start 2022-07-12T08:51:11.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 -- start start 2022-07-12T08:51:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8e414a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e4112610 0x7fa8e4112a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8e4112f60 0x7fa8e40ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4110a20 con 0x7fa8c4003930 2022-07-12T08:51:11.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40b02e0 con 0x7fa8e4112610 2022-07-12T08:51:11.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4110c50 con 0x7fa8e4112f60 2022-07-12T08:51:11.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8e414a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8e414a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34284/0 (socket says 172.21.15.61:34284) 2022-07-12T08:51:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 -- 172.21.15.61:0/4141303307 learned_addr learned my addr 172.21.15.61:0/4141303307 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8daffd700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e4112610 0x7fa8e4112a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e3fff700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8e4112f60 0x7fa8e40ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 -- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8e4112f60 msgr2=0x7fa8e40ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8e4112f60 0x7fa8e40ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 -- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e4112610 msgr2=0x7fa8e4112a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e4112610 0x7fa8e4112a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 -- 172.21.15.61:0/4141303307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e414a7f0 con 0x7fa8c4003930 2022-07-12T08:51:11.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e37fe700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8e414a080 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7fa8d4001430 tx=0x7fa8d4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d4000e90 con 0x7fa8c4003930 2022-07-12T08:51:11.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d400c6d0 con 0x7fa8c4003930 2022-07-12T08:51:11.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e40afd60 con 0x7fa8c4003930 2022-07-12T08:51:11.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d4004710 con 0x7fa8c4003930 2022-07-12T08:51:11.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.839+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e40b17b0 con 0x7fa8c4003930 2022-07-12T08:51:11.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.843+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e40b0450 con 0x7fa8c4003930 2022-07-12T08:51:11.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.843+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa8d4026070 con 0x7fa8c4003930 2022-07-12T08:51:11.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.843+0000 7fa8e17fa700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8c8053af0 0x7fa8c8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:11.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.843+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa8d406c1c0 con 0x7fa8c4003930 2022-07-12T08:51:11.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.843+0000 7fa8daffd700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8c8053af0 0x7fa8c8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:11.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.843+0000 7fa8daffd700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8c8053af0 0x7fa8c8055fb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7fa8cc003000 tx=0x7fa8cc007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:11.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.851+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8e40b0450 con 0x7fa8c4003930 2022-07-12T08:51:11.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.975+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8e40b0450 con 0x7fa8c4003930 2022-07-12T08:51:11.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e17fa700 1 -- 172.21.15.61:0/4141303307 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa8e40b0450 con 0x7fa8c4003930 2022-07-12T08:51:11.984 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:11.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8c8053af0 msgr2=0x7fa8c8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8c8053af0 0x7fa8c8055fb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7fa8cc003000 tx=0x7fa8cc007a00).stop 2022-07-12T08:51:11.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 msgr2=0x7fa8e414a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:11.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8e414a080 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7fa8d4001430 tx=0x7fa8d4015000).stop 2022-07-12T08:51:11.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 shutdown_connections 2022-07-12T08:51:11.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa8e4112f60 0x7fa8e40ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa8c8053af0 0x7fa8c8055fb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8e4112610 0x7fa8e4112a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 --2- 172.21.15.61:0/4141303307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c4003930 0x7fa8e414a080 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:11.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 >> 172.21.15.61:0/4141303307 conn(0x7fa8e4065850 msgr2=0x7fa8e40b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:11.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 shutdown_connections 2022-07-12T08:51:11.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:11.979+0000 7fa8e9bf6700 1 -- 172.21.15.61:0/4141303307 wait complete. 2022-07-12T08:51:11.997 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:51:12.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.107+0000 7f7d59555700 1 Processor -- start 2022-07-12T08:51:12.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.107+0000 7f7d59555700 1 -- start start 2022-07-12T08:51:12.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54140360 0x7f7d541407f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d54067410 con 0x7f7d54067030 2022-07-12T08:51:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5405c970 con 0x7f7d540b2340 2022-07-12T08:51:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540b1840 con 0x7f7d54140360 2022-07-12T08:51:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d53fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7d54067030 0x7f7d540b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50742/0 (socket says 172.21.15.61:50742) 2022-07-12T08:51:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d53fff700 1 -- 172.21.15.61:0/1323929959 learned_addr learned my addr 172.21.15.61:0/1323929959 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d58d54700 1 --2- 172.21.15.61:0/1323929959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54140360 0x7f7d541407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d537fe700 1 -- 172.21.15.61:0/1323929959 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958591346 0 0) 0x7f7d54067410 con 0x7f7d54067030 2022-07-12T08:51:12.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d537fe700 1 -- 172.21.15.61:0/1323929959 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d34003260 con 0x7f7d54067030 2022-07-12T08:51:12.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d58d54700 1 -- 172.21.15.61:0/1323929959 >> v1:172.21.15.111:6789/0 conn(0x7f7d540b2340 legacy=0x7f7d5413fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d58d54700 1 -- 172.21.15.61:0/1323929959 >> v1:172.21.15.61:6789/0 conn(0x7f7d54067030 legacy=0x7f7d540b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d537fe700 1 -- 172.21.15.61:0/1323929959 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272370844 0 0) 0x7f7d5405c970 con 0x7f7d540b2340 2022-07-12T08:51:12.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d58d54700 1 -- 172.21.15.61:0/1323929959 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d54140ce0 con 0x7f7d54140360 2022-07-12T08:51:12.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d58d54700 1 --2- 172.21.15.61:0/1323929959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54140360 0x7f7d541407f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7d4c0032f0 tx=0x7f7d4c0091d0).ready entity=mon.2 client_cookie=a7d76d0f4116ecb0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:12.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d537fe700 1 -- 172.21.15.61:0/1323929959 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d4c00b6c0 con 0x7f7d54140360 2022-07-12T08:51:12.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d537fe700 1 -- 172.21.15.61:0/1323929959 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c00b820 con 0x7f7d54140360 2022-07-12T08:51:12.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d537fe700 1 -- 172.21.15.61:0/1323929959 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d4c00b990 con 0x7f7d54140360 2022-07-12T08:51:12.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- 172.21.15.61:0/1323929959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54140360 msgr2=0x7f7d541407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 --2- 172.21.15.61:0/1323929959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54140360 0x7f7d541407f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f7d4c0032f0 tx=0x7f7d4c0091d0).stop 2022-07-12T08:51:12.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- 172.21.15.61:0/1323929959 shutdown_connections 2022-07-12T08:51:12.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 --2- 172.21.15.61:0/1323929959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54140360 0x7f7d541407f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- 172.21.15.61:0/1323929959 >> 172.21.15.61:0/1323929959 conn(0x7f7d54063ad0 msgr2=0x7f7d54063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:12.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- 172.21.15.61:0/1323929959 shutdown_connections 2022-07-12T08:51:12.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.111+0000 7f7d59555700 1 -- 172.21.15.61:0/1323929959 wait complete. 2022-07-12T08:51:12.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 Processor -- start 2022-07-12T08:51:12.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 -- start start 2022-07-12T08:51:12.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54067030 0x7f7d54144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540b2340 0x7f7d5413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 0x7f7d5413b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540ae8e0 con 0x7f7d54140360 2022-07-12T08:51:12.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d540ae4e0 con 0x7f7d540b2340 2022-07-12T08:51:12.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d59555700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d54150930 con 0x7f7d54067030 2022-07-12T08:51:12.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 0x7f7d5413b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 0x7f7d5413b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34302/0 (socket says 172.21.15.61:34302) 2022-07-12T08:51:12.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 -- 172.21.15.61:0/1982550150 learned_addr learned my addr 172.21.15.61:0/1982550150 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:12.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d53fff700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54067030 0x7f7d54144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d4b7fe700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540b2340 0x7f7d5413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 -- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54067030 msgr2=0x7f7d54144600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54067030 0x7f7d54144600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 -- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540b2340 msgr2=0x7f7d5413aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540b2340 0x7f7d5413aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 -- 172.21.15.61:0/1982550150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c009470 con 0x7f7d54140360 2022-07-12T08:51:12.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.115+0000 7f7d58d54700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 0x7f7d5413b090 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f7d4c0032f0 tx=0x7f7d4c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:12.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d4c00b990 con 0x7f7d54140360 2022-07-12T08:51:12.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c0205f0 con 0x7f7d54140360 2022-07-12T08:51:12.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5413b580 con 0x7f7d54140360 2022-07-12T08:51:12.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d4c00b6c0 con 0x7f7d54140360 2022-07-12T08:51:12.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d540adfd0 con 0x7f7d54140360 2022-07-12T08:51:12.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7d4c027030 con 0x7f7d54140360 2022-07-12T08:51:12.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d540ae6a0 con 0x7f7d54140360 2022-07-12T08:51:12.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d51ffb700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d38053a60 0x7f7d38055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7d4c06cdc0 con 0x7f7d54140360 2022-07-12T08:51:12.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.119+0000 7f7d53fff700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d38053a60 0x7f7d38055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.123+0000 7f7d53fff700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d38053a60 0x7f7d38055f20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f7d4400d000 tx=0x7f7d44009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:12.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.127+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d4c043c10 con 0x7f7d54140360 2022-07-12T08:51:12.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.251+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d54057460 con 0x7f7d54140360 2022-07-12T08:51:12.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.251+0000 7f7d51ffb700 1 -- 172.21.15.61:0/1982550150 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+10 (secure 0 0 0) 0x7f7d4c0150a0 con 0x7f7d54140360 2022-07-12T08:51:12.260 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:12.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d38053a60 msgr2=0x7f7d38055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d38053a60 0x7f7d38055f20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f7d4400d000 tx=0x7f7d44009040).stop 2022-07-12T08:51:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 msgr2=0x7f7d5413b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 0x7f7d5413b090 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7f7d4c0032f0 tx=0x7f7d4c016000).stop 2022-07-12T08:51:12.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 shutdown_connections 2022-07-12T08:51:12.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d54140360 0x7f7d5413b090 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f7d38053a60 0x7f7d38055f20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7d540b2340 0x7f7d5413aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 --2- 172.21.15.61:0/1982550150 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7d54067030 0x7f7d54144600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 >> 172.21.15.61:0/1982550150 conn(0x7f7d54063ad0 msgr2=0x7f7d5405f720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:12.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 shutdown_connections 2022-07-12T08:51:12.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.255+0000 7f7d59555700 1 -- 172.21.15.61:0/1982550150 wait complete. 2022-07-12T08:51:12.269 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:51:12.270 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:12.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 Processor -- start 2022-07-12T08:51:12.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 -- start start 2022-07-12T08:51:12.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2280b6420 0x7ff2280b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff228148960 0x7ff228148db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280b6800 con 0x7ff2280b2220 2022-07-12T08:51:12.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2280b58f0 con 0x7ff2280b6420 2022-07-12T08:51:12.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff22e798700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280b0af0 con 0x7ff228148960 2022-07-12T08:51:12.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff227fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2280b6420 0x7ff2280b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff227fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2280b6420 0x7ff2280b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34306/0 (socket says 172.21.15.61:34306) 2022-07-12T08:51:12.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.367+0000 7ff227fff700 1 -- 172.21.15.61:0/2853783300 learned_addr learned my addr 172.21.15.61:0/2853783300 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:12.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff227fff700 1 -- 172.21.15.61:0/2853783300 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff228148960 msgr2=0x7ff228148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:12.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22cd35700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff228148960 0x7ff228148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff227fff700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff228148960 0x7ff228148db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff227fff700 1 -- 172.21.15.61:0/2853783300 >> v1:172.21.15.111:6789/0 conn(0x7ff2280b2220 legacy=0x7ff2280b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff227fff700 1 -- 172.21.15.61:0/2853783300 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280b0d90 con 0x7ff2280b6420 2022-07-12T08:51:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22cd35700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff228148960 0x7ff228148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff227fff700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2280b4050 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7ff2280b20c0 tx=0x7ff218006eb0).ready entity=mon.0 client_cookie=b32b9e131c822f77 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff226ffd700 1 -- 172.21.15.61:0/2853783300 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21800d570 con 0x7ff2280b6420 2022-07-12T08:51:12.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff226ffd700 1 -- 172.21.15.61:0/2853783300 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff218000e90 con 0x7ff2280b6420 2022-07-12T08:51:12.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff226ffd700 1 -- 172.21.15.61:0/2853783300 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21800d900 con 0x7ff2280b6420 2022-07-12T08:51:12.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 -- 172.21.15.61:0/2853783300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 msgr2=0x7ff2280b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2280b4050 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7ff2280b20c0 tx=0x7ff218006eb0).stop 2022-07-12T08:51:12.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 -- 172.21.15.61:0/2853783300 shutdown_connections 2022-07-12T08:51:12.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff228148960 0x7ff228148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 --2- 172.21.15.61:0/2853783300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2280b4050 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 -- 172.21.15.61:0/2853783300 >> 172.21.15.61:0/2853783300 conn(0x7ff228067020 msgr2=0x7ff228067430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:12.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 -- 172.21.15.61:0/2853783300 shutdown_connections 2022-07-12T08:51:12.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 -- 172.21.15.61:0/2853783300 wait complete. 2022-07-12T08:51:12.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 Processor -- start 2022-07-12T08:51:12.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.371+0000 7ff22e798700 1 -- start start 2022-07-12T08:51:12.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff2280b2220 0x7ff22811c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2281458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff228148960 0x7ff228145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22811f020 con 0x7ff2280b6420 2022-07-12T08:51:12.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff228064fe0 con 0x7ff228148960 2022-07-12T08:51:12.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22811f230 con 0x7ff2280b2220 2022-07-12T08:51:12.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2281458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2281458d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34312/0 (socket says 172.21.15.61:34312) 2022-07-12T08:51:12.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 -- 172.21.15.61:0/1201831981 learned_addr learned my addr 172.21.15.61:0/1201831981 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:12.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22cd35700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff228148960 0x7ff228145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff227fff700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff2280b2220 0x7ff22811c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 -- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff2280b2220 msgr2=0x7ff22811c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff2280b2220 0x7ff22811c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 -- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff228148960 msgr2=0x7ff228145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff228148960 0x7ff228145dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 -- 172.21.15.61:0/1201831981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21800b1e0 con 0x7ff2280b6420 2022-07-12T08:51:12.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22cd35700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff228148960 0x7ff228145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:12.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2277fe700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2281458d0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7ff21c000ed0 tx=0x7ff21c007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:12.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21c00ce80 con 0x7ff2280b6420 2022-07-12T08:51:12.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c00f470 con 0x7ff2280b6420 2022-07-12T08:51:12.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff21c00f7d0 con 0x7ff2280b6420 2022-07-12T08:51:12.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22811b5c0 con 0x7ff2280b6420 2022-07-12T08:51:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22811b2d0 con 0x7ff2280b6420 2022-07-12T08:51:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.375+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22811f3f0 con 0x7ff2280b6420 2022-07-12T08:51:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.379+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff21c019030 con 0x7ff2280b6420 2022-07-12T08:51:12.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.379+0000 7ff2257fa700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff208053a60 0x7ff208055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:12.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.379+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff21c06b300 con 0x7ff2280b6420 2022-07-12T08:51:12.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.379+0000 7ff227fff700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff208053a60 0x7ff208055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:12.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.379+0000 7ff227fff700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff208053a60 0x7ff208055f20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7ff21801a000 tx=0x7ff218015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:12.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.387+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff21c042260 con 0x7ff2280b6420 2022-07-12T08:51:12.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.551+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff228057460 con 0x7ff2280b6420 2022-07-12T08:51:12.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.551+0000 7ff2257fa700 1 -- 172.21.15.61:0/1201831981 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2142 (secure 0 0 0) 0x7ff21c045880 con 0x7ff2280b6420 2022-07-12T08:51:12.556 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:12.556 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1725259132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1725259132},{"type":"v1","addr":"172.21.15.111:6835","nonce":1725259132}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:51:07.632105+0000","modified":"2022-07-12T08:51:08.635747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8775":{"gid":8775,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/502386216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":502386216},{"type":"v1","addr":"172.21.15.61:6833","nonce":502386216}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-12T08:51:12.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff208053a60 msgr2=0x7ff208055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff208053a60 0x7ff208055f20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7ff21801a000 tx=0x7ff218015000).stop 2022-07-12T08:51:12.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 msgr2=0x7ff2281458d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:12.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2281458d0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7ff21c000ed0 tx=0x7ff21c007c50).stop 2022-07-12T08:51:12.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 shutdown_connections 2022-07-12T08:51:12.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff208053a60 0x7ff208055f20 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff228148960 0x7ff228145dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2280b6420 0x7ff2281458d0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 --2- 172.21.15.61:0/1201831981 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff2280b2220 0x7ff22811c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:12.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 >> 172.21.15.61:0/1201831981 conn(0x7ff228067020 msgr2=0x7ff2280b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:12.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 shutdown_connections 2022-07-12T08:51:12.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:12.555+0000 7ff22e798700 1 -- 172.21.15.61:0/1201831981 wait complete. 2022-07-12T08:51:12.564 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-12T08:51:12.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:51:07.632105+0000', 'modified': '2022-07-12T08:51:08.635747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8775': {'gid': 8775, 'name': 'a', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/502386216', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 502386216}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 502386216}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:51:12.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:51:12.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:51:12.896 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8775,"want_state":"up:active","state":"up:active","rank_uptime":5.2502441170000003,"mdsmap_epoch":83,"osdmap_epoch":148,"osdmap_epoch_barrier":147,"uptime":9.6193345519999998} 2022-07-12T08:51:12.897 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8775, "mdsmap_epoch": 83, "osdmap_epoch": 148, "osdmap_epoch_barrier": 147, "rank_uptime": 5.250244117, "state": "up:active", "uptime": 9.619334552, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:51:12.898 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:51:12.899 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-12T08:51:12.899 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi121.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-12T08:51:12.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:12.901 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:51:12.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:51:12.960 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:51:12.961 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:51:13.022 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:51:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:51:13.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:13.026 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:51:13.087 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:51:13.088 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:51:13.103 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2022-07-12T08:51:13.128 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:51:13.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:51:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:2022-07-12T08:51:13.169+0000 7fb61fb0e080 -1 init, newargv = 0x555d2ae787b0 newargc=9 2022-07-12T08:51:13.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22343]: starting ceph client 2022-07-12T08:51:13.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi121.stderr:ceph-fuse[22343]: starting fuse 2022-07-12T08:51:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:51:13.193 INFO:teuthology.orchestra.run.smithi121.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:51:13.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:13.194 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2022-07-12T08:51:13.205 INFO:teuthology.orchestra.run.smithi121.stdout:52 2022-07-12T08:51:13.206 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:51:13.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:13.207 DEBUG:teuthology.orchestra.run.smithi121:> sudo python3 -c ' 2022-07-12T08:51:13.208 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2022-07-12T08:51:13.209 DEBUG:teuthology.orchestra.run.smithi121:> import re 2022-07-12T08:51:13.210 DEBUG:teuthology.orchestra.run.smithi121:> import os 2022-07-12T08:51:13.211 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2022-07-12T08:51:13.211 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:51:13.212 DEBUG:teuthology.orchestra.run.smithi121:> def find_socket(client_name): 2022-07-12T08:51:13.213 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-12T08:51:13.215 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2022-07-12T08:51:13.216 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:51:13.216 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2022-07-12T08:51:13.216 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2022-07-12T08:51:13.217 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2022-07-12T08:51:13.217 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2022-07-12T08:51:13.218 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:51:13.219 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2022-07-12T08:51:13.220 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:51:13.220 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:51:13.221 DEBUG:teuthology.orchestra.run.smithi121:> return f 2022-07-12T08:51:13.222 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:51:13.222 DEBUG:teuthology.orchestra.run.smithi121:> 2022-07-12T08:51:13.223 DEBUG:teuthology.orchestra.run.smithi121:> print(find_socket("client.0")) 2022-07-12T08:51:13.224 DEBUG:teuthology.orchestra.run.smithi121:> ' 2022-07-12T08:51:13.273 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.0.22343.asok 2022-07-12T08:51:13.277 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22343.asok 2022-07-12T08:51:13.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:13.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22343.asok status 2022-07-12T08:51:13.416 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2022-07-12T08:51:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2022-07-12T08:51:13.416 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:51:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:51:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "0", 2022-07-12T08:51:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2022-07-12T08:51:13.417 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-12T08:51:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "22343", 2022-07-12T08:51:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2022-07-12T08:51:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:51:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2022-07-12T08:51:13.418 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2022-07-12T08:51:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 8946, 2022-07-12T08:51:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2022-07-12T08:51:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2022-07-12T08:51:13.419 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2022-07-12T08:51:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 8946 2022-07-12T08:51:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:51:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:51:13.420 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:51:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:51:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 45282928 2022-07-12T08:51:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: } 2022-07-12T08:51:13.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:51:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2022-07-12T08:51:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2022-07-12T08:51:13.422 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2022-07-12T08:51:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 45282928 2022-07-12T08:51:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2022-07-12T08:51:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.8946 172.21.15.121:0/45282928", 2022-07-12T08:51:13.423 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "172.21.15.121:0/45282928", 2022-07-12T08:51:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2022-07-12T08:51:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 83, 2022-07-12T08:51:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 148, 2022-07-12T08:51:13.424 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:51:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "blacklisted": false, 2022-07-12T08:51:13.425 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2022-07-12T08:51:13.425 INFO:teuthology.orchestra.run.smithi121.stdout:} 2022-07-12T08:51:13.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:13.427 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:13.428 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:51:13.436 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2022-07-12T08:51:13.439 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-12T08:51:13.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:13.440 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:51:13.462 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T08:51:13.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 Processor -- start 2022-07-12T08:51:13.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- start start 2022-07-12T08:51:13.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c05c050 0x7f988c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f988c0b01f0 0x7f988c14a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c0ad2f0 con 0x7f988c05cb00 2022-07-12T08:51:13.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0b0a40 con 0x7f988c0b01f0 2022-07-12T08:51:13.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c05c430 con 0x7f988c05c050 2022-07-12T08:51:13.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f988c0b01f0 0x7f988c14a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f988c0b01f0 0x7f988c14a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34320/0 (socket says 172.21.15.61:34320) 2022-07-12T08:51:13.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 -- 172.21.15.61:0/455247800 learned_addr learned my addr 172.21.15.61:0/455247800 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:13.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891e52700 1 --2- 172.21.15.61:0/455247800 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c05c050 0x7f988c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 -- 172.21.15.61:0/455247800 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c05c050 msgr2=0x7f988c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:13.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 --2- 172.21.15.61:0/455247800 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c05c050 0x7f988c0ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 -- 172.21.15.61:0/455247800 >> v1:172.21.15.61:6789/0 conn(0x7f988c05cb00 legacy=0x7f988c1443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 -- 172.21.15.61:0/455247800 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0b2420 con 0x7f988c0b01f0 2022-07-12T08:51:13.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9891651700 1 --2- 172.21.15.61:0/455247800 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c0b01f0 0x7f988c14a8f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f98840032f0 tx=0x7f9884009290).ready entity=mon.0 client_cookie=91b61bce4ee11633 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:13.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9890e50700 1 -- 172.21.15.61:0/455247800 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988400b8a0 con 0x7f988c0b01f0 2022-07-12T08:51:13.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9890e50700 1 -- 172.21.15.61:0/455247800 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988400ba00 con 0x7f988c0b01f0 2022-07-12T08:51:13.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f9890e50700 1 -- 172.21.15.61:0/455247800 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988400bb70 con 0x7f988c0b01f0 2022-07-12T08:51:13.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- 172.21.15.61:0/455247800 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c0b01f0 msgr2=0x7f988c14a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 --2- 172.21.15.61:0/455247800 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c0b01f0 0x7f988c14a8f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 rx=0x7f98840032f0 tx=0x7f9884009290).stop 2022-07-12T08:51:13.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- 172.21.15.61:0/455247800 shutdown_connections 2022-07-12T08:51:13.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 --2- 172.21.15.61:0/455247800 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c0b01f0 0x7f988c14a8f0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 --2- 172.21.15.61:0/455247800 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c05c050 0x7f988c0ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- 172.21.15.61:0/455247800 >> 172.21.15.61:0/455247800 conn(0x7f988c062c00 msgr2=0x7f988c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:13.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- 172.21.15.61:0/455247800 shutdown_connections 2022-07-12T08:51:13.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.575+0000 7f98940b6700 1 -- 172.21.15.61:0/455247800 wait complete. 2022-07-12T08:51:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 Processor -- start 2022-07-12T08:51:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 -- start start 2022-07-12T08:51:13.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f988c05c050 0x7f988c14b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 0x7f988c144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c0b01f0 0x7f988c144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0a1810 con 0x7f988c05cb00 2022-07-12T08:51:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0a1230 con 0x7f988c05c050 2022-07-12T08:51:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0a14d0 con 0x7f988c0b01f0 2022-07-12T08:51:13.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 0x7f988c144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 0x7f988c144880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34328/0 (socket says 172.21.15.61:34328) 2022-07-12T08:51:13.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 -- 172.21.15.61:0/2236661833 learned_addr learned my addr 172.21.15.61:0/2236661833 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:13.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891e52700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f988c05c050 0x7f988c14b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9892653700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c0b01f0 0x7f988c144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 -- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c0b01f0 msgr2=0x7f988c144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c0b01f0 0x7f988c144d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 -- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f988c05c050 msgr2=0x7f988c14b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f988c05c050 0x7f988c14b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 -- 172.21.15.61:0/2236661833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9884009530 con 0x7f988c05cb00 2022-07-12T08:51:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9892653700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c0b01f0 0x7f988c144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9891651700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 0x7f988c144880 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f98840032f0 tx=0x7f9884016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9884021480 con 0x7f988c05cb00 2022-07-12T08:51:13.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98840215e0 con 0x7f988c05cb00 2022-07-12T08:51:13.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c0a1a60 con 0x7f988c05cb00 2022-07-12T08:51:13.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9884021940 con 0x7f988c05cb00 2022-07-12T08:51:13.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.579+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c0a1f00 con 0x7f988c05cb00 2022-07-12T08:51:13.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.583+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f988400f070 con 0x7f988c05cb00 2022-07-12T08:51:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.583+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c057460 con 0x7f988c05cb00 2022-07-12T08:51:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.583+0000 7f9882ffd700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f986c0539b0 0x7f986c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.583+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f988406d4e0 con 0x7f988c05cb00 2022-07-12T08:51:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.583+0000 7f9891e52700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f986c0539b0 0x7f986c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.583+0000 7f9891e52700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f986c0539b0 0x7f986c055e70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f988800c000 tx=0x7f9888008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:13.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.591+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9884044360 con 0x7f988c05cb00 2022-07-12T08:51:13.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.719+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f988c0a2270 con 0x7f988c05cb00 2022-07-12T08:51:13.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.719+0000 7f9882ffd700 1 -- 172.21.15.61:0/2236661833 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f9884015020 con 0x7f988c05cb00 2022-07-12T08:51:13.726 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:13.726 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":148,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:09.648732+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-12T08:51:06.224007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"146","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-12T08:51:06.803491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T08:51:13.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f986c0539b0 msgr2=0x7f986c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f986c0539b0 0x7f986c055e70 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f988800c000 tx=0x7f9888008040).stop 2022-07-12T08:51:13.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 msgr2=0x7f988c144880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 0x7f988c144880 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7f98840032f0 tx=0x7f9884016000).stop 2022-07-12T08:51:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 shutdown_connections 2022-07-12T08:51:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f988c0b01f0 0x7f988c144d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f988c05cb00 0x7f988c144880 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f986c0539b0 0x7f986c055e70 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 --2- 172.21.15.61:0/2236661833 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f988c05c050 0x7f988c14b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 >> 172.21.15.61:0/2236661833 conn(0x7f988c062c00 msgr2=0x7f988c147370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:13.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 shutdown_connections 2022-07-12T08:51:13.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.723+0000 7f98940b6700 1 -- 172.21.15.61:0/2236661833 wait complete. 2022-07-12T08:51:13.740 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-12T08:40:58.382631+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-12T08:41:09.975085+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-12T08:51:06.224007+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '146', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-12T08:51:06.803491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-12T08:51:13.742 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-12T08:51:13.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.855+0000 7f5ca7239700 1 Processor -- start 2022-07-12T08:51:13.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.855+0000 7f5ca7239700 1 -- start start 2022-07-12T08:51:13.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ca00b1000 0x7f5ca00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca0099010 con 0x7f5ca00afdc0 2022-07-12T08:51:13.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca014b780 con 0x7f5ca0060c10 2022-07-12T08:51:13.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca0060ff0 con 0x7f5ca00b1000 2022-07-12T08:51:13.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca6a38700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5ca00afdc0 0x7f5ca009a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50786/0 (socket says 172.21.15.61:50786) 2022-07-12T08:51:13.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca6a38700 1 -- 172.21.15.61:0/1967032790 learned_addr learned my addr 172.21.15.61:0/1967032790 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:13.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5a36700 1 --2- 172.21.15.61:0/1967032790 >> v2:172.21.15.61:3300/0 conn(0x7f5ca00b1000 0x7f5ca00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5235700 1 -- 172.21.15.61:0/1967032790 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923496314 0 0) 0x7f5ca0099010 con 0x7f5ca00afdc0 2022-07-12T08:51:13.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5235700 1 -- 172.21.15.61:0/1967032790 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c003260 con 0x7f5ca00afdc0 2022-07-12T08:51:13.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5235700 1 -- 172.21.15.61:0/1967032790 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855677646 0 0) 0x7f5c90001490 con 0x7f5ca0060c10 2022-07-12T08:51:13.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5a36700 1 -- 172.21.15.61:0/1967032790 >> v1:172.21.15.111:6789/0 conn(0x7f5ca0060c10 legacy=0x7f5ca014b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5a36700 1 -- 172.21.15.61:0/1967032790 >> v1:172.21.15.61:6789/0 conn(0x7f5ca00afdc0 legacy=0x7f5ca009a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5a36700 1 -- 172.21.15.61:0/1967032790 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca014a930 con 0x7f5ca00b1000 2022-07-12T08:51:13.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5a36700 1 --2- 172.21.15.61:0/1967032790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca00b1000 0x7f5ca00af8d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f5c94002f70 tx=0x7f5c94006df0).ready entity=mon.0 client_cookie=880462c18fd1717 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:13.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5235700 1 -- 172.21.15.61:0/1967032790 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9400d610 con 0x7f5ca00b1000 2022-07-12T08:51:13.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5235700 1 -- 172.21.15.61:0/1967032790 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c94000e90 con 0x7f5ca00b1000 2022-07-12T08:51:13.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca5235700 1 -- 172.21.15.61:0/1967032790 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9400d960 con 0x7f5ca00b1000 2022-07-12T08:51:13.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- 172.21.15.61:0/1967032790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca00b1000 msgr2=0x7f5ca00af8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 --2- 172.21.15.61:0/1967032790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca00b1000 0x7f5ca00af8d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f5c94002f70 tx=0x7f5c94006df0).stop 2022-07-12T08:51:13.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- 172.21.15.61:0/1967032790 shutdown_connections 2022-07-12T08:51:13.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 --2- 172.21.15.61:0/1967032790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca00b1000 0x7f5ca00af8d0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- 172.21.15.61:0/1967032790 >> 172.21.15.61:0/1967032790 conn(0x7f5ca005bd10 msgr2=0x7f5ca005c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:13.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- 172.21.15.61:0/1967032790 shutdown_connections 2022-07-12T08:51:13.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.859+0000 7f5ca7239700 1 -- 172.21.15.61:0/1967032790 wait complete. 2022-07-12T08:51:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 Processor -- start 2022-07-12T08:51:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 -- start start 2022-07-12T08:51:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 0x7f5ca0065b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ca00afdc0 0x7f5ca00ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ca00b1000 0x7f5ca0148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca014e010 con 0x7f5ca0060c10 2022-07-12T08:51:13.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca014e2b0 con 0x7f5ca00b1000 2022-07-12T08:51:13.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca014e550 con 0x7f5ca00afdc0 2022-07-12T08:51:13.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 0x7f5ca0065b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 0x7f5ca0065b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34340/0 (socket says 172.21.15.61:34340) 2022-07-12T08:51:13.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 -- 172.21.15.61:0/3794405784 learned_addr learned my addr 172.21.15.61:0/3794405784 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:13.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6a38700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ca00b1000 0x7f5ca0148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca5a36700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ca00afdc0 0x7f5ca00ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 -- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ca00afdc0 msgr2=0x7f5ca00ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ca00afdc0 0x7f5ca00ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 -- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ca00b1000 msgr2=0x7f5ca0148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ca00b1000 0x7f5ca0148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 -- 172.21.15.61:0/3794405784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c9400b180 con 0x7f5ca0060c10 2022-07-12T08:51:13.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca6237700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 0x7f5ca0065b80 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f5c90001360 tx=0x7f5c90007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:13.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9000e4a0 con 0x7f5ca0060c10 2022-07-12T08:51:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9000e600 con 0x7f5ca0060c10 2022-07-12T08:51:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca014e7f0 con 0x7f5ca0060c10 2022-07-12T08:51:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9000e960 con 0x7f5ca0060c10 2022-07-12T08:51:13.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.863+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca00af280 con 0x7f5ca0060c10 2022-07-12T08:51:13.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.867+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca00acd00 con 0x7f5ca0060c10 2022-07-12T08:51:13.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.867+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5c90019030 con 0x7f5ca0060c10 2022-07-12T08:51:13.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.867+0000 7f5c9f7fe700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c800539b0 0x7f5c80055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:13.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.867+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5c9006b360 con 0x7f5ca0060c10 2022-07-12T08:51:13.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.867+0000 7f5ca5a36700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c800539b0 0x7f5c80055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:13.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.867+0000 7f5ca5a36700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c800539b0 0x7f5c80055e70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f5c9401a000 tx=0x7f5c94014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:13.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.871+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ca00acd00 con 0x7f5ca0060c10 2022-07-12T08:51:14.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.995+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5ca014b780 con 0x7f5ca0060c10 2022-07-12T08:51:14.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.995+0000 7f5c9f7fe700 1 -- 172.21.15.61:0/3794405784 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v148) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5ca014b780 con 0x7f5ca0060c10 2022-07-12T08:51:14.001 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-12T08:51:14.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c800539b0 msgr2=0x7f5c80055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c800539b0 0x7f5c80055e70 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f5c9401a000 tx=0x7f5c94014000).stop 2022-07-12T08:51:14.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 msgr2=0x7f5ca0065b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 0x7f5ca0065b80 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f5c90001360 tx=0x7f5c90007d40).stop 2022-07-12T08:51:14.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 shutdown_connections 2022-07-12T08:51:14.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f5c800539b0 0x7f5c80055e70 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5ca00b1000 0x7f5ca0148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f5ca00afdc0 0x7f5ca00ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 --2- 172.21.15.61:0/3794405784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca0060c10 0x7f5ca0065b80 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 >> 172.21.15.61:0/3794405784 conn(0x7f5ca005bd10 msgr2=0x7f5ca009a270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 shutdown_connections 2022-07-12T08:51:14.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:13.999+0000 7f5ca7239700 1 -- 172.21.15.61:0/3794405784 wait complete. 2022-07-12T08:51:14.020 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-12T08:51:14.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe31db0700 1 Processor -- start 2022-07-12T08:51:14.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe31db0700 1 -- start start 2022-07-12T08:51:14.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe31db0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe2c098c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe31db0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c097c80 con 0x7efe2c0b5920 2022-07-12T08:51:14.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe31db0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c0b5d00 con 0x7efe2c143580 2022-07-12T08:51:14.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe31db0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c05c7f0 con 0x7efe2c0987f0 2022-07-12T08:51:14.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe30dae700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efe2c0b5920 0x7efe2c0b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50798/0 (socket says 172.21.15.61:50798) 2022-07-12T08:51:14.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe30dae700 1 -- 172.21.15.61:0/3701089610 learned_addr learned my addr 172.21.15.61:0/3701089610 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:14.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2bfff700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe2c098c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924050095 0 0) 0x7efe2c097c80 con 0x7efe2c0b5920 2022-07-12T08:51:14.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c003260 con 0x7efe2c0b5920 2022-07-12T08:51:14.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369762281 0 0) 0x7efe2c0b5d00 con 0x7efe2c143580 2022-07-12T08:51:14.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe2c097c80 con 0x7efe2c143580 2022-07-12T08:51:14.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571601871 0 0) 0x7efe0c003260 con 0x7efe2c0b5920 2022-07-12T08:51:14.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 msgr2=0x7efe2c098c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe2c098c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 >> v1:172.21.15.111:6789/0 conn(0x7efe2c143580 legacy=0x7efe2c14b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c05cbf0 con 0x7efe2c0b5920 2022-07-12T08:51:14.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7efe1c001660 con 0x7efe2c0b5920 2022-07-12T08:51:14.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2bfff700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe2c098c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:14.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 >> v1:172.21.15.61:6789/0 conn(0x7efe2c0b5920 legacy=0x7efe2c0b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe0c003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe0c003e50 0x7efe0c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe0c0049f0 0x7efe0c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c05c7f0 con 0x7efe0c003e50 2022-07-12T08:51:14.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00b800 con 0x7efe0c0049f0 2022-07-12T08:51:14.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c00b9c0 con 0x7efe2c0987f0 2022-07-12T08:51:14.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe2bfff700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe0c003e50 0x7efe0c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe30dae700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe0c003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.131+0000 7efe315af700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe0c0049f0 0x7efe0c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe30dae700 1 -- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe0c0049f0 msgr2=0x7efe0c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe30dae700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe0c0049f0 0x7efe0c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe30dae700 1 -- 172.21.15.61:0/3701089610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe0c003e50 msgr2=0x7efe0c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe30dae700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe0c003e50 0x7efe0c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe30dae700 1 -- 172.21.15.61:0/3701089610 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1400b180 con 0x7efe2c0987f0 2022-07-12T08:51:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe30dae700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe0c003910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7efe1c001430 tx=0x7efe1c001d00).ready entity=mon.2 client_cookie=7dd18a441854ce9b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe1c0070d0 con 0x7efe2c0987f0 2022-07-12T08:51:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c007230 con 0x7efe2c0987f0 2022-07-12T08:51:14.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe2b7fe700 1 -- 172.21.15.61:0/3701089610 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe1c00c8f0 con 0x7efe2c0987f0 2022-07-12T08:51:14.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe31db0700 1 -- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 msgr2=0x7efe0c003910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe31db0700 1 --2- 172.21.15.61:0/3701089610 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c0987f0 0x7efe0c003910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7efe1c001430 tx=0x7efe1c001d00).stop 2022-07-12T08:51:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe315af700 1 -- 172.21.15.61:0/3701089610 reap_dead start 2022-07-12T08:51:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe31db0700 1 -- 172.21.15.61:0/3701089610 shutdown_connections 2022-07-12T08:51:14.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe31db0700 1 -- 172.21.15.61:0/3701089610 >> 172.21.15.61:0/3701089610 conn(0x7efe2c063340 msgr2=0x7efe2c0b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe31db0700 1 -- 172.21.15.61:0/3701089610 shutdown_connections 2022-07-12T08:51:14.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.135+0000 7efe31db0700 1 -- 172.21.15.61:0/3701089610 wait complete. 2022-07-12T08:51:14.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 Processor -- start 2022-07-12T08:51:14.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 -- start start 2022-07-12T08:51:14.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 0x7efe2c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c13c970 0x7efe2c141350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0ad2c0 0x7efe2c0ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c13d260 con 0x7efe2c0987f0 2022-07-12T08:51:14.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c13e0b0 con 0x7efe2c0ad2c0 2022-07-12T08:51:14.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c13e350 con 0x7efe2c13c970 2022-07-12T08:51:14.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 0x7efe2c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 0x7efe2c0b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34360/0 (socket says 172.21.15.61:34360) 2022-07-12T08:51:14.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe2bfff700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c13c970 0x7efe2c141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 -- 172.21.15.61:0/4134080672 learned_addr learned my addr 172.21.15.61:0/4134080672 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe315af700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0ad2c0 0x7efe2c0ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 -- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c13c970 msgr2=0x7efe2c141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c13c970 0x7efe2c141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 -- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0ad2c0 msgr2=0x7efe2c0ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0ad2c0 0x7efe2c0ad770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 -- 172.21.15.61:0/4134080672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c13cee0 con 0x7efe2c0987f0 2022-07-12T08:51:14.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe30dae700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 0x7efe2c0b3b90 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7efe1c001430 tx=0x7efe1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe1c00cd70 con 0x7efe2c0987f0 2022-07-12T08:51:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c00ced0 con 0x7efe2c0987f0 2022-07-12T08:51:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c141840 con 0x7efe2c0987f0 2022-07-12T08:51:14.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe1c004710 con 0x7efe2c0987f0 2022-07-12T08:51:14.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.139+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c0ade80 con 0x7efe2c0987f0 2022-07-12T08:51:14.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.143+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c1410b0 con 0x7efe2c0987f0 2022-07-12T08:51:14.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.143+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efe1c027070 con 0x7efe2c0987f0 2022-07-12T08:51:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.143+0000 7efe29ffb700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe10053a40 0x7efe10055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.143+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efe1c06c470 con 0x7efe2c0987f0 2022-07-12T08:51:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.143+0000 7efe2bfff700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe10053a40 0x7efe10055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.143+0000 7efe2bfff700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe10053a40 0x7efe10055f00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7efe1400b000 tx=0x7efe14006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.147+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe2c1410b0 con 0x7efe2c0987f0 2022-07-12T08:51:14.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.275+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efe2c1410b0 con 0x7efe2c0987f0 2022-07-12T08:51:14.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.275+0000 7efe29ffb700 1 -- 172.21.15.61:0/4134080672 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v148) v1 ==== 92+0+10 (secure 0 0 0) 0x7efe2c1410b0 con 0x7efe2c0987f0 2022-07-12T08:51:14.284 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:14.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe10053a40 msgr2=0x7efe10055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe10053a40 0x7efe10055f00 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7efe1400b000 tx=0x7efe14006df0).stop 2022-07-12T08:51:14.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 msgr2=0x7efe2c0b3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 0x7efe2c0b3b90 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 rx=0x7efe1c001430 tx=0x7efe1c015000).stop 2022-07-12T08:51:14.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 shutdown_connections 2022-07-12T08:51:14.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efe10053a40 0x7efe10055f00 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe2c0ad2c0 0x7efe2c0ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efe2c13c970 0x7efe2c141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 --2- 172.21.15.61:0/4134080672 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe2c0987f0 0x7efe2c0b3b90 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 >> 172.21.15.61:0/4134080672 conn(0x7efe2c063340 msgr2=0x7efe2c0b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 shutdown_connections 2022-07-12T08:51:14.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.279+0000 7efe31db0700 1 -- 172.21.15.61:0/4134080672 wait complete. 2022-07-12T08:51:14.294 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-12T08:51:14.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.403+0000 7f0849884700 1 Processor -- start 2022-07-12T08:51:14.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.403+0000 7f0849884700 1 -- start start 2022-07-12T08:51:14.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f08440b5480 0x7f08440b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08440b3730 0x7f08440afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f0844142f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08440b3560 con 0x7f08440b3730 2022-07-12T08:51:14.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08440b5b80 con 0x7f08440b5480 2022-07-12T08:51:14.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08440b5e20 con 0x7f08440b03d0 2022-07-12T08:51:14.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08440b3730 0x7f08440afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08440b3730 0x7f08440afee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34370/0 (socket says 172.21.15.61:34370) 2022-07-12T08:51:14.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 -- 172.21.15.61:0/11764342 learned_addr learned my addr 172.21.15.61:0/11764342 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0842ffd700 1 --2- 172.21.15.61:0/11764342 >> v2:172.21.15.111:3300/0 conn(0x7f08440b5480 0x7f08440b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08437fe700 1 --2- 172.21.15.61:0/11764342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f0844142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 -- 172.21.15.61:0/11764342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 msgr2=0x7f0844142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 --2- 172.21.15.61:0/11764342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f0844142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 -- 172.21.15.61:0/11764342 >> v2:172.21.15.111:3300/0 conn(0x7f08440b5480 msgr2=0x7f08440b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 --2- 172.21.15.61:0/11764342 >> v2:172.21.15.111:3300/0 conn(0x7f08440b5480 0x7f08440b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 -- 172.21.15.61:0/11764342 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08440b0800 con 0x7f08440b3730 2022-07-12T08:51:14.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f08427fc700 1 --2- 172.21.15.61:0/11764342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08440b3730 0x7f08440afee0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f082c002f70 tx=0x7f082c006eb0).ready entity=mon.0 client_cookie=52be174430dbe120 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0841ffb700 1 -- 172.21.15.61:0/11764342 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082c00d570 con 0x7f08440b3730 2022-07-12T08:51:14.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0841ffb700 1 -- 172.21.15.61:0/11764342 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c000e90 con 0x7f08440b3730 2022-07-12T08:51:14.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0841ffb700 1 -- 172.21.15.61:0/11764342 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082c00d900 con 0x7f08440b3730 2022-07-12T08:51:14.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- 172.21.15.61:0/11764342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08440b3730 msgr2=0x7f08440afee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- 172.21.15.61:0/11764342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08440b3730 0x7f08440afee0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f082c002f70 tx=0x7f082c006eb0).stop 2022-07-12T08:51:14.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- 172.21.15.61:0/11764342 shutdown_connections 2022-07-12T08:51:14.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- 172.21.15.61:0/11764342 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f0844142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- 172.21.15.61:0/11764342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08440b3730 0x7f08440afee0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 --2- 172.21.15.61:0/11764342 >> v2:172.21.15.111:3300/0 conn(0x7f08440b5480 0x7f08440b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- 172.21.15.61:0/11764342 >> 172.21.15.61:0/11764342 conn(0x7f08440b96b0 msgr2=0x7f08440b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- 172.21.15.61:0/11764342 shutdown_connections 2022-07-12T08:51:14.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.407+0000 7f0849884700 1 -- 172.21.15.61:0/11764342 wait complete. 2022-07-12T08:51:14.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 Processor -- start 2022-07-12T08:51:14.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 -- start start 2022-07-12T08:51:14.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f084405c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08440b5480 0x7f0844141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 0x7f08440ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08441424d0 con 0x7f08441413b0 2022-07-12T08:51:14.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0844140670 con 0x7f08440b5480 2022-07-12T08:51:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0849884700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0844140910 con 0x7f08440b03d0 2022-07-12T08:51:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 0x7f08440ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 0x7f08440ae010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34376/0 (socket says 172.21.15.61:34376) 2022-07-12T08:51:14.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 -- 172.21.15.61:0/3564048861 learned_addr learned my addr 172.21.15.61:0/3564048861 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f0842ffd700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f084405c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08427fc700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08440b5480 0x7f0844141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 -- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 msgr2=0x7f084405c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f084405c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 -- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08440b5480 msgr2=0x7f0844141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08440b5480 0x7f0844141b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08437fe700 1 -- 172.21.15.61:0/3564048861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c00b1e0 con 0x7f08441413b0 2022-07-12T08:51:14.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.411+0000 7f08427fc700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08440b5480 0x7f0844141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:14.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f08437fe700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 0x7f08440ae010 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f083c00c000 tx=0x7f083c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f083c010780 con 0x7f08441413b0 2022-07-12T08:51:14.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c0108e0 con 0x7f08441413b0 2022-07-12T08:51:14.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0844142150 con 0x7f08441413b0 2022-07-12T08:51:14.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f083c010c40 con 0x7f08441413b0 2022-07-12T08:51:14.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08441485f0 con 0x7f08441413b0 2022-07-12T08:51:14.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0844057460 con 0x7f08441413b0 2022-07-12T08:51:14.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f083c01c460 con 0x7f08441413b0 2022-07-12T08:51:14.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.415+0000 7f083bfff700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0828053a60 0x7f0828055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.419+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f083c06c110 con 0x7f08441413b0 2022-07-12T08:51:14.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.419+0000 7f0842ffd700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0828053a60 0x7f0828055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.419+0000 7f0842ffd700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0828053a60 0x7f0828055f20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f083400b000 tx=0x7f08340091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.423+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f083c043010 con 0x7f08441413b0 2022-07-12T08:51:14.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.547+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08441485f0 con 0x7f08441413b0 2022-07-12T08:51:14.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f083bfff700 1 -- 172.21.15.61:0/3564048861 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v148) v1 ==== 104+0+10 (secure 0 0 0) 0x7f08441485f0 con 0x7f08441413b0 2022-07-12T08:51:14.554 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:14.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0828053a60 msgr2=0x7f0828055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0828053a60 0x7f0828055f20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f083400b000 tx=0x7f08340091d0).stop 2022-07-12T08:51:14.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 msgr2=0x7f08440ae010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 0x7f08440ae010 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 rx=0x7f083c00c000 tx=0x7f083c006df0).stop 2022-07-12T08:51:14.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 shutdown_connections 2022-07-12T08:51:14.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08441413b0 0x7f08440ae010 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0828053a60 0x7f0828055f20 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f08440b5480 0x7f0844141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 --2- 172.21.15.61:0/3564048861 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f08440b03d0 0x7f084405c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 >> 172.21.15.61:0/3564048861 conn(0x7f08440b96b0 msgr2=0x7f0844142d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 shutdown_connections 2022-07-12T08:51:14.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.551+0000 7f0849884700 1 -- 172.21.15.61:0/3564048861 wait complete. 2022-07-12T08:51:14.568 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-12T08:51:14.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.675+0000 7fb83de4b700 1 Processor -- start 2022-07-12T08:51:14.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- start start 2022-07-12T08:51:14.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fb8380b6330 0x7fb8380b1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb83813fba0 0x7fb83813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8380b6710 con 0x7fb838065990 2022-07-12T08:51:14.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8380655a0 con 0x7fb83813fba0 2022-07-12T08:51:14.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8381431d0 con 0x7fb8380b6330 2022-07-12T08:51:14.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb83813fba0 0x7fb83813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb83813fba0 0x7fb83813fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34380/0 (socket says 172.21.15.61:34380) 2022-07-12T08:51:14.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 -- 172.21.15.61:0/4092936477 learned_addr learned my addr 172.21.15.61:0/4092936477 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb82ffff700 1 --1- 172.21.15.61:0/4092936477 >> v1:172.21.15.61:6789/0 conn(0x7fb838065990 0x7fb8380b1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50830/0 (socket says 172.21.15.61:50830) 2022-07-12T08:51:14.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83ce49700 1 --2- 172.21.15.61:0/4092936477 >> v2:172.21.15.111:3300/0 conn(0x7fb8380b6330 0x7fb8380b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb82f7fe700 1 -- 172.21.15.61:0/4092936477 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502901508 0 0) 0x7fb8380b6710 con 0x7fb838065990 2022-07-12T08:51:14.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb82f7fe700 1 -- 172.21.15.61:0/4092936477 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb824003260 con 0x7fb838065990 2022-07-12T08:51:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 -- 172.21.15.61:0/4092936477 >> v2:172.21.15.111:3300/0 conn(0x7fb8380b6330 msgr2=0x7fb8380b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 --2- 172.21.15.61:0/4092936477 >> v2:172.21.15.111:3300/0 conn(0x7fb8380b6330 0x7fb8380b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 -- 172.21.15.61:0/4092936477 >> v1:172.21.15.61:6789/0 conn(0x7fb838065990 legacy=0x7fb8380b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 -- 172.21.15.61:0/4092936477 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb838143470 con 0x7fb83813fba0 2022-07-12T08:51:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83ce49700 1 --2- 172.21.15.61:0/4092936477 >> v2:172.21.15.111:3300/0 conn(0x7fb8380b6330 0x7fb8380b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:14.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83d64a700 1 --2- 172.21.15.61:0/4092936477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb83813fff0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7fb8300032f0 tx=0x7fb83000b530).ready entity=mon.0 client_cookie=7de2418d2af424c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb82f7fe700 1 -- 172.21.15.61:0/4092936477 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb83000ddf0 con 0x7fb83813fba0 2022-07-12T08:51:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb82f7fe700 1 -- 172.21.15.61:0/4092936477 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb830010470 con 0x7fb83813fba0 2022-07-12T08:51:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb82f7fe700 1 -- 172.21.15.61:0/4092936477 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8300107d0 con 0x7fb83813fba0 2022-07-12T08:51:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- 172.21.15.61:0/4092936477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 msgr2=0x7fb83813fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 --2- 172.21.15.61:0/4092936477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb83813fff0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7fb8300032f0 tx=0x7fb83000b530).stop 2022-07-12T08:51:14.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- 172.21.15.61:0/4092936477 shutdown_connections 2022-07-12T08:51:14.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 --2- 172.21.15.61:0/4092936477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb83813fff0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 --2- 172.21.15.61:0/4092936477 >> v2:172.21.15.111:3300/0 conn(0x7fb8380b6330 0x7fb8380b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- 172.21.15.61:0/4092936477 >> 172.21.15.61:0/4092936477 conn(0x7fb838063ad0 msgr2=0x7fb838063eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- 172.21.15.61:0/4092936477 shutdown_connections 2022-07-12T08:51:14.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.679+0000 7fb83de4b700 1 -- 172.21.15.61:0/4092936477 wait complete. 2022-07-12T08:51:14.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 Processor -- start 2022-07-12T08:51:14.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 -- start start 2022-07-12T08:51:14.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838065990 0x7fb83805f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb8380b6330 0x7fb838142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb838142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8381382e0 con 0x7fb83813fba0 2022-07-12T08:51:14.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83813d580 con 0x7fb838065990 2022-07-12T08:51:14.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83813d820 con 0x7fb8380b6330 2022-07-12T08:51:14.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb838142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb838142920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34388/0 (socket says 172.21.15.61:34388) 2022-07-12T08:51:14.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 -- 172.21.15.61:0/2483532157 learned_addr learned my addr 172.21.15.61:0/2483532157 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb82ffff700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb8380b6330 0x7fb838142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83ce49700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838065990 0x7fb83805f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 -- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb8380b6330 msgr2=0x7fb838142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb8380b6330 0x7fb838142430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 -- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838065990 msgr2=0x7fb83805f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838065990 0x7fb83805f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 -- 172.21.15.61:0/2483532157 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83000b7d0 con 0x7fb83813fba0 2022-07-12T08:51:14.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83d64a700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb838142920 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fb8300032f0 tx=0x7fb838064ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb83001d070 con 0x7fb83813fba0 2022-07-12T08:51:14.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb830018800 con 0x7fb83813fba0 2022-07-12T08:51:14.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb830018b60 con 0x7fb83813fba0 2022-07-12T08:51:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.683+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83813dac0 con 0x7fb83813fba0 2022-07-12T08:51:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8380604a0 con 0x7fb83813fba0 2022-07-12T08:51:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb838057460 con 0x7fb83813fba0 2022-07-12T08:51:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb830018cc0 con 0x7fb83813fba0 2022-07-12T08:51:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb82dffb700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb818053a40 0x7fb818055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb83006d3d0 con 0x7fb83813fba0 2022-07-12T08:51:14.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb83ce49700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb818053a40 0x7fb818055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.687+0000 7fb83ce49700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb818053a40 0x7fb818055f00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fb83400d000 tx=0x7fb834008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.695+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb830044330 con 0x7fb83813fba0 2022-07-12T08:51:14.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.819+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb83813f430 con 0x7fb83813fba0 2022-07-12T08:51:14.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.819+0000 7fb82dffb700 1 -- 172.21.15.61:0/2483532157 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v148) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb830047950 con 0x7fb83813fba0 2022-07-12T08:51:14.825 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-12T08:51:14.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb818053a40 msgr2=0x7fb818055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb818053a40 0x7fb818055f00 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7fb83400d000 tx=0x7fb834008040).stop 2022-07-12T08:51:14.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 msgr2=0x7fb838142920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb838142920 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7fb8300032f0 tx=0x7fb838064ca0).stop 2022-07-12T08:51:14.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 shutdown_connections 2022-07-12T08:51:14.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb83813fba0 0x7fb838142920 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fb8380b6330 0x7fb838142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fb818053a40 0x7fb818055f00 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 --2- 172.21.15.61:0/2483532157 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb838065990 0x7fb83805f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 >> 172.21.15.61:0/2483532157 conn(0x7fb838063ad0 msgr2=0x7fb838064150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 shutdown_connections 2022-07-12T08:51:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.823+0000 7fb83de4b700 1 -- 172.21.15.61:0/2483532157 wait complete. 2022-07-12T08:51:14.839 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-12T08:51:14.840 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:14.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.947+0000 7fe8c359e700 1 Processor -- start 2022-07-12T08:51:14.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- start start 2022-07-12T08:51:14.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8c40ab290 0x7fe8c40ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40afa60 0x7fe8c40aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c40abc70 con 0x7fe8c405f140 2022-07-12T08:51:14.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c405f580 con 0x7fe8c40ab290 2022-07-12T08:51:14.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c405c720 con 0x7fe8c40afa60 2022-07-12T08:51:14.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8c40ab290 0x7fe8c40ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe8c40ab290 0x7fe8c40ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34392/0 (socket says 172.21.15.61:34392) 2022-07-12T08:51:14.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 -- 172.21.15.61:0/3936044470 learned_addr learned my addr 172.21.15.61:0/3936044470 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c2d9d700 1 --2- 172.21.15.61:0/3936044470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40afa60 0x7fe8c40aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 -- 172.21.15.61:0/3936044470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40afa60 msgr2=0x7fe8c40aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 --2- 172.21.15.61:0/3936044470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40afa60 0x7fe8c40aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 -- 172.21.15.61:0/3936044470 >> v1:172.21.15.111:6789/0 conn(0x7fe8c405f140 legacy=0x7fe8c405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:14.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 -- 172.21.15.61:0/3936044470 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c405c9c0 con 0x7fe8c40ab290 2022-07-12T08:51:14.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c259c700 1 --2- 172.21.15.61:0/3936044470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c40ab290 0x7fe8c40ab670 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7fe8c40615d0 tx=0x7fe8b4009290).ready entity=mon.0 client_cookie=e00e2cce050a1582 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c159a700 1 -- 172.21.15.61:0/3936044470 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8b400b860 con 0x7fe8c40ab290 2022-07-12T08:51:14.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c159a700 1 -- 172.21.15.61:0/3936044470 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b400b9c0 con 0x7fe8c40ab290 2022-07-12T08:51:14.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c159a700 1 -- 172.21.15.61:0/3936044470 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8b400bb70 con 0x7fe8c40ab290 2022-07-12T08:51:14.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- 172.21.15.61:0/3936044470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c40ab290 msgr2=0x7fe8c40ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 --2- 172.21.15.61:0/3936044470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c40ab290 0x7fe8c40ab670 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7fe8c40615d0 tx=0x7fe8b4009290).stop 2022-07-12T08:51:14.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- 172.21.15.61:0/3936044470 shutdown_connections 2022-07-12T08:51:14.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 --2- 172.21.15.61:0/3936044470 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40afa60 0x7fe8c40aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 --2- 172.21.15.61:0/3936044470 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c40ab290 0x7fe8c40ab670 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- 172.21.15.61:0/3936044470 >> 172.21.15.61:0/3936044470 conn(0x7fe8c40b1950 msgr2=0x7fe8c40b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:14.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- 172.21.15.61:0/3936044470 shutdown_connections 2022-07-12T08:51:14.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.951+0000 7fe8c359e700 1 -- 172.21.15.61:0/3936044470 wait complete. 2022-07-12T08:51:14.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 Processor -- start 2022-07-12T08:51:14.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 -- start start 2022-07-12T08:51:14.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 0x7fe8c40b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40ab290 0x7fe8c4143a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c40afa60 0x7fe8c4143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c40ae500 con 0x7fe8c405f140 2022-07-12T08:51:14.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c40671a0 con 0x7fe8c40afa60 2022-07-12T08:51:14.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c4066bc0 con 0x7fe8c40ab290 2022-07-12T08:51:14.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 0x7fe8c40b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 0x7fe8c40b42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34398/0 (socket says 172.21.15.61:34398) 2022-07-12T08:51:14.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.955+0000 7fe8c259c700 1 -- 172.21.15.61:0/2995088718 learned_addr learned my addr 172.21.15.61:0/2995088718 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:14.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c2d9d700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c40afa60 0x7fe8c4143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c1d9b700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40ab290 0x7fe8c4143a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c259c700 1 -- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40ab290 msgr2=0x7fe8c4143a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c259c700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40ab290 0x7fe8c4143a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c259c700 1 -- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c40afa60 msgr2=0x7fe8c4143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:14.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c259c700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c40afa60 0x7fe8c4143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:14.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c259c700 1 -- 172.21.15.61:0/2995088718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4009530 con 0x7fe8c405f140 2022-07-12T08:51:14.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c259c700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 0x7fe8c40b42b0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7fe8c40615d0 tx=0x7fe8b4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8b4021440 con 0x7fe8c405f140 2022-07-12T08:51:14.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b40215a0 con 0x7fe8c405f140 2022-07-12T08:51:14.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c4066e60 con 0x7fe8c405f140 2022-07-12T08:51:14.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8b4021900 con 0x7fe8c405f140 2022-07-12T08:51:14.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c40675f0 con 0x7fe8c405f140 2022-07-12T08:51:14.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c4057460 con 0x7fe8c405f140 2022-07-12T08:51:14.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe8b400f070 con 0x7fe8c405f140 2022-07-12T08:51:14.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8bb7fe700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8a8053a40 0x7fe8a8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:14.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe8b406d8f0 con 0x7fe8c405f140 2022-07-12T08:51:14.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.959+0000 7fe8c1d9b700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8a8053a40 0x7fe8a8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:14.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.963+0000 7fe8c1d9b700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8a8053a40 0x7fe8a8055f00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fe8ac00b000 tx=0x7fe8ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:14.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:14.967+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe8b4044850 con 0x7fe8c405f140 2022-07-12T08:51:15.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.131+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8c4067350 con 0x7fe8c405f140 2022-07-12T08:51:15.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.131+0000 7fe8bb7fe700 1 -- 172.21.15.61:0/2995088718 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fe8b4027090 con 0x7fe8c405f140 2022-07-12T08:51:15.138 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:15.138 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1725259132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1725259132},{"type":"v1","addr":"172.21.15.111:6835","nonce":1725259132}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:51:07.632105+0000","modified":"2022-07-12T08:51:08.635747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8775":{"gid":8775,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/502386216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":502386216},{"type":"v1","addr":"172.21.15.61:6833","nonce":502386216}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-12T08:51:15.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8a8053a40 msgr2=0x7fe8a8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:15.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8a8053a40 0x7fe8a8055f00 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7fe8ac00b000 tx=0x7fe8ac009040).stop 2022-07-12T08:51:15.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 msgr2=0x7fe8c40b42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:15.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 0x7fe8c40b42b0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7fe8c40615d0 tx=0x7fe8b4016000).stop 2022-07-12T08:51:15.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 shutdown_connections 2022-07-12T08:51:15.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe8a8053a40 0x7fe8a8055f00 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:15.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8c40afa60 0x7fe8c4143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:15.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe8c40ab290 0x7fe8c4143a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:15.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 --2- 172.21.15.61:0/2995088718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe8c405f140 0x7fe8c40b42b0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:15.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 >> 172.21.15.61:0/2995088718 conn(0x7fe8c40b1950 msgr2=0x7fe8c4099bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:15.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 shutdown_connections 2022-07-12T08:51:15.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:15.135+0000 7fe8c359e700 1 -- 172.21.15.61:0/2995088718 wait complete. 2022-07-12T08:51:15.147 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-12T08:51:15.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-12T08:51:07.632105+0000', 'modified': '2022-07-12T08:51:08.635747+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8775}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8775': {'gid': 8775, 'name': 'a', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/502386216', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 502386216}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 502386216}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-12T08:51:15.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-12T08:51:15.357 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-12T08:51:15.487 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","whoami":0,"id":8775,"want_state":"up:active","state":"up:active","rank_uptime":7.8389181600000004,"mdsmap_epoch":83,"osdmap_epoch":148,"osdmap_epoch_barrier":147,"uptime":12.208008398} 2022-07-12T08:51:15.487 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", "id": 8775, "mdsmap_epoch": 83, "osdmap_epoch": 148, "osdmap_epoch_barrier": 147, "rank_uptime": 7.83891816, "state": "up:active", "uptime": 12.208008398, "want_state": "up:active", "whoami": 0 } 2022-07-12T08:51:15.488 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-12T08:51:15.488 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-12T08:51:15.488 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi130.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-12T08:51:15.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:15.497 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:15.517 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:51:15.537 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:51:15.538 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:51:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:51:15.613 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:51:15.614 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:15.614 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:51:15.663 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-12T08:51:15.664 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:15.668 DEBUG:teuthology.orchestra.run.smithi130:> sudo modprobe fuse 2022-07-12T08:51:15.696 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-12T08:51:15.696 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-12T08:51:15.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:2022-07-12T08:51:15.740+0000 7f7447397080 -1 init, newargv = 0x5647b8ee8530 newargc=9ceph-fuse[21829]: starting ceph client 2022-07-12T08:51:15.760 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr: 2022-07-12T08:51:15.760 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21829]: starting fuse 2022-07-12T08:51:15.772 INFO:teuthology.orchestra.run.smithi130.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-12T08:51:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-12T08:51:15.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:15.777 DEBUG:teuthology.orchestra.run.smithi130:> ls /sys/fs/fuse/connections 2022-07-12T08:51:15.837 INFO:teuthology.orchestra.run.smithi130.stdout:52 2022-07-12T08:51:15.837 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-12T08:51:15.838 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:15.839 DEBUG:teuthology.orchestra.run.smithi130:> sudo python3 -c ' 2022-07-12T08:51:15.839 DEBUG:teuthology.orchestra.run.smithi130:> import glob 2022-07-12T08:51:15.840 DEBUG:teuthology.orchestra.run.smithi130:> import re 2022-07-12T08:51:15.841 DEBUG:teuthology.orchestra.run.smithi130:> import os 2022-07-12T08:51:15.841 DEBUG:teuthology.orchestra.run.smithi130:> import subprocess 2022-07-12T08:51:15.842 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:51:15.842 DEBUG:teuthology.orchestra.run.smithi130:> def find_socket(client_name): 2022-07-12T08:51:15.843 DEBUG:teuthology.orchestra.run.smithi130:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-12T08:51:15.844 DEBUG:teuthology.orchestra.run.smithi130:> files = glob.glob(asok_path) 2022-07-12T08:51:15.845 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:51:15.845 DEBUG:teuthology.orchestra.run.smithi130:> # Given a non-glob path, it better be there 2022-07-12T08:51:15.845 DEBUG:teuthology.orchestra.run.smithi130:> if "*" not in asok_path: 2022-07-12T08:51:15.846 DEBUG:teuthology.orchestra.run.smithi130:> assert(len(files) == 1) 2022-07-12T08:51:15.846 DEBUG:teuthology.orchestra.run.smithi130:> return files[0] 2022-07-12T08:51:15.847 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:51:15.847 DEBUG:teuthology.orchestra.run.smithi130:> for f in files: 2022-07-12T08:51:15.847 DEBUG:teuthology.orchestra.run.smithi130:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-12T08:51:15.848 DEBUG:teuthology.orchestra.run.smithi130:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-12T08:51:15.849 DEBUG:teuthology.orchestra.run.smithi130:> return f 2022-07-12T08:51:15.849 DEBUG:teuthology.orchestra.run.smithi130:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-12T08:51:15.849 DEBUG:teuthology.orchestra.run.smithi130:> 2022-07-12T08:51:15.850 DEBUG:teuthology.orchestra.run.smithi130:> print(find_socket("client.1")) 2022-07-12T08:51:15.851 DEBUG:teuthology.orchestra.run.smithi130:> ' 2022-07-12T08:51:15.901 INFO:teuthology.orchestra.run.smithi130.stdout:/var/run/ceph/ceph-client.1.21829.asok 2022-07-12T08:51:15.910 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.21829.asok 2022-07-12T08:51:15.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:15.912 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.21829.asok status 2022-07-12T08:51:16.009 INFO:teuthology.orchestra.run.smithi130.stdout:{ 2022-07-12T08:51:16.009 INFO:teuthology.orchestra.run.smithi130.stdout: "metadata": { 2022-07-12T08:51:16.009 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_sha1": "3497be4c2159a20e7a0f23fb5e2730060a4424c0", 2022-07-12T08:51:16.010 INFO:teuthology.orchestra.run.smithi130.stdout: "ceph_version": "ceph version 15.2.16-324-g3497be4c (3497be4c2159a20e7a0f23fb5e2730060a4424c0) octopus (stable)", 2022-07-12T08:51:16.010 INFO:teuthology.orchestra.run.smithi130.stdout: "entity_id": "1", 2022-07-12T08:51:16.011 INFO:teuthology.orchestra.run.smithi130.stdout: "hostname": "smithi130", 2022-07-12T08:51:16.011 INFO:teuthology.orchestra.run.smithi130.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-12T08:51:16.011 INFO:teuthology.orchestra.run.smithi130.stdout: "pid": "21829", 2022-07-12T08:51:16.011 INFO:teuthology.orchestra.run.smithi130.stdout: "root": "/" 2022-07-12T08:51:16.011 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:51:16.012 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_count": 0, 2022-07-12T08:51:16.012 INFO:teuthology.orchestra.run.smithi130.stdout: "dentry_pinned_count": 0, 2022-07-12T08:51:16.013 INFO:teuthology.orchestra.run.smithi130.stdout: "id": 8997, 2022-07-12T08:51:16.013 INFO:teuthology.orchestra.run.smithi130.stdout: "inst": { 2022-07-12T08:51:16.014 INFO:teuthology.orchestra.run.smithi130.stdout: "name": { 2022-07-12T08:51:16.014 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "client", 2022-07-12T08:51:16.015 INFO:teuthology.orchestra.run.smithi130.stdout: "num": 8997 2022-07-12T08:51:16.015 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:51:16.016 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:51:16.016 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:51:16.017 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:51:16.017 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2638838127 2022-07-12T08:51:16.018 INFO:teuthology.orchestra.run.smithi130.stdout: } 2022-07-12T08:51:16.018 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:51:16.019 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": { 2022-07-12T08:51:16.019 INFO:teuthology.orchestra.run.smithi130.stdout: "type": "v1", 2022-07-12T08:51:16.019 INFO:teuthology.orchestra.run.smithi130.stdout: "addr": "172.21.15.130:0", 2022-07-12T08:51:16.020 INFO:teuthology.orchestra.run.smithi130.stdout: "nonce": 2638838127 2022-07-12T08:51:16.020 INFO:teuthology.orchestra.run.smithi130.stdout: }, 2022-07-12T08:51:16.021 INFO:teuthology.orchestra.run.smithi130.stdout: "inst_str": "client.8997 172.21.15.130:0/2638838127", 2022-07-12T08:51:16.021 INFO:teuthology.orchestra.run.smithi130.stdout: "addr_str": "172.21.15.130:0/2638838127", 2022-07-12T08:51:16.023 INFO:teuthology.orchestra.run.smithi130.stdout: "inode_count": 1, 2022-07-12T08:51:16.023 INFO:teuthology.orchestra.run.smithi130.stdout: "mds_epoch": 83, 2022-07-12T08:51:16.024 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch": 148, 2022-07-12T08:51:16.024 INFO:teuthology.orchestra.run.smithi130.stdout: "osd_epoch_barrier": 0, 2022-07-12T08:51:16.025 INFO:teuthology.orchestra.run.smithi130.stdout: "blacklisted": false, 2022-07-12T08:51:16.025 INFO:teuthology.orchestra.run.smithi130.stdout: "fs_name": "cephfs" 2022-07-12T08:51:16.025 INFO:teuthology.orchestra.run.smithi130.stdout:} 2022-07-12T08:51:16.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:16.029 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:16.029 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.045 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:51:16.047 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:51:16.048 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:16.049 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:16.119 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:16.120 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.141 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:51:16.142 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:51:16.143 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:16.144 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:16.145 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.189 INFO:teuthology.orchestra.run.smithi130.stdout:fuseblk 2022-07-12T08:51:16.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-12T08:51:16.193 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi130.front.sepia.ceph.com... 2022-07-12T08:51:16.194 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-12T08:51:16.194 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi130.stderr:ceph-fuse[21829]: fuse finished with error 0 and tester_r 0 2022-07-12T08:51:16.214 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:16.215 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:16.216 DEBUG:teuthology.orchestra.run.smithi130:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.272 INFO:teuthology.orchestra.run.smithi130.stdout:ext2/ext3 2022-07-12T08:51:16.272 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-12T08:51:16.273 INFO:teuthology.orchestra.run:waiting for 900 2022-07-12T08:51:16.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-12T08:51:16.339 DEBUG:teuthology.orchestra.run.smithi130:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-12T08:51:16.347 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-12T08:51:16.348 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-12T08:51:16.380 INFO:teuthology.orchestra.run.smithi121.stdout: File: "." 2022-07-12T08:51:16.381 INFO:teuthology.orchestra.run.smithi121.stdout: ID: 0 Namelen: 255 Type: fuseblk 2022-07-12T08:51:16.392 INFO:teuthology.orchestra.run.smithi121.stdout:Block size: 4194304 Fundamental block size: 4194304 2022-07-12T08:51:16.392 INFO:teuthology.orchestra.run.smithi121.stdout:Blocks: Total: 86148 Free: 86148 Available: 86148 2022-07-12T08:51:16.393 INFO:teuthology.orchestra.run.smithi121.stdout:Inodes: Total: 1 Free: 0 2022-07-12T08:51:16.395 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-12T08:51:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f2194d88700 1 Processor -- start 2022-07-12T08:51:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f2194d88700 1 -- start start 2022-07-12T08:51:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f2194d88700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f2190099f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f2194d88700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2190097bd0 con 0x7f21900b5920 2022-07-12T08:51:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f2194d88700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21900b5d00 con 0x7f2190140e60 2022-07-12T08:51:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f2194d88700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219005c810 con 0x7f21900b09a0 2022-07-12T08:51:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f21900b5920 0x7f2190099820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50858/0 (socket says 172.21.15.61:50858) 2022-07-12T08:51:16.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218f7fe700 1 -- 172.21.15.61:0/4216919200 learned_addr learned my addr 172.21.15.61:0/4216919200 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218effd700 1 --2- 172.21.15.61:0/4216919200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f2190099f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218e7fc700 1 -- 172.21.15.61:0/4216919200 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586752104 0 0) 0x7f2190097bd0 con 0x7f21900b5920 2022-07-12T08:51:16.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218effd700 1 -- 172.21.15.61:0/4216919200 >> v1:172.21.15.111:6789/0 conn(0x7f2190140e60 legacy=0x7f21901412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218effd700 1 -- 172.21.15.61:0/4216919200 >> v1:172.21.15.61:6789/0 conn(0x7f21900b5920 legacy=0x7f2190099820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218effd700 1 -- 172.21.15.61:0/4216919200 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21900991f0 con 0x7f21900b09a0 2022-07-12T08:51:16.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218effd700 1 --2- 172.21.15.61:0/4216919200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f2190099f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2188001040 tx=0x7f21800091d0).ready entity=mon.2 client_cookie=984e254f83b0df40 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218e7fc700 1 -- 172.21.15.61:0/4216919200 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218000b630 con 0x7f21900b09a0 2022-07-12T08:51:16.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218e7fc700 1 -- 172.21.15.61:0/4216919200 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218000b790 con 0x7f21900b09a0 2022-07-12T08:51:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.511+0000 7f218e7fc700 1 -- 172.21.15.61:0/4216919200 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218000b900 con 0x7f21900b09a0 2022-07-12T08:51:16.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- 172.21.15.61:0/4216919200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 msgr2=0x7f2190099f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 --2- 172.21.15.61:0/4216919200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f2190099f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2188001040 tx=0x7f21800091d0).stop 2022-07-12T08:51:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- 172.21.15.61:0/4216919200 shutdown_connections 2022-07-12T08:51:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 --2- 172.21.15.61:0/4216919200 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f2190099f30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:16.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- 172.21.15.61:0/4216919200 >> 172.21.15.61:0/4216919200 conn(0x7f2190063380 msgr2=0x7f21900b4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- 172.21.15.61:0/4216919200 shutdown_connections 2022-07-12T08:51:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- 172.21.15.61:0/4216919200 wait complete. 2022-07-12T08:51:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 Processor -- start 2022-07-12T08:51:16.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- start start 2022-07-12T08:51:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f21900aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 0x7f21900ac550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2190140e60 0x7f21900adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:16.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21900abdf0 con 0x7f21900b5920 2022-07-12T08:51:16.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2190147ef0 con 0x7f2190140e60 2022-07-12T08:51:16.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f2194d88700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2190096d70 con 0x7f21900b09a0 2022-07-12T08:51:16.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f218effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 0x7f21900ac550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:16.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f218f7fe700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f21900aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f218effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 0x7f21900ac550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34414/0 (socket says 172.21.15.61:34414) 2022-07-12T08:51:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f218effd700 1 -- 172.21.15.61:0/1115844959 learned_addr learned my addr 172.21.15.61:0/1115844959 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.515+0000 7f218ffff700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2190140e60 0x7f21900adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218effd700 1 -- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 msgr2=0x7f21900aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:16.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218effd700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f21900aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218effd700 1 -- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2190140e60 msgr2=0x7f21900adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218effd700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2190140e60 0x7f21900adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218effd700 1 -- 172.21.15.61:0/1115844959 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2180009470 con 0x7f21900b5920 2022-07-12T08:51:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218effd700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 0x7f21900ac550 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f2188001040 tx=0x7f2180016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218000b630 con 0x7f21900b5920 2022-07-12T08:51:16.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21900aca40 con 0x7f21900b5920 2022-07-12T08:51:16.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218000beb0 con 0x7f21900b5920 2022-07-12T08:51:16.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218001f5e0 con 0x7f21900b5920 2022-07-12T08:51:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2190147a40 con 0x7f21900b5920 2022-07-12T08:51:16.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2190147d50 con 0x7f21900b5920 2022-07-12T08:51:16.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2180026030 con 0x7f21900b5920 2022-07-12T08:51:16.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218cff9700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2170053a60 0x7f2170055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:16.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f218006cb90 con 0x7f21900b5920 2022-07-12T08:51:16.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.519+0000 7f218f7fe700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2170053a60 0x7f2170055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:16.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.523+0000 7f218f7fe700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2170053a60 0x7f2170055f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f218400d000 tx=0x7f2184009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:16.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.527+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2190147d50 con 0x7f21900b5920 2022-07-12T08:51:16.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.655+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2190147d50 con 0x7f21900b5920 2022-07-12T08:51:16.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f218cff9700 1 -- 172.21.15.61:0/1115844959 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v148) v1 ==== 81+0+33707 (secure 0 0 0) 0x7f2190147d50 con 0x7f21900b5920 2022-07-12T08:51:16.668 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:16.669 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T08:51:16.669 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 148, 2022-07-12T08:51:16.670 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a", 2022-07-12T08:51:16.670 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-12T08:40:54.309679+0000", 2022-07-12T08:51:16.670 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-12T08:51:09.648732+0000", 2022-07-12T08:51:16.671 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:51:16.671 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-12T08:41:03.796038+0000", 2022-07-12T08:51:16.671 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-12T08:51:16.671 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-12T08:51:16.672 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-12T08:51:16.672 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-12T08:51:16.673 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-12T08:51:16.673 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-12T08:51:16.673 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-12T08:51:16.673 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:16.674 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-12T08:51:16.674 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-12T08:51:16.675 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-12T08:51:16.675 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-12T08:51:16.676 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-12T08:51:16.676 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 30, 2022-07-12T08:51:16.677 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-12T08:51:16.677 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-12T08:51:16.678 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-12T08:51:16.678 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-12T08:51:16.678 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-12T08:51:16.679 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.679 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-12T08:51:16.680 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-12T08:51:16.680 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:40:58.382631+0000", 2022-07-12T08:51:16.681 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:51:16.681 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:51:16.682 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:51:16.682 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:51:16.682 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:51:16.683 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:51:16.683 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:51:16.684 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:51:16.684 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-12T08:51:16.685 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-12T08:51:16.685 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-12T08:51:16.685 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-12T08:51:16.686 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-12T08:51:16.687 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:51:16.687 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:51:16.688 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:51:16.688 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:51:16.688 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:51:16.689 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:51:16.701 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:51:16.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.703 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-12T08:51:16.703 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:51:16.704 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:51:16.704 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:51:16.705 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:51:16.705 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:51:16.705 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:51:16.707 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:51:16.707 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:51:16.707 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:51:16.708 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:51:16.708 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:51:16.709 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:51:16.709 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:51:16.709 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:51:16.710 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:51:16.710 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:51:16.711 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:51:16.711 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:51:16.712 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:51:16.712 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:51:16.712 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:51:16.713 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:51:16.713 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:51:16.713 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:51:16.714 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:51:16.714 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:51:16.715 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.715 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:51:16.715 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:51:16.716 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:51:16.716 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:51:16.716 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:51:16.717 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:51:16.717 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:51:16.718 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:51:16.718 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:51:16.719 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:51:16.719 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:51:16.722 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:51:16.722 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-12T08:51:16.723 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.723 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:51:16.723 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-12T08:51:16.723 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.724 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.724 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.725 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-12T08:51:16.725 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-12T08:51:16.725 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:41:09.975085+0000", 2022-07-12T08:51:16.726 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:51:16.727 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:51:16.727 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:51:16.728 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:51:16.728 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:51:16.729 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:51:16.729 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:51:16.729 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:51:16.730 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:51:16.730 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:51:16.731 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:51:16.731 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:51:16.731 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:51:16.732 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:51:16.732 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:51:16.732 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:51:16.733 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:51:16.733 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:51:16.734 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:51:16.734 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:51:16.734 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.734 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-12T08:51:16.735 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:51:16.735 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:51:16.735 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:51:16.735 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:51:16.735 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:51:16.736 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:51:16.736 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:51:16.738 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:51:16.739 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:51:16.739 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:51:16.739 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:51:16.740 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:51:16.740 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:51:16.741 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:51:16.741 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:51:16.742 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:51:16.742 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:51:16.743 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:51:16.743 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:51:16.744 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:51:16.744 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:51:16.745 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:51:16.745 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:51:16.745 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:51:16.746 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:51:16.746 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:51:16.747 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.747 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:51:16.748 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:51:16.749 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:51:16.749 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:51:16.749 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:51:16.750 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:51:16.750 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:51:16.750 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:51:16.751 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:51:16.751 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:51:16.752 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:51:16.752 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:51:16.752 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:51:16.753 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-12T08:51:16.753 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.754 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.754 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.754 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 29, 2022-07-12T08:51:16.755 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-12T08:51:16.755 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:51:06.224007+0000", 2022-07-12T08:51:16.756 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:51:16.756 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:51:16.756 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:51:16.757 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:51:16.757 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:51:16.757 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:51:16.758 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:51:16.758 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:51:16.759 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:51:16.760 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:51:16.760 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:51:16.760 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:51:16.761 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:51:16.761 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:51:16.762 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:51:16.762 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:51:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:51:16.763 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:51:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:51:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:51:16.764 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.765 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "146", 2022-07-12T08:51:16.765 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:51:16.766 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:51:16.766 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:51:16.766 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:51:16.767 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:51:16.768 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:51:16.768 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:51:16.768 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:51:16.768 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:51:16.769 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:51:16.769 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:51:16.770 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:51:16.770 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:51:16.770 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:51:16.771 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:51:16.771 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:51:16.771 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:51:16.772 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:51:16.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:51:16.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:51:16.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:51:16.773 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:51:16.773 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:51:16.773 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:51:16.774 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:51:16.775 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:51:16.776 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.776 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:51:16.776 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:51:16.777 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:51:16.777 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:51:16.778 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:51:16.778 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:51:16.779 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:51:16.779 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:51:16.780 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:51:16.782 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:51:16.782 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:51:16.783 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-12T08:51:16.783 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-12T08:51:16.784 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-12T08:51:16.784 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-12T08:51:16.785 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.785 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:51:16.785 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:51:16.786 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-12T08:51:16.786 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.787 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.788 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.788 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 30, 2022-07-12T08:51:16.789 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-12T08:51:16.789 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-12T08:51:06.803491+0000", 2022-07-12T08:51:16.790 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-12T08:51:16.791 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-12T08:51:16.792 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-12T08:51:16.793 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-12T08:51:16.793 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-12T08:51:16.793 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-12T08:51:16.795 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-12T08:51:16.795 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-12T08:51:16.795 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-12T08:51:16.796 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-12T08:51:16.796 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-12T08:51:16.796 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-12T08:51:16.796 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-12T08:51:16.797 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-12T08:51:16.797 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-12T08:51:16.797 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-12T08:51:16.797 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-12T08:51:16.798 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-12T08:51:16.798 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-12T08:51:16.798 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-12T08:51:16.798 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.799 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "148", 2022-07-12T08:51:16.799 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-12T08:51:16.799 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-12T08:51:16.800 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-12T08:51:16.800 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-12T08:51:16.800 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-12T08:51:16.801 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-12T08:51:16.801 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-12T08:51:16.801 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-12T08:51:16.801 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-12T08:51:16.802 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-12T08:51:16.802 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-12T08:51:16.802 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-12T08:51:16.802 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-12T08:51:16.803 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-12T08:51:16.803 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-12T08:51:16.803 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-12T08:51:16.803 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-12T08:51:16.804 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-12T08:51:16.804 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-12T08:51:16.804 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-12T08:51:16.804 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-12T08:51:16.805 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-12T08:51:16.805 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-12T08:51:16.805 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-12T08:51:16.805 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-12T08:51:16.806 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-12T08:51:16.807 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.807 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-12T08:51:16.807 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-12T08:51:16.807 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-12T08:51:16.808 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-12T08:51:16.808 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-12T08:51:16.808 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-12T08:51:16.808 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-12T08:51:16.809 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-12T08:51:16.809 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-12T08:51:16.809 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-12T08:51:16.809 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-12T08:51:16.810 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-12T08:51:16.810 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-12T08:51:16.811 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-12T08:51:16.811 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-12T08:51:16.811 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.811 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.812 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.812 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:16.812 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-12T08:51:16.812 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.813 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:51:16.813 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "c5e8edcd-71d7-4bc2-84d6-819a6f4be259", 2022-07-12T08:51:16.813 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:16.813 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:16.814 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:16.814 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:16.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:16.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:16.815 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:16.815 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 145, 2022-07-12T08:51:16.815 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:16.815 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:16.816 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:16.816 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.816 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.817 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.817 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-12T08:51:16.817 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.818 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.818 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.818 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.818 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-12T08:51:16.819 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.819 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.819 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.819 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.820 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:16.820 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.820 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.820 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.821 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-12T08:51:16.821 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.821 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.822 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.822 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.822 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-12T08:51:16.822 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.822 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.823 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.823 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.823 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:16.823 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.824 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.824 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-12T08:51:16.824 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.825 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.825 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.825 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.825 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-12T08:51:16.826 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.826 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.826 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.827 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:16.828 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.828 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.828 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.828 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-12T08:51:16.829 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.829 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.829 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.829 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.829 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-12T08:51:16.830 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21668 2022-07-12T08:51:16.830 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.830 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.830 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.831 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/21668", 2022-07-12T08:51:16.831 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/21668", 2022-07-12T08:51:16.831 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/21668", 2022-07-12T08:51:16.831 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/21668", 2022-07-12T08:51:16.832 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:16.832 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:16.832 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:16.832 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.833 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.833 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.839 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:51:16.840 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "78c8bcfb-e525-4fea-a66a-002769a0b1ee", 2022-07-12T08:51:16.840 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:16.840 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:16.841 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:16.841 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:16.841 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:16.841 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:16.842 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:16.842 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 144, 2022-07-12T08:51:16.843 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:16.843 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:16.843 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:16.843 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.844 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.844 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.844 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-12T08:51:16.845 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.845 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.845 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.845 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.846 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-12T08:51:16.846 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.847 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.847 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.847 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.847 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:16.848 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.848 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.848 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-12T08:51:16.850 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.850 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.851 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.851 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.852 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-12T08:51:16.852 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.852 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-12T08:51:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-12T08:51:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-12T08:51:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.860 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.860 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.861 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-12T08:51:16.861 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21713 2022-07-12T08:51:16.862 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.862 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.862 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.863 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/21713", 2022-07-12T08:51:16.863 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/21713", 2022-07-12T08:51:16.863 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6815/21713", 2022-07-12T08:51:16.863 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/21713", 2022-07-12T08:51:16.864 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:16.865 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:16.866 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:16.866 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.867 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.867 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.868 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:51:16.868 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b765f537-14cb-46b5-be53-b78393e087ad", 2022-07-12T08:51:16.869 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:16.870 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:16.870 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:16.871 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:16.871 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:16.871 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:16.872 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:16.872 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 145, 2022-07-12T08:51:16.873 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:16.873 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:16.873 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:16.874 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.875 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.875 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.875 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-12T08:51:16.877 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.878 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.879 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-12T08:51:16.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.880 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.881 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.882 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.882 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:16.883 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.883 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.883 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-12T08:51:16.885 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.886 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.886 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.886 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.887 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-12T08:51:16.887 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.888 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.888 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.889 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.889 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:16.890 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.890 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.890 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.891 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-12T08:51:16.891 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.892 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.893 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.893 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-12T08:51:16.894 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.895 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.895 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.896 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:16.897 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.897 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.898 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.898 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-12T08:51:16.899 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.899 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.900 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.900 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-12T08:51:16.901 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21871 2022-07-12T08:51:16.902 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.903 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.903 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.905 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/21871", 2022-07-12T08:51:16.906 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/21871", 2022-07-12T08:51:16.907 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/21871", 2022-07-12T08:51:16.907 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/21871", 2022-07-12T08:51:16.907 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:16.908 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:16.908 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:16.909 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.910 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.910 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:51:16.911 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "0f043fbb-3135-49f3-9b49-a6ce7e7347b4", 2022-07-12T08:51:16.911 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:16.912 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:16.912 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:16.913 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:16.913 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:16.913 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:16.914 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:16.914 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 146, 2022-07-12T08:51:16.915 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:16.915 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:16.916 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:16.916 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.917 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.917 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-12T08:51:16.918 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.918 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.919 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.919 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.920 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-12T08:51:16.920 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.921 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.921 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.921 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.922 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:16.922 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.923 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.924 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-12T08:51:16.924 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.925 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.925 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.925 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.926 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-12T08:51:16.926 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.927 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.927 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.927 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.928 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:16.928 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.929 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.929 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.929 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-12T08:51:16.930 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.930 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.931 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.931 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.931 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-12T08:51:16.932 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.932 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.933 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.933 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.933 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:16.934 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.934 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.934 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.935 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-12T08:51:16.935 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.935 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.936 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.936 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.937 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-12T08:51:16.938 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21976 2022-07-12T08:51:16.938 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.939 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.939 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.940 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/21976", 2022-07-12T08:51:16.940 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/21976", 2022-07-12T08:51:16.941 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/21976", 2022-07-12T08:51:16.941 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/21976", 2022-07-12T08:51:16.941 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:16.942 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:16.943 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:16.943 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.944 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.944 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.945 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:51:16.945 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b2005cdb-ea05-443e-8816-aaa7dfabfd84", 2022-07-12T08:51:16.946 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:16.946 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:16.947 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:16.947 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:16.947 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:16.948 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:16.948 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:16.949 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 144, 2022-07-12T08:51:16.950 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:16.950 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:16.951 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:16.951 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.952 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.952 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.953 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6802", 2022-07-12T08:51:16.953 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.954 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.954 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.955 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.955 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6803", 2022-07-12T08:51:16.956 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.956 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.957 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.957 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.958 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:16.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.958 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.959 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.959 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6804", 2022-07-12T08:51:16.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.962 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.962 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.963 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.963 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6805", 2022-07-12T08:51:16.964 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.964 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.964 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.964 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.965 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:16.965 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.965 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.966 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.966 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6808", 2022-07-12T08:51:16.967 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.967 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.968 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.968 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.968 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6809", 2022-07-12T08:51:16.969 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.969 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.970 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.970 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.971 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:16.971 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.972 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6806", 2022-07-12T08:51:16.973 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.973 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.974 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.974 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.975 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6807", 2022-07-12T08:51:16.975 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21886 2022-07-12T08:51:16.975 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.976 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.976 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.977 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6803/21886", 2022-07-12T08:51:16.977 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6805/21886", 2022-07-12T08:51:16.978 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6809/21886", 2022-07-12T08:51:16.978 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6807/21886", 2022-07-12T08:51:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:16.979 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.980 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.980 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.980 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:51:16.980 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9b0164a4-42b8-409a-86fb-bafc85e28b29", 2022-07-12T08:51:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:16.981 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 145, 2022-07-12T08:51:16.982 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:16.983 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:16.983 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:16.983 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.983 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6810", 2022-07-12T08:51:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.984 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6811", 2022-07-12T08:51:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.985 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.986 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.986 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.986 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6812", 2022-07-12T08:51:16.987 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.988 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.988 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.988 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.988 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6813", 2022-07-12T08:51:16.988 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.989 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.989 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.989 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.989 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:16.990 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.990 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.990 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.990 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6816", 2022-07-12T08:51:16.990 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.991 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.991 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.991 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.991 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6817", 2022-07-12T08:51:16.992 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.992 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.992 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.992 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.992 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:16.993 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:16.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.993 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:16.993 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6814", 2022-07-12T08:51:16.994 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.994 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.994 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:16.995 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:16.996 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6815", 2022-07-12T08:51:16.996 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 21888 2022-07-12T08:51:16.997 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:16.997 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:16.997 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:16.998 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6811/21888", 2022-07-12T08:51:16.998 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6813/21888", 2022-07-12T08:51:16.999 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6817/21888", 2022-07-12T08:51:16.999 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6815/21888", 2022-07-12T08:51:16.999 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:17.000 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:17.000 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:17.000 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.001 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.001 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:51:17.002 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1718afd4-28d5-4c9e-bbb8-f8b29600a5fa", 2022-07-12T08:51:17.002 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:17.002 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:17.003 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:17.003 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:17.004 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:17.005 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:17.005 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:17.006 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 145, 2022-07-12T08:51:17.006 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:17.006 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:17.007 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:17.007 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.008 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.008 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6818", 2022-07-12T08:51:17.009 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.009 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.010 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.010 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.011 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6819", 2022-07-12T08:51:17.011 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.011 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.013 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6820", 2022-07-12T08:51:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.015 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.015 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6821", 2022-07-12T08:51:17.016 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:17.017 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.018 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.018 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6824", 2022-07-12T08:51:17.019 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.019 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.019 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.020 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6825", 2022-07-12T08:51:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.021 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.022 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.022 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.023 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.024 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6822", 2022-07-12T08:51:17.024 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.025 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6823", 2022-07-12T08:51:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22012 2022-07-12T08:51:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.026 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6819/22012", 2022-07-12T08:51:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6821/22012", 2022-07-12T08:51:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6825/22012", 2022-07-12T08:51:17.027 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6823/22012", 2022-07-12T08:51:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:17.028 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:51:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "87944907-c10c-4187-abd4-fd919e7520be", 2022-07-12T08:51:17.029 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-12T08:51:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-12T08:51:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-12T08:51:17.030 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-12T08:51:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-12T08:51:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-12T08:51:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-12T08:51:17.031 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 145, 2022-07-12T08:51:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-12T08:51:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-12T08:51:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-12T08:51:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.033 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.033 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6826", 2022-07-12T08:51:17.033 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.033 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6827", 2022-07-12T08:51:17.034 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.035 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.035 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-12T08:51:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6828", 2022-07-12T08:51:17.037 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.037 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6829", 2022-07-12T08:51:17.038 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.038 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.038 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.039 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.039 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-12T08:51:17.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6832", 2022-07-12T08:51:17.040 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.041 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.041 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6833", 2022-07-12T08:51:17.042 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.042 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.043 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.043 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.043 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-12T08:51:17.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-12T08:51:17.044 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-12T08:51:17.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6830", 2022-07-12T08:51:17.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.046 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-12T08:51:17.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.111:6831", 2022-07-12T08:51:17.047 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22094 2022-07-12T08:51:17.047 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.047 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.111:6827/22094", 2022-07-12T08:51:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.111:6829/22094", 2022-07-12T08:51:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.111:6833/22094", 2022-07-12T08:51:17.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.111:6831/22094", 2022-07-12T08:51:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-12T08:51:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-12T08:51:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-12T08:51:17.049 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-12T08:51:17.050 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.050 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:17.050 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-12T08:51:17.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.051 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-12T08:51:17.051 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.051 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.051 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.052 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.052 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.052 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.979052+0000", 2022-07-12T08:51:17.053 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.054 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.054 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-12T08:51:17.055 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.055 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.055 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.056 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.056 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.056 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:01.989744+0000", 2022-07-12T08:51:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-12T08:51:17.057 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.058 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.058 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.058 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.058 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.059 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.035397+0000", 2022-07-12T08:51:17.059 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-12T08:51:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.060 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.061 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.061 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.161796+0000", 2022-07-12T08:51:17.061 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.062 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.062 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-12T08:51:17.063 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.063 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.063 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.064 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.064 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.064 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.231661+0000", 2022-07-12T08:51:17.064 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.065 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-12T08:51:17.065 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.066 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.066 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.066 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.067 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.067 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.244487+0000", 2022-07-12T08:51:17.068 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.068 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.069 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-12T08:51:17.069 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.070 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.070 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.071 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.071 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.071 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.291696+0000", 2022-07-12T08:51:17.071 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-12T08:51:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-12T08:51:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-12T08:51:17.072 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-12T08:51:17.073 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-12T08:51:17.073 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-12T08:51:17.073 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-12T08:51:17.073 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-12T08:41:02.247865+0000", 2022-07-12T08:51:17.074 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-12T08:51:17.074 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.074 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-12T08:51:17.074 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-12T08:51:17.074 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-12T08:51:17.075 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-12T08:51:17.075 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-12T08:51:17.075 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-12T08:51:17.075 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-12T08:51:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-12T08:51:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-12T08:51:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-12T08:51:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-12T08:51:17.076 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-12T08:51:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-12T08:51:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-12T08:51:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-12T08:51:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-12T08:51:17.077 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-12T08:51:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-12T08:51:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-12T08:51:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-12T08:51:17.078 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-12T08:51:17.079 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T08:51:17.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2170053a60 msgr2=0x7f2170055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2170053a60 0x7f2170055f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7f218400d000 tx=0x7f2184009040).stop 2022-07-12T08:51:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 msgr2=0x7f21900ac550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 0x7f21900ac550 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f2188001040 tx=0x7f2180016000).stop 2022-07-12T08:51:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 shutdown_connections 2022-07-12T08:51:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2170053a60 0x7f2170055f20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2190140e60 0x7f21900adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21900b5920 0x7f21900ac550 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 --2- 172.21.15.61:0/1115844959 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21900b09a0 0x7f21900aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 >> 172.21.15.61:0/1115844959 conn(0x7f2190063380 msgr2=0x7f2190098390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:17.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 shutdown_connections 2022-07-12T08:51:17.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:16.659+0000 7f2194d88700 1 -- 172.21.15.61:0/1115844959 wait complete. 2022-07-12T08:51:17.084 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:17.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.191+0000 7fd023078700 1 Processor -- start 2022-07-12T08:51:17.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.191+0000 7fd023078700 1 -- start start 2022-07-12T08:51:17.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd01c05c050 0x7fd01c0ac710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c0b01f0 0x7fd01c14a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd01c05cb00 0x7fd01c05cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c0b0a40 con 0x7fd01c05c050 2022-07-12T08:51:17.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01c05d430 con 0x7fd01c05cb00 2022-07-12T08:51:17.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c0ad2f0 con 0x7fd01c0b01f0 2022-07-12T08:51:17.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd01c05c050 0x7fd01c0ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd01c05c050 0x7fd01c0ac710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34422/0 (socket says 172.21.15.61:34422) 2022-07-12T08:51:17.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 -- 172.21.15.61:0/991675978 learned_addr learned my addr 172.21.15.61:0/991675978 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:17.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd01bfff700 1 --2- 172.21.15.61:0/991675978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c0b01f0 0x7fd01c14a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd021615700 1 --2- 172.21.15.61:0/991675978 >> v2:172.21.15.111:3300/0 conn(0x7fd01c05cb00 0x7fd01c05cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 -- 172.21.15.61:0/991675978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c0b01f0 msgr2=0x7fd01c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:17.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 --2- 172.21.15.61:0/991675978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c0b01f0 0x7fd01c14a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 -- 172.21.15.61:0/991675978 >> v2:172.21.15.111:3300/0 conn(0x7fd01c05cb00 msgr2=0x7fd01c05cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:17.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 --2- 172.21.15.61:0/991675978 >> v2:172.21.15.111:3300/0 conn(0x7fd01c05cb00 0x7fd01c05cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 -- 172.21.15.61:0/991675978 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c05c430 con 0x7fd01c05c050 2022-07-12T08:51:17.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd020e14700 1 --2- 172.21.15.61:0/991675978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c05c050 0x7fd01c0ac710 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7fd01c05d570 tx=0x7fd010006eb0).ready entity=mon.0 client_cookie=cea636a2ca0498a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd01b7fe700 1 -- 172.21.15.61:0/991675978 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd01000e610 con 0x7fd01c05c050 2022-07-12T08:51:17.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd01b7fe700 1 -- 172.21.15.61:0/991675978 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd010000e90 con 0x7fd01c05c050 2022-07-12T08:51:17.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd01b7fe700 1 -- 172.21.15.61:0/991675978 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd01000e9a0 con 0x7fd01c05c050 2022-07-12T08:51:17.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- 172.21.15.61:0/991675978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c05c050 msgr2=0x7fd01c0ac710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- 172.21.15.61:0/991675978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c05c050 0x7fd01c0ac710 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7fd01c05d570 tx=0x7fd010006eb0).stop 2022-07-12T08:51:17.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- 172.21.15.61:0/991675978 shutdown_connections 2022-07-12T08:51:17.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- 172.21.15.61:0/991675978 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c0b01f0 0x7fd01c14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- 172.21.15.61:0/991675978 >> v2:172.21.15.111:3300/0 conn(0x7fd01c05cb00 0x7fd01c05cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 --2- 172.21.15.61:0/991675978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c05c050 0x7fd01c0ac710 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- 172.21.15.61:0/991675978 >> 172.21.15.61:0/991675978 conn(0x7fd01c062c00 msgr2=0x7fd01c062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:17.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- 172.21.15.61:0/991675978 shutdown_connections 2022-07-12T08:51:17.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.195+0000 7fd023078700 1 -- 172.21.15.61:0/991675978 wait complete. 2022-07-12T08:51:17.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 Processor -- start 2022-07-12T08:51:17.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 -- start start 2022-07-12T08:51:17.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd01c05c050 0x7fd01c144ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c05cb00 0x7fd01c0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 0x7fd01c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c146760 con 0x7fd01c0b01f0 2022-07-12T08:51:17.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c1464c0 con 0x7fd01c05c050 2022-07-12T08:51:17.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01c159b00 con 0x7fd01c05cb00 2022-07-12T08:51:17.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 0x7fd01c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 0x7fd01c159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34430/0 (socket says 172.21.15.61:34430) 2022-07-12T08:51:17.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 -- 172.21.15.61:0/2773024534 learned_addr learned my addr 172.21.15.61:0/2773024534 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:17.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd01bfff700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c05cb00 0x7fd01c0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd020e14700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd01c05c050 0x7fd01c144ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 -- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c05cb00 msgr2=0x7fd01c0b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c05cb00 0x7fd01c0b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 -- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd01c05c050 msgr2=0x7fd01c144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd01c05c050 0x7fd01c144ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 -- 172.21.15.61:0/2773024534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01000b1e0 con 0x7fd01c0b01f0 2022-07-12T08:51:17.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd020e14700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd01c05c050 0x7fd01c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:17.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd021615700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 0x7fd01c159fa0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7fd01400c000 tx=0x7fd014006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd014010780 con 0x7fd01c0b01f0 2022-07-12T08:51:17.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0140108e0 con 0x7fd01c0b01f0 2022-07-12T08:51:17.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01c143870 con 0x7fd01c0b01f0 2022-07-12T08:51:17.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd014010c40 con 0x7fd01c0b01f0 2022-07-12T08:51:17.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.199+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01c0b5920 con 0x7fd01c0b01f0 2022-07-12T08:51:17.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.203+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01c159cc0 con 0x7fd01c0b01f0 2022-07-12T08:51:17.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.203+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd01401c460 con 0x7fd01c0b01f0 2022-07-12T08:51:17.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.203+0000 7fd019ffb700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcffc053ad0 0x7fcffc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.203+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd01406c490 con 0x7fd01c0b01f0 2022-07-12T08:51:17.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.203+0000 7fd020e14700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcffc053ad0 0x7fcffc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.203+0000 7fd020e14700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcffc053ad0 0x7fcffc055f90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7fd010019000 tx=0x7fd010015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.211+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd014043200 con 0x7fd01c0b01f0 2022-07-12T08:51:17.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.375+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd01c057460 con 0x7fd01c0b01f0 2022-07-12T08:51:17.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.375+0000 7fd019ffb700 1 -- 172.21.15.61:0/2773024534 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fd014046820 con 0x7fd01c0b01f0 2022-07-12T08:51:17.381 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:17.382 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1725259132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1725259132},{"type":"v1","addr":"172.21.15.111:6835","nonce":1725259132}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:51:07.632105+0000","modified":"2022-07-12T08:51:08.635747+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8775":{"gid":8775,"name":"a","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/502386216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":502386216},{"type":"v1","addr":"172.21.15.61:6833","nonce":502386216}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-12T08:51:17.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcffc053ad0 msgr2=0x7fcffc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcffc053ad0 0x7fcffc055f90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7fd010019000 tx=0x7fd010015000).stop 2022-07-12T08:51:17.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 msgr2=0x7fd01c159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 0x7fd01c159fa0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 rx=0x7fd01400c000 tx=0x7fd014006df0).stop 2022-07-12T08:51:17.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 shutdown_connections 2022-07-12T08:51:17.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd01c0b01f0 0x7fd01c159fa0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd01c05cb00 0x7fd01c0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fcffc053ad0 0x7fcffc055f90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 --2- 172.21.15.61:0/2773024534 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd01c05c050 0x7fd01c144ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 >> 172.21.15.61:0/2773024534 conn(0x7fd01c062c00 msgr2=0x7fd01c0ab520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 shutdown_connections 2022-07-12T08:51:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.379+0000 7fd023078700 1 -- 172.21.15.61:0/2773024534 wait complete. 2022-07-12T08:51:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-12T08:51:17.394 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-12T08:51:17.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.511+0000 7f3b52bc6700 1 Processor -- start 2022-07-12T08:51:17.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.511+0000 7f3b52bc6700 1 -- start start 2022-07-12T08:51:17.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f3b4c05f140 0x7f3b4c05ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0abc70 con 0x7f3b4c0afa60 2022-07-12T08:51:17.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c05f580 con 0x7f3b4c05f140 2022-07-12T08:51:17.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c05c720 con 0x7f3b4c0ab290 2022-07-12T08:51:17.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b523c5700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3b4c0afa60 0x7f3b4c14a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50888/0 (socket says 172.21.15.61:50888) 2022-07-12T08:51:17.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b523c5700 1 -- 172.21.15.61:0/1507238649 learned_addr learned my addr 172.21.15.61:0/1507238649 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:17.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b513c3700 1 --2- 172.21.15.61:0/1507238649 >> v2:172.21.15.111:3300/0 conn(0x7f3b4c05f140 0x7f3b4c05ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b51bc4700 1 --2- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b50bc2700 1 -- 172.21.15.61:0/1507238649 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164188227 0 0) 0x7f3b4c0abc70 con 0x7f3b4c0afa60 2022-07-12T08:51:17.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b50bc2700 1 -- 172.21.15.61:0/1507238649 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b34003260 con 0x7f3b4c0afa60 2022-07-12T08:51:17.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b513c3700 1 -- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 msgr2=0x7f3b4c0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:17.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b513c3700 1 --2- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b513c3700 1 -- 172.21.15.61:0/1507238649 >> v1:172.21.15.61:6789/0 conn(0x7f3b4c0afa60 legacy=0x7f3b4c14a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b513c3700 1 -- 172.21.15.61:0/1507238649 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c05c9c0 con 0x7f3b4c05f140 2022-07-12T08:51:17.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b513c3700 1 --2- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c05f140 0x7f3b4c05ce60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f3b480032f0 tx=0x7f3b480091d0).ready entity=mon.1 client_cookie=b7a40e3e88b8e320 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b50bc2700 1 -- 172.21.15.61:0/1507238649 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b4800b600 con 0x7f3b4c05f140 2022-07-12T08:51:17.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b50bc2700 1 -- 172.21.15.61:0/1507238649 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4800b760 con 0x7f3b4c05f140 2022-07-12T08:51:17.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b50bc2700 1 -- 172.21.15.61:0/1507238649 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b4800b8d0 con 0x7f3b4c05f140 2022-07-12T08:51:17.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c05f140 msgr2=0x7f3b4c05ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c05f140 0x7f3b4c05ce60 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f3b480032f0 tx=0x7f3b480091d0).stop 2022-07-12T08:51:17.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1507238649 shutdown_connections 2022-07-12T08:51:17.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c05f140 0x7f3b4c05ce60 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1507238649 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1507238649 >> 172.21.15.61:0/1507238649 conn(0x7f3b4c0b1950 msgr2=0x7f3b4c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:17.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1507238649 shutdown_connections 2022-07-12T08:51:17.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.515+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1507238649 wait complete. 2022-07-12T08:51:17.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 Processor -- start 2022-07-12T08:51:17.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 -- start start 2022-07-12T08:51:17.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 0x7f3b4c0647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c064ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c0afa60 0x7f3b4c14a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0af080 con 0x7f3b4c05f140 2022-07-12T08:51:17.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c0ac810 con 0x7f3b4c0afa60 2022-07-12T08:51:17.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b52bc6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c156fa0 con 0x7f3b4c0ab290 2022-07-12T08:51:17.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b51bc4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 0x7f3b4c0647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b51bc4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 0x7f3b4c0647f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34440/0 (socket says 172.21.15.61:34440) 2022-07-12T08:51:17.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.519+0000 7f3b51bc4700 1 -- 172.21.15.61:0/1081425242 learned_addr learned my addr 172.21.15.61:0/1081425242 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:17.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b523c5700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c0afa60 0x7f3b4c14a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b513c3700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c064ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b51bc4700 1 -- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 msgr2=0x7f3b4c064ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b51bc4700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c064ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b51bc4700 1 -- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c0afa60 msgr2=0x7f3b4c14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b51bc4700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c0afa60 0x7f3b4c14a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b51bc4700 1 -- 172.21.15.61:0/1081425242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b48009470 con 0x7f3b4c05f140 2022-07-12T08:51:17.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b51bc4700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 0x7f3b4c0647f0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f3b3c00c000 tx=0x7f3b3c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b3c010780 con 0x7f3b4c05f140 2022-07-12T08:51:17.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c0108e0 con 0x7f3b4c05f140 2022-07-12T08:51:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c157240 con 0x7f3b4c05f140 2022-07-12T08:51:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b3c010c40 con 0x7f3b4c05f140 2022-07-12T08:51:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c0acbb0 con 0x7f3b4c05f140 2022-07-12T08:51:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3b3c01c460 con 0x7f3b4c05f140 2022-07-12T08:51:17.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4c1575c0 con 0x7f3b4c05f140 2022-07-12T08:51:17.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b3affd700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b2c053af0 0x7f3b2c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b3c06c380 con 0x7f3b4c05f140 2022-07-12T08:51:17.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b513c3700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b2c053af0 0x7f3b2c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.523+0000 7f3b513c3700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b2c053af0 0x7f3b2c055fb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f3b48019000 tx=0x7f3b48016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.527+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b3c069e90 con 0x7f3b4c05f140 2022-07-12T08:51:17.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.691+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3b4c057460 con 0x7f3b4c05f140 2022-07-12T08:51:17.715 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-12T08:51:17.711+0000 7fec569a6700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-12T08:51:17.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.723+0000 7f3b3affd700 1 -- 172.21.15.61:0/1081425242 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3b3c043280 con 0x7f3b4c05f140 2022-07-12T08:51:17.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b2c053af0 msgr2=0x7f3b2c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b2c053af0 0x7f3b2c055fb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f3b48019000 tx=0x7f3b48016000).stop 2022-07-12T08:51:17.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 msgr2=0x7f3b4c0647f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 0x7f3b4c0647f0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f3b3c00c000 tx=0x7f3b3c006df0).stop 2022-07-12T08:51:17.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 shutdown_connections 2022-07-12T08:51:17.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f3b2c053af0 0x7f3b2c055fb0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3b4c0afa60 0x7f3b4c14a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f3b4c0ab290 0x7f3b4c064ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 --2- 172.21.15.61:0/1081425242 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b4c05f140 0x7f3b4c0647f0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 >> 172.21.15.61:0/1081425242 conn(0x7f3b4c0b1950 msgr2=0x7f3b4c0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:17.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 shutdown_connections 2022-07-12T08:51:17.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.727+0000 7f3b52bc6700 1 -- 172.21.15.61:0/1081425242 wait complete. 2022-07-12T08:51:17.734 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-12T08:51:17.747 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-12T08:51:17.751 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-12T08:51:17.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 Processor -- start 2022-07-12T08:51:17.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 -- start start 2022-07-12T08:51:17.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f59f40af280 0x7f59f4060680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f59f409bce0 0x7f59f40af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40aecb0 con 0x7f59f40b3f60 2022-07-12T08:51:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f4143b00 con 0x7f59f409bce0 2022-07-12T08:51:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.839+0000 7f59fa705700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f41435c0 con 0x7f59f40af280 2022-07-12T08:51:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f8f02700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f59f409bce0 0x7f59f40af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f9f04700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f59f40b3f60 0x7f59f40b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50914/0 (socket says 172.21.15.61:50914) 2022-07-12T08:51:17.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f9f04700 1 -- 172.21.15.61:0/2208750469 learned_addr learned my addr 172.21.15.61:0/2208750469 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:17.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f9703700 1 --2- 172.21.15.61:0/2208750469 >> v2:172.21.15.111:3300/0 conn(0x7f59f40af280 0x7f59f4060680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f3fff700 1 -- 172.21.15.61:0/2208750469 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396273298 0 0) 0x7f59f40aecb0 con 0x7f59f40b3f60 2022-07-12T08:51:17.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f3fff700 1 -- 172.21.15.61:0/2208750469 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0003260 con 0x7f59f40b3f60 2022-07-12T08:51:17.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f8f02700 1 -- 172.21.15.61:0/2208750469 >> v2:172.21.15.111:3300/0 conn(0x7f59f40af280 msgr2=0x7f59f4060680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:17.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f8f02700 1 --2- 172.21.15.61:0/2208750469 >> v2:172.21.15.111:3300/0 conn(0x7f59f40af280 0x7f59f4060680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f8f02700 1 -- 172.21.15.61:0/2208750469 >> v1:172.21.15.61:6789/0 conn(0x7f59f40b3f60 legacy=0x7f59f40b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f8f02700 1 -- 172.21.15.61:0/2208750469 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4143860 con 0x7f59f409bce0 2022-07-12T08:51:17.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f9703700 1 --2- 172.21.15.61:0/2208750469 >> v2:172.21.15.111:3300/0 conn(0x7f59f40af280 0x7f59f4060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:17.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f8f02700 1 --2- 172.21.15.61:0/2208750469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f40af660 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f59e8002f70 tx=0x7f59e8006df0).ready entity=mon.0 client_cookie=5903e4cb3935a73b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f3fff700 1 -- 172.21.15.61:0/2208750469 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59e800d4f0 con 0x7f59f409bce0 2022-07-12T08:51:17.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f3fff700 1 -- 172.21.15.61:0/2208750469 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e8000e90 con 0x7f59f409bce0 2022-07-12T08:51:17.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59f3fff700 1 -- 172.21.15.61:0/2208750469 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59e800d840 con 0x7f59f409bce0 2022-07-12T08:51:17.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 -- 172.21.15.61:0/2208750469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 msgr2=0x7f59f40af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 --2- 172.21.15.61:0/2208750469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f40af660 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f59e8002f70 tx=0x7f59e8006df0).stop 2022-07-12T08:51:17.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 -- 172.21.15.61:0/2208750469 shutdown_connections 2022-07-12T08:51:17.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 --2- 172.21.15.61:0/2208750469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f40af660 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 --2- 172.21.15.61:0/2208750469 >> v2:172.21.15.111:3300/0 conn(0x7f59f40af280 0x7f59f4060680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 -- 172.21.15.61:0/2208750469 >> 172.21.15.61:0/2208750469 conn(0x7f59f40b92c0 msgr2=0x7f59f40b96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:17.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 -- 172.21.15.61:0/2208750469 shutdown_connections 2022-07-12T08:51:17.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 -- 172.21.15.61:0/2208750469 wait complete. 2022-07-12T08:51:17.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 Processor -- start 2022-07-12T08:51:17.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.843+0000 7f59fa705700 1 -- start start 2022-07-12T08:51:17.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f41462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f40af280 0x7f59f4141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f59f40b3f60 0x7f59f4155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f41457e0 con 0x7f59f409bce0 2022-07-12T08:51:17.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4142b70 con 0x7f59f40af280 2022-07-12T08:51:17.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f41468d0 con 0x7f59f40b3f60 2022-07-12T08:51:17.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f41462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f41462a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34468/0 (socket says 172.21.15.61:34468) 2022-07-12T08:51:17.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 -- 172.21.15.61:0/3243472950 learned_addr learned my addr 172.21.15.61:0/3243472950 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:17.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9f04700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f59f40b3f60 0x7f59f4155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f8f02700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f40af280 0x7f59f4141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 -- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f59f40b3f60 msgr2=0x7f59f4155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f59f40b3f60 0x7f59f4155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 -- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f40af280 msgr2=0x7f59f4141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:17.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f40af280 0x7f59f4141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:17.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 -- 172.21.15.61:0/3243472950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec009420 con 0x7f59f409bce0 2022-07-12T08:51:17.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f9703700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f41462a0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f59f409e710 tx=0x7f59ec0091a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59ec00c9d0 con 0x7f59f409bce0 2022-07-12T08:51:17.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec00cb30 con 0x7f59f409bce0 2022-07-12T08:51:17.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e800b180 con 0x7f59f409bce0 2022-07-12T08:51:17.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59ec00ce90 con 0x7f59f409bce0 2022-07-12T08:51:17.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f4146e10 con 0x7f59f409bce0 2022-07-12T08:51:17.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f409f210 con 0x7f59f409bce0 2022-07-12T08:51:17.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f59ec01b400 con 0x7f59f409bce0 2022-07-12T08:51:17.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f27fc700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f59d40539d0 0x7f59d4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:17.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f59ec06c9a0 con 0x7f59f409bce0 2022-07-12T08:51:17.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.847+0000 7f59f8f02700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f59d40539d0 0x7f59d4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:17.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.851+0000 7f59f8f02700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f59d40539d0 0x7f59d4055e90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f59e8018000 tx=0x7f59e8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:17.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:17.855+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59f409f210 con 0x7f59f409bce0 2022-07-12T08:51:18.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.015+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59f409f210 con 0x7f59f409bce0 2022-07-12T08:51:18.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.015+0000 7f59f27fc700 1 -- 172.21.15.61:0/3243472950 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1785 (secure 0 0 0) 0x7f59f409f210 con 0x7f59f409bce0 2022-07-12T08:51:18.021 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T08:51:18.022 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":84,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.111:6835/1725259132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6834","nonce":1725259132},{"type":"v1","addr":"172.21.15.111:6835","nonce":1725259132}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[{"mdsmap":{"epoch":84,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-12T08:51:07.632105+0000","modified":"2022-07-12T08:51:17.715302+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":149,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-12T08:51:18.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f59d40539d0 msgr2=0x7f59d4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f59d40539d0 0x7f59d4055e90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f59e8018000 tx=0x7f59e8014000).stop 2022-07-12T08:51:18.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 msgr2=0x7f59f41462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f41462a0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f59f409e710 tx=0x7f59ec0091a0).stop 2022-07-12T08:51:18.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 shutdown_connections 2022-07-12T08:51:18.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f59f40b3f60 0x7f59f4155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f59d40539d0 0x7f59d4055e90 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59f40af280 0x7f59f4141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 --2- 172.21.15.61:0/3243472950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59f409bce0 0x7f59f41462a0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 >> 172.21.15.61:0/3243472950 conn(0x7f59f40b92c0 msgr2=0x7f59f40b7900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:18.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 shutdown_connections 2022-07-12T08:51:18.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.019+0000 7f59fa705700 1 -- 172.21.15.61:0/3243472950 wait complete. 2022-07-12T08:51:18.029 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 84 2022-07-12T08:51:18.033 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-12T08:51:18.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35b59e700 1 Processor -- start 2022-07-12T08:51:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35b59e700 1 -- start start 2022-07-12T08:51:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 0x7fd35c13f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c0ac030 con 0x7fd35c13fee0 2022-07-12T08:51:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35b59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35c0afb60 con 0x7fd35c097a20 2022-07-12T08:51:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0afe00 con 0x7fd35c0abc50 2022-07-12T08:51:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd359d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd35c13fee0 0x7fd35c0aa3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:50924/0 (socket says 172.21.15.61:50924) 2022-07-12T08:51:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd359d9b700 1 -- 172.21.15.61:0/2165686127 learned_addr learned my addr 172.21.15.61:0/2165686127 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T08:51:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35a59c700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 0x7fd35c13f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292727632 0 0) 0x7fd35c0ac030 con 0x7fd35c13fee0 2022-07-12T08:51:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340004370 con 0x7fd35c13fee0 2022-07-12T08:51:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1327556640 0 0) 0x7fd340004370 con 0x7fd35c13fee0 2022-07-12T08:51:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 msgr2=0x7fd35c13f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 0x7fd35c13f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 >> v1:172.21.15.111:6789/0 conn(0x7fd35c097a20 legacy=0x7fd35c13e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c0b0a00 con 0x7fd35c13fee0 2022-07-12T08:51:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 132058336 0 0) 0x7fd34c001660 con 0x7fd35c13fee0 2022-07-12T08:51:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 >> v1:172.21.15.61:6789/0 conn(0x7fd35c13fee0 legacy=0x7fd35c0aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 0x7fd340004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd340004d80 0x7fd340005610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd340005b00 0x7fd340005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0afe00 con 0x7fd340005b00 2022-07-12T08:51:18.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34000c860 con 0x7fd340004d80 2022-07-12T08:51:18.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.147+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34000ca20 con 0x7fd35c0abc50 2022-07-12T08:51:18.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd340005b00 0x7fd340005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35a59c700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 0x7fd340004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd359d9b700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd340004d80 0x7fd340005610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 -- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 msgr2=0x7fd340004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0abc50 0x7fd340004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 -- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd340004d80 msgr2=0x7fd340005610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd340004d80 0x7fd340005610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 -- 172.21.15.61:0/2165686127 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340004370 con 0x7fd340005b00 2022-07-12T08:51:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd340005b00 0x7fd340005f80 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7fd3540013a0 tx=0x7fd354007d60).ready entity=mon.0 client_cookie=44544e90e493f3f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3540016c0 con 0x7fd340005b00 2022-07-12T08:51:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd354006890 con 0x7fd340005b00 2022-07-12T08:51:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35959a700 1 -- 172.21.15.61:0/2165686127 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd354006bf0 con 0x7fd340005b00 2022-07-12T08:51:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 -- 172.21.15.61:0/2165686127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd340005b00 msgr2=0x7fd340005f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 --2- 172.21.15.61:0/2165686127 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd340005b00 0x7fd340005f80 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7fd3540013a0 tx=0x7fd354007d60).stop 2022-07-12T08:51:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35ad9d700 1 -- 172.21.15.61:0/2165686127 reap_dead start 2022-07-12T08:51:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 -- 172.21.15.61:0/2165686127 shutdown_connections 2022-07-12T08:51:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 -- 172.21.15.61:0/2165686127 >> 172.21.15.61:0/2165686127 conn(0x7fd35c0b1a50 msgr2=0x7fd35c0b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 -- 172.21.15.61:0/2165686127 shutdown_connections 2022-07-12T08:51:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 -- 172.21.15.61:0/2165686127 wait complete. 2022-07-12T08:51:18.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 Processor -- start 2022-07-12T08:51:18.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.151+0000 7fd35b59e700 1 -- start start 2022-07-12T08:51:18.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd35c05c950 0x7fd35c1445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0ac980 0x7fd35c144aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 0x7fd35c099650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0ae6d0 con 0x7fd35c09ae30 2022-07-12T08:51:18.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c09aa00 con 0x7fd35c05c950 2022-07-12T08:51:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c0990f0 con 0x7fd35c0ac980 2022-07-12T08:51:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 0x7fd35c099650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 0x7fd35c099650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34490/0 (socket says 172.21.15.61:34490) 2022-07-12T08:51:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 -- 172.21.15.61:0/925713246 learned_addr learned my addr 172.21.15.61:0/925713246 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd359d9b700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0ac980 0x7fd35c144aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35a59c700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd35c05c950 0x7fd35c1445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 -- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0ac980 msgr2=0x7fd35c144aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0ac980 0x7fd35c144aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 -- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd35c05c950 msgr2=0x7fd35c1445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd35c05c950 0x7fd35c1445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 -- 172.21.15.61:0/925713246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c099390 con 0x7fd35c09ae30 2022-07-12T08:51:18.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35a59c700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd35c05c950 0x7fd35c1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T08:51:18.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35ad9d700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 0x7fd35c099650 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7fd35c0984e0 tx=0x7fd354015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:18.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3540016c0 con 0x7fd35c09ae30 2022-07-12T08:51:18.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd354014dc0 con 0x7fd35c09ae30 2022-07-12T08:51:18.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35c096fe0 con 0x7fd35c09ae30 2022-07-12T08:51:18.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd35401f5f0 con 0x7fd35c09ae30 2022-07-12T08:51:18.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35c145130 con 0x7fd35c09ae30 2022-07-12T08:51:18.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.155+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35c09ab40 con 0x7fd35c09ae30 2022-07-12T08:51:18.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.159+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd35401d070 con 0x7fd35c09ae30 2022-07-12T08:51:18.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.159+0000 7fd34b7fe700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd338053af0 0x7fd338055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.159+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd35406c6a0 con 0x7fd35c09ae30 2022-07-12T08:51:18.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.159+0000 7fd35a59c700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd338053af0 0x7fd338055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.159+0000 7fd35a59c700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd338053af0 0x7fd338055fb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fd35000b000 tx=0x7fd350008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:18.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.167+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3540435b0 con 0x7fd35c09ae30 2022-07-12T08:51:18.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.331+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd35c145130 con 0x7fd35c09ae30 2022-07-12T08:51:18.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.731+0000 7fd34b7fe700 1 -- 172.21.15.61:0/925713246 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd35c145130 con 0x7fd35c09ae30 2022-07-12T08:51:18.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd338053af0 msgr2=0x7fd338055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd338053af0 0x7fd338055fb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7fd35000b000 tx=0x7fd350008040).stop 2022-07-12T08:51:18.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 msgr2=0x7fd35c099650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 0x7fd35c099650 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7fd35c0984e0 tx=0x7fd354015000).stop 2022-07-12T08:51:18.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 shutdown_connections 2022-07-12T08:51:18.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd35c09ae30 0x7fd35c099650 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd35c0ac980 0x7fd35c144aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd338053af0 0x7fd338055fb0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 --2- 172.21.15.61:0/925713246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd35c05c950 0x7fd35c1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 >> 172.21.15.61:0/925713246 conn(0x7fd35c0b1a50 msgr2=0x7fd35c0607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:18.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 shutdown_connections 2022-07-12T08:51:18.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.735+0000 7fd35b59e700 1 -- 172.21.15.61:0/925713246 wait complete. 2022-07-12T08:51:18.756 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-12T08:51:18.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 Processor -- start 2022-07-12T08:51:18.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 -- start start 2022-07-12T08:51:18.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62640b1210 0x7f626413f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f626405f340 0x7f626413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626405d220 con 0x7f626405c740 2022-07-12T08:51:18.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6264098780 con 0x7f626405f340 2022-07-12T08:51:18.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6269de6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62640aaa00 con 0x7f62640b1210 2022-07-12T08:51:18.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f62695e5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f626405f340 0x7f626413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f62695e5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f626405f340 0x7f626413f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34498/0 (socket says 172.21.15.61:34498) 2022-07-12T08:51:18.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f62695e5700 1 -- 172.21.15.61:0/2080066515 learned_addr learned my addr 172.21.15.61:0/2080066515 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:18.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.875+0000 7f6263fff700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62640b1210 0x7f626413f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62637fe700 1 -- 172.21.15.61:0/2080066515 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783530202 0 0) 0x7f626405d220 con 0x7f626405c740 2022-07-12T08:51:18.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62637fe700 1 -- 172.21.15.61:0/2080066515 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6244003260 con 0x7f626405c740 2022-07-12T08:51:18.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62695e5700 1 -- 172.21.15.61:0/2080066515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62640b1210 msgr2=0x7f626413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:18.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62695e5700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62640b1210 0x7f626413f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62695e5700 1 -- 172.21.15.61:0/2080066515 >> v1:172.21.15.111:6789/0 conn(0x7f626405c740 legacy=0x7f6264098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62695e5700 1 -- 172.21.15.61:0/2080066515 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626409aca0 con 0x7f626405f340 2022-07-12T08:51:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6263fff700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62640b1210 0x7f626413f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T08:51:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62695e5700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405f340 0x7f626413f5e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f625c0032f0 tx=0x7f625c009290).ready entity=mon.0 client_cookie=922a346dd90d28c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62637fe700 1 -- 172.21.15.61:0/2080066515 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f625c00b6c0 con 0x7f626405f340 2022-07-12T08:51:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62637fe700 1 -- 172.21.15.61:0/2080066515 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c00b820 con 0x7f626405f340 2022-07-12T08:51:18.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f62637fe700 1 -- 172.21.15.61:0/2080066515 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f625c00b990 con 0x7f626405f340 2022-07-12T08:51:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 -- 172.21.15.61:0/2080066515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405f340 msgr2=0x7f626413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405f340 0x7f626413f5e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 rx=0x7f625c0032f0 tx=0x7f625c009290).stop 2022-07-12T08:51:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 -- 172.21.15.61:0/2080066515 shutdown_connections 2022-07-12T08:51:18.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405f340 0x7f626413f5e0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 --2- 172.21.15.61:0/2080066515 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f62640b1210 0x7f626413f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 -- 172.21.15.61:0/2080066515 >> 172.21.15.61:0/2080066515 conn(0x7f62640b1950 msgr2=0x7f62640b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:18.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 -- 172.21.15.61:0/2080066515 shutdown_connections 2022-07-12T08:51:18.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 -- 172.21.15.61:0/2080066515 wait complete. 2022-07-12T08:51:18.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 Processor -- start 2022-07-12T08:51:18.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.879+0000 7f6269de6700 1 -- start start 2022-07-12T08:51:18.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 0x7f6264064400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f626405f340 0x7f62640aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62640b1210 0x7f6264152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6264067650 con 0x7f626405c740 2022-07-12T08:51:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626413e4b0 con 0x7f62640b1210 2022-07-12T08:51:18.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626413e750 con 0x7f626405f340 2022-07-12T08:51:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 0x7f6264064400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 0x7f6264064400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34500/0 (socket says 172.21.15.61:34500) 2022-07-12T08:51:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 -- 172.21.15.61:0/1498131180 learned_addr learned my addr 172.21.15.61:0/1498131180 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:18.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f62695e5700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62640b1210 0x7f6264152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6263fff700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f626405f340 0x7f62640aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 -- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f626405f340 msgr2=0x7f62640aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f626405f340 0x7f62640aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 -- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62640b1210 msgr2=0x7f6264152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:18.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62640b1210 0x7f6264152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:18.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 -- 172.21.15.61:0/1498131180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625400b180 con 0x7f626405c740 2022-07-12T08:51:18.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6268de4700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 0x7f6264064400 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f624c001110 tx=0x7f624c007b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:18.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f624c012400 con 0x7f626405c740 2022-07-12T08:51:18.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c012560 con 0x7f626405c740 2022-07-12T08:51:18.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625c009530 con 0x7f626405c740 2022-07-12T08:51:18.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f624c0128c0 con 0x7f626405c740 2022-07-12T08:51:18.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.883+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626413ee90 con 0x7f626405c740 2022-07-12T08:51:18.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.887+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626413f080 con 0x7f626405c740 2022-07-12T08:51:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.887+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f624c010070 con 0x7f626405c740 2022-07-12T08:51:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.887+0000 7f6261ffb700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62480539d0 0x7f6248055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.887+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f624c06b000 con 0x7f626405c740 2022-07-12T08:51:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.887+0000 7f6263fff700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62480539d0 0x7f6248055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:18.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.887+0000 7f6263fff700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62480539d0 0x7f6248055e90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f625400b000 tx=0x7f6254006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:18.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:18.891+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f624c041f10 con 0x7f626405c740 2022-07-12T08:51:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.019+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6264057460 con 0x7f626405c740 2022-07-12T08:51:19.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.735+0000 7f6261ffb700 1 -- 172.21.15.61:0/1498131180 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v150) v1 ==== 185+0+0 (secure 0 0 0) 0x7f624c045530 con 0x7f626405c740 2022-07-12T08:51:19.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62480539d0 msgr2=0x7f6248055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:19.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62480539d0 0x7f6248055e90 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7f625400b000 tx=0x7f6254006df0).stop 2022-07-12T08:51:19.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 msgr2=0x7f6264064400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:19.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 0x7f6264064400 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f624c001110 tx=0x7f624c007b20).stop 2022-07-12T08:51:19.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 shutdown_connections 2022-07-12T08:51:19.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f62480539d0 0x7f6248055e90 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62640b1210 0x7f6264152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f626405f340 0x7f62640aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 --2- 172.21.15.61:0/1498131180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f626405c740 0x7f6264064400 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.739+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 >> 172.21.15.61:0/1498131180 conn(0x7f62640b1950 msgr2=0x7f62640b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:19.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.743+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 shutdown_connections 2022-07-12T08:51:19.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.743+0000 7f6269de6700 1 -- 172.21.15.61:0/1498131180 wait complete. 2022-07-12T08:51:19.772 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-12T08:51:19.775 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-12T08:51:19.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.899+0000 7f105e7e6700 1 Processor -- start 2022-07-12T08:51:19.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.899+0000 7f105e7e6700 1 -- start start 2022-07-12T08:51:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f10581440a0 0x7f105805bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1058144480 0x7f10581413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580b3010 con 0x7f10581418d0 2022-07-12T08:51:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10580624b0 con 0x7f1058144480 2022-07-12T08:51:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1058144da0 con 0x7f10581440a0 2022-07-12T08:51:19.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1058144480 0x7f10581413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:19.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1058144480 0x7f10581413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34508/0 (socket says 172.21.15.61:34508) 2022-07-12T08:51:19.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 -- 172.21.15.61:0/1135747730 learned_addr learned my addr 172.21.15.61:0/1135747730 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:19.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105d7e4700 1 --2- 172.21.15.61:0/1135747730 >> v2:172.21.15.111:3300/0 conn(0x7f10581440a0 0x7f105805bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 -- 172.21.15.61:0/1135747730 >> v2:172.21.15.111:3300/0 conn(0x7f10581440a0 msgr2=0x7f105805bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T08:51:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 --2- 172.21.15.61:0/1135747730 >> v2:172.21.15.111:3300/0 conn(0x7f10581440a0 0x7f105805bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 -- 172.21.15.61:0/1135747730 >> v1:172.21.15.111:6789/0 conn(0x7f10581418d0 legacy=0x7f105813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:19.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 -- 172.21.15.61:0/1135747730 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580b2d70 con 0x7f1058144480 2022-07-12T08:51:19.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105d7e4700 1 --2- 172.21.15.61:0/1135747730 >> v2:172.21.15.111:3300/0 conn(0x7f10581440a0 0x7f105805bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-12T08:51:19.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105cfe3700 1 --2- 172.21.15.61:0/1135747730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1058144480 0x7f10581413e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f1048002f70 tx=0x7f1048006eb0).ready entity=mon.0 client_cookie=e46721624be888bf server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:19.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f1047fff700 1 -- 172.21.15.61:0/1135747730 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f104800d610 con 0x7f1058144480 2022-07-12T08:51:19.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f1047fff700 1 -- 172.21.15.61:0/1135747730 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1048000e90 con 0x7f1058144480 2022-07-12T08:51:19.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f1047fff700 1 -- 172.21.15.61:0/1135747730 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f104800d9a0 con 0x7f1058144480 2022-07-12T08:51:19.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- 172.21.15.61:0/1135747730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1058144480 msgr2=0x7f10581413e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:19.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1135747730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1058144480 0x7f10581413e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f1048002f70 tx=0x7f1048006eb0).stop 2022-07-12T08:51:19.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- 172.21.15.61:0/1135747730 shutdown_connections 2022-07-12T08:51:19.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1135747730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1058144480 0x7f10581413e0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1135747730 >> v2:172.21.15.111:3300/0 conn(0x7f10581440a0 0x7f105805bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- 172.21.15.61:0/1135747730 >> 172.21.15.61:0/1135747730 conn(0x7f10580b70b0 msgr2=0x7f10580b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:19.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- 172.21.15.61:0/1135747730 shutdown_connections 2022-07-12T08:51:19.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.903+0000 7f105e7e6700 1 -- 172.21.15.61:0/1135747730 wait complete. 2022-07-12T08:51:19.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 Processor -- start 2022-07-12T08:51:19.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 -- start start 2022-07-12T08:51:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 0x7f105809b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10581440a0 0x7f1058144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1058144480 0x7f1058145470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580ae4f0 con 0x7f10581418d0 2022-07-12T08:51:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580ae720 con 0x7f1058144480 2022-07-12T08:51:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105e7e6700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580ae9c0 con 0x7f10581440a0 2022-07-12T08:51:19.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105d7e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 0x7f105809b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:19.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105d7e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 0x7f105809b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34514/0 (socket says 172.21.15.61:34514) 2022-07-12T08:51:19.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.907+0000 7f105d7e4700 1 -- 172.21.15.61:0/1938907984 learned_addr learned my addr 172.21.15.61:0/1938907984 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T08:51:19.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105cfe3700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10581440a0 0x7f1058144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:19.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105dfe5700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1058144480 0x7f1058145470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:19.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105d7e4700 1 -- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10581440a0 msgr2=0x7f1058144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:19.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105d7e4700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10581440a0 0x7f1058144f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105d7e4700 1 -- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1058144480 msgr2=0x7f1058145470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:19.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105d7e4700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1058144480 0x7f1058145470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:19.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105d7e4700 1 -- 172.21.15.61:0/1938907984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104800b1e0 con 0x7f10581418d0 2022-07-12T08:51:19.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105d7e4700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 0x7f105809b3f0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f1054007000 tx=0x7f1054008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:19.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1054010580 con 0x7f10581418d0 2022-07-12T08:51:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10540106e0 con 0x7f10581418d0 2022-07-12T08:51:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10581465a0 con 0x7f10581418d0 2022-07-12T08:51:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1054010a40 con 0x7f10581418d0 2022-07-12T08:51:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10580af080 con 0x7f10581418d0 2022-07-12T08:51:19.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058146bb0 con 0x7f10581418d0 2022-07-12T08:51:19.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.911+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1054010ba0 con 0x7f10581418d0 2022-07-12T08:51:19.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.915+0000 7f10467fc700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1038053920 0x7f1038055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T08:51:19.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.915+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f105406bdf0 con 0x7f10581418d0 2022-07-12T08:51:19.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.915+0000 7f105cfe3700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1038053920 0x7f1038055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T08:51:19.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.915+0000 7f105cfe3700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1038053920 0x7f1038055de0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f104801c000 tx=0x7f1048015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T08:51:19.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:19.919+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1054042e50 con 0x7f10581418d0 2022-07-12T08:51:20.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.043+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1058057460 con 0x7f10581418d0 2022-07-12T08:51:20.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.751+0000 7f10467fc700 1 -- 172.21.15.61:0/1938907984 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v151) v1 ==== 173+0+0 (secure 0 0 0) 0x7f1054046470 con 0x7f10581418d0 2022-07-12T08:51:20.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1038053920 msgr2=0x7f1038055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:20.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1038053920 0x7f1038055de0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f104801c000 tx=0x7f1048015000).stop 2022-07-12T08:51:20.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 msgr2=0x7f105809b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T08:51:20.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 0x7f105809b3f0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f1054007000 tx=0x7f1054008040).stop 2022-07-12T08:51:20.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 shutdown_connections 2022-07-12T08:51:20.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1038053920 0x7f1038055de0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:20.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1058144480 0x7f1058145470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:20.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f10581440a0 0x7f1058144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:20.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 --2- 172.21.15.61:0/1938907984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10581418d0 0x7f105809b3f0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T08:51:20.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 >> 172.21.15.61:0/1938907984 conn(0x7f10580b70b0 msgr2=0x7f10580b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T08:51:20.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 shutdown_connections 2022-07-12T08:51:20.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T08:51:20.755+0000 7f105e7e6700 1 -- 172.21.15.61:0/1938907984 wait complete. 2022-07-12T08:51:20.766 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-12T08:51:20.773 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-07-12T08:51:20.773 DEBUG:teuthology.orchestra.run.smithi121:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-12T08:51:33.829 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:33.850 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:33.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:33.865 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:33.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:51:50.894 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T08:51:50.896 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T08:51:50.900 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T08:52:04.053 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:04.070 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:04.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:04.108 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:04.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:20.809 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T08:52:20.865 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-12T08:52:34.182 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:34.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:34.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:34.259 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:52:34.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:04.328 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:04.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:04.343 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:04.357 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:04.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:34.443 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:34.463 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:34.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:34.490 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:53:34.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:04.638 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:04.648 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:04.658 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:04.680 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:04.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:34.781 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:34.790 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:34.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:34.802 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:54:34.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:04.883 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:04.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:04.893 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:04.902 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:04.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:34.998 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:35.008 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:35.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:35.027 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:55:35.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:05.117 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:05.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:05.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:05.246 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:05.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:35.365 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:35.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:35.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:35.378 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:56:35.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:05.473 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:05.478 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:05.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:05.488 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:05.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:35.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:35.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:35.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:35.584 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:57:35.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:05.676 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:05.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:05.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:05.706 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:05.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:35.791 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:35.808 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:35.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:35.822 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:58:35.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:05.896 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:05.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:05.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:05.908 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:05.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:35.982 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:35.992 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:35.999 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:36.011 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T08:59:36.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:06.069 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:06.072 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:06.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:06.080 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:06.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:36.165 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:36.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:36.178 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:36.185 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:00:36.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:06.272 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:06.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:06.283 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:06.295 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:06.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:36.356 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:36.753 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:36.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:36.776 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:01:36.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:06.858 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:06.871 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:06.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:06.888 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:06.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:36.950 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:36.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:36.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:36.961 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:02:36.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:07.061 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:07.068 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:07.087 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:07.101 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:07.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:37.179 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:37.189 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:37.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:37.197 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:03:37.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:07.303 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:07.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:07.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:07.314 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:07.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:37.376 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:37.381 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:37.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:37.389 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:04:37.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:07.523 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:07.546 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:07.566 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:07.574 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:07.579 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:37.710 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:37.729 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:37.731 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:37.746 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:05:37.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:07.924 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:07.931 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:07.936 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:07.940 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:07.954 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:38.082 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:38.090 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:38.094 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:38.099 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:38.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:06:50.932 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T09:06:50.933 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-12T09:06:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:06:50.937 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-12T09:06:50.938 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T09:06:50.967 DEBUG:teuthology.orchestra.run.smithi111:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T09:06:50.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T09:06:51.015 DEBUG:teuthology.orchestra.run.smithi130:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T09:06:51.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-12T09:06:51.091 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-12T09:06:51.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T09:06:51.093 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T09:06:51.104 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T09:07:08.284 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:08.288 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:08.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:08.294 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:08.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:38.414 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:38.439 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:38.443 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:38.446 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:07:38.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:08.565 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:08.569 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:08.573 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:08.577 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:08.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:38.683 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:38.686 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:38.691 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:38.700 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:08:38.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:08.820 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:08.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:08.831 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:08.834 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:08.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:38.902 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:38.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:38.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:38.917 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:09:38.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:09.050 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:09.054 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:09.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:09.060 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:09.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:39.197 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:39.206 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:39.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:39.212 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:10:39.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:09.368 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:09.379 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:09.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:09.385 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:09.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:39.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:39.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:39.529 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:39.532 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:11:39.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:09.655 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:09.660 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:09.672 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:09.676 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:09.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:39.802 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:39.806 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:39.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:39.812 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:12:39.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:09.939 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:09.943 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:09.946 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:09.948 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:09.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:40.077 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:40.082 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:40.086 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:40.089 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:13:40.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:10.208 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:10.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:10.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:10.222 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:10.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:40.378 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:40.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:40.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:40.407 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:14:40.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:10.526 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:10.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:10.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:10.551 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:10.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:40.924 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:40.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:40.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:40.934 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:15:40.938 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:11.110 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:11.119 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:11.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:11.129 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:11.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:41.254 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:41.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:41.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:41.266 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:16:41.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:11.409 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:11.413 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:11.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:11.435 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:11.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:41.570 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:41.575 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:41.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:41.584 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:17:41.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:11.717 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:11.722 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:11.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:11.728 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:11.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:41.881 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:41.885 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:41.889 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:41.894 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:18:41.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:12.180 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:12.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:12.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:12.197 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:12.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:42.326 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:42.330 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:42.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:42.338 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:19:42.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:12.488 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:12.492 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:12.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:12.507 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:12.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:42.643 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:42.648 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:42.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:42.657 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:20:42.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:12.818 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:12.822 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:12.824 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:12.827 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:12.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:42.959 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:42.975 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:42.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:42.982 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:42.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:21:51.110 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T09:21:51.112 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-12T09:21:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:21:51.118 INFO:tasks.cephfs_test_runner:test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) ... ERROR 2022-07-12T09:21:51.119 INFO:tasks.cephfs_test_runner:ERROR 2022-07-12T09:21:51.120 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.120 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-12T09:21:51.121 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T09:21:51.121 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-12T09:21:51.121 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.122 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-12T09:21:51.122 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-12T09:21:51.122 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-12T09:21:51.123 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-12T09:21:51.123 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-12T09:21:51.123 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.123 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-12T09:21:51.124 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.124 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.124 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-12T09:21:51.125 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-12T09:21:51.125 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-12T09:21:51.125 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-12T09:21:51.126 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-12T09:21:51.126 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-12T09:21:51.126 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait 2022-07-12T09:21:51.127 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-12T09:21:51.127 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-12T09:21:51.128 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-12T09:21:51.128 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-12T09:21:51.128 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-12T09:21:51.129 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-12T09:21:51.129 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-12T09:21:51.129 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-12T09:21:51.129 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-12T09:21:51.130 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-12T09:21:51.130 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-12T09:21:51.130 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-12T09:21:51.131 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-12T09:21:51.131 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-12T09:21:51.131 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-12T09:21:51.132 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-12T09:21:51.132 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-12T09:21:51.132 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-12T09:21:51.133 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-12T09:21:51.133 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.133 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-12T09:21:51.134 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T09:21:51.134 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-12T09:21:51.134 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.135 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-12T09:21:51.135 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-12T09:21:51.135 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-12T09:21:51.136 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-12T09:21:51.136 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-12T09:21:51.136 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.137 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-12T09:21:51.137 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.137 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.137 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-12T09:21:51.138 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-12T09:21:51.138 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-12T09:21:51.138 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-12T09:21:51.138 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-12T09:21:51.139 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-12T09:21:51.139 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-12T09:21:51.139 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-12T09:21:51.140 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait 2022-07-12T09:21:51.140 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-12T09:21:51.140 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-12T09:21:51.140 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-12T09:21:51.141 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-12T09:21:51.141 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-12T09:21:51.141 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-12T09:21:51.142 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-12T09:21:51.142 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-12T09:21:51.142 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-12T09:21:51.143 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-12T09:21:51.143 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-12T09:21:51.143 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-12T09:21:51.144 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-12T09:21:51.144 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-12T09:21:51.144 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-12T09:21:51.144 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-12T09:21:51.145 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-12T09:21:51.145 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-12T09:21:51.145 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-12T09:21:51.146 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.146 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-12T09:21:51.146 INFO:tasks.cephfs_test_runner:Ran 12 tests in 2407.860s 2022-07-12T09:21:51.147 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.147 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2022-07-12T09:21:51.147 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.148 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-12T09:21:51.148 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T09:21:51.148 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-12T09:21:51.148 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.149 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-12T09:21:51.149 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-12T09:21:51.150 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-12T09:21:51.150 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-12T09:21:51.150 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-12T09:21:51.151 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.151 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-12T09:21:51.151 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.152 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.152 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-12T09:21:51.153 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-12T09:21:51.153 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-12T09:21:51.153 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-12T09:21:51.153 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-12T09:21:51.154 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-12T09:21:51.154 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait 2022-07-12T09:21:51.154 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-12T09:21:51.155 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-12T09:21:51.155 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-12T09:21:51.155 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-12T09:21:51.156 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-12T09:21:51.156 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-12T09:21:51.156 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-12T09:21:51.156 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-12T09:21:51.157 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-12T09:21:51.157 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-12T09:21:51.157 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-12T09:21:51.158 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-12T09:21:51.158 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-12T09:21:51.158 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-12T09:21:51.159 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-12T09:21:51.159 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-12T09:21:51.159 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-12T09:21:51.160 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-12T09:21:51.160 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-12T09:21:51.160 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.161 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-12T09:21:51.161 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T09:21:51.161 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-12T09:21:51.162 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.162 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-12T09:21:51.162 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-12T09:21:51.162 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-12T09:21:51.163 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-12T09:21:51.163 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-12T09:21:51.163 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.163 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-12T09:21:51.164 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.164 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-12T09:21:51.164 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-12T09:21:51.165 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-12T09:21:51.165 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-12T09:21:51.165 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-12T09:21:51.165 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-12T09:21:51.166 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-12T09:21:51.166 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-12T09:21:51.166 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-12T09:21:51.167 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait 2022-07-12T09:21:51.167 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-12T09:21:51.167 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-12T09:21:51.168 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-12T09:21:51.168 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-12T09:21:51.168 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-12T09:21:51.169 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-12T09:21:51.169 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-12T09:21:51.169 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-12T09:21:51.169 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-12T09:21:51.170 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-12T09:21:51.170 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-12T09:21:51.170 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-12T09:21:51.171 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-12T09:21:51.171 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-12T09:21:51.171 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-12T09:21:51.171 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-12T09:21:51.172 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-12T09:21:51.172 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-12T09:21:51.172 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-12T09:21:51.172 INFO:tasks.cephfs_test_runner: 2022-07-12T09:21:51.173 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T09:21:51.321 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dc843196a3884eee9392d504b7b56b09 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-12T09:21:51.324 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2022-07-12T09:21:51.333 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2022-07-12T09:21:51.344 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2022-07-12T09:21:51.345 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T09:21:51.346 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T09:21:51.347 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T09:22:13.105 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:13.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:13.113 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:13.116 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:13.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:43.241 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:43.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:43.248 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:43.251 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:22:43.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:13.389 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:13.394 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:13.397 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:13.400 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:13.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:43.544 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:43.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:43.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:43.553 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:23:43.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:13.688 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:13.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:13.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:13.699 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:13.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:43.844 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:43.849 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:43.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:43.855 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:24:43.858 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:13.988 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:13.994 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:13.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:14.003 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:14.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:44.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:44.153 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:44.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:44.165 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:25:44.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:14.365 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:14.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:14.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:14.380 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:14.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:44.494 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:44.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:44.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:44.508 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:26:44.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:14.631 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:14.637 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:14.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:14.645 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:14.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:44.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:44.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:44.781 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:44.785 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:27:44.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:14.895 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:14.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:14.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:14.912 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:14.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:45.054 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:45.058 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:45.061 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:45.064 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:28:45.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:15.167 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:15.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:15.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:15.179 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:15.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:45.316 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:45.322 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:45.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:45.329 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:29:45.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:15.450 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:15.456 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:15.462 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:15.466 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:15.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:45.596 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:45.602 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:45.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:45.614 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:30:45.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:15.913 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:15.925 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:15.929 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:15.935 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:15.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:46.075 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:46.080 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:46.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:46.106 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:31:46.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:16.253 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:16.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:16.261 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:16.265 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:16.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:46.587 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:46.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:46.594 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:46.597 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:32:46.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:16.856 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:16.860 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:16.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:16.866 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:16.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:47.175 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:47.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:47.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:47.186 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:33:47.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:17.310 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:17.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:17.323 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:17.327 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:17.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:47.474 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:47.477 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:47.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:47.483 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:34:47.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:17.604 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:17.608 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:17.610 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:17.614 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:17.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:47.741 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:47.746 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:47.749 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:47.752 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:35:47.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:17.898 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:17.904 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:17.908 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:17.911 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:17.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:48.031 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:48.035 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:48.038 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:48.042 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:48.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:36:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:36:51.380 INFO:teuthology.orchestra.run.smithi121.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Connection timed out 2022-07-12T09:36:51.382 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-12T09:36:51.384 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-12T09:36:51.385 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-12T09:36:51.386 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-12T09:37:18.202 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:18.206 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:18.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:18.212 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:18.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:48.331 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:48.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:48.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:48.340 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:37:48.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:18.477 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:18.481 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:18.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:18.486 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:18.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:48.632 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:48.635 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:48.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:48.641 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:38:48.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:18.761 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:18.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:18.768 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:18.770 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:18.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:48.966 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:48.970 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:48.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:48.976 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:39:48.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:19.095 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:19.100 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:19.102 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:19.105 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:19.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:49.290 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:49.294 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:49.297 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:49.299 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:40:49.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:19.472 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:19.476 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:19.478 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:19.481 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:19.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:49.624 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:49.629 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:49.632 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:49.635 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:41:49.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:19.782 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:19.787 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:19.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:19.793 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:19.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:50.034 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:50.038 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:50.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:50.043 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:42:50.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:20.188 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:20.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:20.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:20.197 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:20.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:50.325 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:50.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:50.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:50.342 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:43:50.345 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:20.475 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:20.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:20.497 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:20.502 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:20.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:50.638 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:50.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:50.647 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:50.651 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:44:50.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:20.806 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:20.812 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:20.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:20.820 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:20.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:51.027 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:51.031 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:51.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:51.052 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:45:51.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:21.198 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:21.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:21.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:21.208 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:21.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:51.346 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:51.351 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:51.353 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:51.356 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:46:51.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:21.491 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:21.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:21.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:21.502 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:21.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:51.637 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:51.641 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:51.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:51.647 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:47:51.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:21.777 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:21.781 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:21.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:21.786 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:21.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:52.026 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:52.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:52.033 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:52.038 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:48:52.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:22.150 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:22.154 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:22.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:22.159 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:22.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:52.285 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:52.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:52.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:52.295 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:49:52.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:22.475 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:22.479 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:22.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:22.486 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:22.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:52.654 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:52.659 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:52.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:52.666 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:50:52.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:22.808 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:22.812 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:22.816 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:22.819 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:22.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:51.392 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T09:51:51.417 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-12T09:51:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:51:51.456 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T09:51:51.476 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-07-12T09:51:51.488 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2022-07-12T09:51:51.490 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T09:51:51.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 Processor -- start 2022-07-12T09:51:51.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 -- start start 2022-07-12T09:51:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fad940ab670 0x7fad9405f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fad940afa60 0x7fad940afe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad940abc70 con 0x7fad940ab290 2022-07-12T09:51:51.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad940aba90 con 0x7fad940ab670 2022-07-12T09:51:51.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad98b46700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad9405c720 con 0x7fad940afa60 2022-07-12T09:51:51.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad92ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fad940ab670 0x7fad9405f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad937fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fad940ab290 0x7fad9405ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51030/0 (socket says 172.21.15.61:51030) 2022-07-12T09:51:51.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad937fe700 1 -- 172.21.15.61:0/1314500263 learned_addr learned my addr 172.21.15.61:0/1314500263 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:51.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad93fff700 1 --2- 172.21.15.61:0/1314500263 >> v2:172.21.15.111:3300/0 conn(0x7fad940afa60 0x7fad940afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad92ffd700 1 -- 172.21.15.61:0/1314500263 >> v2:172.21.15.111:3300/0 conn(0x7fad940afa60 msgr2=0x7fad940afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:51.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad92ffd700 1 --2- 172.21.15.61:0/1314500263 >> v2:172.21.15.111:3300/0 conn(0x7fad940afa60 0x7fad940afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad92ffd700 1 -- 172.21.15.61:0/1314500263 >> v1:172.21.15.61:6789/0 conn(0x7fad940ab290 legacy=0x7fad9405ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad92ffd700 1 -- 172.21.15.61:0/1314500263 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9405c9c0 con 0x7fad940ab670 2022-07-12T09:51:51.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad92ffd700 1 --2- 172.21.15.61:0/1314500263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad9405f140 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7fad840032f0 tx=0x7fad840091d0).ready entity=mon.0 client_cookie=57a5f98e44af6f32 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:51.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad927fc700 1 -- 172.21.15.61:0/1314500263 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad84009ea0 con 0x7fad940ab670 2022-07-12T09:51:51.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.611+0000 7fad927fc700 1 -- 172.21.15.61:0/1314500263 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8400b870 con 0x7fad940ab670 2022-07-12T09:51:51.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad927fc700 1 -- 172.21.15.61:0/1314500263 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad8400b9e0 con 0x7fad940ab670 2022-07-12T09:51:51.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 -- 172.21.15.61:0/1314500263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 msgr2=0x7fad9405f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 --2- 172.21.15.61:0/1314500263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad9405f140 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 rx=0x7fad840032f0 tx=0x7fad840091d0).stop 2022-07-12T09:51:51.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 -- 172.21.15.61:0/1314500263 shutdown_connections 2022-07-12T09:51:51.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 --2- 172.21.15.61:0/1314500263 >> v2:172.21.15.111:3300/0 conn(0x7fad940afa60 0x7fad940afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 --2- 172.21.15.61:0/1314500263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad9405f140 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 -- 172.21.15.61:0/1314500263 >> 172.21.15.61:0/1314500263 conn(0x7fad940b1950 msgr2=0x7fad940b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:51.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 -- 172.21.15.61:0/1314500263 shutdown_connections 2022-07-12T09:51:51.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 -- 172.21.15.61:0/1314500263 wait complete. 2022-07-12T09:51:51.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 Processor -- start 2022-07-12T09:51:51.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.615+0000 7fad98b46700 1 -- start start 2022-07-12T09:51:51.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad940ab290 0x7fad94149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad94066870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad940afa60 0x7fad94155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9414a210 con 0x7fad940ab670 2022-07-12T09:51:51.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad94148720 con 0x7fad940ab290 2022-07-12T09:51:51.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad941489c0 con 0x7fad940afa60 2022-07-12T09:51:51.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad94066870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad94066870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34588/0 (socket says 172.21.15.61:34588) 2022-07-12T09:51:51.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 -- 172.21.15.61:0/2086375519 learned_addr learned my addr 172.21.15.61:0/2086375519 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:51.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad937fe700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad940ab290 0x7fad94149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad93fff700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad940afa60 0x7fad94155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 -- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad940afa60 msgr2=0x7fad94155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad940afa60 0x7fad94155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 -- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad940ab290 msgr2=0x7fad94149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad940ab290 0x7fad94149100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 -- 172.21.15.61:0/2086375519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84009470 con 0x7fad940ab670 2022-07-12T09:51:51.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad92ffd700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad94066870 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7fad840032f0 tx=0x7fad84016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:51.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad8400b9e0 con 0x7fad940ab670 2022-07-12T09:51:51.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8401f3f0 con 0x7fad940ab670 2022-07-12T09:51:51.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad94148c60 con 0x7fad940ab670 2022-07-12T09:51:51.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad8401f750 con 0x7fad940ab670 2022-07-12T09:51:51.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad94066d60 con 0x7fad940ab670 2022-07-12T09:51:51.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.619+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad940678b0 con 0x7fad940ab670 2022-07-12T09:51:51.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.623+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fad84026030 con 0x7fad940ab670 2022-07-12T09:51:51.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.623+0000 7fad90ff9700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad74053a40 0x7fad74055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.623+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fad8406c780 con 0x7fad940ab670 2022-07-12T09:51:51.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.623+0000 7fad937fe700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad74053a40 0x7fad74055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.623+0000 7fad937fe700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad74053a40 0x7fad74055f00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7fad8800d000 tx=0x7fad88009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:51.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.627+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad940678b0 con 0x7fad940ab670 2022-07-12T09:51:51.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.755+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad940678b0 con 0x7fad940ab670 2022-07-12T09:51:51.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad90ff9700 1 -- 172.21.15.61:0/2086375519 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+13366 (secure 0 0 0) 0x7fad940678b0 con 0x7fad940ab670 2022-07-12T09:51:51.759 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T09:51:51.759 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":151,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:20.736506+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.61:6833/502386216":"2022-07-13T08:51:17.697391+0000","172.21.15.61:6832/502386216":"2022-07-13T08:51:17.697391+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T09:51:51.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad74053a40 msgr2=0x7fad74055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad74053a40 0x7fad74055f00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7fad8800d000 tx=0x7fad88009040).stop 2022-07-12T09:51:51.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 msgr2=0x7fad94066870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad94066870 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7fad840032f0 tx=0x7fad84016000).stop 2022-07-12T09:51:51.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 shutdown_connections 2022-07-12T09:51:51.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fad940afa60 0x7fad94155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad940ab670 0x7fad94066870 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fad74053a40 0x7fad74055f00 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 --2- 172.21.15.61:0/2086375519 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fad940ab290 0x7fad94149100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 >> 172.21.15.61:0/2086375519 conn(0x7fad940b1950 msgr2=0x7fad940afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:51.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 shutdown_connections 2022-07-12T09:51:51.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.759+0000 7fad98b46700 1 -- 172.21.15.61:0/2086375519 wait complete. 2022-07-12T09:51:51.772 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-12T09:51:51.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.875+0000 7f29a72fb700 1 Processor -- start 2022-07-12T09:51:51.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.875+0000 7f29a72fb700 1 -- start start 2022-07-12T09:51:51.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f29a005c8e0 0x7f29a00ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29a00ad8b0 0x7f29a00aed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a013f350 con 0x7f29a00af240 2022-07-12T09:51:51.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a00b1c60 con 0x7f29a00ad8b0 2022-07-12T09:51:51.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a00af650 con 0x7f29a005c8e0 2022-07-12T09:51:51.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29a00ad8b0 0x7f29a00aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29a00ad8b0 0x7f29a00aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34594/0 (socket says 172.21.15.61:34594) 2022-07-12T09:51:51.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 -- 172.21.15.61:0/3264185059 learned_addr learned my addr 172.21.15.61:0/3264185059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:51.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a6afa700 1 --1- 172.21.15.61:0/3264185059 >> v1:172.21.15.61:6789/0 conn(0x7f29a00af240 0x7f29a0142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51048/0 (socket says 172.21.15.61:51048) 2022-07-12T09:51:51.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a62f9700 1 --2- 172.21.15.61:0/3264185059 >> v2:172.21.15.111:3300/0 conn(0x7f29a005c8e0 0x7f29a00ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 -- 172.21.15.61:0/3264185059 >> v2:172.21.15.111:3300/0 conn(0x7f29a005c8e0 msgr2=0x7f29a00ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:51.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 --2- 172.21.15.61:0/3264185059 >> v2:172.21.15.111:3300/0 conn(0x7f29a005c8e0 0x7f29a00ae860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 -- 172.21.15.61:0/3264185059 >> v1:172.21.15.61:6789/0 conn(0x7f29a00af240 legacy=0x7f29a0142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 -- 172.21.15.61:0/3264185059 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a013f5f0 con 0x7f29a00ad8b0 2022-07-12T09:51:51.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a62f9700 1 --2- 172.21.15.61:0/3264185059 >> v2:172.21.15.111:3300/0 conn(0x7f29a005c8e0 0x7f29a00ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:51.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a5af8700 1 --2- 172.21.15.61:0/3264185059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00ad8b0 0x7f29a00aed50 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f2994002f70 tx=0x7f2994006e40).ready entity=mon.0 client_cookie=64b99a9817c42780 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:51.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a52f7700 1 -- 172.21.15.61:0/3264185059 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f299400bcf0 con 0x7f29a00ad8b0 2022-07-12T09:51:51.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a52f7700 1 -- 172.21.15.61:0/3264185059 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994000e90 con 0x7f29a00ad8b0 2022-07-12T09:51:51.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a52f7700 1 -- 172.21.15.61:0/3264185059 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29940155e0 con 0x7f29a00ad8b0 2022-07-12T09:51:51.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- 172.21.15.61:0/3264185059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00ad8b0 msgr2=0x7f29a00aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 --2- 172.21.15.61:0/3264185059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00ad8b0 0x7f29a00aed50 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7f2994002f70 tx=0x7f2994006e40).stop 2022-07-12T09:51:51.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- 172.21.15.61:0/3264185059 shutdown_connections 2022-07-12T09:51:51.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 --2- 172.21.15.61:0/3264185059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00ad8b0 0x7f29a00aed50 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 --2- 172.21.15.61:0/3264185059 >> v2:172.21.15.111:3300/0 conn(0x7f29a005c8e0 0x7f29a00ae860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- 172.21.15.61:0/3264185059 >> 172.21.15.61:0/3264185059 conn(0x7f29a00b5780 msgr2=0x7f29a00b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:51.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- 172.21.15.61:0/3264185059 shutdown_connections 2022-07-12T09:51:51.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- 172.21.15.61:0/3264185059 wait complete. 2022-07-12T09:51:51.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 Processor -- start 2022-07-12T09:51:51.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.879+0000 7f29a72fb700 1 -- start start 2022-07-12T09:51:51.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29a005c8e0 0x7f29a0110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a00ad8b0 0x7f29a0110d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00af240 0x7f29a0139b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a0111920 con 0x7f29a00af240 2022-07-12T09:51:51.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a010f090 con 0x7f29a00ad8b0 2022-07-12T09:51:51.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a0113330 con 0x7f29a005c8e0 2022-07-12T09:51:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a6afa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00af240 0x7f29a0139b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a6afa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00af240 0x7f29a0139b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34602/0 (socket says 172.21.15.61:34602) 2022-07-12T09:51:51.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a6afa700 1 -- 172.21.15.61:0/1928977193 learned_addr learned my addr 172.21.15.61:0/1928977193 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:51.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29a005c8e0 0x7f29a0110860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a5af8700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a00ad8b0 0x7f29a0110d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 -- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a00ad8b0 msgr2=0x7f29a0110d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a00ad8b0 0x7f29a0110d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 -- 172.21.15.61:0/1928977193 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00af240 msgr2=0x7f29a0139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:51.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00af240 0x7f29a0139b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:51.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 -- 172.21.15.61:0/1928977193 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299400d4d0 con 0x7f29a005c8e0 2022-07-12T09:51:51.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a62f9700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29a005c8e0 0x7f29a0110860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f299c00a000 tx=0x7f299c008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:51.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f299c011780 con 0x7f29a005c8e0 2022-07-12T09:51:51.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c0118e0 con 0x7f29a005c8e0 2022-07-12T09:51:51.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a010ede0 con 0x7f29a005c8e0 2022-07-12T09:51:51.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f299c011c40 con 0x7f29a005c8e0 2022-07-12T09:51:51.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a010f540 con 0x7f29a005c8e0 2022-07-12T09:51:51.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.883+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a010f200 con 0x7f29a005c8e0 2022-07-12T09:51:51.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.887+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mon.2 v2:172.21.15.111:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f299c012420 con 0x7f29a005c8e0 2022-07-12T09:51:51.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.887+0000 7f29937fe700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2980053a40 0x7f2980055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:51.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.887+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mon.2 v2:172.21.15.111:3301/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f299c06b7f0 con 0x7f29a005c8e0 2022-07-12T09:51:51.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.887+0000 7f29a5af8700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2980053a40 0x7f2980055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:51.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.887+0000 7f29a5af8700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2980053a40 0x7f2980055f00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f299401b000 tx=0x7f2994018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:51.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:51.895+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mon.2 v2:172.21.15.111:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29a010f200 con 0x7f29a005c8e0 2022-07-12T09:51:52.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.019+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29a010f200 con 0x7f2980053a40 2022-07-12T09:51:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.027+0000 7f29937fe700 1 -- 172.21.15.61:0/1928977193 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+67760 (secure 0 0 0) 0x7f29a010f200 con 0x7f2980053a40 2022-07-12T09:51:52.030 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T09:51:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2980053a40 msgr2=0x7f2980055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2980053a40 0x7f2980055f00 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 rx=0x7f299401b000 tx=0x7f2994018000).stop 2022-07-12T09:51:52.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29a005c8e0 msgr2=0x7f29a0110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29a005c8e0 0x7f29a0110860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f299c00a000 tx=0x7f299c008040).stop 2022-07-12T09:51:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 shutdown_connections 2022-07-12T09:51:52.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29a00af240 0x7f29a0139b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f2980053a40 0x7f2980055f00 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29a00ad8b0 0x7f29a0110d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 --2- 172.21.15.61:0/1928977193 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f29a005c8e0 0x7f29a0110860 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 >> 172.21.15.61:0/1928977193 conn(0x7f29a00b5780 msgr2=0x7f29a00b5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 shutdown_connections 2022-07-12T09:51:52.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.031+0000 7f29a72fb700 1 -- 172.21.15.61:0/1928977193 wait complete. 2022-07-12T09:51:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-12T09:51:52.048 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2275,"stamp":"2022-07-12T09:51:50.259224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385120,"kb_used_data":743209920,"kb_used_omap":27846,"kb_used_meta":0,"kb_avail":743209920,"statfs":{"total":767585320960,"available":761046958080,"internally_reserved":0,"allocated":761046958080,"data_stored":761046958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28515244,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.475725"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":138,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853075+0000","last_change":"2022-07-12T08:41:04.955012+0000","last_active":"2022-07-12T08:51:20.853075+0000","last_peered":"2022-07-12T08:51:20.853075+0000","last_clean":"2022-07-12T08:51:20.853075+0000","last_became_active":"2022-07-12T08:41:04.954339+0000","last_became_peered":"2022-07-12T08:41:04.954339+0000","last_unstale":"2022-07-12T08:51:20.853075+0000","last_undegraded":"2022-07-12T08:51:20.853075+0000","last_fullsized":"2022-07-12T08:51:20.853075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_clean_scrub_stamp":"2022-07-12T08:40:58.647421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:19.737924+0000","last_change":"2022-07-12T08:41:11.067479+0000","last_active":"2022-07-12T08:51:19.737924+0000","last_peered":"2022-07-12T08:51:19.737924+0000","last_clean":"2022-07-12T08:51:19.737924+0000","last_became_active":"2022-07-12T08:41:11.067215+0000","last_became_peered":"2022-07-12T08:41:11.067215+0000","last_unstale":"2022-07-12T08:51:19.737924+0000","last_undegraded":"2022-07-12T08:51:19.737924+0000","last_fullsized":"2022-07-12T08:51:19.737924+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.852812+0000","last_change":"2022-07-12T08:41:11.064070+0000","last_active":"2022-07-12T08:51:20.852812+0000","last_peered":"2022-07-12T08:51:20.852812+0000","last_clean":"2022-07-12T08:51:20.852812+0000","last_became_active":"2022-07-12T08:41:11.063329+0000","last_became_peered":"2022-07-12T08:41:11.063329+0000","last_unstale":"2022-07-12T08:51:20.852812+0000","last_undegraded":"2022-07-12T08:51:20.852812+0000","last_fullsized":"2022-07-12T08:51:20.852812+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-07-12T08:51:17.742997+0000","last_change":"2022-07-12T08:41:11.071303+0000","last_active":"2022-07-12T08:51:17.742997+0000","last_peered":"2022-07-12T08:51:17.742997+0000","last_clean":"2022-07-12T08:51:17.742997+0000","last_became_active":"2022-07-12T08:41:11.070549+0000","last_became_peered":"2022-07-12T08:41:11.070549+0000","last_unstale":"2022-07-12T08:51:17.742997+0000","last_undegraded":"2022-07-12T08:51:17.742997+0000","last_fullsized":"2022-07-12T08:51:17.742997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:19.737982+0000","last_change":"2022-07-12T08:41:11.066220+0000","last_active":"2022-07-12T08:51:19.737982+0000","last_peered":"2022-07-12T08:51:19.737982+0000","last_clean":"2022-07-12T08:51:19.737982+0000","last_became_active":"2022-07-12T08:41:11.065911+0000","last_became_peered":"2022-07-12T08:41:11.065911+0000","last_unstale":"2022-07-12T08:51:19.737982+0000","last_undegraded":"2022-07-12T08:51:19.737982+0000","last_fullsized":"2022-07-12T08:51:19.737982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853881+0000","last_change":"2022-07-12T08:41:11.958581+0000","last_active":"2022-07-12T08:51:20.853881+0000","last_peered":"2022-07-12T08:51:20.853881+0000","last_clean":"2022-07-12T08:51:20.853881+0000","last_became_active":"2022-07-12T08:41:11.958006+0000","last_became_peered":"2022-07-12T08:41:11.958006+0000","last_unstale":"2022-07-12T08:51:20.853881+0000","last_undegraded":"2022-07-12T08:51:20.853881+0000","last_fullsized":"2022-07-12T08:51:20.853881+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853703+0000","last_change":"2022-07-12T08:41:11.954669+0000","last_active":"2022-07-12T08:51:20.853703+0000","last_peered":"2022-07-12T08:51:20.853703+0000","last_clean":"2022-07-12T08:51:20.853703+0000","last_became_active":"2022-07-12T08:41:11.954063+0000","last_became_peered":"2022-07-12T08:41:11.954063+0000","last_unstale":"2022-07-12T08:51:20.853703+0000","last_undegraded":"2022-07-12T08:51:20.853703+0000","last_fullsized":"2022-07-12T08:51:20.853703+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853929+0000","last_change":"2022-07-12T08:41:11.069495+0000","last_active":"2022-07-12T08:51:20.853929+0000","last_peered":"2022-07-12T08:51:20.853929+0000","last_clean":"2022-07-12T08:51:20.853929+0000","last_became_active":"2022-07-12T08:41:11.068746+0000","last_became_peered":"2022-07-12T08:41:11.068746+0000","last_unstale":"2022-07-12T08:51:20.853929+0000","last_undegraded":"2022-07-12T08:51:20.853929+0000","last_fullsized":"2022-07-12T08:51:20.853929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-07-12T08:51:17.743069+0000","last_change":"2022-07-12T08:41:11.064835+0000","last_active":"2022-07-12T08:51:17.743069+0000","last_peered":"2022-07-12T08:51:17.743069+0000","last_clean":"2022-07-12T08:51:17.743069+0000","last_became_active":"2022-07-12T08:41:11.064473+0000","last_became_peered":"2022-07-12T08:41:11.064473+0000","last_unstale":"2022-07-12T08:51:17.743069+0000","last_undegraded":"2022-07-12T08:51:17.743069+0000","last_fullsized":"2022-07-12T08:51:17.743069+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575699,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3617,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3704129,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88200000000000001,"15min":0.88400000000000001},"min":{"1min":0.68500000000000005,"5min":0.53300000000000003,"15min":0.50700000000000001},"max":{"1min":1.0369999999999999,"5min":1.137,"15min":1.1479999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86199999999999999,"15min":0.879},"min":{"1min":0.72599999999999998,"5min":0.623,"15min":0.60799999999999998},"max":{"1min":0.998,"5min":1.1040000000000001,"15min":1.111},"last":0.96999999999999997}]},{"osd":1,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.56499999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0820000000000001,"5min":1.1659999999999999,"15min":1.214},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.64000000000000001,"5min":0.60199999999999998,"15min":0.58099999999999996},"max":{"1min":1.0700000000000001,"5min":1.109,"15min":1.1279999999999999},"last":0.88}]},{"osd":2,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.82599999999999996,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.016,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.86399999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1220000000000001,"5min":1.242,"15min":1.242},"last":1.0840000000000001}]},{"osd":3,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.58499999999999996},"max":{"1min":1.1080000000000001,"5min":1.232,"15min":1.232},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.89300000000000002,"15min":0.90200000000000002},"min":{"1min":0.65200000000000002,"5min":0.60299999999999998,"15min":0.57899999999999996},"max":{"1min":1.1599999999999999,"5min":1.1859999999999999,"15min":1.22},"last":0.92200000000000004}]},{"osd":4,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.67900000000000005,"15min":0.71499999999999997},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.47199999999999998},"max":{"1min":0.94999999999999996,"5min":1.145,"15min":1.1819999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88400000000000001,"15min":0.89100000000000001},"min":{"1min":0.70599999999999996,"5min":0.64600000000000002,"15min":0.44600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1759999999999999,"15min":1.2070000000000001},"last":0.996}]},{"osd":5,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":1.198,"5min":1.395,"15min":1.395},"last":1.125},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.78300000000000003,"5min":0.56100000000000005,"15min":0.46100000000000002},"max":{"1min":1.121,"5min":1.1639999999999999,"15min":1.194},"last":1.0980000000000001}]},{"osd":6,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.91100000000000003,"15min":0.91400000000000003},"min":{"1min":0.65200000000000002,"5min":0.55500000000000005,"15min":0.434},"max":{"1min":1.077,"5min":1.1279999999999999,"15min":1.2410000000000001},"last":1.0569999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.67000000000000004,"5min":0.52000000000000002,"15min":0.39600000000000002},"max":{"1min":1.1830000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.069}]}]},{"osd":6,"up_from":13,"seq":55834575699,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":4018,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4115239,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80100000000000005,"15min":0.78300000000000003},"min":{"1min":0.63,"5min":0.54300000000000004,"15min":0.52700000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"max":{"1min":1.012,"5min":1.0580000000000001,"15min":1.2130000000000001},"last":0.74099999999999999}]},{"osd":1,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85099999999999998,"15min":0.874},"min":{"1min":0.66500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0549999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.71899999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1160000000000001,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.85799999999999998}]},{"osd":2,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.874,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.55900000000000005},"max":{"1min":1.0900000000000001,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.873},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.96899999999999997,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.997}]},{"osd":3,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85399999999999998,"15min":0.84399999999999997},"min":{"1min":0.66200000000000003,"5min":0.65400000000000003,"15min":0.621},"max":{"1min":1.077,"5min":1.1140000000000001,"15min":1.179},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.91300000000000003,"15min":0.92200000000000004},"min":{"1min":0.70799999999999996,"5min":0.63700000000000001,"15min":0.61599999999999999},"max":{"1min":1.054,"5min":1.125,"15min":1.2150000000000001},"last":0.79200000000000004}]},{"osd":4,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80900000000000005,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0529999999999999,"5min":1.1950000000000001,"15min":1.2010000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.47899999999999998},"max":{"1min":1.139,"5min":1.175,"15min":1.413},"last":1.022}]},{"osd":5,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88500000000000001,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.57999999999999996,"15min":0.54600000000000004},"max":{"1min":1.1160000000000001,"5min":1.591,"15min":1.591},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.0069999999999999,"15min":0.996},"min":{"1min":0.77900000000000003,"5min":0.77300000000000002,"15min":0.626},"max":{"1min":1.1899999999999999,"5min":1.246,"15min":1.2589999999999999},"last":0.89700000000000002}]},{"osd":7,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.97799999999999998,"15min":0.96499999999999997},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.57999999999999996},"max":{"1min":1.135,"5min":1.365,"15min":1.4350000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.96499999999999997,"15min":0.95799999999999996},"min":{"1min":0.73899999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1639999999999999,"5min":1.77,"15min":1.77},"last":0.80800000000000005}]}]},{"osd":4,"up_from":13,"seq":55834575699,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798028,"kb_used_data":92901352,"kb_used_omap":3571,"kb_used_meta":0,"kb_avail":92901352,"statfs":{"total":95948165120,"available":95130984448,"internally_reserved":0,"allocated":95130984448,"data_stored":95130984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3657716,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84599999999999997,"15min":0.82999999999999996},"min":{"1min":0.66800000000000004,"5min":0.621,"15min":0.499},"max":{"1min":1.079,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82199999999999995,"15min":0.81399999999999995},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.51400000000000001},"max":{"1min":1.071,"5min":1.1499999999999999,"15min":1.1830000000000001},"last":0.83199999999999996}]},{"osd":1,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.84099999999999997,"15min":0.82799999999999996},"min":{"1min":0.63,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.089,"5min":1.149,"15min":1.149},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.85199999999999998,"15min":0.85999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.55400000000000005},"max":{"1min":1.002,"5min":1.0860000000000001,"15min":1.1499999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.85399999999999998,"15min":0.85199999999999998},"min":{"1min":0.64900000000000002,"5min":0.63200000000000001,"15min":0.59299999999999997},"max":{"1min":1.1879999999999999,"5min":1.272,"15min":1.272},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88,"15min":0.88100000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.59599999999999997},"max":{"1min":1.077,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.93899999999999995}]},{"osd":3,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.68300000000000005,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":0.997,"5min":1.0529999999999999,"15min":1.45},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.90500000000000003,"15min":0.90300000000000002},"min":{"1min":0.66600000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.4079999999999999},"last":1.0669999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.80800000000000005,"15min":0.80400000000000005},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.51800000000000002},"max":{"1min":0.95399999999999996,"5min":1.2010000000000001,"15min":1.258},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85399999999999998,"15min":0.86299999999999999},"min":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.51200000000000001},"max":{"1min":1.163,"5min":1.2010000000000001,"15min":1.367},"last":0.88900000000000001}]},{"osd":6,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.86399999999999999,"15min":0.86299999999999999},"min":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.56000000000000005},"max":{"1min":1.258,"5min":1.258,"15min":1.349},"last":1.034},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.93899999999999995},"min":{"1min":0.70499999999999996,"5min":0.623,"15min":0.59399999999999997},"max":{"1min":1.159,"5min":1.282,"15min":1.3109999999999999},"last":1.2}]},{"osd":7,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93600000000000005,"15min":0.94099999999999995},"min":{"1min":0.61399999999999999,"5min":0.61299999999999999,"15min":0.60099999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.4199999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0489999999999999,"15min":1.044},"min":{"1min":0.71699999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.343,"5min":1.3740000000000001,"15min":1.4390000000000001},"last":1.1100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575699,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3339,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3420122,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.86299999999999999,"5min":1.0800000000000001,"15min":1.266},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.63100000000000001,"5min":0.57499999999999996,"15min":0.57299999999999995},"max":{"1min":0.996,"5min":1.0589999999999999,"15min":1.1180000000000001},"last":0.79800000000000004}]},{"osd":1,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.54100000000000004,"5min":0.47399999999999998,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":1.0009999999999999,"15min":1.8169999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.76000000000000001},"min":{"1min":0.53800000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.96399999999999997,"5min":1.0780000000000001,"15min":1.5369999999999999},"last":0.97299999999999998}]},{"osd":2,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.68500000000000005,"15min":0.68899999999999995},"min":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.5},"max":{"1min":1.034,"5min":1.248,"15min":1.248},"last":0.627},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.72299999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.5720000000000001},"last":0.72999999999999998}]},{"osd":4,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.64200000000000002},"max":{"1min":1.149,"5min":1.236,"15min":1.405},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.92400000000000004,"15min":0.93500000000000005},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68100000000000005},"max":{"1min":1.155,"5min":1.3360000000000001,"15min":1.708},"last":0.95099999999999996}]},{"osd":5,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.92700000000000005,"15min":0.93600000000000005},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65000000000000002},"max":{"1min":1.1439999999999999,"5min":1.2949999999999999,"15min":1.4319999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95999999999999996,"15min":0.96899999999999997},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.1950000000000001,"5min":1.3580000000000001,"15min":1.5129999999999999},"last":1.133}]},{"osd":6,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.026,"15min":1.01},"min":{"1min":0.78700000000000003,"5min":0.73899999999999999,"15min":0.70399999999999996},"max":{"1min":1.238,"5min":1.966,"15min":1.966},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0649999999999999,"15min":1.0620000000000001},"min":{"1min":0.89900000000000002,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.6519999999999999},"last":1.087}]},{"osd":7,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0509999999999999,"15min":1.0600000000000001},"min":{"1min":0.78400000000000003,"5min":0.75700000000000001,"15min":0.73099999999999998},"max":{"1min":1.25,"5min":1.2989999999999999,"15min":1.468},"last":1.181},{"interface":"front","average":{"1min":1.083,"5min":1.085,"15min":1.0840000000000001},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.2749999999999999,"5min":1.5920000000000001,"15min":1.706},"last":1.0529999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575699,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2801,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2868423,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.50700000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.155},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70099999999999996,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.48399999999999999,"15min":0.47999999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.1339999999999999},"last":0.64100000000000001}]},{"osd":1,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.2529999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54400000000000004,"5min":0.54300000000000004,"15min":0.53000000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.85599999999999998}]},{"osd":3,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72499999999999998,"15min":0.74199999999999999},"min":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.48499999999999999},"max":{"1min":0.92900000000000005,"5min":1.069,"15min":1.1839999999999999},"last":1.042},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.82499999999999996,"15min":0.82799999999999996},"min":{"1min":0.66800000000000004,"5min":0.60599999999999998,"15min":0.57899999999999996},"max":{"1min":1.087,"5min":1.0900000000000001,"15min":1.123},"last":1.181}]},{"osd":4,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.95599999999999996,"15min":0.95999999999999996},"min":{"1min":0.80900000000000005,"5min":0.70199999999999996,"15min":0.69299999999999995},"max":{"1min":1.256,"5min":1.256,"15min":1.3680000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.94199999999999995,"15min":0.93999999999999995},"min":{"1min":0.83599999999999997,"5min":0.69399999999999995,"15min":0.67100000000000004},"max":{"1min":1.262,"5min":1.262,"15min":1.3109999999999999},"last":0.89300000000000002}]},{"osd":5,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.97699999999999998,"15min":0.99099999999999999},"min":{"1min":0.76600000000000001,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2589999999999999,"5min":1.321,"15min":1.341},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.97699999999999998,"15min":0.98799999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.66100000000000003},"max":{"1min":1.3360000000000001,"5min":1.476,"15min":1.476},"last":0.84099999999999997}]},{"osd":6,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.99099999999999999,"15min":0.98599999999999999},"min":{"1min":0.86599999999999999,"5min":0.73499999999999999,"15min":0.67600000000000005},"max":{"1min":1.298,"5min":1.425,"15min":1.425},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.079,"5min":1.0569999999999999,"15min":1.0589999999999999},"min":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.2490000000000001,"5min":1.379,"15min":1.4059999999999999},"last":1.262}]},{"osd":7,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.05,"15min":1.073},"min":{"1min":0.85399999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.4099999999999999,"5min":1.591,"15min":1.591},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.079,"15min":1.073},"min":{"1min":0.96399999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.2669999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.02}]}]},{"osd":1,"up_from":13,"seq":55834575699,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3966,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4061662,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":1.046,"15min":1.216},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.502,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.089,"5min":1.1779999999999999,"15min":1.226},"last":0.68200000000000005}]},{"osd":2,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.44},"max":{"1min":0.78400000000000003,"5min":1.143,"15min":1.1879999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.76600000000000001,"15min":0.76500000000000001},"min":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.075,"5min":1.149,"15min":1.3620000000000001},"last":0.85499999999999998}]},{"osd":3,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81899999999999995,"15min":0.83599999999999997},"min":{"1min":0.63500000000000001,"5min":0.60599999999999998,"15min":0.56599999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.8779999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76100000000000001,"15min":0.78300000000000003},"min":{"1min":0.59799999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92800000000000005,"5min":1.147,"15min":1.258},"last":0.82499999999999996}]},{"osd":4,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.89900000000000002,"15min":0.90200000000000002},"min":{"1min":0.77100000000000002,"5min":0.64100000000000001,"15min":0.63700000000000001},"max":{"1min":1.1830000000000001,"5min":1.278,"15min":1.3},"last":1.03},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0089999999999999,"15min":1.012},"min":{"1min":0.88800000000000001,"5min":0.79400000000000004,"15min":0.68999999999999995},"max":{"1min":1.2330000000000001,"5min":1.321,"15min":1.444},"last":0.94499999999999995}]},{"osd":5,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.995,"15min":0.98699999999999999},"min":{"1min":0.85199999999999998,"5min":0.73399999999999999,"15min":0.70799999999999996},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0009999999999999,"15min":0.99199999999999999},"min":{"1min":0.76400000000000001,"5min":0.749,"15min":0.63300000000000001},"max":{"1min":1.28,"5min":1.28,"15min":1.397},"last":1.1799999999999999}]},{"osd":6,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.004,"15min":1.002},"min":{"1min":0.70599999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.341,"5min":1.341,"15min":1.349},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.042,"5min":1.022,"15min":1.016},"min":{"1min":0.80000000000000004,"5min":0.753,"15min":0.68600000000000005},"max":{"1min":1.359,"5min":1.359,"15min":1.5189999999999999},"last":1.1200000000000001}]},{"osd":7,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0960000000000001},"min":{"1min":0.93100000000000005,"5min":0.78400000000000003,"15min":0.73099999999999998},"max":{"1min":1.319,"5min":1.3939999999999999,"15min":1.5049999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.071,"15min":1.085},"min":{"1min":0.90500000000000003,"5min":0.78500000000000003,"15min":0.72399999999999998},"max":{"1min":1.228,"5min":1.3680000000000001,"15min":1.583},"last":1.1459999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575699,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3791,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3882329,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.49299999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77600000000000002,"5min":0.89200000000000002,"15min":0.99399999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70699999999999996,"15min":0.70799999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0880000000000001,"5min":1.095,"15min":1.1819999999999999},"last":0.77400000000000002}]},{"osd":2,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.753,"15min":0.72199999999999998},"min":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.51900000000000002},"max":{"1min":1.0660000000000001,"5min":1.2210000000000001,"15min":1.2270000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.64700000000000002,"15min":0.66900000000000004},"min":{"1min":0.51000000000000001,"5min":0.48399999999999999,"15min":0.42699999999999999},"max":{"1min":1.0600000000000001,"5min":1.129,"15min":1.224},"last":0.65000000000000002}]},{"osd":3,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94899999999999995,"5min":1.0640000000000001,"15min":1.1799999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.876,"15min":0.877},"min":{"1min":0.72599999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0109999999999999,"5min":1.2130000000000001,"15min":1.246},"last":0.85999999999999999}]},{"osd":4,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.1180000000000001,"5min":1.133,"15min":1.391},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94999999999999996,"15min":0.94299999999999995},"min":{"1min":0.76200000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.0469999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":0.95999999999999996},"min":{"1min":0.748,"5min":0.67200000000000004,"15min":0.66500000000000004},"max":{"1min":1.361,"5min":1.395,"15min":1.48},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.98299999999999998,"15min":0.97999999999999998},"min":{"1min":0.81299999999999994,"5min":0.70299999999999996,"15min":0.69999999999999996},"max":{"1min":1.2330000000000001,"5min":1.258,"15min":1.498},"last":0.93999999999999995}]},{"osd":6,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.02,"15min":1.022},"min":{"1min":0.78100000000000003,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.238,"5min":1.3169999999999999,"15min":1.522},"last":1.002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96399999999999997,"15min":0.99099999999999999},"min":{"1min":0.74199999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.137,"5min":1.3759999999999999,"15min":1.4650000000000001},"last":1.0669999999999999}]},{"osd":7,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0049999999999999,"15min":1.0309999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.68799999999999994},"max":{"1min":1.179,"5min":1.4550000000000001,"15min":1.5489999999999999},"last":1.155},{"interface":"front","average":{"1min":1.032,"5min":1.079,"15min":1.087},"min":{"1min":0.83999999999999997,"5min":0.79300000000000004,"15min":0.77200000000000002},"max":{"1min":1.5,"5min":1.524,"15min":1.569},"last":1.1759999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575699,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2739,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2805624,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.63600000000000001,"5min":0.63100000000000001,"15min":0.58099999999999996},"max":{"1min":0.98999999999999999,"5min":1.194,"15min":1.224},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90200000000000002,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.65100000000000002,"15min":0.59999999999999998},"max":{"1min":1.0089999999999999,"5min":1.161,"15min":1.208},"last":0.84099999999999997}]},{"osd":1,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.5},"max":{"1min":1.073,"5min":1.198,"15min":1.22},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.51600000000000001},"max":{"1min":1.052,"5min":1.109,"15min":1.109},"last":0.59699999999999998}]},{"osd":2,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.66800000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0429999999999999,"5min":1.127,"15min":1.1819999999999999},"last":0.746},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.67300000000000004,"5min":0.61599999999999999,"15min":0.55800000000000005},"max":{"1min":0.99099999999999999,"5min":1.1759999999999999,"15min":1.1859999999999999},"last":0.77300000000000002}]},{"osd":3,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.57799999999999996},"max":{"1min":1.0609999999999999,"5min":1.151,"15min":1.3360000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.879,"15min":0.88200000000000001},"min":{"1min":0.63700000000000001,"5min":0.61199999999999999,"15min":0.54100000000000004},"max":{"1min":1.069,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.81499999999999995}]},{"osd":4,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70999999999999996,"15min":0.72199999999999998},"min":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.39000000000000001},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.1719999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.94299999999999995,"15min":0.93999999999999995},"min":{"1min":0.70099999999999996,"5min":0.68799999999999994,"15min":0.58599999999999997},"max":{"1min":1.073,"5min":1.212,"15min":1.4930000000000001},"last":0.92600000000000005}]},{"osd":6,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.879},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91700000000000004,"15min":0.90900000000000003},"min":{"1min":0.72299999999999998,"5min":0.57599999999999996,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.232,"15min":1.276},"last":0.67800000000000005}]},{"osd":7,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89500000000000002,"15min":0.89700000000000002},"min":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0389999999999999,"5min":1.2809999999999999,"15min":2.1200000000000001},"last":1.042},{"interface":"front","average":{"1min":1.002,"5min":1.0169999999999999,"15min":1.0129999999999999},"min":{"1min":0.84199999999999997,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.482,"5min":1.482,"15min":2.1019999999999999},"last":0.97499999999999998}]}]}]}} 2022-07-12T09:51:52.051 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-12T09:51:52.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 Processor -- start 2022-07-12T09:51:52.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- start start 2022-07-12T09:51:52.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff9fc060630 0x7ff9fc0ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:52.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1425c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:52.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9fc0b03b0 con 0x7ff9fc142ab0 2022-07-12T09:51:52.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9fc061300 con 0x7ff9fc060630 2022-07-12T09:51:52.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0ac820 con 0x7ff9fc061460 2022-07-12T09:51:52.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff9fc060630 0x7ff9fc0ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:52.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff9fc060630 0x7ff9fc0ad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34608/0 (socket says 172.21.15.61:34608) 2022-07-12T09:51:52.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2474662643 learned_addr learned my addr 172.21.15.61:0/2474662643 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:52.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9f9d9b700 1 --2- 172.21.15.61:0/2474662643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:52.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2474662643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 msgr2=0x7ff9fc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:52.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 --2- 172.21.15.61:0/2474662643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1425c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2474662643 >> v1:172.21.15.111:6789/0 conn(0x7ff9fc142ab0 legacy=0x7ff9fc13e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2474662643 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc0b0650 con 0x7ff9fc060630 2022-07-12T09:51:52.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9fa59c700 1 --2- 172.21.15.61:0/2474662643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc0ad3b0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7ff9ec002f70 tx=0x7ff9ec006f40).ready entity=mon.0 client_cookie=1bf2b0d7c4a604c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:52.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9f959a700 1 -- 172.21.15.61:0/2474662643 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9ec00dc70 con 0x7ff9fc060630 2022-07-12T09:51:52.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9f959a700 1 -- 172.21.15.61:0/2474662643 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec000e90 con 0x7ff9fc060630 2022-07-12T09:51:52.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ff9f959a700 1 -- 172.21.15.61:0/2474662643 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9ec00be90 con 0x7ff9fc060630 2022-07-12T09:51:52.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- 172.21.15.61:0/2474662643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 msgr2=0x7ff9fc0ad3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2474662643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc0ad3b0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7ff9ec002f70 tx=0x7ff9ec006f40).stop 2022-07-12T09:51:52.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- 172.21.15.61:0/2474662643 shutdown_connections 2022-07-12T09:51:52.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2474662643 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1425c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2474662643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc0ad3b0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- 172.21.15.61:0/2474662643 >> 172.21.15.61:0/2474662643 conn(0x7ff9fc05f140 msgr2=0x7ff9fc05f550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:52.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- 172.21.15.61:0/2474662643 shutdown_connections 2022-07-12T09:51:52.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.155+0000 7ffa00c91700 1 -- 172.21.15.61:0/2474662643 wait complete. 2022-07-12T09:51:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 Processor -- start 2022-07-12T09:51:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 -- start start 2022-07-12T09:51:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc064080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9fc142ab0 0x7ff9fc1445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:52.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc146110 con 0x7ff9fc060630 2022-07-12T09:51:52.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc145eb0 con 0x7ff9fc142ab0 2022-07-12T09:51:52.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc066ff0 con 0x7ff9fc061460 2022-07-12T09:51:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc064080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc064080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34614/0 (socket says 172.21.15.61:34614) 2022-07-12T09:51:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2382746879 learned_addr learned my addr 172.21.15.61:0/2382746879 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:52.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9f9d9b700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:52.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fad9d700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9fc142ab0 0x7ff9fc1445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:52.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 msgr2=0x7ff9fc1440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9fc142ab0 msgr2=0x7ff9fc1445b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9fc142ab0 0x7ff9fc1445b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 -- 172.21.15.61:0/2382746879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec00b460 con 0x7ff9fc060630 2022-07-12T09:51:52.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9fa59c700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc064080 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7ff9ec002f70 tx=0x7ff9ec006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:52.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9ec01d070 con 0x7ff9fc060630 2022-07-12T09:51:52.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec022600 con 0x7ff9fc060630 2022-07-12T09:51:52.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc13dcf0 con 0x7ff9fc060630 2022-07-12T09:51:52.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9ec010ea0 con 0x7ff9fc060630 2022-07-12T09:51:52.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.159+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9fc146380 con 0x7ff9fc060630 2022-07-12T09:51:52.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.163+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9fc057460 con 0x7ff9fc060630 2022-07-12T09:51:52.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.163+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff9ec029070 con 0x7ff9fc060630 2022-07-12T09:51:52.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.163+0000 7ff9f37fe700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9e0053ad0 0x7ff9e0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:52.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.163+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff9ec06d140 con 0x7ff9fc060630 2022-07-12T09:51:52.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.163+0000 7ff9f9d9b700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9e0053ad0 0x7ff9e0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:52.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.163+0000 7ff9f9d9b700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9e0053ad0 0x7ff9e0055f90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7ff9f400d000 tx=0x7ff9f4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:52.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.171+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff9ec044380 con 0x7ff9fc060630 2022-07-12T09:51:52.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.299+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9fc0647e0 con 0x7ff9fc060630 2022-07-12T09:51:52.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ff9f37fe700 1 -- 172.21.15.61:0/2382746879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+13366 (secure 0 0 0) 0x7ff9ec0479a0 con 0x7ff9fc060630 2022-07-12T09:51:52.304 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T09:51:52.304 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":151,"fsid":"2d9e4a1e-6d05-4ca6-8b2a-89282c026e3a","created":"2022-07-12T08:40:54.309679+0000","modified":"2022-07-12T08:51:20.736506+0000","last_up_change":"2022-07-12T08:41:03.796038+0000","last_in_change":"2022-07-12T08:41:03.796038+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-12T08:40:58.382631+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-12T08:41:09.975085+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c5e8edcd-71d7-4bc2-84d6-819a6f4be259","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":21668},{"type":"v1","addr":"172.21.15.61:6801","nonce":21668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":21668},{"type":"v1","addr":"172.21.15.61:6803","nonce":21668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":21668},{"type":"v1","addr":"172.21.15.61:6807","nonce":21668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":21668},{"type":"v1","addr":"172.21.15.61:6805","nonce":21668}]},"public_addr":"172.21.15.61:6801/21668","cluster_addr":"172.21.15.61:6803/21668","heartbeat_back_addr":"172.21.15.61:6807/21668","heartbeat_front_addr":"172.21.15.61:6805/21668","state":["exists","up"]},{"osd":1,"uuid":"78c8bcfb-e525-4fea-a66a-002769a0b1ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":21713},{"type":"v1","addr":"172.21.15.61:6809","nonce":21713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":21713},{"type":"v1","addr":"172.21.15.61:6811","nonce":21713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":21713},{"type":"v1","addr":"172.21.15.61:6815","nonce":21713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":21713},{"type":"v1","addr":"172.21.15.61:6813","nonce":21713}]},"public_addr":"172.21.15.61:6809/21713","cluster_addr":"172.21.15.61:6811/21713","heartbeat_back_addr":"172.21.15.61:6815/21713","heartbeat_front_addr":"172.21.15.61:6813/21713","state":["exists","up"]},{"osd":2,"uuid":"b765f537-14cb-46b5-be53-b78393e087ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6816","nonce":21871},{"type":"v1","addr":"172.21.15.61:6817","nonce":21871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":21871},{"type":"v1","addr":"172.21.15.61:6819","nonce":21871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":21871},{"type":"v1","addr":"172.21.15.61:6823","nonce":21871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":21871},{"type":"v1","addr":"172.21.15.61:6821","nonce":21871}]},"public_addr":"172.21.15.61:6817/21871","cluster_addr":"172.21.15.61:6819/21871","heartbeat_back_addr":"172.21.15.61:6823/21871","heartbeat_front_addr":"172.21.15.61:6821/21871","state":["exists","up"]},{"osd":3,"uuid":"0f043fbb-3135-49f3-9b49-a6ce7e7347b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":21976},{"type":"v1","addr":"172.21.15.61:6825","nonce":21976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":21976},{"type":"v1","addr":"172.21.15.61:6827","nonce":21976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":21976},{"type":"v1","addr":"172.21.15.61:6831","nonce":21976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":21976},{"type":"v1","addr":"172.21.15.61:6829","nonce":21976}]},"public_addr":"172.21.15.61:6825/21976","cluster_addr":"172.21.15.61:6827/21976","heartbeat_back_addr":"172.21.15.61:6831/21976","heartbeat_front_addr":"172.21.15.61:6829/21976","state":["exists","up"]},{"osd":4,"uuid":"b2005cdb-ea05-443e-8816-aaa7dfabfd84","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":144,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":21886},{"type":"v1","addr":"172.21.15.111:6803","nonce":21886}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":21886},{"type":"v1","addr":"172.21.15.111:6805","nonce":21886}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":21886},{"type":"v1","addr":"172.21.15.111:6809","nonce":21886}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":21886},{"type":"v1","addr":"172.21.15.111:6807","nonce":21886}]},"public_addr":"172.21.15.111:6803/21886","cluster_addr":"172.21.15.111:6805/21886","heartbeat_back_addr":"172.21.15.111:6809/21886","heartbeat_front_addr":"172.21.15.111:6807/21886","state":["exists","up"]},{"osd":5,"uuid":"9b0164a4-42b8-409a-86fb-bafc85e28b29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":21888},{"type":"v1","addr":"172.21.15.111:6811","nonce":21888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":21888},{"type":"v1","addr":"172.21.15.111:6813","nonce":21888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":21888},{"type":"v1","addr":"172.21.15.111:6817","nonce":21888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":21888},{"type":"v1","addr":"172.21.15.111:6815","nonce":21888}]},"public_addr":"172.21.15.111:6811/21888","cluster_addr":"172.21.15.111:6813/21888","heartbeat_back_addr":"172.21.15.111:6817/21888","heartbeat_front_addr":"172.21.15.111:6815/21888","state":["exists","up"]},{"osd":6,"uuid":"1718afd4-28d5-4c9e-bbb8-f8b29600a5fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":22012},{"type":"v1","addr":"172.21.15.111:6819","nonce":22012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":22012},{"type":"v1","addr":"172.21.15.111:6821","nonce":22012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":22012},{"type":"v1","addr":"172.21.15.111:6825","nonce":22012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":22012},{"type":"v1","addr":"172.21.15.111:6823","nonce":22012}]},"public_addr":"172.21.15.111:6819/22012","cluster_addr":"172.21.15.111:6821/22012","heartbeat_back_addr":"172.21.15.111:6825/22012","heartbeat_front_addr":"172.21.15.111:6823/22012","state":["exists","up"]},{"osd":7,"uuid":"87944907-c10c-4187-abd4-fd919e7520be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":22094},{"type":"v1","addr":"172.21.15.111:6827","nonce":22094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":22094},{"type":"v1","addr":"172.21.15.111:6829","nonce":22094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":22094},{"type":"v1","addr":"172.21.15.111:6833","nonce":22094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":22094},{"type":"v1","addr":"172.21.15.111:6831","nonce":22094}]},"public_addr":"172.21.15.111:6827/22094","cluster_addr":"172.21.15.111:6829/22094","heartbeat_back_addr":"172.21.15.111:6833/22094","heartbeat_front_addr":"172.21.15.111:6831/22094","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.979052+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:01.989744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.035397+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.161796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.231661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.244487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.291696+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-12T08:41:02.247865+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.61:6833/502386216":"2022-07-13T08:51:17.697391+0000","172.21.15.61:6832/502386216":"2022-07-13T08:51:17.697391+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-12T09:51:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9e0053ad0 msgr2=0x7ff9e0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9e0053ad0 0x7ff9e0055f90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7ff9f400d000 tx=0x7ff9f4008040).stop 2022-07-12T09:51:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 msgr2=0x7ff9fc064080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc064080 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 rx=0x7ff9ec002f70 tx=0x7ff9ec006f40).stop 2022-07-12T09:51:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 shutdown_connections 2022-07-12T09:51:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7ff9e0053ad0 0x7ff9e0055f90 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff9fc142ab0 0x7ff9fc1445b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7ff9fc061460 0x7ff9fc1440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 --2- 172.21.15.61:0/2382746879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9fc060630 0x7ff9fc064080 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:52.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 >> 172.21.15.61:0/2382746879 conn(0x7ff9fc05f140 msgr2=0x7ff9fc0b50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:52.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 shutdown_connections 2022-07-12T09:51:52.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:52.303+0000 7ffa00c91700 1 -- 172.21.15.61:0/2382746879 wait complete. 2022-07-12T09:51:52.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:52.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:52.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:52.961 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:52.964 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:51:53.318 INFO:tasks.ceph:Scrubbing osd.0 2022-07-12T09:51:53.319 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:53.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.435+0000 7fa9db488700 1 Processor -- start 2022-07-12T09:51:53.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- start start 2022-07-12T09:51:53.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9d40ad8b0 0x7fa9d40aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d413f350 con 0x7fa9d413e120 2022-07-12T09:51:53.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d413f5f0 con 0x7fa9d405c8e0 2022-07-12T09:51:53.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d40b1c60 con 0x7fa9d40ad8b0 2022-07-12T09:51:53.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9d40ad8b0 0x7fa9d40aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9d40ad8b0 0x7fa9d40aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34624/0 (socket says 172.21.15.61:34624) 2022-07-12T09:51:53.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 -- 172.21.15.61:0/1640863092 learned_addr learned my addr 172.21.15.61:0/1640863092 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:53.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 -- 172.21.15.61:0/1640863092 >> v1:172.21.15.111:6789/0 conn(0x7fa9d405c8e0 legacy=0x7fa9d40ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 -- 172.21.15.61:0/1640863092 >> v1:172.21.15.61:6789/0 conn(0x7fa9d413e120 legacy=0x7fa9d413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 -- 172.21.15.61:0/1640863092 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d413f890 con 0x7fa9d40ad8b0 2022-07-12T09:51:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9484700 1 -- 172.21.15.61:0/1640863092 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263729456 0 0) 0x7fa9d413f350 con 0x7fa9d413e120 2022-07-12T09:51:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9c85700 1 --2- 172.21.15.61:0/1640863092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d40ad8b0 0x7fa9d40aef70 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7fa9c4002f70 tx=0x7fa9c4006eb0).ready entity=mon.0 client_cookie=90e5f758100de79c server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9484700 1 -- 172.21.15.61:0/1640863092 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c400d660 con 0x7fa9d40ad8b0 2022-07-12T09:51:53.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9484700 1 -- 172.21.15.61:0/1640863092 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c4000e90 con 0x7fa9d40ad8b0 2022-07-12T09:51:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9d9484700 1 -- 172.21.15.61:0/1640863092 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9c400d9b0 con 0x7fa9d40ad8b0 2022-07-12T09:51:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- 172.21.15.61:0/1640863092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d40ad8b0 msgr2=0x7fa9d40aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 --2- 172.21.15.61:0/1640863092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d40ad8b0 0x7fa9d40aef70 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7fa9c4002f70 tx=0x7fa9c4006eb0).stop 2022-07-12T09:51:53.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- 172.21.15.61:0/1640863092 shutdown_connections 2022-07-12T09:51:53.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 --2- 172.21.15.61:0/1640863092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d40ad8b0 0x7fa9d40aef70 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- 172.21.15.61:0/1640863092 >> 172.21.15.61:0/1640863092 conn(0x7fa9d40b5780 msgr2=0x7fa9d40b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- 172.21.15.61:0/1640863092 shutdown_connections 2022-07-12T09:51:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.439+0000 7fa9db488700 1 -- 172.21.15.61:0/1640863092 wait complete. 2022-07-12T09:51:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 Processor -- start 2022-07-12T09:51:53.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 -- start start 2022-07-12T09:51:53.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d405c8e0 0x7fa9d4060e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9d40ad8b0 0x7fa9d4139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d413e120 0x7fa9d413b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d413ada0 con 0x7fa9d413e120 2022-07-12T09:51:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d413a2f0 con 0x7fa9d405c8e0 2022-07-12T09:51:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9db488700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d4138b30 con 0x7fa9d40ad8b0 2022-07-12T09:51:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d413e120 0x7fa9d413b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d413e120 0x7fa9d413b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34632/0 (socket says 172.21.15.61:34632) 2022-07-12T09:51:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 -- 172.21.15.61:0/2189061494 learned_addr learned my addr 172.21.15.61:0/2189061494 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9da486700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d405c8e0 0x7fa9d4060e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9d9c85700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9d40ad8b0 0x7fa9d4139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 -- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9d40ad8b0 msgr2=0x7fa9d4139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fa9d40ad8b0 0x7fa9d4139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 -- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d405c8e0 msgr2=0x7fa9d4060e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa9d405c8e0 0x7fa9d4060e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 -- 172.21.15.61:0/2189061494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c400b1e0 con 0x7fa9d413e120 2022-07-12T09:51:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.443+0000 7fa9dac87700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d413e120 0x7fa9d413b190 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7fa9d4067230 tx=0x7fa9cc007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9cc00f470 con 0x7fa9d413e120 2022-07-12T09:51:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc00f5d0 con 0x7fa9d413e120 2022-07-12T09:51:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d4138dd0 con 0x7fa9d413e120 2022-07-12T09:51:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9cc00f970 con 0x7fa9d413e120 2022-07-12T09:51:53.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d41387b0 con 0x7fa9d413e120 2022-07-12T09:51:53.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa9d4065860 con 0x7fa9d413e120 2022-07-12T09:51:53.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa9cc019030 con 0x7fa9d413e120 2022-07-12T09:51:53.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9b40539b0 0x7fa9b4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa9cc06b050 con 0x7fa9d413e120 2022-07-12T09:51:53.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fa9b405a0a0 0x7fa9b405c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 --> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9b405cb80 con 0x7fa9b405a0a0 2022-07-12T09:51:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9cc06d6e0 con 0x7fa9d413e120 2022-07-12T09:51:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9da486700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9b40539b0 0x7fa9b4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.447+0000 7fa9d9c85700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fa9b405a0a0 0x7fa9b405c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.451+0000 7fa9d9c85700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fa9b405a0a0 0x7fa9b405c500 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.451+0000 7fa9da486700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9b40539b0 0x7fa9b4055e70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fa9d000d000 tx=0x7fa9d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.451+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== osd.0 v2:172.21.15.61:6800/21668 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fa9b405cb80 con 0x7fa9b405a0a0 2022-07-12T09:51:53.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 --> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa9d413aef0 con 0x7fa9b405a0a0 2022-07-12T09:51:53.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9c37fe700 1 -- 172.21.15.61:0/2189061494 <== osd.0 v2:172.21.15.61:6800/21668 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa9c4006a60 con 0x7fa9b405a0a0 2022-07-12T09:51:53.480 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:53.480 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:53.481 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:53.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fa9b405a0a0 msgr2=0x7fa9b405c500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:6800/21668,v1:172.21.15.61:6801/21668] conn(0x7fa9b405a0a0 0x7fa9b405c500 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9b40539b0 msgr2=0x7fa9b4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fa9b40539b0 0x7fa9b4055e70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fa9d000d000 tx=0x7fa9d0009040).stop 2022-07-12T09:51:53.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d413e120 msgr2=0x7fa9d413b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 --2- 172.21.15.61:0/2189061494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9d413e120 0x7fa9d413b190 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 rx=0x7fa9d4067230 tx=0x7fa9cc007ef0).stop 2022-07-12T09:51:53.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9dac87700 1 -- 172.21.15.61:0/2189061494 reap_dead start 2022-07-12T09:51:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 shutdown_connections 2022-07-12T09:51:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 >> 172.21.15.61:0/2189061494 conn(0x7fa9d40b5780 msgr2=0x7fa9d40ae400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 shutdown_connections 2022-07-12T09:51:53.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.467+0000 7fa9db488700 1 -- 172.21.15.61:0/2189061494 wait complete. 2022-07-12T09:51:53.486 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2022-07-12T09:51:53.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.599+0000 7f1828d13700 1 Processor -- start 2022-07-12T09:51:53.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.599+0000 7f1828d13700 1 -- start start 2022-07-12T09:51:53.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1828d13700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18240b6a40 0x7f1824143100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1828d13700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f18240678b0 0x7f18241435f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1828d13700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1824060110 0x7f1824143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1828d13700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824147100 con 0x7f18240b6a40 2022-07-12T09:51:53.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1828d13700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240a9c20 con 0x7f18240678b0 2022-07-12T09:51:53.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1828d13700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1824147560 con 0x7f1824060110 2022-07-12T09:51:53.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18240b6a40 0x7f1824143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f181bfff700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f18240678b0 0x7f18241435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18240b6a40 0x7f1824143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34638/0 (socket says 172.21.15.61:34638) 2022-07-12T09:51:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 -- 172.21.15.61:0/4153790251 learned_addr learned my addr 172.21.15.61:0/4153790251 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:53.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1822d9d700 1 --2- 172.21.15.61:0/4153790251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1824060110 0x7f1824143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 -- 172.21.15.61:0/4153790251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1824060110 msgr2=0x7f1824143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:53.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 --2- 172.21.15.61:0/4153790251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1824060110 0x7f1824143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 -- 172.21.15.61:0/4153790251 >> v2:172.21.15.111:3300/0 conn(0x7f18240678b0 msgr2=0x7f18241435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:53.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 --2- 172.21.15.61:0/4153790251 >> v2:172.21.15.111:3300/0 conn(0x7f18240678b0 0x7f18241435f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 -- 172.21.15.61:0/4153790251 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240a9980 con 0x7f18240b6a40 2022-07-12T09:51:53.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f181bfff700 1 --2- 172.21.15.61:0/4153790251 >> v2:172.21.15.111:3300/0 conn(0x7f18240678b0 0x7f18241435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T09:51:53.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f182259c700 1 --2- 172.21.15.61:0/4153790251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f1824143100 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f1824065310 tx=0x7f1814006eb0).ready entity=mon.0 client_cookie=a39de5f1185309a server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1821d9b700 1 -- 172.21.15.61:0/4153790251 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181400e610 con 0x7f18240b6a40 2022-07-12T09:51:53.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1821d9b700 1 -- 172.21.15.61:0/4153790251 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814000e90 con 0x7f18240b6a40 2022-07-12T09:51:53.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.603+0000 7f1821d9b700 1 -- 172.21.15.61:0/4153790251 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181400e9a0 con 0x7f18240b6a40 2022-07-12T09:51:53.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 -- 172.21.15.61:0/4153790251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 msgr2=0x7f1824143100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 --2- 172.21.15.61:0/4153790251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f1824143100 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 rx=0x7f1824065310 tx=0x7f1814006eb0).stop 2022-07-12T09:51:53.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 -- 172.21.15.61:0/4153790251 shutdown_connections 2022-07-12T09:51:53.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 --2- 172.21.15.61:0/4153790251 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1824060110 0x7f1824143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 --2- 172.21.15.61:0/4153790251 >> v2:172.21.15.111:3300/0 conn(0x7f18240678b0 0x7f18241435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 --2- 172.21.15.61:0/4153790251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f1824143100 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 -- 172.21.15.61:0/4153790251 >> 172.21.15.61:0/4153790251 conn(0x7f18240645c0 msgr2=0x7f18240b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:53.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 -- 172.21.15.61:0/4153790251 shutdown_connections 2022-07-12T09:51:53.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 -- 172.21.15.61:0/4153790251 wait complete. 2022-07-12T09:51:53.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 Processor -- start 2022-07-12T09:51:53.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.607+0000 7f1828d13700 1 -- start start 2022-07-12T09:51:53.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1824060110 0x7f1824151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f18240678b0 0x7f1824146270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f18241556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1824155280 con 0x7f18240b6a40 2022-07-12T09:51:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240ad9a0 con 0x7f1824060110 2022-07-12T09:51:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240adc40 con 0x7f18240678b0 2022-07-12T09:51:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f18241556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f18241556b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34648/0 (socket says 172.21.15.61:34648) 2022-07-12T09:51:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 -- 172.21.15.61:0/4064328343 learned_addr learned my addr 172.21.15.61:0/4064328343 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:53.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f182259c700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1824060110 0x7f1824151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f181bfff700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f18240678b0 0x7f1824146270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 -- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f18240678b0 msgr2=0x7f1824146270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f18240678b0 0x7f1824146270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 -- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1824060110 msgr2=0x7f1824151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1824060110 0x7f1824151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 -- 172.21.15.61:0/4064328343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c00b180 con 0x7f18240b6a40 2022-07-12T09:51:53.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1822d9d700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f18241556b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f181c00c000 tx=0x7f181c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c012600 con 0x7f18240b6a40 2022-07-12T09:51:53.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c012760 con 0x7f18240b6a40 2022-07-12T09:51:53.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181400b1e0 con 0x7f18240b6a40 2022-07-12T09:51:53.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c012ac0 con 0x7f18240b6a40 2022-07-12T09:51:53.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18240ad500 con 0x7f18240b6a40 2022-07-12T09:51:53.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.611+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1824057460 con 0x7f18240b6a40 2022-07-12T09:51:53.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.615+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f181c022070 con 0x7f18240b6a40 2022-07-12T09:51:53.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.615+0000 7f181b7fe700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1808053a40 0x7f1808055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.615+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f181c06c140 con 0x7f18240b6a40 2022-07-12T09:51:53.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.615+0000 7f182259c700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1808053a40 0x7f1808055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.615+0000 7f182259c700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1808053a40 0x7f1808055f00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f181401c000 tx=0x7f1814015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.619+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f181c043380 con 0x7f18240b6a40 2022-07-12T09:51:53.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.747+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f182406ac10 con 0x7f1808053a40 2022-07-12T09:51:53.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.747+0000 7f181b7fe700 1 -- 172.21.15.61:0/4064328343 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f182406ac10 con 0x7f1808053a40 2022-07-12T09:51:53.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1808053a40 msgr2=0x7f1808055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1808053a40 0x7f1808055f00 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f181401c000 tx=0x7f1814015000).stop 2022-07-12T09:51:53.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 msgr2=0x7f18241556b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f18241556b0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f181c00c000 tx=0x7f181c0091d0).stop 2022-07-12T09:51:53.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 shutdown_connections 2022-07-12T09:51:53.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18240b6a40 0x7f18241556b0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f18240678b0 0x7f1824146270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1808053a40 0x7f1808055f00 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 --2- 172.21.15.61:0/4064328343 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1824060110 0x7f1824151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 >> 172.21.15.61:0/4064328343 conn(0x7f18240645c0 msgr2=0x7f182405c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:53.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 shutdown_connections 2022-07-12T09:51:53.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.751+0000 7f1828d13700 1 -- 172.21.15.61:0/4064328343 wait complete. 2022-07-12T09:51:53.763 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 0 to deep-scrub 2022-07-12T09:51:53.769 INFO:tasks.ceph:Scrubbing osd.1 2022-07-12T09:51:53.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:53.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 Processor -- start 2022-07-12T09:51:53.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 -- start start 2022-07-12T09:51:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f21d80b5480 0x7f21d80b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d80afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80b3560 con 0x7f21d80b03d0 2022-07-12T09:51:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80b5b80 con 0x7f21d80b5480 2022-07-12T09:51:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21ddd32700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80b5e20 con 0x7f21d80b3730 2022-07-12T09:51:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f21d80b03d0 0x7f21d8142f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51106/0 (socket says 172.21.15.61:51106) 2022-07-12T09:51:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d7fff700 1 -- 172.21.15.61:0/3977041425 learned_addr learned my addr 172.21.15.61:0/3977041425 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d6ffd700 1 --2- 172.21.15.61:0/3977041425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d80afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d77fe700 1 --2- 172.21.15.61:0/3977041425 >> v2:172.21.15.111:3300/0 conn(0x7f21d80b5480 0x7f21d80b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d67fc700 1 -- 172.21.15.61:0/3977041425 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130613291 0 0) 0x7f21d80b3560 con 0x7f21d80b03d0 2022-07-12T09:51:53.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d67fc700 1 -- 172.21.15.61:0/3977041425 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b8003260 con 0x7f21d80b03d0 2022-07-12T09:51:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d6ffd700 1 -- 172.21.15.61:0/3977041425 >> v2:172.21.15.111:3300/0 conn(0x7f21d80b5480 msgr2=0x7f21d80b45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d6ffd700 1 --2- 172.21.15.61:0/3977041425 >> v2:172.21.15.111:3300/0 conn(0x7f21d80b5480 0x7f21d80b45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d6ffd700 1 -- 172.21.15.61:0/3977041425 >> v1:172.21.15.61:6789/0 conn(0x7f21d80b03d0 legacy=0x7f21d8142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d6ffd700 1 -- 172.21.15.61:0/3977041425 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80b0820 con 0x7f21d80b3730 2022-07-12T09:51:53.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d6ffd700 1 --2- 172.21.15.61:0/3977041425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d80afee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f21c00032f0 tx=0x7f21c00091d0).ready entity=mon.2 client_cookie=e4590d5793955e28 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d67fc700 1 -- 172.21.15.61:0/3977041425 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c000b600 con 0x7f21d80b3730 2022-07-12T09:51:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d67fc700 1 -- 172.21.15.61:0/3977041425 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c000b760 con 0x7f21d80b3730 2022-07-12T09:51:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.887+0000 7f21d67fc700 1 -- 172.21.15.61:0/3977041425 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c000b8d0 con 0x7f21d80b3730 2022-07-12T09:51:53.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 -- 172.21.15.61:0/3977041425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 msgr2=0x7f21d80afee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 --2- 172.21.15.61:0/3977041425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d80afee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f21c00032f0 tx=0x7f21c00091d0).stop 2022-07-12T09:51:53.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 -- 172.21.15.61:0/3977041425 shutdown_connections 2022-07-12T09:51:53.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 --2- 172.21.15.61:0/3977041425 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d80afee0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 --2- 172.21.15.61:0/3977041425 >> v2:172.21.15.111:3300/0 conn(0x7f21d80b5480 0x7f21d80b45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 -- 172.21.15.61:0/3977041425 >> 172.21.15.61:0/3977041425 conn(0x7f21d80b96b0 msgr2=0x7f21d80b9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:53.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 -- 172.21.15.61:0/3977041425 shutdown_connections 2022-07-12T09:51:53.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 -- 172.21.15.61:0/3977041425 wait complete. 2022-07-12T09:51:53.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 Processor -- start 2022-07-12T09:51:53.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.891+0000 7f21ddd32700 1 -- start start 2022-07-12T09:51:53.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21d80b03d0 0x7f21d81128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d8146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d80b5480 0x7f21d81499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d81471b0 con 0x7f21d80b03d0 2022-07-12T09:51:53.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8147450 con 0x7f21d80b5480 2022-07-12T09:51:53.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d81444a0 con 0x7f21d80b3730 2022-07-12T09:51:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21d80b03d0 0x7f21d81128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21d80b03d0 0x7f21d81128a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34658/0 (socket says 172.21.15.61:34658) 2022-07-12T09:51:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 -- 172.21.15.61:0/114467560 learned_addr learned my addr 172.21.15.61:0/114467560 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d6ffd700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d8146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d7fff700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d80b5480 0x7f21d81499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 -- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 msgr2=0x7f21d8146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f21d80b3730 0x7f21d8146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 -- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d80b5480 msgr2=0x7f21d81499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f21d80b5480 0x7f21d81499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 -- 172.21.15.61:0/114467560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0009470 con 0x7f21d80b03d0 2022-07-12T09:51:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d77fe700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21d80b03d0 0x7f21d81128a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f21c800c000 tx=0x7f21c8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c8011750 con 0x7f21d80b03d0 2022-07-12T09:51:53.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c80118b0 con 0x7f21d80b03d0 2022-07-12T09:51:53.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d8146260 con 0x7f21d80b03d0 2022-07-12T09:51:53.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21c8011c10 con 0x7f21d80b03d0 2022-07-12T09:51:53.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d8152140 con 0x7f21d80b03d0 2022-07-12T09:51:53.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.895+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21d8147030 con 0x7f21d80b03d0 2022-07-12T09:51:53.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f21c8011d70 con 0x7f21d80b03d0 2022-07-12T09:51:53.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21bc053920 0x7f21bc055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f21c806bf30 con 0x7f21d80b03d0 2022-07-12T09:51:53.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d6ffd700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21bc053920 0x7f21bc055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f21bc05a0b0 0x7f21bc05c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:53.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 --> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21bc05cb90 con 0x7f21bc05a0b0 2022-07-12T09:51:53.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21c806e560 con 0x7f21d80b03d0 2022-07-12T09:51:53.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d7fff700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f21bc05a0b0 0x7f21bc05c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:53.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d7fff700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f21bc05a0b0 0x7f21bc05c510 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d6ffd700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21bc053920 0x7f21bc055de0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f21c001b000 tx=0x7f21c0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:53.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.899+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== osd.1 v2:172.21.15.61:6808/21713 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f21bc05cb90 con 0x7f21bc05a0b0 2022-07-12T09:51:53.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.915+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 --> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f21d8057460 con 0x7f21bc05a0b0 2022-07-12T09:51:53.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.915+0000 7f21d4ff9700 1 -- 172.21.15.61:0/114467560 <== osd.1 v2:172.21.15.61:6808/21713 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f21d8057460 con 0x7f21bc05a0b0 2022-07-12T09:51:53.924 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:53.924 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:53.924 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:53.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f21bc05a0b0 msgr2=0x7f21bc05c510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:6808/21713,v1:172.21.15.61:6809/21713] conn(0x7f21bc05a0b0 0x7f21bc05c510 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:53.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21bc053920 msgr2=0x7f21bc055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f21bc053920 0x7f21bc055de0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f21c001b000 tx=0x7f21c0016000).stop 2022-07-12T09:51:53.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21d80b03d0 msgr2=0x7f21d81128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 --2- 172.21.15.61:0/114467560 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21d80b03d0 0x7f21d81128a0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f21c800c000 tx=0x7f21c8006df0).stop 2022-07-12T09:51:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21d7fff700 1 -- 172.21.15.61:0/114467560 reap_dead start 2022-07-12T09:51:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 shutdown_connections 2022-07-12T09:51:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 >> 172.21.15.61:0/114467560 conn(0x7f21d80b96b0 msgr2=0x7f21d80b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 shutdown_connections 2022-07-12T09:51:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:53.919+0000 7f21ddd32700 1 -- 172.21.15.61:0/114467560 wait complete. 2022-07-12T09:51:53.935 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2022-07-12T09:51:54.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.039+0000 7f96bef0f700 1 Processor -- start 2022-07-12T09:51:54.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.039+0000 7f96bef0f700 1 -- start start 2022-07-12T09:51:54.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f96b805c050 0x7f96b805d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b805c430 con 0x7f96b8144730 2022-07-12T09:51:54.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b80b3e30 con 0x7f96b80b3800 2022-07-12T09:51:54.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b80b40d0 con 0x7f96b805c050 2022-07-12T09:51:54.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bd4ac700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f96b8144730 0x7f96b80acd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51122/0 (socket says 172.21.15.61:51122) 2022-07-12T09:51:54.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bd4ac700 1 -- 172.21.15.61:0/536096518 learned_addr learned my addr 172.21.15.61:0/536096518 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:54.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bccab700 1 --2- 172.21.15.61:0/536096518 >> v2:172.21.15.111:3300/0 conn(0x7f96b805c050 0x7f96b805d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96b77fe700 1 -- 172.21.15.61:0/536096518 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976842742 0 0) 0x7f96b805c430 con 0x7f96b8144730 2022-07-12T09:51:54.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96b77fe700 1 -- 172.21.15.61:0/536096518 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698003260 con 0x7f96b8144730 2022-07-12T09:51:54.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bccab700 1 -- 172.21.15.61:0/536096518 >> v1:172.21.15.111:6789/0 conn(0x7f96b80b3800 legacy=0x7f96b8144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96b77fe700 1 -- 172.21.15.61:0/536096518 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550683129 0 0) 0x7f96b80b3e30 con 0x7f96b80b3800 2022-07-12T09:51:54.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bccab700 1 -- 172.21.15.61:0/536096518 >> v1:172.21.15.61:6789/0 conn(0x7f96b8144730 legacy=0x7f96b80acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bccab700 1 -- 172.21.15.61:0/536096518 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b80b4370 con 0x7f96b805c050 2022-07-12T09:51:54.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bccab700 1 --2- 172.21.15.61:0/536096518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b805c050 0x7f96b805d0e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f96b80b36a0 tx=0x7f96a80091d0).ready entity=mon.1 client_cookie=3ab57ef7d8705b35 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96b77fe700 1 -- 172.21.15.61:0/536096518 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96a800b680 con 0x7f96b805c050 2022-07-12T09:51:54.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96b77fe700 1 -- 172.21.15.61:0/536096518 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a800b7e0 con 0x7f96b805c050 2022-07-12T09:51:54.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96b77fe700 1 -- 172.21.15.61:0/536096518 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96a800b950 con 0x7f96b805c050 2022-07-12T09:51:54.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- 172.21.15.61:0/536096518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b805c050 msgr2=0x7f96b805d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 --2- 172.21.15.61:0/536096518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b805c050 0x7f96b805d0e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f96b80b36a0 tx=0x7f96a80091d0).stop 2022-07-12T09:51:54.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- 172.21.15.61:0/536096518 shutdown_connections 2022-07-12T09:51:54.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 --2- 172.21.15.61:0/536096518 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b805c050 0x7f96b805d0e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- 172.21.15.61:0/536096518 >> 172.21.15.61:0/536096518 conn(0x7f96b8064b90 msgr2=0x7f96b8064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- 172.21.15.61:0/536096518 shutdown_connections 2022-07-12T09:51:54.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.043+0000 7f96bef0f700 1 -- 172.21.15.61:0/536096518 wait complete. 2022-07-12T09:51:54.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 Processor -- start 2022-07-12T09:51:54.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 -- start start 2022-07-12T09:51:54.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f96b805c050 0x7f96b814f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b80b3800 0x7f96b8143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 0x7f96b813c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b80b00a0 con 0x7f96b8144730 2022-07-12T09:51:54.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b814fe20 con 0x7f96b80b3800 2022-07-12T09:51:54.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b81426e0 con 0x7f96b805c050 2022-07-12T09:51:54.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 0x7f96b813c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 0x7f96b813c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34678/0 (socket says 172.21.15.61:34678) 2022-07-12T09:51:54.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 -- 172.21.15.61:0/4198693813 learned_addr learned my addr 172.21.15.61:0/4198693813 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96b7fff700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b80b3800 0x7f96b8143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bccab700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f96b805c050 0x7f96b814f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 -- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f96b805c050 msgr2=0x7f96b814f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f96b805c050 0x7f96b814f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 -- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b80b3800 msgr2=0x7f96b8143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b80b3800 0x7f96b8143760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 -- 172.21.15.61:0/4198693813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8009470 con 0x7f96b8144730 2022-07-12T09:51:54.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bd4ac700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 0x7f96b813c710 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f96b0001110 tx=0x7f96b0007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b000cda0 con 0x7f96b8144730 2022-07-12T09:51:54.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b00070d0 con 0x7f96b8144730 2022-07-12T09:51:54.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b8142980 con 0x7f96b8144730 2022-07-12T09:51:54.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b0007430 con 0x7f96b8144730 2022-07-12T09:51:54.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.047+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b8142ce0 con 0x7f96b8144730 2022-07-12T09:51:54.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.051+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f96b001a060 con 0x7f96b8144730 2022-07-12T09:51:54.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.051+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b8057460 con 0x7f96b8144730 2022-07-12T09:51:54.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.051+0000 7f96b5ffb700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f969c053a40 0x7f969c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.051+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f96b006b2e0 con 0x7f96b8144730 2022-07-12T09:51:54.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.051+0000 7f96bccab700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f969c053a40 0x7f969c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.051+0000 7f96bccab700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f969c053a40 0x7f969c055f00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f96a801b000 tx=0x7f96a8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.055+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f96b0042410 con 0x7f96b8144730 2022-07-12T09:51:54.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.183+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f96b8143580 con 0x7f969c053a40 2022-07-12T09:51:54.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.183+0000 7f96b5ffb700 1 -- 172.21.15.61:0/4198693813 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f96a800bac0 con 0x7f969c053a40 2022-07-12T09:51:54.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f969c053a40 msgr2=0x7f969c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f969c053a40 0x7f969c055f00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f96a801b000 tx=0x7f96a8016000).stop 2022-07-12T09:51:54.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 msgr2=0x7f96b813c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 0x7f96b813c710 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7f96b0001110 tx=0x7f96b0007a80).stop 2022-07-12T09:51:54.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 shutdown_connections 2022-07-12T09:51:54.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96b8144730 0x7f96b813c710 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f969c053a40 0x7f969c055f00 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96b80b3800 0x7f96b8143760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 --2- 172.21.15.61:0/4198693813 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f96b805c050 0x7f96b814f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 >> 172.21.15.61:0/4198693813 conn(0x7f96b8064b90 msgr2=0x7f96b8141240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 shutdown_connections 2022-07-12T09:51:54.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.187+0000 7f96bef0f700 1 -- 172.21.15.61:0/4198693813 wait complete. 2022-07-12T09:51:54.195 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 1 to deep-scrub 2022-07-12T09:51:54.204 INFO:tasks.ceph:Scrubbing osd.2 2022-07-12T09:51:54.206 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:54.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.319+0000 7f0e1359e700 1 Processor -- start 2022-07-12T09:51:54.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.319+0000 7f0e1359e700 1 -- start start 2022-07-12T09:51:54.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.319+0000 7f0e1359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e140b26a0 con 0x7f0e14142f40 2022-07-12T09:51:54.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e140b7ee0 con 0x7f0e140628e0 2022-07-12T09:51:54.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140ae2e0 con 0x7f0e140b7b00 2022-07-12T09:51:54.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e12d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0e14142f40 0x7f0e141433d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51136/0 (socket says 172.21.15.61:51136) 2022-07-12T09:51:54.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e12d9d700 1 -- 172.21.15.61:0/1126714940 learned_addr learned my addr 172.21.15.61:0/1126714940 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:54.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1259c700 1 --2- 172.21.15.61:0/1126714940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887680364 0 0) 0x7f0e140b26a0 con 0x7f0e14142f40 2022-07-12T09:51:54.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0df4003260 con 0x7f0e14142f40 2022-07-12T09:51:54.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381320684 0 0) 0x7f0e140b7ee0 con 0x7f0e140628e0 2022-07-12T09:51:54.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e140b26a0 con 0x7f0e140628e0 2022-07-12T09:51:54.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1259c700 1 -- 172.21.15.61:0/1126714940 >> v1:172.21.15.111:6789/0 conn(0x7f0e140628e0 legacy=0x7f0e140b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1259c700 1 -- 172.21.15.61:0/1126714940 >> v1:172.21.15.61:6789/0 conn(0x7f0e14142f40 legacy=0x7f0e141433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1259c700 1 -- 172.21.15.61:0/1126714940 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e140b2d30 con 0x7f0e140b7b00 2022-07-12T09:51:54.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1259c700 1 --2- 172.21.15.61:0/1126714940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140ac870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f0e04002f70 tx=0x7f0e04006df0).ready entity=mon.2 client_cookie=5da3e862714cfa9f server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0400d5c0 con 0x7f0e140b7b00 2022-07-12T09:51:54.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04000e90 con 0x7f0e140b7b00 2022-07-12T09:51:54.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1159a700 1 -- 172.21.15.61:0/1126714940 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0400d910 con 0x7f0e140b7b00 2022-07-12T09:51:54.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- 172.21.15.61:0/1126714940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 msgr2=0x7f0e140ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 --2- 172.21.15.61:0/1126714940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140ac870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f0e04002f70 tx=0x7f0e04006df0).stop 2022-07-12T09:51:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- 172.21.15.61:0/1126714940 shutdown_connections 2022-07-12T09:51:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 --2- 172.21.15.61:0/1126714940 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140ac870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- 172.21.15.61:0/1126714940 >> 172.21.15.61:0/1126714940 conn(0x7f0e14065550 msgr2=0x7f0e140b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- 172.21.15.61:0/1126714940 shutdown_connections 2022-07-12T09:51:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.323+0000 7f0e1359e700 1 -- 172.21.15.61:0/1126714940 wait complete. 2022-07-12T09:51:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 Processor -- start 2022-07-12T09:51:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 -- start start 2022-07-12T09:51:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e140628e0 0x7f0e1414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e14142f40 0x7f0e140b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140b0110 con 0x7f0e14142f40 2022-07-12T09:51:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140b03b0 con 0x7f0e140628e0 2022-07-12T09:51:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140b0650 con 0x7f0e140b7b00 2022-07-12T09:51:54.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e14142f40 0x7f0e140b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e14142f40 0x7f0e140b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34692/0 (socket says 172.21.15.61:34692) 2022-07-12T09:51:54.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 -- 172.21.15.61:0/2447944232 learned_addr learned my addr 172.21.15.61:0/2447944232 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e11d9b700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1259c700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e140628e0 0x7f0e1414f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 -- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 msgr2=0x7f0e140aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0e140b7b00 0x7f0e140aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 -- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e140628e0 msgr2=0x7f0e1414f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0e140628e0 0x7f0e1414f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 -- 172.21.15.61:0/2447944232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0400b180 con 0x7f0e14142f40 2022-07-12T09:51:54.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e12d9d700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e14142f40 0x7f0e140b0af0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f0e0c001110 tx=0x7f0e0c007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0c00cdb0 con 0x7f0e14142f40 2022-07-12T09:51:54.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0c0070d0 con 0x7f0e14142f40 2022-07-12T09:51:54.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1409c000 con 0x7f0e14142f40 2022-07-12T09:51:54.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0c007430 con 0x7f0e14142f40 2022-07-12T09:51:54.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.327+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e140afce0 con 0x7f0e14142f40 2022-07-12T09:51:54.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e140b0810 con 0x7f0e14142f40 2022-07-12T09:51:54.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0e0c01a060 con 0x7f0e14142f40 2022-07-12T09:51:54.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e0b7fe700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0df80539b0 0x7f0df8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0e0c06b2e0 con 0x7f0e14142f40 2022-07-12T09:51:54.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e0b7fe700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7f0df805a0a0 0x7f0df805c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 --> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0df805cb80 con 0x7f0df805a0a0 2022-07-12T09:51:54.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e0c06d8a0 con 0x7f0e14142f40 2022-07-12T09:51:54.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e1259c700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0df80539b0 0x7f0df8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.331+0000 7f0e11d9b700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7f0df805a0a0 0x7f0df805c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.335+0000 7f0e11d9b700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7f0df805a0a0 0x7f0df805c500 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.335+0000 7f0e1259c700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0df80539b0 0x7f0df8055e70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f0e04018000 tx=0x7f0e04014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.335+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== osd.2 v2:172.21.15.61:6816/21871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f0df805cb80 con 0x7f0df805a0a0 2022-07-12T09:51:54.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 --> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0e1409a840 con 0x7f0df805a0a0 2022-07-12T09:51:54.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e0b7fe700 1 -- 172.21.15.61:0/2447944232 <== osd.2 v2:172.21.15.61:6816/21871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0e1409a840 con 0x7f0df805a0a0 2022-07-12T09:51:54.370 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:54.371 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:54.371 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:54.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7f0df805a0a0 msgr2=0x7f0df805c500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:6816/21871,v1:172.21.15.61:6817/21871] conn(0x7f0df805a0a0 0x7f0df805c500 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0df80539b0 msgr2=0x7f0df8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0df80539b0 0x7f0df8055e70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f0e04018000 tx=0x7f0e04014000).stop 2022-07-12T09:51:54.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e14142f40 msgr2=0x7f0e140b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 --2- 172.21.15.61:0/2447944232 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0e14142f40 0x7f0e140b0af0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f0e0c001110 tx=0x7f0e0c007a80).stop 2022-07-12T09:51:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e12d9d700 1 -- 172.21.15.61:0/2447944232 reap_dead start 2022-07-12T09:51:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 shutdown_connections 2022-07-12T09:51:54.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 >> 172.21.15.61:0/2447944232 conn(0x7f0e14065550 msgr2=0x7f0e140ac3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 shutdown_connections 2022-07-12T09:51:54.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.351+0000 7f0e1359e700 1 -- 172.21.15.61:0/2447944232 wait complete. 2022-07-12T09:51:54.379 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2022-07-12T09:51:54.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.487+0000 7f200ba8c700 1 Processor -- start 2022-07-12T09:51:54.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.487+0000 7f200ba8c700 1 -- start start 2022-07-12T09:51:54.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f20040b1f80 0x7f20040ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20040abdf0 0x7f200405bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f200405c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200405d480 con 0x7f20040abdf0 2022-07-12T09:51:54.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20040b16d0 con 0x7f20040b1f80 2022-07-12T09:51:54.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20040b0fc0 con 0x7f20040afc70 2022-07-12T09:51:54.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20040abdf0 0x7f200405bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20040abdf0 0x7f200405bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34698/0 (socket says 172.21.15.61:34698) 2022-07-12T09:51:54.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 -- 172.21.15.61:0/1865571797 learned_addr learned my addr 172.21.15.61:0/1865571797 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200a029700 1 --2- 172.21.15.61:0/1865571797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f200405c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009828700 1 --2- 172.21.15.61:0/1865571797 >> v2:172.21.15.111:3300/0 conn(0x7f20040b1f80 0x7f20040ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 -- 172.21.15.61:0/1865571797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 msgr2=0x7f200405c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:54.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 --2- 172.21.15.61:0/1865571797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f200405c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 -- 172.21.15.61:0/1865571797 >> v2:172.21.15.111:3300/0 conn(0x7f20040b1f80 msgr2=0x7f20040ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:54.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 --2- 172.21.15.61:0/1865571797 >> v2:172.21.15.111:3300/0 conn(0x7f20040b1f80 0x7f20040ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 -- 172.21.15.61:0/1865571797 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040b2360 con 0x7f20040abdf0 2022-07-12T09:51:54.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009828700 1 --2- 172.21.15.61:0/1865571797 >> v2:172.21.15.111:3300/0 conn(0x7f20040b1f80 0x7f20040ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:54.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2009027700 1 --2- 172.21.15.61:0/1865571797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200405bd10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f1ff4002f70 tx=0x7f1ff4006eb0).ready entity=mon.0 client_cookie=a69aaebc50236e6d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2008826700 1 -- 172.21.15.61:0/1865571797 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ff400d570 con 0x7f20040abdf0 2022-07-12T09:51:54.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2008826700 1 -- 172.21.15.61:0/1865571797 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4000e90 con 0x7f20040abdf0 2022-07-12T09:51:54.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f2008826700 1 -- 172.21.15.61:0/1865571797 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ff400d900 con 0x7f20040abdf0 2022-07-12T09:51:54.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- 172.21.15.61:0/1865571797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 msgr2=0x7f200405bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- 172.21.15.61:0/1865571797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200405bd10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f1ff4002f70 tx=0x7f1ff4006eb0).stop 2022-07-12T09:51:54.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- 172.21.15.61:0/1865571797 shutdown_connections 2022-07-12T09:51:54.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- 172.21.15.61:0/1865571797 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f200405c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- 172.21.15.61:0/1865571797 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200405bd10 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 --2- 172.21.15.61:0/1865571797 >> v2:172.21.15.111:3300/0 conn(0x7f20040b1f80 0x7f20040ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- 172.21.15.61:0/1865571797 >> 172.21.15.61:0/1865571797 conn(0x7f2004062c00 msgr2=0x7f2004062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- 172.21.15.61:0/1865571797 shutdown_connections 2022-07-12T09:51:54.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.491+0000 7f200ba8c700 1 -- 172.21.15.61:0/1865571797 wait complete. 2022-07-12T09:51:54.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 Processor -- start 2022-07-12T09:51:54.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 -- start start 2022-07-12T09:51:54.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200413e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f2004145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20040b1f80 0x7f2004148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200413f110 con 0x7f20040abdf0 2022-07-12T09:51:54.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2004065e80 con 0x7f20040b1f80 2022-07-12T09:51:54.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200ba8c700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20041449c0 con 0x7f20040afc70 2022-07-12T09:51:54.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200413e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200413e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34704/0 (socket says 172.21.15.61:34704) 2022-07-12T09:51:54.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 -- 172.21.15.61:0/513122918 learned_addr learned my addr 172.21.15.61:0/513122918 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f200a029700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20040b1f80 0x7f2004148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009027700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f2004145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 -- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 msgr2=0x7f2004145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f2004145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 -- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20040b1f80 msgr2=0x7f2004148b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20040b1f80 0x7f2004148b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 -- 172.21.15.61:0/513122918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff400b1e0 con 0x7f20040abdf0 2022-07-12T09:51:54.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009027700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f2004145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:54.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f2009828700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200413e5f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f20041447c0 tx=0x7f2000008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.495+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2000011780 con 0x7f20040abdf0 2022-07-12T09:51:54.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20000118e0 con 0x7f20040abdf0 2022-07-12T09:51:54.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2000011c40 con 0x7f20040abdf0 2022-07-12T09:51:54.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2004144c60 con 0x7f20040abdf0 2022-07-12T09:51:54.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20041455f0 con 0x7f20040abdf0 2022-07-12T09:51:54.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2004057460 con 0x7f20040abdf0 2022-07-12T09:51:54.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2000012420 con 0x7f20040abdf0 2022-07-12T09:51:54.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f1ff2ffd700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1fe4053af0 0x7f1fe4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f200006c3d0 con 0x7f20040abdf0 2022-07-12T09:51:54.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.499+0000 7f2009027700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1fe4053af0 0x7f1fe4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.503+0000 7f2009027700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1fe4053af0 0x7f1fe4055fb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f1ff401a000 tx=0x7f1ff4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.507+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20000435b0 con 0x7f20040abdf0 2022-07-12T09:51:54.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.635+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2004144df0 con 0x7f1fe4053af0 2022-07-12T09:51:54.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f1ff2ffd700 1 -- 172.21.15.61:0/513122918 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2004144df0 con 0x7f1fe4053af0 2022-07-12T09:51:54.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1fe4053af0 msgr2=0x7f1fe4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1fe4053af0 0x7f1fe4055fb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f1ff401a000 tx=0x7f1ff4015000).stop 2022-07-12T09:51:54.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 msgr2=0x7f200413e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200413e5f0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7f20041447c0 tx=0x7f2000008040).stop 2022-07-12T09:51:54.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 shutdown_connections 2022-07-12T09:51:54.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1fe4053af0 0x7f1fe4055fb0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20040b1f80 0x7f2004148b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f20040afc70 0x7f2004145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 --2- 172.21.15.61:0/513122918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20040abdf0 0x7f200413e5f0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 >> 172.21.15.61:0/513122918 conn(0x7f2004062c00 msgr2=0x7f20040b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 shutdown_connections 2022-07-12T09:51:54.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.639+0000 7f200ba8c700 1 -- 172.21.15.61:0/513122918 wait complete. 2022-07-12T09:51:54.645 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 2 to deep-scrub 2022-07-12T09:51:54.652 INFO:tasks.ceph:Scrubbing osd.3 2022-07-12T09:51:54.653 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fb59e700 1 Processor -- start 2022-07-12T09:51:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fb59e700 1 -- start start 2022-07-12T09:51:54.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fb59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc7fc064300 0x7fc7fc0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc0adb80 con 0x7fc7fc142a20 2022-07-12T09:51:54.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fb59e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7fc0b03d0 con 0x7fc7fc0ad7a0 2022-07-12T09:51:54.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fb59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7fc146ad0 con 0x7fc7fc064300 2022-07-12T09:51:54.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc7fc064300 0x7fc7fc0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc7fc064300 0x7fc7fc0b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34716/0 (socket says 172.21.15.61:34716) 2022-07-12T09:51:54.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 -- 172.21.15.61:0/1180790798 learned_addr learned my addr 172.21.15.61:0/1180790798 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7f9d9b700 1 --1- 172.21.15.61:0/1180790798 >> v1:172.21.15.61:6789/0 conn(0x7fc7fc142a20 0x7fc7fc141990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51164/0 (socket says 172.21.15.61:51164) 2022-07-12T09:51:54.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 -- 172.21.15.61:0/1180790798 >> v1:172.21.15.111:6789/0 conn(0x7fc7fc0ad7a0 legacy=0x7fc7fc0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 -- 172.21.15.61:0/1180790798 >> v1:172.21.15.61:6789/0 conn(0x7fc7fc142a20 legacy=0x7fc7fc141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 -- 172.21.15.61:0/1180790798 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7fc146d70 con 0x7fc7fc064300 2022-07-12T09:51:54.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.767+0000 7fc7fad9d700 1 --2- 172.21.15.61:0/1180790798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc064300 0x7fc7fc0b0ed0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7fc7f40032f0 tx=0x7fc7f4009290).ready entity=mon.0 client_cookie=23ff4179064a8033 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7f959a700 1 -- 172.21.15.61:0/1180790798 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7f400b8a0 con 0x7fc7fc064300 2022-07-12T09:51:54.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7f959a700 1 -- 172.21.15.61:0/1180790798 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f400ba00 con 0x7fc7fc064300 2022-07-12T09:51:54.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7f959a700 1 -- 172.21.15.61:0/1180790798 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7f400bb70 con 0x7fc7fc064300 2022-07-12T09:51:54.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 -- 172.21.15.61:0/1180790798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc064300 msgr2=0x7fc7fc0b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 --2- 172.21.15.61:0/1180790798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc064300 0x7fc7fc0b0ed0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 rx=0x7fc7f40032f0 tx=0x7fc7f4009290).stop 2022-07-12T09:51:54.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 -- 172.21.15.61:0/1180790798 shutdown_connections 2022-07-12T09:51:54.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 --2- 172.21.15.61:0/1180790798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc064300 0x7fc7fc0b0ed0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 -- 172.21.15.61:0/1180790798 >> 172.21.15.61:0/1180790798 conn(0x7fc7fc061e70 msgr2=0x7fc7fc063a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 -- 172.21.15.61:0/1180790798 shutdown_connections 2022-07-12T09:51:54.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 -- 172.21.15.61:0/1180790798 wait complete. 2022-07-12T09:51:54.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 Processor -- start 2022-07-12T09:51:54.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.771+0000 7fc7fb59e700 1 -- start start 2022-07-12T09:51:54.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc7fc064300 0x7fc7fc1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7fc0ad7a0 0x7fc7fc065870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc142a20 0x7fc7fc147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc064a50 con 0x7fc7fc142a20 2022-07-12T09:51:54.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc064c80 con 0x7fc7fc0ad7a0 2022-07-12T09:51:54.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7fc064f20 con 0x7fc7fc064300 2022-07-12T09:51:54.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc142a20 0x7fc7fc147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc142a20 0x7fc7fc147680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34722/0 (socket says 172.21.15.61:34722) 2022-07-12T09:51:54.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 -- 172.21.15.61:0/3613785679 learned_addr learned my addr 172.21.15.61:0/3613785679 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fa59c700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc7fc064300 0x7fc7fc1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7f9d9b700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7fc0ad7a0 0x7fc7fc065870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 -- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc7fc064300 msgr2=0x7fc7fc1554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fc7fc064300 0x7fc7fc1554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 -- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7fc0ad7a0 msgr2=0x7fc7fc065870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7fc0ad7a0 0x7fc7fc065870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 -- 172.21.15.61:0/3613785679 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f4009530 con 0x7fc7fc142a20 2022-07-12T09:51:54.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fad9d700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc142a20 0x7fc7fc147680 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7fc7f40032f0 tx=0x7fc7f4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7f401f3f0 con 0x7fc7fc142a20 2022-07-12T09:51:54.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7fc145f00 con 0x7fc7fc142a20 2022-07-12T09:51:54.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f401f550 con 0x7fc7fc142a20 2022-07-12T09:51:54.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7fc113e60 con 0x7fc7fc142a20 2022-07-12T09:51:54.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc7f401f8b0 con 0x7fc7fc142a20 2022-07-12T09:51:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc7fc0650e0 con 0x7fc7fc142a20 2022-07-12T09:51:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc7f4026030 con 0x7fc7fc142a20 2022-07-12T09:51:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.775+0000 7fc7eb7fe700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc7d8053920 0x7fc7d8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc7f406ce10 con 0x7fc7fc142a20 2022-07-12T09:51:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7eb7fe700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fc7d805a0b0 0x7fc7d805c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 --> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7d805cb90 con 0x7fc7d805a0b0 2022-07-12T09:51:54.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7f406f480 con 0x7fc7fc142a20 2022-07-12T09:51:54.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7fa59c700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc7d8053920 0x7fc7d8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7f9d9b700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fc7d805a0b0 0x7fc7d805c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7fa59c700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc7d8053920 0x7fc7d8055de0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fc7f000b000 tx=0x7fc7f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7f9d9b700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fc7d805a0b0 0x7fc7d805c510 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.779+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== osd.3 v2:172.21.15.61:6824/21976 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fc7d805cb90 con 0x7fc7d805a0b0 2022-07-12T09:51:54.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.795+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 --> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc7fc057460 con 0x7fc7d805a0b0 2022-07-12T09:51:54.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.795+0000 7fc7eb7fe700 1 -- 172.21.15.61:0/3613785679 <== osd.3 v2:172.21.15.61:6824/21976 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc7fc057460 con 0x7fc7d805a0b0 2022-07-12T09:51:54.809 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:54.809 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:54.810 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:54.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.795+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fc7d805a0b0 msgr2=0x7fc7d805c510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.795+0000 7fc7fb59e700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:6824/21976,v1:172.21.15.61:6825/21976] conn(0x7fc7d805a0b0 0x7fc7d805c510 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc7d8053920 msgr2=0x7fc7d8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fc7d8053920 0x7fc7d8055de0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7fc7f000b000 tx=0x7fc7f0009040).stop 2022-07-12T09:51:54.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc142a20 msgr2=0x7fc7fc147680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 --2- 172.21.15.61:0/3613785679 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc7fc142a20 0x7fc7fc147680 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7fc7f40032f0 tx=0x7fc7f4016000).stop 2022-07-12T09:51:54.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fad9d700 1 -- 172.21.15.61:0/3613785679 reap_dead start 2022-07-12T09:51:54.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 shutdown_connections 2022-07-12T09:51:54.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 >> 172.21.15.61:0/3613785679 conn(0x7fc7fc061e70 msgr2=0x7fc7fc05fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 shutdown_connections 2022-07-12T09:51:54.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.799+0000 7fc7fb59e700 1 -- 172.21.15.61:0/3613785679 wait complete. 2022-07-12T09:51:54.816 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2022-07-12T09:51:54.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 Processor -- start 2022-07-12T09:51:54.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 -- start start 2022-07-12T09:51:54.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f0ad40b6ac0 0x7f0ad4146220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0ad405f550 0x7f0ad4060430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40b6ea0 con 0x7f0ad41420b0 2022-07-12T09:51:54.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad4146710 con 0x7f0ad405f550 2022-07-12T09:51:54.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad359e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad4060200 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad2d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0ad41420b0 0x7f0ad4142510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51182/0 (socket says 172.21.15.61:51182) 2022-07-12T09:51:54.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad2d9d700 1 -- 172.21.15.61:0/3342310583 learned_addr learned my addr 172.21.15.61:0/3342310583 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:54.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad1d9b700 1 --2- 172.21.15.61:0/3342310583 >> v2:172.21.15.61:3300/0 conn(0x7f0ad405f550 0x7f0ad4060430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad259c700 1 --2- 172.21.15.61:0/3342310583 >> v2:172.21.15.111:3300/0 conn(0x7f0ad40b6ac0 0x7f0ad4146220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad159a700 1 -- 172.21.15.61:0/3342310583 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384471381 0 0) 0x7f0ad40b6ea0 con 0x7f0ad41420b0 2022-07-12T09:51:54.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad159a700 1 -- 172.21.15.61:0/3342310583 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab4003260 con 0x7f0ad41420b0 2022-07-12T09:51:54.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/3342310583 >> v2:172.21.15.111:3300/0 conn(0x7f0ad40b6ac0 msgr2=0x7f0ad4146220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:54.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad1d9b700 1 --2- 172.21.15.61:0/3342310583 >> v2:172.21.15.111:3300/0 conn(0x7f0ad40b6ac0 0x7f0ad4146220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/3342310583 >> v1:172.21.15.61:6789/0 conn(0x7f0ad41420b0 legacy=0x7f0ad4142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/3342310583 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40b1360 con 0x7f0ad405f550 2022-07-12T09:51:54.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad1d9b700 1 --2- 172.21.15.61:0/3342310583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad405f550 0x7f0ad4060430 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f0abc002f70 tx=0x7f0abc006df0).ready entity=mon.0 client_cookie=1262106f7ec2233d server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.923+0000 7f0ad159a700 1 -- 172.21.15.61:0/3342310583 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc00d4f0 con 0x7f0ad405f550 2022-07-12T09:51:54.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad159a700 1 -- 172.21.15.61:0/3342310583 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc000e90 con 0x7f0ad405f550 2022-07-12T09:51:54.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad159a700 1 -- 172.21.15.61:0/3342310583 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc00d840 con 0x7f0ad405f550 2022-07-12T09:51:54.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- 172.21.15.61:0/3342310583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad405f550 msgr2=0x7f0ad4060430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 --2- 172.21.15.61:0/3342310583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad405f550 0x7f0ad4060430 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7f0abc002f70 tx=0x7f0abc006df0).stop 2022-07-12T09:51:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- 172.21.15.61:0/3342310583 shutdown_connections 2022-07-12T09:51:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 --2- 172.21.15.61:0/3342310583 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad405f550 0x7f0ad4060430 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 --2- 172.21.15.61:0/3342310583 >> v2:172.21.15.111:3300/0 conn(0x7f0ad40b6ac0 0x7f0ad4146220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- 172.21.15.61:0/3342310583 >> 172.21.15.61:0/3342310583 conn(0x7f0ad4064570 msgr2=0x7f0ad40b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:54.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- 172.21.15.61:0/3342310583 shutdown_connections 2022-07-12T09:51:54.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- 172.21.15.61:0/3342310583 wait complete. 2022-07-12T09:51:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 Processor -- start 2022-07-12T09:51:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- start start 2022-07-12T09:51:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad405f550 0x7f0ad4151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 0x7f0ad4148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41420b0 0x7f0ad41450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4149530 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41497d0 con 0x7f0ad41420b0 2022-07-12T09:51:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4149a70 con 0x7f0ad405f550 2022-07-12T09:51:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 0x7f0ad4148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 0x7f0ad4148890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34736/0 (socket says 172.21.15.61:34736) 2022-07-12T09:51:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/1308900980 learned_addr learned my addr 172.21.15.61:0/1308900980 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad2d9d700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41420b0 0x7f0ad41450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad259c700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad405f550 0x7f0ad4151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad405f550 msgr2=0x7f0ad4151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad405f550 0x7f0ad4151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41420b0 msgr2=0x7f0ad41450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41420b0 0x7f0ad41450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 -- 172.21.15.61:0/1308900980 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc00b180 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad2d9d700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41420b0 0x7f0ad41450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.927+0000 7f0ad1d9b700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 0x7f0ad4148890 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f0abc002f70 tx=0x7f0abc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc00de60 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc01e400 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad414a020 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0abc01e760 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41455a0 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0abc025030 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad4149320 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0acb7fe700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0ab8053a40 0x7f0ab8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0abc06c800 con 0x7f0ad40b6ac0 2022-07-12T09:51:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0ad259c700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0ab8053a40 0x7f0ab8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:54.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.931+0000 7f0ad259c700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0ab8053a40 0x7f0ab8055f00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f0ac400b000 tx=0x7f0ac4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:54.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:54.939+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0abc043e10 con 0x7f0ad40b6ac0 2022-07-12T09:51:55.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ad405d8b0 con 0x7f0ab8053a40 2022-07-12T09:51:55.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0acb7fe700 1 -- 172.21.15.61:0/1308900980 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ad405d8b0 con 0x7f0ab8053a40 2022-07-12T09:51:55.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0ab8053a40 msgr2=0x7f0ab8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0ab8053a40 0x7f0ab8055f00 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f0ac400b000 tx=0x7f0ac4008040).stop 2022-07-12T09:51:55.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 msgr2=0x7f0ad4148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 0x7f0ad4148890 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f0abc002f70 tx=0x7f0abc014000).stop 2022-07-12T09:51:55.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 shutdown_connections 2022-07-12T09:51:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0ab8053a40 0x7f0ab8055f00 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0ad41420b0 0x7f0ad41450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad40b6ac0 0x7f0ad4148890 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 --2- 172.21.15.61:0/1308900980 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f0ad405f550 0x7f0ad4151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 >> 172.21.15.61:0/1308900980 conn(0x7f0ad4064570 msgr2=0x7f0ad4146850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 shutdown_connections 2022-07-12T09:51:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.067+0000 7f0ad359e700 1 -- 172.21.15.61:0/1308900980 wait complete. 2022-07-12T09:51:55.075 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 3 to deep-scrub 2022-07-12T09:51:55.082 INFO:tasks.ceph:Scrubbing osd.4 2022-07-12T09:51:55.083 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:55.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.191+0000 7f9894ef7700 1 Processor -- start 2022-07-12T09:51:55.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.191+0000 7f9894ef7700 1 -- start start 2022-07-12T09:51:55.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f98900b1320 0x7f98900ae500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b0120 0x7f98900ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9890142c80 0x7f9890143110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98900acee0 con 0x7f9890142c80 2022-07-12T09:51:55.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98900ad180 con 0x7f98900b1320 2022-07-12T09:51:55.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98900aeee0 con 0x7f98900b0120 2022-07-12T09:51:55.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9890142c80 0x7f9890143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9890142c80 0x7f9890143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34746/0 (socket says 172.21.15.61:34746) 2022-07-12T09:51:55.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 -- 172.21.15.61:0/843800306 learned_addr learned my addr 172.21.15.61:0/843800306 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:55.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9886ffd700 1 --2- 172.21.15.61:0/843800306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b0120 0x7f98900ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988f7fe700 1 --2- 172.21.15.61:0/843800306 >> v2:172.21.15.111:3300/0 conn(0x7f98900b1320 0x7f98900ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 -- 172.21.15.61:0/843800306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b0120 msgr2=0x7f98900ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:55.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 --2- 172.21.15.61:0/843800306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b0120 0x7f98900ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 -- 172.21.15.61:0/843800306 >> v2:172.21.15.111:3300/0 conn(0x7f98900b1320 msgr2=0x7f98900ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 --2- 172.21.15.61:0/843800306 >> v2:172.21.15.111:3300/0 conn(0x7f98900b1320 0x7f98900ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 -- 172.21.15.61:0/843800306 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900acb60 con 0x7f9890142c80 2022-07-12T09:51:55.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988f7fe700 1 --2- 172.21.15.61:0/843800306 >> v2:172.21.15.111:3300/0 conn(0x7f98900b1320 0x7f98900ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988ffff700 1 --2- 172.21.15.61:0/843800306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f9890143110 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f98880032f0 tx=0x7f9888009290).ready entity=mon.0 client_cookie=47e40ea97b6e49d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988effd700 1 -- 172.21.15.61:0/843800306 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988800b870 con 0x7f9890142c80 2022-07-12T09:51:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988effd700 1 -- 172.21.15.61:0/843800306 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988800b9d0 con 0x7f9890142c80 2022-07-12T09:51:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f988effd700 1 -- 172.21.15.61:0/843800306 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988800bb40 con 0x7f9890142c80 2022-07-12T09:51:55.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- 172.21.15.61:0/843800306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 msgr2=0x7f9890143110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- 172.21.15.61:0/843800306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f9890143110 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f98880032f0 tx=0x7f9888009290).stop 2022-07-12T09:51:55.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- 172.21.15.61:0/843800306 shutdown_connections 2022-07-12T09:51:55.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- 172.21.15.61:0/843800306 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b0120 0x7f98900ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- 172.21.15.61:0/843800306 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f9890143110 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 --2- 172.21.15.61:0/843800306 >> v2:172.21.15.111:3300/0 conn(0x7f98900b1320 0x7f98900ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- 172.21.15.61:0/843800306 >> 172.21.15.61:0/843800306 conn(0x7f98900b5780 msgr2=0x7f98900b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- 172.21.15.61:0/843800306 shutdown_connections 2022-07-12T09:51:55.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- 172.21.15.61:0/843800306 wait complete. 2022-07-12T09:51:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 Processor -- start 2022-07-12T09:51:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.195+0000 7f9894ef7700 1 -- start start 2022-07-12T09:51:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98900b0120 0x7f9890065f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b1320 0x7f98901528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f989013b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9890066cf0 con 0x7f9890142c80 2022-07-12T09:51:55.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98900667b0 con 0x7f98900b0120 2022-07-12T09:51:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9890066a50 con 0x7f98900b1320 2022-07-12T09:51:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f989013b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f989013b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34752/0 (socket says 172.21.15.61:34752) 2022-07-12T09:51:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 -- 172.21.15.61:0/3376245851 learned_addr learned my addr 172.21.15.61:0/3376245851 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9886ffd700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b1320 0x7f98901528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988f7fe700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98900b0120 0x7f9890065f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 -- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b1320 msgr2=0x7f98901528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f98900b1320 0x7f98901528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 -- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98900b0120 msgr2=0x7f9890065f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98900b0120 0x7f9890065f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 -- 172.21.15.61:0/3376245851 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888009530 con 0x7f9890142c80 2022-07-12T09:51:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988f7fe700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f98900b0120 0x7f9890065f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988ffff700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f989013b9a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f98880032f0 tx=0x7f9888016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f988800b870 con 0x7f9890142c80 2022-07-12T09:51:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9888020430 con 0x7f9890142c80 2022-07-12T09:51:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9890065000 con 0x7f9890142c80 2022-07-12T09:51:55.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9888020790 con 0x7f9890142c80 2022-07-12T09:51:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9890064910 con 0x7f9890142c80 2022-07-12T09:51:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.199+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9890066630 con 0x7f9890142c80 2022-07-12T09:51:55.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9888026020 con 0x7f9890142c80 2022-07-12T09:51:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988d7fa700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9874053940 0x7f9874055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f988806c8f0 con 0x7f9890142c80 2022-07-12T09:51:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988d7fa700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f987405a0d0 0x7f987405c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 --> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987405cbb0 con 0x7f987405a0d0 2022-07-12T09:51:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9888071240 con 0x7f9890142c80 2022-07-12T09:51:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988f7fe700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9874053940 0x7f9874055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f9886ffd700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f987405a0d0 0x7f987405c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f988f7fe700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9874053940 0x7f9874055e00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f988000c000 tx=0x7f9880008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.203+0000 7f9886ffd700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f987405a0d0 0x7f987405c530 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.207+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== osd.4 v2:172.21.15.111:6802/21886 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f987405cbb0 con 0x7f987405a0d0 2022-07-12T09:51:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 --> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9890065190 con 0x7f987405a0d0 2022-07-12T09:51:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f988d7fa700 1 -- 172.21.15.61:0/3376245851 <== osd.4 v2:172.21.15.111:6802/21886 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9890065190 con 0x7f987405a0d0 2022-07-12T09:51:55.232 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:55.232 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:55.233 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:55.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f987405a0d0 msgr2=0x7f987405c530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6802/21886,v1:172.21.15.111:6803/21886] conn(0x7f987405a0d0 0x7f987405c530 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9874053940 msgr2=0x7f9874055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f9874053940 0x7f9874055e00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f988000c000 tx=0x7f9880008040).stop 2022-07-12T09:51:55.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 msgr2=0x7f989013b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 --2- 172.21.15.61:0/3376245851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9890142c80 0x7f989013b9a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f98880032f0 tx=0x7f9888016000).stop 2022-07-12T09:51:55.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f988ffff700 1 -- 172.21.15.61:0/3376245851 reap_dead start 2022-07-12T09:51:55.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 shutdown_connections 2022-07-12T09:51:55.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 >> 172.21.15.61:0/3376245851 conn(0x7f98900b5780 msgr2=0x7f98900af170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 shutdown_connections 2022-07-12T09:51:55.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.223+0000 7f9894ef7700 1 -- 172.21.15.61:0/3376245851 wait complete. 2022-07-12T09:51:55.239 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-07-12T09:51:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 Processor -- start 2022-07-12T09:51:55.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- start start 2022-07-12T09:51:55.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f409c0ab670 0x7f409c05f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0abc70 con 0x7f409c0afa60 2022-07-12T09:51:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c05c720 con 0x7f409c0ab290 2022-07-12T09:51:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409c0aba90 con 0x7f409c0ab670 2022-07-12T09:51:55.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a1a9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f409c0ab670 0x7f409c05f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a2a9f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f409c0afa60 0x7f409c147530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51212/0 (socket says 172.21.15.61:51212) 2022-07-12T09:51:55.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a2a9f700 1 -- 172.21.15.61:0/3342039068 learned_addr learned my addr 172.21.15.61:0/3342039068 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a1a9d700 1 -- 172.21.15.61:0/3342039068 >> v1:172.21.15.111:6789/0 conn(0x7f409c0ab290 legacy=0x7f409c05ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a1a9d700 1 -- 172.21.15.61:0/3342039068 >> v1:172.21.15.61:6789/0 conn(0x7f409c0afa60 legacy=0x7f409c147530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a1a9d700 1 -- 172.21.15.61:0/3342039068 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c05c9c0 con 0x7f409c0ab670 2022-07-12T09:51:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a129c700 1 -- 172.21.15.61:0/3342039068 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549280937 0 0) 0x7f409c05c720 con 0x7f409c0ab290 2022-07-12T09:51:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a1a9d700 1 --2- 172.21.15.61:0/3342039068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab670 0x7f409c05f140 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f40940032f0 tx=0x7f40940091d0).ready entity=mon.0 client_cookie=13a1eee18a0cc779 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a129c700 1 -- 172.21.15.61:0/3342039068 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4094009ea0 con 0x7f409c0ab670 2022-07-12T09:51:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a129c700 1 -- 172.21.15.61:0/3342039068 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409400b8a0 con 0x7f409c0ab670 2022-07-12T09:51:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a129c700 1 -- 172.21.15.61:0/3342039068 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f409400ba10 con 0x7f409c0ab670 2022-07-12T09:51:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- 172.21.15.61:0/3342039068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab670 msgr2=0x7f409c05f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3342039068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab670 0x7f409c05f140 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 rx=0x7f40940032f0 tx=0x7f40940091d0).stop 2022-07-12T09:51:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- 172.21.15.61:0/3342039068 shutdown_connections 2022-07-12T09:51:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3342039068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab670 0x7f409c05f140 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- 172.21.15.61:0/3342039068 >> 172.21.15.61:0/3342039068 conn(0x7f409c0b1950 msgr2=0x7f409c0b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- 172.21.15.61:0/3342039068 shutdown_connections 2022-07-12T09:51:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.351+0000 7f40a32a0700 1 -- 172.21.15.61:0/3342039068 wait complete. 2022-07-12T09:51:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 Processor -- start 2022-07-12T09:51:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 -- start start 2022-07-12T09:51:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 0x7f409c149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f409c0ab670 0x7f409c0640f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f409c0afa60 0x7f409c146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c066970 con 0x7f409c0ab290 2022-07-12T09:51:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c148790 con 0x7f409c0afa60 2022-07-12T09:51:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f409c148a30 con 0x7f409c0ab670 2022-07-12T09:51:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 0x7f409c149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 0x7f409c149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34764/0 (socket says 172.21.15.61:34764) 2022-07-12T09:51:55.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 -- 172.21.15.61:0/3596698832 learned_addr learned my addr 172.21.15.61:0/3596698832 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a1a9d700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f409c0ab670 0x7f409c0640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a2a9f700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f409c0afa60 0x7f409c146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 -- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f409c0ab670 msgr2=0x7f409c0640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f409c0ab670 0x7f409c0640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 -- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f409c0afa60 msgr2=0x7f409c146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f409c0afa60 0x7f409c146910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 -- 172.21.15.61:0/3596698832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094009470 con 0x7f409c0ab290 2022-07-12T09:51:55.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a229e700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 0x7f409c149700 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f4090000bc0 tx=0x7f4090007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f409000ce50 con 0x7f409c0ab290 2022-07-12T09:51:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409000f470 con 0x7f409c0ab290 2022-07-12T09:51:55.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c066dd0 con 0x7f409c0ab290 2022-07-12T09:51:55.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f409000f7d0 con 0x7f409c0ab290 2022-07-12T09:51:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.355+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c067630 con 0x7f409c0ab290 2022-07-12T09:51:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.359+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c057460 con 0x7f409c0ab290 2022-07-12T09:51:55.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.359+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4090019030 con 0x7f409c0ab290 2022-07-12T09:51:55.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.359+0000 7f408f7fe700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f407c053a40 0x7f407c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.359+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f409006afa0 con 0x7f409c0ab290 2022-07-12T09:51:55.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.359+0000 7f40a1a9d700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f407c053a40 0x7f407c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.359+0000 7f40a1a9d700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f407c053a40 0x7f407c055f00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f4094019000 tx=0x7f4094016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.367+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f40900421f0 con 0x7f409c0ab290 2022-07-12T09:51:55.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.491+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f409c066b30 con 0x7f407c053a40 2022-07-12T09:51:55.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.491+0000 7f408f7fe700 1 -- 172.21.15.61:0/3596698832 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4094009ea0 con 0x7f407c053a40 2022-07-12T09:51:55.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f407c053a40 msgr2=0x7f407c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f407c053a40 0x7f407c055f00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f4094019000 tx=0x7f4094016000).stop 2022-07-12T09:51:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 msgr2=0x7f409c149700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 0x7f409c149700 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 rx=0x7f4090000bc0 tx=0x7f4090007af0).stop 2022-07-12T09:51:55.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 shutdown_connections 2022-07-12T09:51:55.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f407c053a40 0x7f407c055f00 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f409c0afa60 0x7f409c146910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f409c0ab670 0x7f409c0640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 --2- 172.21.15.61:0/3596698832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f409c0ab290 0x7f409c149700 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 >> 172.21.15.61:0/3596698832 conn(0x7f409c0b1950 msgr2=0x7f409c0b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 shutdown_connections 2022-07-12T09:51:55.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.495+0000 7f40a32a0700 1 -- 172.21.15.61:0/3596698832 wait complete. 2022-07-12T09:51:55.503 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 4 to deep-scrub 2022-07-12T09:51:55.507 INFO:tasks.ceph:Scrubbing osd.5 2022-07-12T09:51:55.508 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:55.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.615+0000 7fd752c35700 1 Processor -- start 2022-07-12T09:51:55.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.615+0000 7fd752c35700 1 -- start start 2022-07-12T09:51:55.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd74c0b9eb0 0x7fd74c0b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fd74c0b3180 0x7fd74c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd74c0ba290 con 0x7fd74c0b2630 2022-07-12T09:51:55.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0b1410 con 0x7fd74c0b9eb0 2022-07-12T09:51:55.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd74c0b4010 con 0x7fd74c0b3180 2022-07-12T09:51:55.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7509d1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd74c0b9eb0 0x7fd74c0b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7511d2700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd74c0b2630 0x7fd74c05bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51226/0 (socket says 172.21.15.61:51226) 2022-07-12T09:51:55.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7511d2700 1 -- 172.21.15.61:0/3372987941 learned_addr learned my addr 172.21.15.61:0/3372987941 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:55.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd74bfff700 1 --2- 172.21.15.61:0/3372987941 >> v2:172.21.15.111:3300/0 conn(0x7fd74c0b3180 0x7fd74c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd74b7fe700 1 -- 172.21.15.61:0/3372987941 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187102901 0 0) 0x7fd74c0ba290 con 0x7fd74c0b2630 2022-07-12T09:51:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd74b7fe700 1 -- 172.21.15.61:0/3372987941 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd72c003260 con 0x7fd74c0b2630 2022-07-12T09:51:55.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7509d1700 1 -- 172.21.15.61:0/3372987941 >> v2:172.21.15.111:3300/0 conn(0x7fd74c0b3180 msgr2=0x7fd74c0b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:55.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7509d1700 1 --2- 172.21.15.61:0/3372987941 >> v2:172.21.15.111:3300/0 conn(0x7fd74c0b3180 0x7fd74c0b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7509d1700 1 -- 172.21.15.61:0/3372987941 >> v1:172.21.15.61:6789/0 conn(0x7fd74c0b2630 legacy=0x7fd74c05bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7509d1700 1 -- 172.21.15.61:0/3372987941 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c0b42b0 con 0x7fd74c0b9eb0 2022-07-12T09:51:55.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd7509d1700 1 --2- 172.21.15.61:0/3372987941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b9eb0 0x7fd74c0b0a30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7fd73c002f70 tx=0x7fd73c006df0).ready entity=mon.0 client_cookie=a20be0a1eb644697 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd74b7fe700 1 -- 172.21.15.61:0/3372987941 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd73c00e470 con 0x7fd74c0b9eb0 2022-07-12T09:51:55.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd74b7fe700 1 -- 172.21.15.61:0/3372987941 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73c000e90 con 0x7fd74c0b9eb0 2022-07-12T09:51:55.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd74b7fe700 1 -- 172.21.15.61:0/3372987941 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd73c00e7c0 con 0x7fd74c0b9eb0 2022-07-12T09:51:55.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- 172.21.15.61:0/3372987941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b9eb0 msgr2=0x7fd74c0b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 --2- 172.21.15.61:0/3372987941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b9eb0 0x7fd74c0b0a30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 rx=0x7fd73c002f70 tx=0x7fd73c006df0).stop 2022-07-12T09:51:55.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- 172.21.15.61:0/3372987941 shutdown_connections 2022-07-12T09:51:55.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 --2- 172.21.15.61:0/3372987941 >> v2:172.21.15.111:3300/0 conn(0x7fd74c0b3180 0x7fd74c0b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 --2- 172.21.15.61:0/3372987941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b9eb0 0x7fd74c0b0a30 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- 172.21.15.61:0/3372987941 >> 172.21.15.61:0/3372987941 conn(0x7fd74c0678c0 msgr2=0x7fd74c0b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.619+0000 7fd752c35700 1 -- 172.21.15.61:0/3372987941 shutdown_connections 2022-07-12T09:51:55.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 -- 172.21.15.61:0/3372987941 wait complete. 2022-07-12T09:51:55.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 Processor -- start 2022-07-12T09:51:55.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 -- start start 2022-07-12T09:51:55.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd74c0b2630 0x7fd74c13ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b3180 0x7fd74c0acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd74c0b9eb0 0x7fd74c0ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1529c0 con 0x7fd74c0b3180 2022-07-12T09:51:55.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c13fd80 con 0x7fd74c0b9eb0 2022-07-12T09:51:55.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd752c35700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c13f3a0 con 0x7fd74c0b2630 2022-07-12T09:51:55.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b3180 0x7fd74c0acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b3180 0x7fd74c0acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34780/0 (socket says 172.21.15.61:34780) 2022-07-12T09:51:55.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 -- 172.21.15.61:0/3053206710 learned_addr learned my addr 172.21.15.61:0/3053206710 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd7509d1700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd74c0b2630 0x7fd74c13ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd7511d2700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd74c0b9eb0 0x7fd74c0ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 -- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd74c0b2630 msgr2=0x7fd74c13ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fd74c0b2630 0x7fd74c13ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 -- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd74c0b9eb0 msgr2=0x7fd74c0ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd74c0b9eb0 0x7fd74c0ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.623+0000 7fd74bfff700 1 -- 172.21.15.61:0/3053206710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c00b120 con 0x7fd74c0b3180 2022-07-12T09:51:55.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd74bfff700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b3180 0x7fd74c0acb60 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7fd73400c000 tx=0x7fd734006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd734010780 con 0x7fd74c0b3180 2022-07-12T09:51:55.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7340108e0 con 0x7fd74c0b3180 2022-07-12T09:51:55.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c13f640 con 0x7fd74c0b3180 2022-07-12T09:51:55.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd734010c40 con 0x7fd74c0b3180 2022-07-12T09:51:55.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c0ad540 con 0x7fd74c0b3180 2022-07-12T09:51:55.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd74c152b10 con 0x7fd74c0b3180 2022-07-12T09:51:55.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd73401c460 con 0x7fd74c0b3180 2022-07-12T09:51:55.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.627+0000 7fd749ffb700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd730053ad0 0x7fd730055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd73406c030 con 0x7fd74c0b3180 2022-07-12T09:51:55.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd749ffb700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7fd73005a1c0 0x7fd73005c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 --> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd73005cc00 con 0x7fd73005a1c0 2022-07-12T09:51:55.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd73406e6a0 con 0x7fd74c0b3180 2022-07-12T09:51:55.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd7509d1700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd730053ad0 0x7fd730055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd7511d2700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7fd73005a1c0 0x7fd73005c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd7511d2700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7fd73005a1c0 0x7fd73005c620 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd7509d1700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd730053ad0 0x7fd730055f90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fd73c01a000 tx=0x7fd73c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.631+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== osd.5 v2:172.21.15.111:6810/21888 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fd73005cc00 con 0x7fd73005a1c0 2022-07-12T09:51:55.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 --> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd74c057460 con 0x7fd73005a1c0 2022-07-12T09:51:55.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd749ffb700 1 -- 172.21.15.61:0/3053206710 <== osd.5 v2:172.21.15.111:6810/21888 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fd74c057460 con 0x7fd73005a1c0 2022-07-12T09:51:55.698 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:55.698 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:55.699 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:55.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7fd73005a1c0 msgr2=0x7fd73005c620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6810/21888,v1:172.21.15.111:6811/21888] conn(0x7fd73005a1c0 0x7fd73005c620 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd730053ad0 msgr2=0x7fd730055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fd730053ad0 0x7fd730055f90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fd73c01a000 tx=0x7fd73c014000).stop 2022-07-12T09:51:55.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b3180 msgr2=0x7fd74c0acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 --2- 172.21.15.61:0/3053206710 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd74c0b3180 0x7fd74c0acb60 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 rx=0x7fd73400c000 tx=0x7fd734006df0).stop 2022-07-12T09:51:55.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd7511d2700 1 -- 172.21.15.61:0/3053206710 reap_dead start 2022-07-12T09:51:55.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 shutdown_connections 2022-07-12T09:51:55.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 >> 172.21.15.61:0/3053206710 conn(0x7fd74c0678c0 msgr2=0x7fd74c0b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 shutdown_connections 2022-07-12T09:51:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.647+0000 7fd752c35700 1 -- 172.21.15.61:0/3053206710 wait complete. 2022-07-12T09:51:55.708 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2022-07-12T09:51:55.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 Processor -- start 2022-07-12T09:51:55.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 -- start start 2022-07-12T09:51:55.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80b1e40 0x7feee805d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feee80ac710 0x7feee80ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feee80ad370 0x7feee81401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee813fd50 con 0x7feee80ac710 2022-07-12T09:51:55.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee8067900 con 0x7feee80ad370 2022-07-12T09:51:55.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.819+0000 7feeeca90700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee8067ba0 con 0x7feee80b1e40 2022-07-12T09:51:55.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feee80ac710 0x7feee80ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feee80ac710 0x7feee80ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34788/0 (socket says 172.21.15.61:34788) 2022-07-12T09:51:55.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee6d9d700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7feee80ad370 0x7feee81401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 -- 172.21.15.61:0/3384708664 learned_addr learned my addr 172.21.15.61:0/3384708664 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:55.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee659c700 1 --2- 172.21.15.61:0/3384708664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80b1e40 0x7feee805d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 -- 172.21.15.61:0/3384708664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80b1e40 msgr2=0x7feee805d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:55.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 --2- 172.21.15.61:0/3384708664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80b1e40 0x7feee805d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 -- 172.21.15.61:0/3384708664 >> v2:172.21.15.111:3300/0 conn(0x7feee80ad370 msgr2=0x7feee81401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:55.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 --2- 172.21.15.61:0/3384708664 >> v2:172.21.15.111:3300/0 conn(0x7feee80ad370 0x7feee81401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 -- 172.21.15.61:0/3384708664 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee80ad7e0 con 0x7feee80ac710 2022-07-12T09:51:55.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee5d9b700 1 --2- 172.21.15.61:0/3384708664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80ac710 0x7feee80ace80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7feed80032f0 tx=0x7feed8009290).ready entity=mon.0 client_cookie=d7583e8fe7893301 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee559a700 1 -- 172.21.15.61:0/3384708664 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feed800b680 con 0x7feee80ac710 2022-07-12T09:51:55.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee559a700 1 -- 172.21.15.61:0/3384708664 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed800b7e0 con 0x7feee80ac710 2022-07-12T09:51:55.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feee559a700 1 -- 172.21.15.61:0/3384708664 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feed800b950 con 0x7feee80ac710 2022-07-12T09:51:55.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 -- 172.21.15.61:0/3384708664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80ac710 msgr2=0x7feee80ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 --2- 172.21.15.61:0/3384708664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80ac710 0x7feee80ace80 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 rx=0x7feed80032f0 tx=0x7feed8009290).stop 2022-07-12T09:51:55.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 -- 172.21.15.61:0/3384708664 shutdown_connections 2022-07-12T09:51:55.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 --2- 172.21.15.61:0/3384708664 >> v2:172.21.15.111:3300/0 conn(0x7feee80ad370 0x7feee81401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 --2- 172.21.15.61:0/3384708664 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80ac710 0x7feee80ace80 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 --2- 172.21.15.61:0/3384708664 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80b1e40 0x7feee805d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 -- 172.21.15.61:0/3384708664 >> 172.21.15.61:0/3384708664 conn(0x7feee8062c00 msgr2=0x7feee8062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 -- 172.21.15.61:0/3384708664 shutdown_connections 2022-07-12T09:51:55.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.823+0000 7feeeca90700 1 -- 172.21.15.61:0/3384708664 wait complete. 2022-07-12T09:51:55.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 Processor -- start 2022-07-12T09:51:55.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- start start 2022-07-12T09:51:55.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee80ac710 0x7feee813cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80ad370 0x7feee8136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 0x7feee8138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee8065e20 con 0x7feee80b1e40 2022-07-12T09:51:55.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee80660c0 con 0x7feee80ac710 2022-07-12T09:51:55.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee8065aa0 con 0x7feee80ad370 2022-07-12T09:51:55.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 0x7feee8138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee80ac710 0x7feee813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee5d9b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80ad370 0x7feee8136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee80ac710 0x7feee813cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:41600/0 (socket says 172.21.15.61:41600) 2022-07-12T09:51:55.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 0x7feee8138ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34798/0 (socket says 172.21.15.61:34798) 2022-07-12T09:51:55.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee659c700 1 -- 172.21.15.61:0/2858782840 learned_addr learned my addr 172.21.15.61:0/2858782840 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:55.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 -- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80ad370 msgr2=0x7feee8136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80ad370 0x7feee8136360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 -- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee80ac710 msgr2=0x7feee813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee80ac710 0x7feee813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 -- 172.21.15.61:0/2858782840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed8009530 con 0x7feee80b1e40 2022-07-12T09:51:55.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feee6d9d700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 0x7feee8138ae0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7feee000c000 tx=0x7feee0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feee0011750 con 0x7feee80b1e40 2022-07-12T09:51:55.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee00118b0 con 0x7feee80b1e40 2022-07-12T09:51:55.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee8149d30 con 0x7feee80b1e40 2022-07-12T09:51:55.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feee0011c10 con 0x7feee80b1e40 2022-07-12T09:51:55.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee8066e00 con 0x7feee80b1e40 2022-07-12T09:51:55.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.827+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee8057460 con 0x7feee80b1e40 2022-07-12T09:51:55.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.831+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feee0011d70 con 0x7feee80b1e40 2022-07-12T09:51:55.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.831+0000 7feed77fe700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feec4053a40 0x7feec4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:55.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.831+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7feee006be40 con 0x7feee80b1e40 2022-07-12T09:51:55.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.831+0000 7feee659c700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feec4053a40 0x7feec4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:55.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.831+0000 7feee659c700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feec4053a40 0x7feec4055f00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7feedc00b000 tx=0x7feedc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:55.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.835+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feee0043080 con 0x7feee80b1e40 2022-07-12T09:51:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.959+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7feee80447a0 con 0x7feec4053a40 2022-07-12T09:51:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feed77fe700 1 -- 172.21.15.61:0/2858782840 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feee80447a0 con 0x7feec4053a40 2022-07-12T09:51:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feec4053a40 msgr2=0x7feec4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feec4053a40 0x7feec4055f00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7feedc00b000 tx=0x7feedc006df0).stop 2022-07-12T09:51:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 msgr2=0x7feee8138ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 0x7feee8138ae0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 rx=0x7feee000c000 tx=0x7feee0006df0).stop 2022-07-12T09:51:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 shutdown_connections 2022-07-12T09:51:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feee80b1e40 0x7feee8138ae0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7feee80ad370 0x7feee8136360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7feec4053a40 0x7feec4055f00 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 --2- 172.21.15.61:0/2858782840 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feee80ac710 0x7feee813cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:55.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 >> 172.21.15.61:0/2858782840 conn(0x7feee8062c00 msgr2=0x7feee80b5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:55.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 shutdown_connections 2022-07-12T09:51:55.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:55.963+0000 7feeeca90700 1 -- 172.21.15.61:0/2858782840 wait complete. 2022-07-12T09:51:55.970 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 5 to deep-scrub 2022-07-12T09:51:55.976 INFO:tasks.ceph:Scrubbing osd.6 2022-07-12T09:51:55.978 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:56.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 Processor -- start 2022-07-12T09:51:56.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 -- start start 2022-07-12T09:51:56.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe09005d060 0x7fe090145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7fe090067860 0x7fe090146760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0900afca0 con 0x7fe0900b3c60 2022-07-12T09:51:56.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09005d440 con 0x7fe090067860 2022-07-12T09:51:56.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.091+0000 7fe0948a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0900aff00 con 0x7fe09005d060 2022-07-12T09:51:56.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08e59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe0900b3c60 0x7fe090145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51254/0 (socket says 172.21.15.61:51254) 2022-07-12T09:51:56.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08e59c700 1 -- 172.21.15.61:0/3960081953 learned_addr learned my addr 172.21.15.61:0/3960081953 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:56.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08f59e700 1 --2- 172.21.15.61:0/3960081953 >> v2:172.21.15.111:3300/0 conn(0x7fe090067860 0x7fe090146760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe09005d060 0x7fe090145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08dd9b700 1 -- 172.21.15.61:0/3960081953 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257395627 0 0) 0x7fe0900afca0 con 0x7fe0900b3c60 2022-07-12T09:51:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08dd9b700 1 -- 172.21.15.61:0/3960081953 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe070003260 con 0x7fe0900b3c60 2022-07-12T09:51:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08f59e700 1 -- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe09005d060 msgr2=0x7fe090145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08f59e700 1 --2- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe09005d060 0x7fe090145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08f59e700 1 -- 172.21.15.61:0/3960081953 >> v1:172.21.15.61:6789/0 conn(0x7fe0900b3c60 legacy=0x7fe090145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08f59e700 1 -- 172.21.15.61:0/3960081953 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0900b4200 con 0x7fe090067860 2022-07-12T09:51:56.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08f59e700 1 --2- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe090067860 0x7fe090146760 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fe0880032f0 tx=0x7fe0880091d0).ready entity=mon.1 client_cookie=b9c62d2e1c6aa6b1 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08dd9b700 1 -- 172.21.15.61:0/3960081953 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe08800b600 con 0x7fe090067860 2022-07-12T09:51:56.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08dd9b700 1 -- 172.21.15.61:0/3960081953 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08800b760 con 0x7fe090067860 2022-07-12T09:51:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe08dd9b700 1 -- 172.21.15.61:0/3960081953 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe08800b8d0 con 0x7fe090067860 2022-07-12T09:51:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 -- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe090067860 msgr2=0x7fe090146760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 --2- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe090067860 0x7fe090146760 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7fe0880032f0 tx=0x7fe0880091d0).stop 2022-07-12T09:51:56.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 -- 172.21.15.61:0/3960081953 shutdown_connections 2022-07-12T09:51:56.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 --2- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe090067860 0x7fe090146760 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 --2- 172.21.15.61:0/3960081953 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe09005d060 0x7fe090145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 -- 172.21.15.61:0/3960081953 >> 172.21.15.61:0/3960081953 conn(0x7fe0900b67c0 msgr2=0x7fe090066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 -- 172.21.15.61:0/3960081953 shutdown_connections 2022-07-12T09:51:56.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.095+0000 7fe0948a3700 1 -- 172.21.15.61:0/3960081953 wait complete. 2022-07-12T09:51:56.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 Processor -- start 2022-07-12T09:51:56.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 -- start start 2022-07-12T09:51:56.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe09005d060 0x7fe0901525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe090067860 0x7fe09009cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900b3c60 0x7fe09009eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe090152b90 con 0x7fe0900b3c60 2022-07-12T09:51:56.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09009d6c0 con 0x7fe09005d060 2022-07-12T09:51:56.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09009e170 con 0x7fe090067860 2022-07-12T09:51:56.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900b3c60 0x7fe09009eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900b3c60 0x7fe09009eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34810/0 (socket says 172.21.15.61:34810) 2022-07-12T09:51:56.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 -- 172.21.15.61:0/2220689639 learned_addr learned my addr 172.21.15.61:0/2220689639 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:56.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe09005d060 0x7fe0901525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08e59c700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe090067860 0x7fe09009cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 -- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe090067860 msgr2=0x7fe09009cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fe090067860 0x7fe09009cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 -- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe09005d060 msgr2=0x7fe0901525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe09005d060 0x7fe0901525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 -- 172.21.15.61:0/2220689639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe088009470 con 0x7fe0900b3c60 2022-07-12T09:51:56.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe09005d060 0x7fe0901525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T09:51:56.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900b3c60 0x7fe09009eaf0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 rx=0x7fe0880032f0 tx=0x7fe088016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe08800b8d0 con 0x7fe0900b3c60 2022-07-12T09:51:56.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08801f3f0 con 0x7fe0900b3c60 2022-07-12T09:51:56.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09009e410 con 0x7fe0900b3c60 2022-07-12T09:51:56.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe08800b600 con 0x7fe0900b3c60 2022-07-12T09:51:56.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.099+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09009f180 con 0x7fe0900b3c60 2022-07-12T09:51:56.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe09009e5d0 con 0x7fe0900b3c60 2022-07-12T09:51:56.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe088026030 con 0x7fe0900b3c60 2022-07-12T09:51:56.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe087fff700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe074053a60 0x7fe074055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe08806c660 con 0x7fe0900b3c60 2022-07-12T09:51:56.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe087fff700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fe07405a150 0x7fe07405c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 --> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe07405cb90 con 0x7fe07405a150 2022-07-12T09:51:56.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe08806ecf0 con 0x7fe0900b3c60 2022-07-12T09:51:56.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe074053a60 0x7fe074055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe08e59c700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fe07405a150 0x7fe07405c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe074053a60 0x7fe074055f20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fe08000b000 tx=0x7fe080006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.103+0000 7fe08e59c700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fe07405a150 0x7fe07405c5b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.107+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== osd.6 v2:172.21.15.111:6818/22012 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fe07405cb90 con 0x7fe07405a150 2022-07-12T09:51:56.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 --> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe0900af770 con 0x7fe07405a150 2022-07-12T09:51:56.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe087fff700 1 -- 172.21.15.61:0/2220689639 <== osd.6 v2:172.21.15.111:6818/22012 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe0900af770 con 0x7fe07405a150 2022-07-12T09:51:56.129 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:56.129 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:56.130 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:56.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fe07405a150 msgr2=0x7fe07405c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6818/22012,v1:172.21.15.111:6819/22012] conn(0x7fe07405a150 0x7fe07405c5b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe074053a60 msgr2=0x7fe074055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fe074053a60 0x7fe074055f20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fe08000b000 tx=0x7fe080006df0).stop 2022-07-12T09:51:56.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900b3c60 msgr2=0x7fe09009eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 --2- 172.21.15.61:0/2220689639 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900b3c60 0x7fe09009eaf0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 rx=0x7fe0880032f0 tx=0x7fe088016000).stop 2022-07-12T09:51:56.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe08f59e700 1 -- 172.21.15.61:0/2220689639 reap_dead start 2022-07-12T09:51:56.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 shutdown_connections 2022-07-12T09:51:56.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.119+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 >> 172.21.15.61:0/2220689639 conn(0x7fe0900b67c0 msgr2=0x7fe0900b7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.123+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 shutdown_connections 2022-07-12T09:51:56.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.123+0000 7fe0948a3700 1 -- 172.21.15.61:0/2220689639 wait complete. 2022-07-12T09:51:56.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2022-07-12T09:51:56.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.255+0000 7fbb35552700 1 Processor -- start 2022-07-12T09:51:56.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 -- start start 2022-07-12T09:51:56.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300b1e40 0x7fbb3005d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb300ac710 0x7fbb300ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3013fd50 con 0x7fbb300ad370 2022-07-12T09:51:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb30067900 con 0x7fbb300ac710 2022-07-12T09:51:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30067ba0 con 0x7fbb300b1e40 2022-07-12T09:51:56.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb300ac710 0x7fbb300ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbb300ac710 0x7fbb300ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34818/0 (socket says 172.21.15.61:34818) 2022-07-12T09:51:56.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 -- 172.21.15.61:0/3822550648 learned_addr learned my addr 172.21.15.61:0/3822550648 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:56.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb2effd700 1 --2- 172.21.15.61:0/3822550648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300b1e40 0x7fbb3005d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 -- 172.21.15.61:0/3822550648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300b1e40 msgr2=0x7fbb3005d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:56.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 --2- 172.21.15.61:0/3822550648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300b1e40 0x7fbb3005d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 -- 172.21.15.61:0/3822550648 >> v1:172.21.15.111:6789/0 conn(0x7fbb300ad370 legacy=0x7fbb301401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 -- 172.21.15.61:0/3822550648 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb300ad7e0 con 0x7fbb300ac710 2022-07-12T09:51:56.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb27fff700 1 --2- 172.21.15.61:0/3822550648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300ac710 0x7fbb300ace80 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7fbb18002f70 tx=0x7fbb18006eb0).ready entity=mon.0 client_cookie=23f70e2851a937c0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb2e7fc700 1 -- 172.21.15.61:0/3822550648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb1800d5c0 con 0x7fbb300ac710 2022-07-12T09:51:56.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb2e7fc700 1 -- 172.21.15.61:0/3822550648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb18000e90 con 0x7fbb300ac710 2022-07-12T09:51:56.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb2e7fc700 1 -- 172.21.15.61:0/3822550648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb1800d950 con 0x7fbb300ac710 2022-07-12T09:51:56.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 -- 172.21.15.61:0/3822550648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300ac710 msgr2=0x7fbb300ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.259+0000 7fbb35552700 1 --2- 172.21.15.61:0/3822550648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300ac710 0x7fbb300ace80 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 rx=0x7fbb18002f70 tx=0x7fbb18006eb0).stop 2022-07-12T09:51:56.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 -- 172.21.15.61:0/3822550648 shutdown_connections 2022-07-12T09:51:56.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 --2- 172.21.15.61:0/3822550648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300ac710 0x7fbb300ace80 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 --2- 172.21.15.61:0/3822550648 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300b1e40 0x7fbb3005d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 -- 172.21.15.61:0/3822550648 >> 172.21.15.61:0/3822550648 conn(0x7fbb30062c00 msgr2=0x7fbb30062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 -- 172.21.15.61:0/3822550648 shutdown_connections 2022-07-12T09:51:56.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 -- 172.21.15.61:0/3822550648 wait complete. 2022-07-12T09:51:56.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 Processor -- start 2022-07-12T09:51:56.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.263+0000 7fbb35552700 1 -- start start 2022-07-12T09:51:56.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300ac710 0x7fbb300649f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb300ad370 0x7fbb3013a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 0x7fbb30135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3013ab80 con 0x7fbb300b1e40 2022-07-12T09:51:56.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30150370 con 0x7fbb300ad370 2022-07-12T09:51:56.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb30150030 con 0x7fbb300ac710 2022-07-12T09:51:56.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 0x7fbb30135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 0x7fbb30135130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34828/0 (socket says 172.21.15.61:34828) 2022-07-12T09:51:56.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 -- 172.21.15.61:0/1025772201 learned_addr learned my addr 172.21.15.61:0/1025772201 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:56.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2effd700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300ac710 0x7fbb300649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb27fff700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb300ad370 0x7fbb3013a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 -- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300ac710 msgr2=0x7fbb300649f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300ac710 0x7fbb300649f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 -- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb300ad370 msgr2=0x7fbb3013a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb300ad370 0x7fbb3013a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 -- 172.21.15.61:0/1025772201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1800b1e0 con 0x7fbb300b1e40 2022-07-12T09:51:56.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb27fff700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb300ad370 0x7fbb3013a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-12T09:51:56.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2f7fe700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 0x7fbb30135130 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7fbb3005ce90 tx=0x7fbb28007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2800e4a0 con 0x7fbb300b1e40 2022-07-12T09:51:56.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2800e600 con 0x7fbb300b1e40 2022-07-12T09:51:56.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3013a840 con 0x7fbb300b1e40 2022-07-12T09:51:56.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb2800e960 con 0x7fbb300b1e40 2022-07-12T09:51:56.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3005bd10 con 0x7fbb300b1e40 2022-07-12T09:51:56.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb30057460 con 0x7fbb300b1e40 2022-07-12T09:51:56.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbb28019030 con 0x7fbb300b1e40 2022-07-12T09:51:56.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.267+0000 7fbb2cff9700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb14053a40 0x7fbb14055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.271+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fbb2806b310 con 0x7fbb300b1e40 2022-07-12T09:51:56.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.271+0000 7fbb2effd700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb14053a40 0x7fbb14055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.271+0000 7fbb2effd700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb14053a40 0x7fbb14055f00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fbb2000c000 tx=0x7fbb20008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.271+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb28042450 con 0x7fbb300b1e40 2022-07-12T09:51:56.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.399+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb300447a0 con 0x7fbb14053a40 2022-07-12T09:51:56.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.399+0000 7fbb2cff9700 1 -- 172.21.15.61:0/1025772201 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbb300447a0 con 0x7fbb14053a40 2022-07-12T09:51:56.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb14053a40 msgr2=0x7fbb14055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb14053a40 0x7fbb14055f00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7fbb2000c000 tx=0x7fbb20008040).stop 2022-07-12T09:51:56.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 msgr2=0x7fbb30135130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 0x7fbb30135130 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7fbb3005ce90 tx=0x7fbb28007f40).stop 2022-07-12T09:51:56.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 shutdown_connections 2022-07-12T09:51:56.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb300b1e40 0x7fbb30135130 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7fbb14053a40 0x7fbb14055f00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbb300ad370 0x7fbb3013a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 --2- 172.21.15.61:0/1025772201 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7fbb300ac710 0x7fbb300649f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 >> 172.21.15.61:0/1025772201 conn(0x7fbb30062c00 msgr2=0x7fbb300b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 shutdown_connections 2022-07-12T09:51:56.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.403+0000 7fbb35552700 1 -- 172.21.15.61:0/1025772201 wait complete. 2022-07-12T09:51:56.424 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 6 to deep-scrub 2022-07-12T09:51:56.425 INFO:tasks.ceph:Scrubbing osd.7 2022-07-12T09:51:56.426 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2022-07-12T09:51:56.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.539+0000 7f796359e700 1 Processor -- start 2022-07-12T09:51:56.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.539+0000 7f796359e700 1 -- start start 2022-07-12T09:51:56.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f796359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7964144480 0x7f796405bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f796359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79640b3010 con 0x7f79641440a0 2022-07-12T09:51:56.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f796359e700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7964144da0 con 0x7f79640b3630 2022-07-12T09:51:56.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f796359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79640624b0 con 0x7f7964144480 2022-07-12T09:51:56.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f796259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f79641440a0 0x7f79641413e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51282/0 (socket says 172.21.15.61:51282) 2022-07-12T09:51:56.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f796259c700 1 -- 172.21.15.61:0/567653751 learned_addr learned my addr 172.21.15.61:0/567653751 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:56.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7959d9b700 1 --2- 172.21.15.61:0/567653751 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7964144480 0x7f796405bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952335429 0 0) 0x7f79640b3010 con 0x7f79641440a0 2022-07-12T09:51:56.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7948003260 con 0x7f79641440a0 2022-07-12T09:51:56.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607035550 0 0) 0x7f7964144da0 con 0x7f79640b3630 2022-07-12T09:51:56.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79640b3010 con 0x7f79640b3630 2022-07-12T09:51:56.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7959d9b700 1 -- 172.21.15.61:0/567653751 >> v1:172.21.15.111:6789/0 conn(0x7f79640b3630 legacy=0x7f796413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7959d9b700 1 -- 172.21.15.61:0/567653751 >> v1:172.21.15.61:6789/0 conn(0x7f79641440a0 legacy=0x7f79641413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7959d9b700 1 -- 172.21.15.61:0/567653751 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79640b2d70 con 0x7f7964144480 2022-07-12T09:51:56.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7959d9b700 1 --2- 172.21.15.61:0/567653751 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7964144480 0x7f796405bd10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f794c002f70 tx=0x7f794c006df0).ready entity=mon.2 client_cookie=7015ead69d8360b8 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f794c00d5c0 con 0x7f7964144480 2022-07-12T09:51:56.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c000e90 con 0x7f7964144480 2022-07-12T09:51:56.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.543+0000 7f7961d9b700 1 -- 172.21.15.61:0/567653751 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f794c00d910 con 0x7f7964144480 2022-07-12T09:51:56.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 -- 172.21.15.61:0/567653751 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7964144480 msgr2=0x7f796405bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 --2- 172.21.15.61:0/567653751 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7964144480 0x7f796405bd10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f794c002f70 tx=0x7f794c006df0).stop 2022-07-12T09:51:56.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 -- 172.21.15.61:0/567653751 shutdown_connections 2022-07-12T09:51:56.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 --2- 172.21.15.61:0/567653751 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f7964144480 0x7f796405bd10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 -- 172.21.15.61:0/567653751 >> 172.21.15.61:0/567653751 conn(0x7f79640b70b0 msgr2=0x7f79640b7490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 -- 172.21.15.61:0/567653751 shutdown_connections 2022-07-12T09:51:56.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 -- 172.21.15.61:0/567653751 wait complete. 2022-07-12T09:51:56.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 Processor -- start 2022-07-12T09:51:56.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.547+0000 7f796359e700 1 -- start start 2022-07-12T09:51:56.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79640b3630 0x7f7964142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641440a0 0x7f7964155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7964144480 0x7f7964155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796405f4c0 con 0x7f7964144480 2022-07-12T09:51:56.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796405f760 con 0x7f79641440a0 2022-07-12T09:51:56.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796405fa00 con 0x7f79640b3630 2022-07-12T09:51:56.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7964144480 0x7f7964155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796259c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79640b3630 0x7f7964142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7959d9b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641440a0 0x7f7964155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7964144480 0x7f7964155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34842/0 (socket says 172.21.15.61:34842) 2022-07-12T09:51:56.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 -- 172.21.15.61:0/4001880755 learned_addr learned my addr 172.21.15.61:0/4001880755 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:56.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 -- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79640b3630 msgr2=0x7f7964142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f79640b3630 0x7f7964142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 -- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641440a0 msgr2=0x7f7964155770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f79641440a0 0x7f7964155770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 -- 172.21.15.61:0/4001880755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794c00b180 con 0x7f7964144480 2022-07-12T09:51:56.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f7962d9d700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7964144480 0x7f7964155c60 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f795c001490 tx=0x7f795c007e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f795c002310 con 0x7f7964144480 2022-07-12T09:51:56.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c002470 con 0x7f7964144480 2022-07-12T09:51:56.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7964143420 con 0x7f7964144480 2022-07-12T09:51:56.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f795c001840 con 0x7f7964144480 2022-07-12T09:51:56.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7964147ce0 con 0x7f7964144480 2022-07-12T09:51:56.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.551+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f796405fbc0 con 0x7f7964144480 2022-07-12T09:51:56.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f795c019020 con 0x7f7964144480 2022-07-12T09:51:56.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f793c053ad0 0x7f793c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f795c06bb10 con 0x7f7964144480 2022-07-12T09:51:56.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f796259c700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f793c053ad0 0x7f793c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f793c05a1c0 0x7f793c05c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 --> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f793c05cc00 con 0x7f793c05a1c0 2022-07-12T09:51:56.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f795c070240 con 0x7f7964144480 2022-07-12T09:51:56.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f7959d9b700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f793c05a1c0 0x7f793c05c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f796259c700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f793c053ad0 0x7f793c055f90 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f795400d000 tx=0x7f7954009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f7959d9b700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f793c05a1c0 0x7f793c05c620 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.555+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== osd.7 v2:172.21.15.111:6826/22094 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f793c05cc00 con 0x7f793c05a1c0 2022-07-12T09:51:56.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 --> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f796409bd70 con 0x7f793c05a1c0 2022-07-12T09:51:56.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f795bfff700 1 -- 172.21.15.61:0/4001880755 <== osd.7 v2:172.21.15.111:6826/22094 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f796409bd70 con 0x7f793c05a1c0 2022-07-12T09:51:56.603 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-12T09:51:56.604 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-12T09:51:56.604 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-12T09:51:56.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f793c05a1c0 msgr2=0x7f793c05c620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6826/22094,v1:172.21.15.111:6827/22094] conn(0x7f793c05a1c0 0x7f793c05c620 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f793c053ad0 msgr2=0x7f793c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f793c053ad0 0x7f793c055f90 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f795400d000 tx=0x7f7954009040).stop 2022-07-12T09:51:56.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7964144480 msgr2=0x7f7964155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 --2- 172.21.15.61:0/4001880755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7964144480 0x7f7964155c60 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 rx=0x7f795c001490 tx=0x7f795c007e30).stop 2022-07-12T09:51:56.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f7962d9d700 1 -- 172.21.15.61:0/4001880755 reap_dead start 2022-07-12T09:51:56.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 shutdown_connections 2022-07-12T09:51:56.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 >> 172.21.15.61:0/4001880755 conn(0x7f79640b70b0 msgr2=0x7f79640ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 shutdown_connections 2022-07-12T09:51:56.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.571+0000 7f796359e700 1 -- 172.21.15.61:0/4001880755 wait complete. 2022-07-12T09:51:56.614 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2022-07-12T09:51:56.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 Processor -- start 2022-07-12T09:51:56.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 -- start start 2022-07-12T09:51:56.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f8cd00ab290 0x7f8cd00ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd00afa60 0x7f8cd00aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00abc70 con 0x7f8cd005f140 2022-07-12T09:51:56.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd005f580 con 0x7f8cd00ab290 2022-07-12T09:51:56.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd621e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd005c720 con 0x7f8cd00afa60 2022-07-12T09:51:56.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd4a1b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8cd005f140 0x7f8cd005ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51298/0 (socket says 172.21.15.61:51298) 2022-07-12T09:51:56.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.723+0000 7f8cd4a1b700 1 -- 172.21.15.61:0/2621184918 learned_addr learned my addr 172.21.15.61:0/2621184918 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:56.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd521c700 1 --2- 172.21.15.61:0/2621184918 >> v2:172.21.15.111:3300/0 conn(0x7f8cd00ab290 0x7f8cd00ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd5a1d700 1 --2- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd00afa60 0x7f8cd00aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cc7fff700 1 -- 172.21.15.61:0/2621184918 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903389401 0 0) 0x7f8cd00abc70 con 0x7f8cd005f140 2022-07-12T09:51:56.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cc7fff700 1 -- 172.21.15.61:0/2621184918 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb4003260 con 0x7f8cd005f140 2022-07-12T09:51:56.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd521c700 1 -- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd00afa60 msgr2=0x7f8cd00aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:51:56.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd521c700 1 --2- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd00afa60 0x7f8cd00aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd521c700 1 -- 172.21.15.61:0/2621184918 >> v1:172.21.15.61:6789/0 conn(0x7f8cd005f140 legacy=0x7f8cd005ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd521c700 1 -- 172.21.15.61:0/2621184918 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd005c9c0 con 0x7f8cd00ab290 2022-07-12T09:51:56.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd521c700 1 --2- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00ab290 0x7f8cd00ab670 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f8cc0002f70 tx=0x7f8cc0006df0).ready entity=mon.1 client_cookie=b9647a1b0189e829 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cc7fff700 1 -- 172.21.15.61:0/2621184918 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc000d4a0 con 0x7f8cd00ab290 2022-07-12T09:51:56.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cc7fff700 1 -- 172.21.15.61:0/2621184918 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0000e90 con 0x7f8cd00ab290 2022-07-12T09:51:56.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cc7fff700 1 -- 172.21.15.61:0/2621184918 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cc000d7f0 con 0x7f8cd00ab290 2022-07-12T09:51:56.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 -- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00ab290 msgr2=0x7f8cd00ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 --2- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00ab290 0x7f8cd00ab670 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f8cc0002f70 tx=0x7f8cc0006df0).stop 2022-07-12T09:51:56.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 -- 172.21.15.61:0/2621184918 shutdown_connections 2022-07-12T09:51:56.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 --2- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd00afa60 0x7f8cd00aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 --2- 172.21.15.61:0/2621184918 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00ab290 0x7f8cd00ab670 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 -- 172.21.15.61:0/2621184918 >> 172.21.15.61:0/2621184918 conn(0x7f8cd00b1950 msgr2=0x7f8cd00b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 -- 172.21.15.61:0/2621184918 shutdown_connections 2022-07-12T09:51:56.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 -- 172.21.15.61:0/2621184918 wait complete. 2022-07-12T09:51:56.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 Processor -- start 2022-07-12T09:51:56.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.727+0000 7f8cd621e700 1 -- start start 2022-07-12T09:51:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd005f140 0x7f8cd0099b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 0x7f8cd00647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00afa60 0x7f8cd0064ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0067830 con 0x7f8cd00ab290 2022-07-12T09:51:56.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0148720 con 0x7f8cd00afa60 2022-07-12T09:51:56.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd01489c0 con 0x7f8cd005f140 2022-07-12T09:51:56.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 0x7f8cd00647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 0x7f8cd00647f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34856/0 (socket says 172.21.15.61:34856) 2022-07-12T09:51:56.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd5a1d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00afa60 0x7f8cd0064ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd521c700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd005f140 0x7f8cd0099b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 -- 172.21.15.61:0/4103844685 learned_addr learned my addr 172.21.15.61:0/4103844685 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:56.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 -- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd005f140 msgr2=0x7f8cd0099b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd005f140 0x7f8cd0099b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 -- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00afa60 msgr2=0x7f8cd0064ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00afa60 0x7f8cd0064ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 -- 172.21.15.61:0/4103844685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc000b180 con 0x7f8cd00ab290 2022-07-12T09:51:56.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd4a1b700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 0x7f8cd00647f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7f8ccc001490 tx=0x7f8ccc007ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ccc005ba0 con 0x7f8cd00ab290 2022-07-12T09:51:56.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc005d00 con 0x7f8cd00ab290 2022-07-12T09:51:56.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ccc0072c0 con 0x7f8cd00ab290 2022-07-12T09:51:56.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd00674b0 con 0x7f8cd00ab290 2022-07-12T09:51:56.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.731+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0066b10 con 0x7f8cd00ab290 2022-07-12T09:51:56.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.735+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8ccc010070 con 0x7f8cd00ab290 2022-07-12T09:51:56.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.735+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd0057460 con 0x7f8cd00ab290 2022-07-12T09:51:56.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.735+0000 7f8cc67fc700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8cb80539b0 0x7f8cb8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:56.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.735+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8ccc06b4f0 con 0x7f8cd00ab290 2022-07-12T09:51:56.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.735+0000 7f8cd521c700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8cb80539b0 0x7f8cb8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:56.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.735+0000 7f8cd521c700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8cb80539b0 0x7f8cb8055e70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f8cc001a000 tx=0x7f8cc0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:56.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.743+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ccc0425a0 con 0x7f8cd00ab290 2022-07-12T09:51:56.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cd0066b10 con 0x7f8cb80539b0 2022-07-12T09:51:56.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cc67fc700 1 -- 172.21.15.61:0/4103844685 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8cd0066b10 con 0x7f8cb80539b0 2022-07-12T09:51:56.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8cb80539b0 msgr2=0x7f8cb8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8cb80539b0 0x7f8cb8055e70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7f8cc001a000 tx=0x7f8cc0014000).stop 2022-07-12T09:51:56.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 msgr2=0x7f8cd00647f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:56.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 0x7f8cd00647f0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 rx=0x7f8ccc001490 tx=0x7f8ccc007ea0).stop 2022-07-12T09:51:56.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 shutdown_connections 2022-07-12T09:51:56.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f8cb80539b0 0x7f8cb8055e70 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8cd00afa60 0x7f8cd0064ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd00ab290 0x7f8cd00647f0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 --2- 172.21.15.61:0/4103844685 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f8cd005f140 0x7f8cd0099b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:56.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 >> 172.21.15.61:0/4103844685 conn(0x7f8cd00b1950 msgr2=0x7f8cd00b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:56.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.871+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 shutdown_connections 2022-07-12T09:51:56.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.875+0000 7f8cd621e700 1 -- 172.21.15.61:0/4103844685 wait complete. 2022-07-12T09:51:56.880 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 7 to deep-scrub 2022-07-12T09:51:56.888 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-12T09:51:57.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.987+0000 7faaba5a3700 1 Processor -- start 2022-07-12T09:51:57.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.987+0000 7faaba5a3700 1 -- start start 2022-07-12T09:51:57.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faab40b1f80 0x7faab40ab900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:57.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab405d480 con 0x7faab40abdf0 2022-07-12T09:51:57.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40b16d0 con 0x7faab40afc70 2022-07-12T09:51:57.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40b0fc0 con 0x7faab40b1f80 2022-07-12T09:51:57.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaabfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7faab40abdf0 0x7faab405bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51312/0 (socket says 172.21.15.61:51312) 2022-07-12T09:51:57.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaabfff700 1 -- 172.21.15.61:0/3395590048 learned_addr learned my addr 172.21.15.61:0/3395590048 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:51:57.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab3fff700 1 --2- 172.21.15.61:0/3395590048 >> v2:172.21.15.61:3300/0 conn(0x7faab40b1f80 0x7faab40ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:57.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354948494 0 0) 0x7faab405d480 con 0x7faab40abdf0 2022-07-12T09:51:57.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa94003260 con 0x7faab40abdf0 2022-07-12T09:51:57.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098041435 0 0) 0x7faab40b16d0 con 0x7faab40afc70 2022-07-12T09:51:57.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab405d480 con 0x7faab40afc70 2022-07-12T09:51:57.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab3fff700 1 -- 172.21.15.61:0/3395590048 >> v1:172.21.15.111:6789/0 conn(0x7faab40afc70 legacy=0x7faab405c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab3fff700 1 -- 172.21.15.61:0/3395590048 >> v1:172.21.15.61:6789/0 conn(0x7faab40abdf0 legacy=0x7faab405bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236315852 0 0) 0x7faa94003260 con 0x7faab40abdf0 2022-07-12T09:51:57.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab3fff700 1 -- 172.21.15.61:0/3395590048 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40b2360 con 0x7faab40b1f80 2022-07-12T09:51:57.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab3fff700 1 --2- 172.21.15.61:0/3395590048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b1f80 0x7faab40ab900 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7faa9c001110 tx=0x7faaa4006df0).ready entity=mon.0 client_cookie=2619c36eb274b3d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:57.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa9c001660 con 0x7faab40b1f80 2022-07-12T09:51:57.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa4000e90 con 0x7faab40b1f80 2022-07-12T09:51:57.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faab37fe700 1 -- 172.21.15.61:0/3395590048 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaa4012460 con 0x7faab40b1f80 2022-07-12T09:51:57.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- 172.21.15.61:0/3395590048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b1f80 msgr2=0x7faab40ab900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 --2- 172.21.15.61:0/3395590048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b1f80 0x7faab40ab900 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 rx=0x7faa9c001110 tx=0x7faaa4006df0).stop 2022-07-12T09:51:57.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- 172.21.15.61:0/3395590048 shutdown_connections 2022-07-12T09:51:57.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 --2- 172.21.15.61:0/3395590048 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40b1f80 0x7faab40ab900 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- 172.21.15.61:0/3395590048 >> 172.21.15.61:0/3395590048 conn(0x7faab4062c00 msgr2=0x7faab4062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:57.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- 172.21.15.61:0/3395590048 shutdown_connections 2022-07-12T09:51:57.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.991+0000 7faaba5a3700 1 -- 172.21.15.61:0/3395590048 wait complete. 2022-07-12T09:51:57.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 Processor -- start 2022-07-12T09:51:57.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- start start 2022-07-12T09:51:57.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 0x7faab4144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:57.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab40afc70 0x7faab4065550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:57.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b1f80 0x7faab4148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:57.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4148e00 con 0x7faab40abdf0 2022-07-12T09:51:57.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4144bd0 con 0x7faab40b1f80 2022-07-12T09:51:57.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab4066560 con 0x7faab40afc70 2022-07-12T09:51:57.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 0x7faab4144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:57.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 0x7faab4144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34868/0 (socket says 172.21.15.61:34868) 2022-07-12T09:51:57.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 -- 172.21.15.61:0/1991165116 learned_addr learned my addr 172.21.15.61:0/1991165116 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:51:57.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaabfff700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab40afc70 0x7faab4065550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:57.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab8b40700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b1f80 0x7faab4148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:57.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 -- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab40afc70 msgr2=0x7faab4065550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab40afc70 0x7faab4065550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 -- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b1f80 msgr2=0x7faab4148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b1f80 0x7faab4148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 -- 172.21.15.61:0/1991165116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa400b180 con 0x7faab40abdf0 2022-07-12T09:51:57.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab3fff700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 0x7faab4144630 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7faa9c001110 tx=0x7faaa4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:57.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaa4016070 con 0x7faab40abdf0 2022-07-12T09:51:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab4066800 con 0x7faab40abdf0 2022-07-12T09:51:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4066ec0 con 0x7faab40abdf0 2022-07-12T09:51:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa400bec0 con 0x7faab40abdf0 2022-07-12T09:51:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaa4012460 con 0x7faab40abdf0 2022-07-12T09:51:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.995+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c003410 con 0x7faab40abdf0 2022-07-12T09:51:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.999+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faaa400f030 con 0x7faab40abdf0 2022-07-12T09:51:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.999+0000 7faab1ffb700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa98053ad0 0x7faa98055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:51:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.999+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7faaa406c990 con 0x7faab40abdf0 2022-07-12T09:51:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.999+0000 7faaabfff700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa98053ad0 0x7faa98055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:51:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:56.999+0000 7faaabfff700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa98053ad0 0x7faa98055f90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7faa9c00d000 tx=0x7faa9c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:51:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.003+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faaa4043a40 con 0x7faab40abdf0 2022-07-12T09:51:57.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.127+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa8c002070 con 0x7faa98053ad0 2022-07-12T09:51:57.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faab1ffb700 1 -- 172.21.15.61:0/1991165116 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+67903 (secure 0 0 0) 0x7faa8c002070 con 0x7faa98053ad0 2022-07-12T09:51:57.137 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T09:51:57.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa98053ad0 msgr2=0x7faa98055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa98053ad0 0x7faa98055f90 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7faa9c00d000 tx=0x7faa9c000bc0).stop 2022-07-12T09:51:57.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 msgr2=0x7faab4144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:51:57.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 0x7faab4144630 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 rx=0x7faa9c001110 tx=0x7faaa4014000).stop 2022-07-12T09:51:57.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 shutdown_connections 2022-07-12T09:51:57.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7faa98053ad0 0x7faa98055f90 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faab40b1f80 0x7faab4148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7faab40afc70 0x7faab4065550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 --2- 172.21.15.61:0/1991165116 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faab40abdf0 0x7faab4144630 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:51:57.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 >> 172.21.15.61:0/1991165116 conn(0x7faab4062c00 msgr2=0x7faab40b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:51:57.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 shutdown_connections 2022-07-12T09:51:57.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:51:57.135+0000 7faaba5a3700 1 -- 172.21.15.61:0/1991165116 wait complete. 2022-07-12T09:51:57.146 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-12T09:51:57.150 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2278,"stamp":"2022-07-12T09:51:56.261075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385120,"kb_used_data":743209920,"kb_used_omap":27846,"kb_used_meta":0,"kb_avail":743209920,"statfs":{"total":767585320960,"available":761046958080,"internally_reserved":0,"allocated":761046958080,"data_stored":761046958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28515244,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.475725"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":138,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853075+0000","last_change":"2022-07-12T08:41:04.955012+0000","last_active":"2022-07-12T08:51:20.853075+0000","last_peered":"2022-07-12T08:51:20.853075+0000","last_clean":"2022-07-12T08:51:20.853075+0000","last_became_active":"2022-07-12T08:41:04.954339+0000","last_became_peered":"2022-07-12T08:41:04.954339+0000","last_unstale":"2022-07-12T08:51:20.853075+0000","last_undegraded":"2022-07-12T08:51:20.853075+0000","last_fullsized":"2022-07-12T08:51:20.853075+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:40:58.647421+0000","last_clean_scrub_stamp":"2022-07-12T08:40:58.647421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:19.737924+0000","last_change":"2022-07-12T08:41:11.067479+0000","last_active":"2022-07-12T08:51:19.737924+0000","last_peered":"2022-07-12T08:51:19.737924+0000","last_clean":"2022-07-12T08:51:19.737924+0000","last_became_active":"2022-07-12T08:41:11.067215+0000","last_became_peered":"2022-07-12T08:41:11.067215+0000","last_unstale":"2022-07-12T08:51:19.737924+0000","last_undegraded":"2022-07-12T08:51:19.737924+0000","last_fullsized":"2022-07-12T08:51:19.737924+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.852812+0000","last_change":"2022-07-12T08:41:11.064070+0000","last_active":"2022-07-12T08:51:20.852812+0000","last_peered":"2022-07-12T08:51:20.852812+0000","last_clean":"2022-07-12T08:51:20.852812+0000","last_became_active":"2022-07-12T08:41:11.063329+0000","last_became_peered":"2022-07-12T08:41:11.063329+0000","last_unstale":"2022-07-12T08:51:20.852812+0000","last_undegraded":"2022-07-12T08:51:20.852812+0000","last_fullsized":"2022-07-12T08:51:20.852812+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":133,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-07-12T08:51:17.742997+0000","last_change":"2022-07-12T08:41:11.071303+0000","last_active":"2022-07-12T08:51:17.742997+0000","last_peered":"2022-07-12T08:51:17.742997+0000","last_clean":"2022-07-12T08:51:17.742997+0000","last_became_active":"2022-07-12T08:41:11.070549+0000","last_became_peered":"2022-07-12T08:41:11.070549+0000","last_unstale":"2022-07-12T08:51:17.742997+0000","last_undegraded":"2022-07-12T08:51:17.742997+0000","last_fullsized":"2022-07-12T08:51:17.742997+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:19.737982+0000","last_change":"2022-07-12T08:41:11.066220+0000","last_active":"2022-07-12T08:51:19.737982+0000","last_peered":"2022-07-12T08:51:19.737982+0000","last_clean":"2022-07-12T08:51:19.737982+0000","last_became_active":"2022-07-12T08:41:11.065911+0000","last_became_peered":"2022-07-12T08:41:11.065911+0000","last_unstale":"2022-07-12T08:51:19.737982+0000","last_undegraded":"2022-07-12T08:51:19.737982+0000","last_fullsized":"2022-07-12T08:51:19.737982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853881+0000","last_change":"2022-07-12T08:41:11.958581+0000","last_active":"2022-07-12T08:51:20.853881+0000","last_peered":"2022-07-12T08:51:20.853881+0000","last_clean":"2022-07-12T08:51:20.853881+0000","last_became_active":"2022-07-12T08:41:11.958006+0000","last_became_peered":"2022-07-12T08:41:11.958006+0000","last_unstale":"2022-07-12T08:51:20.853881+0000","last_undegraded":"2022-07-12T08:51:20.853881+0000","last_fullsized":"2022-07-12T08:51:20.853881+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853703+0000","last_change":"2022-07-12T08:41:11.954669+0000","last_active":"2022-07-12T08:51:20.853703+0000","last_peered":"2022-07-12T08:51:20.853703+0000","last_clean":"2022-07-12T08:51:20.853703+0000","last_became_active":"2022-07-12T08:41:11.954063+0000","last_became_peered":"2022-07-12T08:41:11.954063+0000","last_unstale":"2022-07-12T08:51:20.853703+0000","last_undegraded":"2022-07-12T08:51:20.853703+0000","last_fullsized":"2022-07-12T08:51:20.853703+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853929+0000","last_change":"2022-07-12T08:41:11.069495+0000","last_active":"2022-07-12T08:51:20.853929+0000","last_peered":"2022-07-12T08:51:20.853929+0000","last_clean":"2022-07-12T08:51:20.853929+0000","last_became_active":"2022-07-12T08:41:11.068746+0000","last_became_peered":"2022-07-12T08:41:11.068746+0000","last_unstale":"2022-07-12T08:51:20.853929+0000","last_undegraded":"2022-07-12T08:51:20.853929+0000","last_fullsized":"2022-07-12T08:51:20.853929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":149,"state":"active+clean","last_fresh":"2022-07-12T08:51:17.743069+0000","last_change":"2022-07-12T08:41:11.064835+0000","last_active":"2022-07-12T08:51:17.743069+0000","last_peered":"2022-07-12T08:51:17.743069+0000","last_clean":"2022-07-12T08:51:17.743069+0000","last_became_active":"2022-07-12T08:41:11.064473+0000","last_became_peered":"2022-07-12T08:41:11.064473+0000","last_unstale":"2022-07-12T08:51:17.743069+0000","last_undegraded":"2022-07-12T08:51:17.743069+0000","last_fullsized":"2022-07-12T08:51:17.743069+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575700,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3617,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3704129,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88200000000000001,"15min":0.88400000000000001},"min":{"1min":0.68500000000000005,"5min":0.53300000000000003,"15min":0.50700000000000001},"max":{"1min":1.0369999999999999,"5min":1.137,"15min":1.1479999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86199999999999999,"15min":0.879},"min":{"1min":0.72599999999999998,"5min":0.623,"15min":0.60799999999999998},"max":{"1min":0.998,"5min":1.1040000000000001,"15min":1.111},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.56499999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0820000000000001,"5min":1.1659999999999999,"15min":1.214},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.64000000000000001,"5min":0.60199999999999998,"15min":0.58099999999999996},"max":{"1min":1.0700000000000001,"5min":1.109,"15min":1.1279999999999999},"last":0.96299999999999997}]},{"osd":2,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.82599999999999996,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.016,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.86399999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1220000000000001,"5min":1.242,"15min":1.242},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.58499999999999996},"max":{"1min":1.1080000000000001,"5min":1.232,"15min":1.232},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.89300000000000002,"15min":0.90200000000000002},"min":{"1min":0.65200000000000002,"5min":0.60299999999999998,"15min":0.57899999999999996},"max":{"1min":1.1599999999999999,"5min":1.1859999999999999,"15min":1.22},"last":0.73799999999999999}]},{"osd":4,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.67900000000000005,"15min":0.71499999999999997},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.47199999999999998},"max":{"1min":0.94999999999999996,"5min":1.145,"15min":1.1819999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88400000000000001,"15min":0.89100000000000001},"min":{"1min":0.70599999999999996,"5min":0.64600000000000002,"15min":0.44600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1759999999999999,"15min":1.2070000000000001},"last":0.85599999999999998}]},{"osd":5,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":1.198,"5min":1.395,"15min":1.395},"last":0.871},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.78300000000000003,"5min":0.56100000000000005,"15min":0.46100000000000002},"max":{"1min":1.121,"5min":1.1639999999999999,"15min":1.194},"last":1.0249999999999999}]},{"osd":6,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.91100000000000003,"15min":0.91400000000000003},"min":{"1min":0.65200000000000002,"5min":0.55500000000000005,"15min":0.434},"max":{"1min":1.077,"5min":1.1279999999999999,"15min":1.2410000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.67000000000000004,"5min":0.52000000000000002,"15min":0.39600000000000002},"max":{"1min":1.1830000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.89200000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":4018,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4115239,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80100000000000005,"15min":0.78300000000000003},"min":{"1min":0.63,"5min":0.54300000000000004,"15min":0.52700000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"max":{"1min":1.012,"5min":1.0580000000000001,"15min":1.2130000000000001},"last":0.95999999999999996}]},{"osd":1,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85099999999999998,"15min":0.874},"min":{"1min":0.66500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0549999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.71899999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1160000000000001,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.91500000000000004}]},{"osd":2,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.874,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.55900000000000005},"max":{"1min":1.0900000000000001,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.96899999999999997,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.86799999999999999}]},{"osd":3,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85399999999999998,"15min":0.84399999999999997},"min":{"1min":0.66200000000000003,"5min":0.65400000000000003,"15min":0.621},"max":{"1min":1.077,"5min":1.1140000000000001,"15min":1.179},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.91300000000000003,"15min":0.92200000000000004},"min":{"1min":0.70799999999999996,"5min":0.63700000000000001,"15min":0.61599999999999999},"max":{"1min":1.054,"5min":1.125,"15min":1.2150000000000001},"last":0.89200000000000002}]},{"osd":4,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80900000000000005,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0529999999999999,"5min":1.1950000000000001,"15min":1.2010000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.47899999999999998},"max":{"1min":1.139,"5min":1.175,"15min":1.413},"last":0.78200000000000003}]},{"osd":5,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88500000000000001,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.57999999999999996,"15min":0.54600000000000004},"max":{"1min":1.1160000000000001,"5min":1.591,"15min":1.591},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.006,"5min":1.0069999999999999,"15min":0.996},"min":{"1min":0.77900000000000003,"5min":0.77300000000000002,"15min":0.626},"max":{"1min":1.1899999999999999,"5min":1.246,"15min":1.2589999999999999},"last":0.82999999999999996}]},{"osd":7,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.97799999999999998,"15min":0.96499999999999997},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.57999999999999996},"max":{"1min":1.135,"5min":1.365,"15min":1.4350000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.96499999999999997,"15min":0.95799999999999996},"min":{"1min":0.73899999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1639999999999999,"5min":1.77,"15min":1.77},"last":0.98399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575700,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798028,"kb_used_data":92901352,"kb_used_omap":3571,"kb_used_meta":0,"kb_avail":92901352,"statfs":{"total":95948165120,"available":95130984448,"internally_reserved":0,"allocated":95130984448,"data_stored":95130984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3657716,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84599999999999997,"15min":0.82999999999999996},"min":{"1min":0.66800000000000004,"5min":0.621,"15min":0.499},"max":{"1min":1.079,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82199999999999995,"15min":0.81399999999999995},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.51400000000000001},"max":{"1min":1.071,"5min":1.1499999999999999,"15min":1.1830000000000001},"last":0.66900000000000004}]},{"osd":1,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.84099999999999997,"15min":0.82799999999999996},"min":{"1min":0.63,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.089,"5min":1.149,"15min":1.149},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.85199999999999998,"15min":0.85999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.55400000000000005},"max":{"1min":1.002,"5min":1.0860000000000001,"15min":1.1499999999999999},"last":0.70699999999999996}]},{"osd":2,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.85399999999999998,"15min":0.85199999999999998},"min":{"1min":0.64900000000000002,"5min":0.63200000000000001,"15min":0.59299999999999997},"max":{"1min":1.1879999999999999,"5min":1.272,"15min":1.272},"last":0.75},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88,"15min":0.88100000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.59599999999999997},"max":{"1min":1.077,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.85799999999999998}]},{"osd":3,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.68300000000000005,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":0.997,"5min":1.0529999999999999,"15min":1.45},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.90500000000000003,"15min":0.90300000000000002},"min":{"1min":0.66600000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.4079999999999999},"last":0.82599999999999996}]},{"osd":5,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.80800000000000005,"15min":0.80400000000000005},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.51800000000000002},"max":{"1min":0.95399999999999996,"5min":1.2010000000000001,"15min":1.258},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85399999999999998,"15min":0.86299999999999999},"min":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.51200000000000001},"max":{"1min":1.163,"5min":1.2010000000000001,"15min":1.367},"last":0.88300000000000001}]},{"osd":6,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.86399999999999999,"15min":0.86299999999999999},"min":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.56000000000000005},"max":{"1min":1.258,"5min":1.258,"15min":1.349},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.93899999999999995},"min":{"1min":0.70499999999999996,"5min":0.623,"15min":0.59399999999999997},"max":{"1min":1.159,"5min":1.282,"15min":1.3109999999999999},"last":0.93500000000000005}]},{"osd":7,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93600000000000005,"15min":0.94099999999999995},"min":{"1min":0.61399999999999999,"5min":0.61299999999999999,"15min":0.60099999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.4199999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.044,"5min":1.0489999999999999,"15min":1.044},"min":{"1min":0.71699999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.343,"5min":1.3740000000000001,"15min":1.4390000000000001},"last":0.98299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834575700,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3339,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3420122,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.86299999999999999,"5min":1.0800000000000001,"15min":1.266},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.63100000000000001,"5min":0.57499999999999996,"15min":0.57299999999999995},"max":{"1min":0.996,"5min":1.0589999999999999,"15min":1.1180000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.54100000000000004,"5min":0.47399999999999998,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":1.0009999999999999,"15min":1.8169999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.76000000000000001},"min":{"1min":0.53800000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.96399999999999997,"5min":1.0780000000000001,"15min":1.5369999999999999},"last":0.63}]},{"osd":2,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.68500000000000005,"15min":0.68899999999999995},"min":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.5},"max":{"1min":1.034,"5min":1.248,"15min":1.248},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.72299999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.5720000000000001},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.64200000000000002},"max":{"1min":1.149,"5min":1.236,"15min":1.405},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.92400000000000004,"15min":0.93500000000000005},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68100000000000005},"max":{"1min":1.155,"5min":1.3360000000000001,"15min":1.708},"last":0.65900000000000003}]},{"osd":5,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.92700000000000005,"15min":0.93600000000000005},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65000000000000002},"max":{"1min":1.1439999999999999,"5min":1.2949999999999999,"15min":1.4319999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95999999999999996,"15min":0.96899999999999997},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.1950000000000001,"5min":1.3580000000000001,"15min":1.5129999999999999},"last":0.77500000000000002}]},{"osd":6,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.026,"15min":1.01},"min":{"1min":0.78700000000000003,"5min":0.73899999999999999,"15min":0.70399999999999996},"max":{"1min":1.238,"5min":1.966,"15min":1.966},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0649999999999999,"15min":1.0620000000000001},"min":{"1min":0.89900000000000002,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.6519999999999999},"last":0.873}]},{"osd":7,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0509999999999999,"15min":1.0600000000000001},"min":{"1min":0.78400000000000003,"5min":0.75700000000000001,"15min":0.73099999999999998},"max":{"1min":1.25,"5min":1.2989999999999999,"15min":1.468},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.083,"5min":1.085,"15min":1.0840000000000001},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.2749999999999999,"5min":1.5920000000000001,"15min":1.706},"last":0.83799999999999997}]}]},{"osd":2,"up_from":13,"seq":55834575700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2801,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2868423,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.50700000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":1.155},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70099999999999996,"15min":0.70799999999999996},"min":{"1min":0.49399999999999999,"5min":0.48399999999999999,"15min":0.47999999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.1339999999999999},"last":0.623}]},{"osd":1,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.46200000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.2529999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.54400000000000004,"5min":0.54300000000000004,"15min":0.53000000000000003},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.92000000000000004}]},{"osd":3,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72499999999999998,"15min":0.74199999999999999},"min":{"1min":0.52800000000000002,"5min":0.51000000000000001,"15min":0.48499999999999999},"max":{"1min":0.92900000000000005,"5min":1.069,"15min":1.1839999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.82499999999999996,"15min":0.82799999999999996},"min":{"1min":0.66800000000000004,"5min":0.60599999999999998,"15min":0.57899999999999996},"max":{"1min":1.087,"5min":1.0900000000000001,"15min":1.123},"last":0.70699999999999996}]},{"osd":4,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.95599999999999996,"15min":0.95999999999999996},"min":{"1min":0.80900000000000005,"5min":0.70199999999999996,"15min":0.69299999999999995},"max":{"1min":1.256,"5min":1.256,"15min":1.3680000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0169999999999999,"5min":0.94199999999999995,"15min":0.93999999999999995},"min":{"1min":0.83599999999999997,"5min":0.69399999999999995,"15min":0.67100000000000004},"max":{"1min":1.262,"5min":1.262,"15min":1.3109999999999999},"last":0.87}]},{"osd":5,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.97699999999999998,"15min":0.99099999999999999},"min":{"1min":0.76600000000000001,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.2589999999999999,"5min":1.321,"15min":1.341},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.97699999999999998,"15min":0.98799999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.66100000000000003},"max":{"1min":1.3360000000000001,"5min":1.476,"15min":1.476},"last":0.81799999999999995}]},{"osd":6,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.99099999999999999,"15min":0.98599999999999999},"min":{"1min":0.86599999999999999,"5min":0.73499999999999999,"15min":0.67600000000000005},"max":{"1min":1.298,"5min":1.425,"15min":1.425},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.0569999999999999,"15min":1.0589999999999999},"min":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.2490000000000001,"5min":1.379,"15min":1.4059999999999999},"last":0.97399999999999998}]},{"osd":7,"last update":"Tue Jul 12 09:51:09 2022","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.05,"15min":1.073},"min":{"1min":0.85399999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.4099999999999999,"5min":1.591,"15min":1.591},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.079,"15min":1.073},"min":{"1min":0.96399999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.2669999999999999,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.107}]}]},{"osd":1,"up_from":13,"seq":55834575700,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3966,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4061662,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":1.046,"15min":1.216},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.502,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.089,"5min":1.1779999999999999,"15min":1.226},"last":0.46400000000000002}]},{"osd":2,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.44},"max":{"1min":0.78400000000000003,"5min":1.143,"15min":1.1879999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.76600000000000001,"15min":0.76500000000000001},"min":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.075,"5min":1.149,"15min":1.3620000000000001},"last":0.79200000000000004}]},{"osd":3,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81899999999999995,"15min":0.83599999999999997},"min":{"1min":0.63500000000000001,"5min":0.60599999999999998,"15min":0.56599999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.8779999999999999},"last":0.754},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76100000000000001,"15min":0.78300000000000003},"min":{"1min":0.59799999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92800000000000005,"5min":1.147,"15min":1.258},"last":0.73099999999999998}]},{"osd":4,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.89900000000000002,"15min":0.90200000000000002},"min":{"1min":0.77100000000000002,"5min":0.64100000000000001,"15min":0.63700000000000001},"max":{"1min":1.1830000000000001,"5min":1.278,"15min":1.3},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0089999999999999,"15min":1.012},"min":{"1min":0.88800000000000001,"5min":0.79400000000000004,"15min":0.68999999999999995},"max":{"1min":1.2330000000000001,"5min":1.321,"15min":1.444},"last":0.95599999999999996}]},{"osd":5,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.995,"15min":0.98699999999999999},"min":{"1min":0.85199999999999998,"5min":0.73399999999999999,"15min":0.70799999999999996},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0009999999999999,"15min":0.99199999999999999},"min":{"1min":0.76400000000000001,"5min":0.749,"15min":0.63300000000000001},"max":{"1min":1.28,"5min":1.28,"15min":1.397},"last":0.69299999999999995}]},{"osd":6,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.004,"15min":1.002},"min":{"1min":0.70599999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.341,"5min":1.341,"15min":1.349},"last":1},{"interface":"front","average":{"1min":1.042,"5min":1.022,"15min":1.016},"min":{"1min":0.80000000000000004,"5min":0.753,"15min":0.68600000000000005},"max":{"1min":1.359,"5min":1.359,"15min":1.5189999999999999},"last":0.90000000000000002}]},{"osd":7,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0960000000000001},"min":{"1min":0.93100000000000005,"5min":0.78400000000000003,"15min":0.73099999999999998},"max":{"1min":1.319,"5min":1.3939999999999999,"15min":1.5049999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.071,"15min":1.085},"min":{"1min":0.90500000000000003,"5min":0.78500000000000003,"15min":0.72399999999999998},"max":{"1min":1.228,"5min":1.3680000000000001,"15min":1.583},"last":1.0409999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575700,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3791,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3882329,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.49299999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77600000000000002,"5min":0.89200000000000002,"15min":0.99399999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70699999999999996,"15min":0.70799999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0880000000000001,"5min":1.095,"15min":1.1819999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.753,"15min":0.72199999999999998},"min":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.51900000000000002},"max":{"1min":1.0660000000000001,"5min":1.2210000000000001,"15min":1.2270000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.64700000000000002,"15min":0.66900000000000004},"min":{"1min":0.51000000000000001,"5min":0.48399999999999999,"15min":0.42699999999999999},"max":{"1min":1.0600000000000001,"5min":1.129,"15min":1.224},"last":0.64700000000000002}]},{"osd":3,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94899999999999995,"5min":1.0640000000000001,"15min":1.1799999999999999},"last":0.995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.876,"15min":0.877},"min":{"1min":0.72599999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0109999999999999,"5min":1.2130000000000001,"15min":1.246},"last":1.1020000000000001}]},{"osd":4,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.1180000000000001,"5min":1.133,"15min":1.391},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94999999999999996,"15min":0.94299999999999995},"min":{"1min":0.76200000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.92800000000000005}]},{"osd":5,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":0.95999999999999996},"min":{"1min":0.748,"5min":0.67200000000000004,"15min":0.66500000000000004},"max":{"1min":1.361,"5min":1.395,"15min":1.48},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.98299999999999998,"15min":0.97999999999999998},"min":{"1min":0.81299999999999994,"5min":0.70299999999999996,"15min":0.69999999999999996},"max":{"1min":1.2330000000000001,"5min":1.258,"15min":1.498},"last":0.89800000000000002}]},{"osd":6,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.02,"15min":1.022},"min":{"1min":0.78100000000000003,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.238,"5min":1.3169999999999999,"15min":1.522},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96399999999999997,"15min":0.99099999999999999},"min":{"1min":0.74199999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.137,"5min":1.3759999999999999,"15min":1.4650000000000001},"last":1.0129999999999999}]},{"osd":7,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0049999999999999,"15min":1.0309999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.68799999999999994},"max":{"1min":1.179,"5min":1.4550000000000001,"15min":1.5489999999999999},"last":1.151},{"interface":"front","average":{"1min":1.032,"5min":1.079,"15min":1.087},"min":{"1min":0.83999999999999997,"5min":0.79300000000000004,"15min":0.77200000000000002},"max":{"1min":1.5,"5min":1.524,"15min":1.569},"last":1.0880000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2739,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2805624,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.63600000000000001,"5min":0.63100000000000001,"15min":0.58099999999999996},"max":{"1min":0.98999999999999999,"5min":1.194,"15min":1.224},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90200000000000002,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.65100000000000002,"15min":0.59999999999999998},"max":{"1min":1.0089999999999999,"5min":1.161,"15min":1.208},"last":0.83699999999999997}]},{"osd":1,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.5},"max":{"1min":1.073,"5min":1.198,"15min":1.22},"last":1.042},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.51600000000000001},"max":{"1min":1.052,"5min":1.109,"15min":1.109},"last":0.77400000000000002}]},{"osd":2,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.66800000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0429999999999999,"5min":1.127,"15min":1.1819999999999999},"last":1},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.67300000000000004,"5min":0.61599999999999999,"15min":0.55800000000000005},"max":{"1min":0.99099999999999999,"5min":1.1759999999999999,"15min":1.1859999999999999},"last":0.66300000000000003}]},{"osd":3,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.57799999999999996},"max":{"1min":1.0609999999999999,"5min":1.151,"15min":1.3360000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.879,"15min":0.88200000000000001},"min":{"1min":0.63700000000000001,"5min":0.61199999999999999,"15min":0.54100000000000004},"max":{"1min":1.069,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.69199999999999995}]},{"osd":4,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70999999999999996,"15min":0.72199999999999998},"min":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.39000000000000001},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.1719999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.94299999999999995,"15min":0.93999999999999995},"min":{"1min":0.70099999999999996,"5min":0.68799999999999994,"15min":0.58599999999999997},"max":{"1min":1.073,"5min":1.212,"15min":1.4930000000000001},"last":0.91800000000000004}]},{"osd":6,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91700000000000004,"15min":0.90900000000000003},"min":{"1min":0.72299999999999998,"5min":0.57599999999999996,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.232,"15min":1.276},"last":0.74299999999999999}]},{"osd":7,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89500000000000002,"15min":0.89700000000000002},"min":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0389999999999999,"5min":1.2809999999999999,"15min":2.1200000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.002,"5min":1.0169999999999999,"15min":1.0129999999999999},"min":{"1min":0.84199999999999997,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.482,"5min":1.482,"15min":2.1019999999999999},"last":0.93700000000000006}]}]}]}} 2022-07-12T09:51:57.161 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-07-12T08:40:58.647421+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=40, tm_sec=58, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.162 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.163 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.165 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.166 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.167 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.168 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.169 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.171 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:51:57.172 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2022-07-12T09:52:17.174 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-12T09:52:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 Processor -- start 2022-07-12T09:52:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- start start 2022-07-12T09:52:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f09700afc70 0x7f09701433e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097005d480 con 0x7f09700b1f80 2022-07-12T09:52:17.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09700b16d0 con 0x7f09700adb00 2022-07-12T09:52:17.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09700b0fc0 con 0x7f09700afc70 2022-07-12T09:52:17.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f09700afc70 0x7f09701433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:17.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f09700afc70 0x7f09701433e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34876/0 (socket says 172.21.15.61:34876) 2022-07-12T09:52:17.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 -- 172.21.15.61:0/296982549 learned_addr learned my addr 172.21.15.61:0/296982549 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:52:17.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 -- 172.21.15.61:0/296982549 >> v1:172.21.15.111:6789/0 conn(0x7f09700adb00 legacy=0x7f09701438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-12T09:52:17.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 -- 172.21.15.61:0/296982549 >> v1:172.21.15.61:6789/0 conn(0x7f09700b1f80 legacy=0x7f09700ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:17.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 -- 172.21.15.61:0/296982549 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09700b2360 con 0x7f09700afc70 2022-07-12T09:52:17.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09759db700 1 --2- 172.21.15.61:0/296982549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f09701433e0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7f09700b30c0 tx=0x7f0960006eb0).ready entity=mon.0 client_cookie=f12564aad75663ec server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:17.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09751da700 1 -- 172.21.15.61:0/296982549 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f096000d5c0 con 0x7f09700afc70 2022-07-12T09:52:17.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09751da700 1 -- 172.21.15.61:0/296982549 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0960000e90 con 0x7f09700afc70 2022-07-12T09:52:17.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f09751da700 1 -- 172.21.15.61:0/296982549 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f096000d950 con 0x7f09700afc70 2022-07-12T09:52:17.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- 172.21.15.61:0/296982549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 msgr2=0x7f09701433e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:17.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 --2- 172.21.15.61:0/296982549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f09701433e0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 rx=0x7f09700b30c0 tx=0x7f0960006eb0).stop 2022-07-12T09:52:17.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- 172.21.15.61:0/296982549 shutdown_connections 2022-07-12T09:52:17.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 --2- 172.21.15.61:0/296982549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f09701433e0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- 172.21.15.61:0/296982549 >> 172.21.15.61:0/296982549 conn(0x7f0970062c00 msgr2=0x7f0970062fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:52:17.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- 172.21.15.61:0/296982549 shutdown_connections 2022-07-12T09:52:17.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.291+0000 7f0978440700 1 -- 172.21.15.61:0/296982549 wait complete. 2022-07-12T09:52:17.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 Processor -- start 2022-07-12T09:52:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 -- start start 2022-07-12T09:52:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09700adb00 0x7f0970153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f0970065b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09700b1f80 0x7f0970147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970146a60 con 0x7f09700afc70 2022-07-12T09:52:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0970066910 con 0x7f09700b1f80 2022-07-12T09:52:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0970067490 con 0x7f09700adb00 2022-07-12T09:52:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f0970065b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f0970065b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34884/0 (socket says 172.21.15.61:34884) 2022-07-12T09:52:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 -- 172.21.15.61:0/2340360558 learned_addr learned my addr 172.21.15.61:0/2340360558 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:52:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09769dd700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09700b1f80 0x7f0970147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09761dc700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09700adb00 0x7f0970153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:17.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 -- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09700adb00 msgr2=0x7f0970153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:17.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09700adb00 0x7f0970153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 -- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09700b1f80 msgr2=0x7f0970147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:17.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09700b1f80 0x7f0970147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 -- 172.21.15.61:0/2340360558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096000b1e0 con 0x7f09700afc70 2022-07-12T09:52:17.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09761dc700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09700adb00 0x7f0970153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T09:52:17.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f09759db700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f0970065b80 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7f09700b30c0 tx=0x7f0960015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:17.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0960021070 con 0x7f09700afc70 2022-07-12T09:52:17.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09600117e0 con 0x7f09700afc70 2022-07-12T09:52:17.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0960011b40 con 0x7f09700afc70 2022-07-12T09:52:17.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0970067730 con 0x7f09700afc70 2022-07-12T09:52:17.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.295+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0970066210 con 0x7f09700afc70 2022-07-12T09:52:17.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.299+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09700678c0 con 0x7f09700afc70 2022-07-12T09:52:17.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.299+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f096000f070 con 0x7f09700afc70 2022-07-12T09:52:17.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.299+0000 7f095f7fe700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0950053ad0 0x7f0950055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:17.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.299+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f096006cf40 con 0x7f09700afc70 2022-07-12T09:52:17.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.299+0000 7f09761dc700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0950053ad0 0x7f0950055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:17.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.299+0000 7f09761dc700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0950053ad0 0x7f0950055f90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f096c00d000 tx=0x7f096c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:17.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.303+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0960044070 con 0x7f09700afc70 2022-07-12T09:52:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.427+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0970153520 con 0x7f0950053ad0 2022-07-12T09:52:17.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.435+0000 7f095f7fe700 1 -- 172.21.15.61:0/2340360558 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+67766 (secure 0 0 0) 0x7f096c00f620 con 0x7f0950053ad0 2022-07-12T09:52:17.439 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T09:52:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0950053ad0 msgr2=0x7f0950055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0950053ad0 0x7f0950055f90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f096c00d000 tx=0x7f096c009040).stop 2022-07-12T09:52:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 msgr2=0x7f0970065b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:17.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f0970065b80 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 rx=0x7f09700b30c0 tx=0x7f0960015000).stop 2022-07-12T09:52:17.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 shutdown_connections 2022-07-12T09:52:17.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f0950053ad0 0x7f0950055f90 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09700b1f80 0x7f0970147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09700afc70 0x7f0970065b80 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 --2- 172.21.15.61:0/2340360558 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f09700adb00 0x7f0970153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:17.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 >> 172.21.15.61:0/2340360558 conn(0x7f0970062c00 msgr2=0x7f09700b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:52:17.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 shutdown_connections 2022-07-12T09:52:17.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:17.439+0000 7f0978440700 1 -- 172.21.15.61:0/2340360558 wait complete. 2022-07-12T09:52:17.446 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-12T09:52:17.453 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2288,"stamp":"2022-07-12T09:52:16.266563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385132,"kb_used_data":743209920,"kb_used_omap":27865,"kb_used_meta":0,"kb_avail":743209908,"statfs":{"total":767585320960,"available":761046945792,"internally_reserved":12288,"allocated":761046958080,"data_stored":761046958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28533974,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.475725"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":141,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:52:09.274612+0000","last_change":"2022-07-12T09:52:09.274713+0000","last_active":"2022-07-12T09:52:09.274612+0000","last_peered":"2022-07-12T09:52:09.274612+0000","last_clean":"2022-07-12T09:52:09.274612+0000","last_became_active":"2022-07-12T08:41:04.954339+0000","last_became_peered":"2022-07-12T08:41:04.954339+0000","last_unstale":"2022-07-12T09:52:09.274612+0000","last_undegraded":"2022-07-12T09:52:09.274612+0000","last_fullsized":"2022-07-12T09:52:09.274612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:52:09.274611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:52:09.274611+0000","last_clean_scrub_stamp":"2022-07-12T09:52:09.274611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:58.018240+0000","last_change":"2022-07-12T09:51:58.018344+0000","last_active":"2022-07-12T09:51:58.018240+0000","last_peered":"2022-07-12T09:51:58.018240+0000","last_clean":"2022-07-12T09:51:58.018240+0000","last_became_active":"2022-07-12T08:41:11.067215+0000","last_became_peered":"2022-07-12T08:41:11.067215+0000","last_unstale":"2022-07-12T09:51:58.018240+0000","last_undegraded":"2022-07-12T09:51:58.018240+0000","last_fullsized":"2022-07-12T09:51:58.018240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:58.018240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:58.018240+0000","last_clean_scrub_stamp":"2022-07-12T09:51:58.018240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:52:12.194717+0000","last_change":"2022-07-12T09:52:12.194857+0000","last_active":"2022-07-12T09:52:12.194717+0000","last_peered":"2022-07-12T09:52:12.194717+0000","last_clean":"2022-07-12T09:52:12.194717+0000","last_became_active":"2022-07-12T08:41:11.063329+0000","last_became_peered":"2022-07-12T08:41:11.063329+0000","last_unstale":"2022-07-12T09:52:12.194717+0000","last_undegraded":"2022-07-12T09:52:12.194717+0000","last_fullsized":"2022-07-12T09:52:12.194717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:52:12.194716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:52:12.194716+0000","last_clean_scrub_stamp":"2022-07-12T09:52:12.194716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:59.102250+0000","last_change":"2022-07-12T09:51:59.102345+0000","last_active":"2022-07-12T09:51:59.102250+0000","last_peered":"2022-07-12T09:51:59.102250+0000","last_clean":"2022-07-12T09:51:59.102250+0000","last_became_active":"2022-07-12T08:41:11.070549+0000","last_became_peered":"2022-07-12T08:41:11.070549+0000","last_unstale":"2022-07-12T09:51:59.102250+0000","last_undegraded":"2022-07-12T09:51:59.102250+0000","last_fullsized":"2022-07-12T09:51:59.102250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:59.102249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:59.102249+0000","last_clean_scrub_stamp":"2022-07-12T09:51:59.102249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:56.968610+0000","last_change":"2022-07-12T09:51:56.968708+0000","last_active":"2022-07-12T09:51:56.968610+0000","last_peered":"2022-07-12T09:51:56.968610+0000","last_clean":"2022-07-12T09:51:56.968610+0000","last_became_active":"2022-07-12T08:41:11.065911+0000","last_became_peered":"2022-07-12T08:41:11.065911+0000","last_unstale":"2022-07-12T09:51:56.968610+0000","last_undegraded":"2022-07-12T09:51:56.968610+0000","last_fullsized":"2022-07-12T09:51:56.968610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:56.968609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:56.968609+0000","last_clean_scrub_stamp":"2022-07-12T09:51:56.968609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:57.989977+0000","last_change":"2022-07-12T09:51:57.990085+0000","last_active":"2022-07-12T09:51:57.989977+0000","last_peered":"2022-07-12T09:51:57.989977+0000","last_clean":"2022-07-12T09:51:57.989977+0000","last_became_active":"2022-07-12T08:41:11.958006+0000","last_became_peered":"2022-07-12T08:41:11.958006+0000","last_unstale":"2022-07-12T09:51:57.989977+0000","last_undegraded":"2022-07-12T09:51:57.989977+0000","last_fullsized":"2022-07-12T09:51:57.989977+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:57.989977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:57.989977+0000","last_clean_scrub_stamp":"2022-07-12T09:51:57.989977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":135,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-12T08:51:20.853703+0000","last_change":"2022-07-12T08:41:11.954669+0000","last_active":"2022-07-12T08:51:20.853703+0000","last_peered":"2022-07-12T08:51:20.853703+0000","last_clean":"2022-07-12T08:51:20.853703+0000","last_became_active":"2022-07-12T08:41:11.954063+0000","last_became_peered":"2022-07-12T08:41:11.954063+0000","last_unstale":"2022-07-12T08:51:20.853703+0000","last_undegraded":"2022-07-12T08:51:20.853703+0000","last_fullsized":"2022-07-12T08:51:20.853703+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T08:41:10.037594+0000","last_clean_scrub_stamp":"2022-07-12T08:41:10.037594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:58.966829+0000","last_change":"2022-07-12T09:51:58.966935+0000","last_active":"2022-07-12T09:51:58.966829+0000","last_peered":"2022-07-12T09:51:58.966829+0000","last_clean":"2022-07-12T09:51:58.966829+0000","last_became_active":"2022-07-12T08:41:11.068746+0000","last_became_peered":"2022-07-12T08:41:11.068746+0000","last_unstale":"2022-07-12T09:51:58.966829+0000","last_undegraded":"2022-07-12T09:51:58.966829+0000","last_fullsized":"2022-07-12T09:51:58.966829+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:58.966828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:58.966828+0000","last_clean_scrub_stamp":"2022-07-12T09:51:58.966828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:57.393190+0000","last_change":"2022-07-12T09:51:57.393283+0000","last_active":"2022-07-12T09:51:57.393190+0000","last_peered":"2022-07-12T09:51:57.393190+0000","last_clean":"2022-07-12T09:51:57.393190+0000","last_became_active":"2022-07-12T08:41:11.064473+0000","last_became_peered":"2022-07-12T08:41:11.064473+0000","last_unstale":"2022-07-12T09:51:57.393190+0000","last_undegraded":"2022-07-12T09:51:57.393190+0000","last_fullsized":"2022-07-12T09:51:57.393190+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:57.393189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:57.393189+0000","last_clean_scrub_stamp":"2022-07-12T09:51:57.393189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575704,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798164,"kb_used_data":92901224,"kb_used_omap":3620,"kb_used_meta":0,"kb_avail":92901216,"statfs":{"total":95948165120,"available":95130845184,"internally_reserved":8192,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3707620,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88200000000000001,"15min":0.88400000000000001},"min":{"1min":0.68500000000000005,"5min":0.53300000000000003,"15min":0.50700000000000001},"max":{"1min":1.0369999999999999,"5min":1.137,"15min":1.1479999999999999},"last":1},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86199999999999999,"15min":0.879},"min":{"1min":0.72599999999999998,"5min":0.623,"15min":0.60799999999999998},"max":{"1min":0.998,"5min":1.1040000000000001,"15min":1.111},"last":0.81499999999999995}]},{"osd":1,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.56499999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0820000000000001,"5min":1.1659999999999999,"15min":1.214},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.64000000000000001,"5min":0.60199999999999998,"15min":0.58099999999999996},"max":{"1min":1.0700000000000001,"5min":1.109,"15min":1.1279999999999999},"last":0.73699999999999999}]},{"osd":2,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.82599999999999996,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.016,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.86399999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1220000000000001,"5min":1.242,"15min":1.242},"last":1.1479999999999999}]},{"osd":3,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.58499999999999996},"max":{"1min":1.1080000000000001,"5min":1.232,"15min":1.232},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.89300000000000002,"15min":0.90200000000000002},"min":{"1min":0.65200000000000002,"5min":0.60299999999999998,"15min":0.57899999999999996},"max":{"1min":1.1599999999999999,"5min":1.1859999999999999,"15min":1.22},"last":1.2030000000000001}]},{"osd":4,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.67900000000000005,"15min":0.71499999999999997},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.47199999999999998},"max":{"1min":0.94999999999999996,"5min":1.145,"15min":1.1819999999999999},"last":0.874},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88400000000000001,"15min":0.89100000000000001},"min":{"1min":0.70599999999999996,"5min":0.64600000000000002,"15min":0.44600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1759999999999999,"15min":1.2070000000000001},"last":1.0289999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":1.198,"5min":1.395,"15min":1.395},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.78300000000000003,"5min":0.56100000000000005,"15min":0.46100000000000002},"max":{"1min":1.121,"5min":1.1639999999999999,"15min":1.194},"last":1.0549999999999999}]},{"osd":6,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.91100000000000003,"15min":0.91400000000000003},"min":{"1min":0.65200000000000002,"5min":0.55500000000000005,"15min":0.434},"max":{"1min":1.077,"5min":1.1279999999999999,"15min":1.2410000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.67000000000000004,"5min":0.52000000000000002,"15min":0.39600000000000002},"max":{"1min":1.1830000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":1.099}]}]},{"osd":6,"up_from":13,"seq":55834575704,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":4021,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4117574,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.80100000000000005,"15min":0.78300000000000003},"min":{"1min":0.63,"5min":0.54300000000000004,"15min":0.52700000000000002},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.79600000000000004,"15min":0.82099999999999995},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.55700000000000005},"max":{"1min":1.012,"5min":1.0580000000000001,"15min":1.2130000000000001},"last":0.68500000000000005}]},{"osd":1,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85099999999999998,"15min":0.874},"min":{"1min":0.66500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0549999999999999,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"min":{"1min":0.71899999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1160000000000001,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.88700000000000001}]},{"osd":2,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.874,"15min":0.85299999999999998},"min":{"1min":0.63600000000000001,"5min":0.58799999999999997,"15min":0.55900000000000005},"max":{"1min":1.0900000000000001,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.82699999999999996,"15min":0.82999999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":0.96899999999999997,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.98499999999999999}]},{"osd":3,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85399999999999998,"15min":0.84399999999999997},"min":{"1min":0.66200000000000003,"5min":0.65400000000000003,"15min":0.621},"max":{"1min":1.077,"5min":1.1140000000000001,"15min":1.179},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.91300000000000003,"15min":0.92200000000000004},"min":{"1min":0.70799999999999996,"5min":0.63700000000000001,"15min":0.61599999999999999},"max":{"1min":1.054,"5min":1.125,"15min":1.2150000000000001},"last":0.95899999999999996}]},{"osd":4,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.80900000000000005,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0529999999999999,"5min":1.1950000000000001,"15min":1.2010000000000001},"last":1.141},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.86899999999999999,"15min":0.86499999999999999},"min":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.47899999999999998},"max":{"1min":1.139,"5min":1.175,"15min":1.413},"last":0.85699999999999998}]},{"osd":5,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88500000000000001,"15min":0.85399999999999998},"min":{"1min":0.60099999999999998,"5min":0.57999999999999996,"15min":0.54600000000000004},"max":{"1min":1.1160000000000001,"5min":1.591,"15min":1.591},"last":1.103},{"interface":"front","average":{"1min":1.006,"5min":1.0069999999999999,"15min":0.996},"min":{"1min":0.77900000000000003,"5min":0.77300000000000002,"15min":0.626},"max":{"1min":1.1899999999999999,"5min":1.246,"15min":1.2589999999999999},"last":1.0169999999999999}]},{"osd":7,"last update":"Tue Jul 12 09:51:25 2022","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.97799999999999998,"15min":0.96499999999999997},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.57999999999999996},"max":{"1min":1.135,"5min":1.365,"15min":1.4350000000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.96499999999999997,"15min":0.95799999999999996},"min":{"1min":0.73899999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1639999999999999,"5min":1.77,"15min":1.77},"last":1.0640000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575704,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798028,"kb_used_data":92901352,"kb_used_omap":3571,"kb_used_meta":0,"kb_avail":92901352,"statfs":{"total":95948165120,"available":95130984448,"internally_reserved":0,"allocated":95130984448,"data_stored":95130984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3657716,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84599999999999997,"15min":0.82999999999999996},"min":{"1min":0.66800000000000004,"5min":0.621,"15min":0.499},"max":{"1min":1.079,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.82199999999999995,"15min":0.81399999999999995},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.51400000000000001},"max":{"1min":1.071,"5min":1.1499999999999999,"15min":1.1830000000000001},"last":0.77000000000000002}]},{"osd":1,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.84099999999999997,"15min":0.82799999999999996},"min":{"1min":0.63,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.089,"5min":1.149,"15min":1.149},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.85199999999999998,"15min":0.85999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60499999999999998,"15min":0.55400000000000005},"max":{"1min":1.002,"5min":1.0860000000000001,"15min":1.1499999999999999},"last":1.04}]},{"osd":2,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.85399999999999998,"15min":0.85199999999999998},"min":{"1min":0.64900000000000002,"5min":0.63200000000000001,"15min":0.59299999999999997},"max":{"1min":1.1879999999999999,"5min":1.272,"15min":1.272},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.88,"15min":0.88100000000000001},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.59599999999999997},"max":{"1min":1.077,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.94899999999999995}]},{"osd":3,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.68300000000000005,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":0.997,"5min":1.0529999999999999,"15min":1.45},"last":0.87},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.90500000000000003,"15min":0.90300000000000002},"min":{"1min":0.66600000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.4079999999999999},"last":1.014}]},{"osd":5,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.80800000000000005,"15min":0.80400000000000005},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.51800000000000002},"max":{"1min":0.95399999999999996,"5min":1.2010000000000001,"15min":1.258},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85399999999999998,"15min":0.86299999999999999},"min":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.51200000000000001},"max":{"1min":1.163,"5min":1.2010000000000001,"15min":1.367},"last":0.91200000000000003}]},{"osd":6,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.86399999999999999,"15min":0.86299999999999999},"min":{"1min":0.59699999999999998,"5min":0.57499999999999996,"15min":0.56000000000000005},"max":{"1min":1.258,"5min":1.258,"15min":1.349},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.94199999999999995,"15min":0.93899999999999995},"min":{"1min":0.70499999999999996,"5min":0.623,"15min":0.59399999999999997},"max":{"1min":1.159,"5min":1.282,"15min":1.3109999999999999},"last":1.095}]},{"osd":7,"last update":"Tue Jul 12 09:51:10 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93600000000000005,"15min":0.94099999999999995},"min":{"1min":0.61399999999999999,"5min":0.61299999999999999,"15min":0.60099999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.4199999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.044,"5min":1.0489999999999999,"15min":1.044},"min":{"1min":0.71699999999999997,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.343,"5min":1.3740000000000001,"15min":1.4390000000000001},"last":1.0740000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3341,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3421351,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.63800000000000001,"15min":0.63900000000000001},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.86299999999999999,"5min":1.0800000000000001,"15min":1.266},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.63100000000000001,"5min":0.57499999999999996,"15min":0.57299999999999995},"max":{"1min":0.996,"5min":1.0589999999999999,"15min":1.1180000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.64800000000000002,"15min":0.65500000000000003},"min":{"1min":0.54100000000000004,"5min":0.47399999999999998,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":1.0009999999999999,"15min":1.8169999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.74299999999999999,"15min":0.76000000000000001},"min":{"1min":0.53800000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.96399999999999997,"5min":1.0780000000000001,"15min":1.5369999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.68500000000000005,"15min":0.68899999999999995},"min":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.5},"max":{"1min":1.034,"5min":1.248,"15min":1.248},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.78800000000000003,"15min":0.78500000000000003},"min":{"1min":0.72299999999999998,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.5720000000000001},"last":0.56599999999999995}]},{"osd":4,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.64200000000000002},"max":{"1min":1.149,"5min":1.236,"15min":1.405},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.92400000000000004,"15min":0.93500000000000005},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68100000000000005},"max":{"1min":1.155,"5min":1.3360000000000001,"15min":1.708},"last":1.0129999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.92700000000000005,"15min":0.93600000000000005},"min":{"1min":0.77200000000000002,"5min":0.65100000000000002,"15min":0.65000000000000002},"max":{"1min":1.1439999999999999,"5min":1.2949999999999999,"15min":1.4319999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95999999999999996,"15min":0.96899999999999997},"min":{"1min":0.754,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.1950000000000001,"5min":1.3580000000000001,"15min":1.5129999999999999},"last":0.96099999999999997}]},{"osd":6,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.026,"15min":1.01},"min":{"1min":0.78700000000000003,"5min":0.73899999999999999,"15min":0.70399999999999996},"max":{"1min":1.238,"5min":1.966,"15min":1.966},"last":0.996},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0649999999999999,"15min":1.0620000000000001},"min":{"1min":0.89900000000000002,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.6519999999999999},"last":1.113}]},{"osd":7,"last update":"Tue Jul 12 09:51:14 2022","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0509999999999999,"15min":1.0600000000000001},"min":{"1min":0.78400000000000003,"5min":0.75700000000000001,"15min":0.73099999999999998},"max":{"1min":1.25,"5min":1.2989999999999999,"15min":1.468},"last":1.141},{"interface":"front","average":{"1min":1.083,"5min":1.085,"15min":1.0840000000000001},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.2749999999999999,"5min":1.5920000000000001,"15min":1.706},"last":1.0449999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575704,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2803,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870758,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.621,"15min":0.62},"min":{"1min":0.496,"5min":0.443,"15min":0.438},"max":{"1min":1.153,"5min":1.153,"15min":1.155},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.56399999999999995,"5min":0.48399999999999999,"15min":0.47999999999999998},"max":{"1min":1.0189999999999999,"5min":1.026,"15min":1.1339999999999999},"last":0.82699999999999996}]},{"osd":1,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81100000000000005,"5min":1.1379999999999999,"15min":1.2529999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76600000000000001,"15min":0.76000000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53000000000000003},"max":{"1min":0.92000000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.66300000000000003}]},{"osd":3,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.48499999999999999},"max":{"1min":1.042,"5min":1.069,"15min":1.1839999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.74299999999999999}]},{"osd":4,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94199999999999995,"15min":0.95799999999999996},"min":{"1min":0.74199999999999999,"5min":0.70199999999999996,"15min":0.69299999999999995},"max":{"1min":1.1599999999999999,"5min":1.256,"15min":1.3680000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.93600000000000005,"15min":0.93500000000000005},"min":{"1min":0.76000000000000001,"5min":0.69399999999999995,"15min":0.67200000000000004},"max":{"1min":1.135,"5min":1.262,"15min":1.3109999999999999},"last":0.88400000000000001}]},{"osd":5,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.68300000000000005,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.341},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.95999999999999996,"15min":0.98099999999999998},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.66100000000000003},"max":{"1min":1.2310000000000001,"5min":1.3360000000000001,"15min":1.476},"last":1.0669999999999999}]},{"osd":6,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.98499999999999999,"15min":0.98299999999999998},"min":{"1min":0.80600000000000005,"5min":0.74299999999999999,"15min":0.67600000000000005},"max":{"1min":1.204,"5min":1.425,"15min":1.425},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.135,"5min":1.0700000000000001,"15min":1.0640000000000001},"min":{"1min":0.84499999999999997,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.91300000000000003}]},{"osd":7,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0489999999999999,"15min":1.0720000000000001},"min":{"1min":0.83699999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.2829999999999999,"5min":1.591,"15min":1.591},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0740000000000001,"15min":1.0720000000000001},"min":{"1min":0.88700000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.27,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.96599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575704,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798160,"kb_used_data":92901224,"kb_used_omap":3970,"kb_used_meta":0,"kb_avail":92901220,"statfs":{"total":95948165120,"available":95130849280,"internally_reserved":4096,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4066282,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63300000000000001},"min":{"1min":0.48499999999999999,"5min":0.48099999999999998,"15min":0.46899999999999997},"max":{"1min":0.97399999999999998,"5min":1.046,"15min":1.216},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.502,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.089,"5min":1.1779999999999999,"15min":1.226},"last":0.63300000000000001}]},{"osd":2,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.44},"max":{"1min":0.78400000000000003,"5min":1.143,"15min":1.1879999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.76600000000000001,"15min":0.76500000000000001},"min":{"1min":0.56299999999999994,"5min":0.56200000000000006,"15min":0.502},"max":{"1min":1.075,"5min":1.149,"15min":1.3620000000000001},"last":0.93400000000000005}]},{"osd":3,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.81899999999999995,"15min":0.83599999999999997},"min":{"1min":0.63500000000000001,"5min":0.60599999999999998,"15min":0.56599999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.8779999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76100000000000001,"15min":0.78300000000000003},"min":{"1min":0.59799999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.92800000000000005,"5min":1.147,"15min":1.258},"last":0.58399999999999996}]},{"osd":4,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.89900000000000002,"15min":0.90200000000000002},"min":{"1min":0.77100000000000002,"5min":0.64100000000000001,"15min":0.63700000000000001},"max":{"1min":1.1830000000000001,"5min":1.278,"15min":1.3},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0089999999999999,"15min":1.012},"min":{"1min":0.88800000000000001,"5min":0.79400000000000004,"15min":0.68999999999999995},"max":{"1min":1.2330000000000001,"5min":1.321,"15min":1.444},"last":1.4750000000000001}]},{"osd":5,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.995,"15min":0.98699999999999999},"min":{"1min":0.85199999999999998,"5min":0.73399999999999999,"15min":0.70799999999999996},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0009999999999999,"15min":0.99199999999999999},"min":{"1min":0.76400000000000001,"5min":0.749,"15min":0.63300000000000001},"max":{"1min":1.28,"5min":1.28,"15min":1.397},"last":1.012}]},{"osd":6,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.004,"15min":1.002},"min":{"1min":0.70599999999999996,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.341,"5min":1.341,"15min":1.349},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.042,"5min":1.022,"15min":1.016},"min":{"1min":0.80000000000000004,"5min":0.753,"15min":0.68600000000000005},"max":{"1min":1.359,"5min":1.359,"15min":1.5189999999999999},"last":0.874}]},{"osd":7,"last update":"Tue Jul 12 09:51:17 2022","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0920000000000001,"15min":1.0960000000000001},"min":{"1min":0.93100000000000005,"5min":0.78400000000000003,"15min":0.73099999999999998},"max":{"1min":1.319,"5min":1.3939999999999999,"15min":1.5049999999999999},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.071,"15min":1.085},"min":{"1min":0.90500000000000003,"5min":0.78500000000000003,"15min":0.72399999999999998},"max":{"1min":1.228,"5min":1.3680000000000001,"15min":1.583},"last":0.85899999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575704,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3793,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3884787,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59399999999999997,"15min":0.60299999999999998},"min":{"1min":0.49299999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.77600000000000002,"5min":0.89200000000000002,"15min":0.99399999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.70699999999999996,"15min":0.70799999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.0880000000000001,"5min":1.095,"15min":1.1819999999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.753,"15min":0.72199999999999998},"min":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.51900000000000002},"max":{"1min":1.0660000000000001,"5min":1.2210000000000001,"15min":1.2270000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.64700000000000002,"15min":0.66900000000000004},"min":{"1min":0.51000000000000001,"5min":0.48399999999999999,"15min":0.42699999999999999},"max":{"1min":1.0600000000000001,"5min":1.129,"15min":1.224},"last":0.66200000000000003}]},{"osd":3,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.745,"15min":0.75900000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.94899999999999995,"5min":1.0640000000000001,"15min":1.1799999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.876,"15min":0.877},"min":{"1min":0.72599999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0109999999999999,"5min":1.2130000000000001,"15min":1.246},"last":0.90100000000000002}]},{"osd":4,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.70999999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.1180000000000001,"5min":1.133,"15min":1.391},"last":0.871},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.94999999999999996,"15min":0.94299999999999995},"min":{"1min":0.76200000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":1.1100000000000001}]},{"osd":5,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":0.95999999999999996},"min":{"1min":0.748,"5min":0.67200000000000004,"15min":0.66500000000000004},"max":{"1min":1.361,"5min":1.395,"15min":1.48},"last":1.21},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.98299999999999998,"15min":0.97999999999999998},"min":{"1min":0.81299999999999994,"5min":0.70299999999999996,"15min":0.69999999999999996},"max":{"1min":1.2330000000000001,"5min":1.258,"15min":1.498},"last":0.96699999999999997}]},{"osd":6,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.02,"15min":1.022},"min":{"1min":0.78100000000000003,"5min":0.77400000000000002,"15min":0.70999999999999996},"max":{"1min":1.238,"5min":1.3169999999999999,"15min":1.522},"last":1.026},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96399999999999997,"15min":0.99099999999999999},"min":{"1min":0.74199999999999999,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.137,"5min":1.3759999999999999,"15min":1.4650000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0049999999999999,"15min":1.0309999999999999},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.68799999999999994},"max":{"1min":1.179,"5min":1.4550000000000001,"15min":1.5489999999999999},"last":1.163},{"interface":"front","average":{"1min":1.032,"5min":1.079,"15min":1.087},"min":{"1min":0.83999999999999997,"5min":0.79300000000000004,"15min":0.77200000000000002},"max":{"1min":1.5,"5min":1.524,"15min":1.569},"last":1.2569999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575704,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2742,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2807886,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.84599999999999997,"15min":0.85399999999999998},"min":{"1min":0.63600000000000001,"5min":0.63100000000000001,"15min":0.58099999999999996},"max":{"1min":0.98999999999999999,"5min":1.194,"15min":1.224},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.90200000000000002,"15min":0.91400000000000003},"min":{"1min":0.69399999999999995,"5min":0.65100000000000002,"15min":0.59999999999999998},"max":{"1min":1.0089999999999999,"5min":1.161,"15min":1.208},"last":0.85599999999999998}]},{"osd":1,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81499999999999995},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.5},"max":{"1min":1.073,"5min":1.198,"15min":1.22},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.51600000000000001},"max":{"1min":1.052,"5min":1.109,"15min":1.109},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.66800000000000004,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.0429999999999999,"5min":1.127,"15min":1.1819999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82999999999999996,"15min":0.82799999999999996},"min":{"1min":0.67300000000000004,"5min":0.61599999999999999,"15min":0.55800000000000005},"max":{"1min":0.99099999999999999,"5min":1.1759999999999999,"15min":1.1859999999999999},"last":0.72099999999999997}]},{"osd":3,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.63900000000000001,"5min":0.59999999999999998,"15min":0.57799999999999996},"max":{"1min":1.0609999999999999,"5min":1.151,"15min":1.3360000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.879,"15min":0.88200000000000001},"min":{"1min":0.63700000000000001,"5min":0.61199999999999999,"15min":0.54100000000000004},"max":{"1min":1.069,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.83099999999999996}]},{"osd":4,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.70999999999999996,"15min":0.72199999999999998},"min":{"1min":0.54100000000000004,"5min":0.52200000000000002,"15min":0.39000000000000001},"max":{"1min":0.98499999999999999,"5min":1.137,"15min":1.1719999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.94299999999999995,"15min":0.93999999999999995},"min":{"1min":0.70099999999999996,"5min":0.68799999999999994,"15min":0.58599999999999997},"max":{"1min":1.073,"5min":1.212,"15min":1.4930000000000001},"last":0.67600000000000005}]},{"osd":6,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.872,"15min":0.86799999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.752},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91700000000000004,"15min":0.90900000000000003},"min":{"1min":0.72299999999999998,"5min":0.57599999999999996,"15min":0.48199999999999998},"max":{"1min":1.069,"5min":1.232,"15min":1.276},"last":0.86799999999999999}]},{"osd":7,"last update":"Tue Jul 12 09:51:20 2022","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89500000000000002,"15min":0.89700000000000002},"min":{"1min":0.57499999999999996,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0389999999999999,"5min":1.2809999999999999,"15min":2.1200000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.002,"5min":1.0169999999999999,"15min":1.0129999999999999},"min":{"1min":0.84199999999999997,"5min":0.623,"15min":0.59799999999999998},"max":{"1min":1.482,"5min":1.482,"15min":2.1019999999999999},"last":0.95199999999999996}]}]}]}} 2022-07-12T09:52:17.456 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-07-12T08:41:10.037594+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=8, tm_min=41, tm_sec=10, tm_wday=1, tm_yday=193, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=12, tm_hour=9, tm_min=51, tm_sec=52, tm_wday=1, tm_yday=193, tm_isdst=0) 2022-07-12T09:52:17.457 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2022-07-12T09:52:23.184 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:23.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:23.191 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:23.195 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:23.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:37.459 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-12T09:52:37.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.574+0000 7efc35e1b700 1 Processor -- start 2022-07-12T09:52:37.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.574+0000 7efc35e1b700 1 -- start start 2022-07-12T09:52:37.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc300b1210 0x7efc3013f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3005d220 con 0x7efc3005f340 2022-07-12T09:52:37.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3009aca0 con 0x7efc3005c740 2022-07-12T09:52:37.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc30098780 con 0x7efc300b1210 2022-07-12T09:52:37.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc3561a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efc3005f340 0x7efc3013f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51342/0 (socket says 172.21.15.61:51342) 2022-07-12T09:52:37.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc3561a700 1 -- 172.21.15.61:0/3850140105 learned_addr learned my addr 172.21.15.61:0/3850140105 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:52:37.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc27fff700 1 --2- 172.21.15.61:0/3850140105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc300b1210 0x7efc3013f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387974447 0 0) 0x7efc3005d220 con 0x7efc3005f340 2022-07-12T09:52:37.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1c003260 con 0x7efc3005f340 2022-07-12T09:52:37.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 <== mon.1 v1:172.21.15.111:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206299593 0 0) 0x7efc3009aca0 con 0x7efc3005c740 2022-07-12T09:52:37.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 --> v1:172.21.15.111:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3005d220 con 0x7efc3005c740 2022-07-12T09:52:37.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc27fff700 1 -- 172.21.15.61:0/3850140105 >> v1:172.21.15.111:6789/0 conn(0x7efc3005c740 legacy=0x7efc30098070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc27fff700 1 -- 172.21.15.61:0/3850140105 >> v1:172.21.15.61:6789/0 conn(0x7efc3005f340 legacy=0x7efc3013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc27fff700 1 -- 172.21.15.61:0/3850140105 --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3005cb20 con 0x7efc300b1210 2022-07-12T09:52:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc27fff700 1 --2- 172.21.15.61:0/3850140105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc300b1210 0x7efc3013f0f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7efc20002f70 tx=0x7efc20006e80).ready entity=mon.2 client_cookie=69d022f918113bc2 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 <== mon.2 v2:172.21.15.111:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc20014070 con 0x7efc300b1210 2022-07-12T09:52:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 <== mon.2 v2:172.21.15.111:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc20000e90 con 0x7efc300b1210 2022-07-12T09:52:37.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc277fe700 1 -- 172.21.15.61:0/3850140105 <== mon.2 v2:172.21.15.111:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc200107f0 con 0x7efc300b1210 2022-07-12T09:52:37.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- 172.21.15.61:0/3850140105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc300b1210 msgr2=0x7efc3013f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 --2- 172.21.15.61:0/3850140105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc300b1210 0x7efc3013f0f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7efc20002f70 tx=0x7efc20006e80).stop 2022-07-12T09:52:37.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- 172.21.15.61:0/3850140105 shutdown_connections 2022-07-12T09:52:37.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 --2- 172.21.15.61:0/3850140105 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc300b1210 0x7efc3013f0f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- 172.21.15.61:0/3850140105 >> 172.21.15.61:0/3850140105 conn(0x7efc300b1950 msgr2=0x7efc300b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:52:37.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- 172.21.15.61:0/3850140105 shutdown_connections 2022-07-12T09:52:37.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.578+0000 7efc35e1b700 1 -- 172.21.15.61:0/3850140105 wait complete. 2022-07-12T09:52:37.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 Processor -- start 2022-07-12T09:52:37.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 -- start start 2022-07-12T09:52:37.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 0x7efc30153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 0x7efc30064cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 0x7efc3014f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30067700 con 0x7efc300b1210 2022-07-12T09:52:37.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30090dd0 con 0x7efc3005f340 2022-07-12T09:52:37.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc35e1b700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc30152fe0 con 0x7efc3005c740 2022-07-12T09:52:37.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 0x7efc3014f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc27fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 0x7efc30064cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc34e19700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 0x7efc30153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 0x7efc3014f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34900/0 (socket says 172.21.15.61:34900) 2022-07-12T09:52:37.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc34e19700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 0x7efc30153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3301/0 says I am v2:172.21.15.61:45324/0 (socket says 172.21.15.61:45324) 2022-07-12T09:52:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc27fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 0x7efc30064cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.61:41704/0 (socket says 172.21.15.61:41704) 2022-07-12T09:52:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 -- 172.21.15.61:0/2722425359 learned_addr learned my addr 172.21.15.61:0/2722425359 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:52:37.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 -- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 msgr2=0x7efc30153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 0x7efc30153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 -- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 msgr2=0x7efc30064cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 0x7efc30064cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 -- 172.21.15.61:0/2722425359 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2000c310 con 0x7efc300b1210 2022-07-12T09:52:37.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc27fff700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 0x7efc30064cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T09:52:37.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc34e19700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 0x7efc30153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T09:52:37.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.582+0000 7efc3561a700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 0x7efc3014f980 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7efc28001490 tx=0x7efc28007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:37.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc28005d40 con 0x7efc300b1210 2022-07-12T09:52:37.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc280070d0 con 0x7efc300b1210 2022-07-12T09:52:37.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc30153c10 con 0x7efc300b1210 2022-07-12T09:52:37.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc28005ea0 con 0x7efc300b1210 2022-07-12T09:52:37.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc30090950 con 0x7efc300b1210 2022-07-12T09:52:37.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc30057460 con 0x7efc300b1210 2022-07-12T09:52:37.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efc28010070 con 0x7efc300b1210 2022-07-12T09:52:37.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc25ffb700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7efc2806b740 con 0x7efc300b1210 2022-07-12T09:52:37.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.586+0000 7efc34e19700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.590+0000 7efc34e19700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7efc2c00d000 tx=0x7efc2c007b30).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:37.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.594+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efc28042870 con 0x7efc300b1210 2022-07-12T09:52:37.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.714+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 --> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc3006b2b0 con 0x7efc10053ad0 2022-07-12T09:52:37.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.722+0000 7efc25ffb700 1 -- 172.21.15.61:0/2722425359 <== mgr.4103 v2:172.21.15.111:6800/21499 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68024 (secure 0 0 0) 0x7efc3006b2b0 con 0x7efc10053ad0 2022-07-12T09:52:37.726 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-12T09:52:37.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.722+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 msgr2=0x7efc10055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.722+0000 7efc35e1b700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7efc2c00d000 tx=0x7efc2c007b30).stop 2022-07-12T09:52:37.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.722+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 msgr2=0x7efc3014f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.722+0000 7efc35e1b700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 0x7efc3014f980 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 rx=0x7efc28001490 tx=0x7efc28007e00).stop 2022-07-12T09:52:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 shutdown_connections 2022-07-12T09:52:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efc300b1210 0x7efc3014f980 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7efc10053ad0 0x7efc10055f90 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc3005f340 0x7efc30064cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 --2- 172.21.15.61:0/2722425359 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7efc3005c740 0x7efc30153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 >> 172.21.15.61:0/2722425359 conn(0x7efc300b1950 msgr2=0x7efc300b2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:52:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 shutdown_connections 2022-07-12T09:52:37.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.726+0000 7efc35e1b700 1 -- 172.21.15.61:0/2722425359 wait complete. 2022-07-12T09:52:37.733 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-12T09:52:37.742 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2298,"stamp":"2022-07-12T09:52:36.272562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385120,"kb_used_data":743209920,"kb_used_omap":27867,"kb_used_meta":0,"kb_avail":743209920,"statfs":{"total":767585320960,"available":761046958080,"internally_reserved":0,"allocated":761046958080,"data_stored":761046958080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28536409,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.475725"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":141,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:52:09.274612+0000","last_change":"2022-07-12T09:52:09.274713+0000","last_active":"2022-07-12T09:52:09.274612+0000","last_peered":"2022-07-12T09:52:09.274612+0000","last_clean":"2022-07-12T09:52:09.274612+0000","last_became_active":"2022-07-12T08:41:04.954339+0000","last_became_peered":"2022-07-12T08:41:04.954339+0000","last_unstale":"2022-07-12T09:52:09.274612+0000","last_undegraded":"2022-07-12T09:52:09.274612+0000","last_fullsized":"2022-07-12T09:52:09.274612+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:52:09.274611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:52:09.274611+0000","last_clean_scrub_stamp":"2022-07-12T09:52:09.274611+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:58.018240+0000","last_change":"2022-07-12T09:51:58.018344+0000","last_active":"2022-07-12T09:51:58.018240+0000","last_peered":"2022-07-12T09:51:58.018240+0000","last_clean":"2022-07-12T09:51:58.018240+0000","last_became_active":"2022-07-12T08:41:11.067215+0000","last_became_peered":"2022-07-12T08:41:11.067215+0000","last_unstale":"2022-07-12T09:51:58.018240+0000","last_undegraded":"2022-07-12T09:51:58.018240+0000","last_fullsized":"2022-07-12T09:51:58.018240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:58.018240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:58.018240+0000","last_clean_scrub_stamp":"2022-07-12T09:51:58.018240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:52:12.194717+0000","last_change":"2022-07-12T09:52:12.194857+0000","last_active":"2022-07-12T09:52:12.194717+0000","last_peered":"2022-07-12T09:52:12.194717+0000","last_clean":"2022-07-12T09:52:12.194717+0000","last_became_active":"2022-07-12T08:41:11.063329+0000","last_became_peered":"2022-07-12T08:41:11.063329+0000","last_unstale":"2022-07-12T09:52:12.194717+0000","last_undegraded":"2022-07-12T09:52:12.194717+0000","last_fullsized":"2022-07-12T09:52:12.194717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:52:12.194716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:52:12.194716+0000","last_clean_scrub_stamp":"2022-07-12T09:52:12.194716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:59.102250+0000","last_change":"2022-07-12T09:51:59.102345+0000","last_active":"2022-07-12T09:51:59.102250+0000","last_peered":"2022-07-12T09:51:59.102250+0000","last_clean":"2022-07-12T09:51:59.102250+0000","last_became_active":"2022-07-12T08:41:11.070549+0000","last_became_peered":"2022-07-12T08:41:11.070549+0000","last_unstale":"2022-07-12T09:51:59.102250+0000","last_undegraded":"2022-07-12T09:51:59.102250+0000","last_fullsized":"2022-07-12T09:51:59.102250+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:59.102249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:59.102249+0000","last_clean_scrub_stamp":"2022-07-12T09:51:59.102249+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:56.968610+0000","last_change":"2022-07-12T09:51:56.968708+0000","last_active":"2022-07-12T09:51:56.968610+0000","last_peered":"2022-07-12T09:51:56.968610+0000","last_clean":"2022-07-12T09:51:56.968610+0000","last_became_active":"2022-07-12T08:41:11.065911+0000","last_became_peered":"2022-07-12T08:41:11.065911+0000","last_unstale":"2022-07-12T09:51:56.968610+0000","last_undegraded":"2022-07-12T09:51:56.968610+0000","last_fullsized":"2022-07-12T09:51:56.968610+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:56.968609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:56.968609+0000","last_clean_scrub_stamp":"2022-07-12T09:51:56.968609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:57.989977+0000","last_change":"2022-07-12T09:51:57.990085+0000","last_active":"2022-07-12T09:51:57.989977+0000","last_peered":"2022-07-12T09:51:57.989977+0000","last_clean":"2022-07-12T09:51:57.989977+0000","last_became_active":"2022-07-12T08:41:11.958006+0000","last_became_peered":"2022-07-12T08:41:11.958006+0000","last_unstale":"2022-07-12T09:51:57.989977+0000","last_undegraded":"2022-07-12T09:51:57.989977+0000","last_fullsized":"2022-07-12T09:51:57.989977+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:57.989977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:57.989977+0000","last_clean_scrub_stamp":"2022-07-12T09:51:57.989977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:52:15.217869+0000","last_change":"2022-07-12T09:52:15.217971+0000","last_active":"2022-07-12T09:52:15.217869+0000","last_peered":"2022-07-12T09:52:15.217869+0000","last_clean":"2022-07-12T09:52:15.217869+0000","last_became_active":"2022-07-12T08:41:11.954063+0000","last_became_peered":"2022-07-12T08:41:11.954063+0000","last_unstale":"2022-07-12T09:52:15.217869+0000","last_undegraded":"2022-07-12T09:52:15.217869+0000","last_fullsized":"2022-07-12T09:52:15.217869+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:52:15.217868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:52:15.217868+0000","last_clean_scrub_stamp":"2022-07-12T09:52:15.217868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:58.966829+0000","last_change":"2022-07-12T09:51:58.966935+0000","last_active":"2022-07-12T09:51:58.966829+0000","last_peered":"2022-07-12T09:51:58.966829+0000","last_clean":"2022-07-12T09:51:58.966829+0000","last_became_active":"2022-07-12T08:41:11.068746+0000","last_became_peered":"2022-07-12T08:41:11.068746+0000","last_unstale":"2022-07-12T09:51:58.966829+0000","last_undegraded":"2022-07-12T09:51:58.966829+0000","last_fullsized":"2022-07-12T09:51:58.966829+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:58.966828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:58.966828+0000","last_clean_scrub_stamp":"2022-07-12T09:51:58.966828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":138,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-12T09:51:57.393190+0000","last_change":"2022-07-12T09:51:57.393283+0000","last_active":"2022-07-12T09:51:57.393190+0000","last_peered":"2022-07-12T09:51:57.393190+0000","last_clean":"2022-07-12T09:51:57.393190+0000","last_became_active":"2022-07-12T08:41:11.064473+0000","last_became_peered":"2022-07-12T08:41:11.064473+0000","last_unstale":"2022-07-12T09:51:57.393190+0000","last_undegraded":"2022-07-12T09:51:57.393190+0000","last_fullsized":"2022-07-12T09:51:57.393190+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-12T09:51:57.393189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-12T09:51:57.393189+0000","last_clean_scrub_stamp":"2022-07-12T09:51:57.393189+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575708,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3621,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3708776,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.88200000000000001,"15min":0.88400000000000001},"min":{"1min":0.68500000000000005,"5min":0.53300000000000003,"15min":0.50700000000000001},"max":{"1min":1.0369999999999999,"5min":1.137,"15min":1.1479999999999999},"last":1.052},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86199999999999999,"15min":0.879},"min":{"1min":0.72599999999999998,"5min":0.623,"15min":0.60799999999999998},"max":{"1min":0.998,"5min":1.1040000000000001,"15min":1.111},"last":1.0740000000000001}]},{"osd":1,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81599999999999995},"min":{"1min":0.56499999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0820000000000001,"5min":1.1659999999999999,"15min":1.214},"last":1.006},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.83399999999999996,"15min":0.82099999999999995},"min":{"1min":0.64000000000000001,"5min":0.60199999999999998,"15min":0.58099999999999996},"max":{"1min":1.0700000000000001,"5min":1.109,"15min":1.1279999999999999},"last":0.91000000000000003}]},{"osd":2,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.82599999999999996,"15min":0.80800000000000005},"min":{"1min":0.61599999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.016,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.629},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.86399999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1220000000000001,"5min":1.242,"15min":1.242},"last":0.876}]},{"osd":3,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82199999999999995,"15min":0.82099999999999995},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.58499999999999996},"max":{"1min":1.1080000000000001,"5min":1.232,"15min":1.232},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.89300000000000002,"15min":0.90200000000000002},"min":{"1min":0.65200000000000002,"5min":0.60299999999999998,"15min":0.57899999999999996},"max":{"1min":1.1599999999999999,"5min":1.1859999999999999,"15min":1.22},"last":0.94399999999999995}]},{"osd":4,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.67900000000000005,"15min":0.71499999999999997},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.47199999999999998},"max":{"1min":0.94999999999999996,"5min":1.145,"15min":1.1819999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88400000000000001,"15min":0.89100000000000001},"min":{"1min":0.70599999999999996,"5min":0.64600000000000002,"15min":0.44600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1759999999999999,"15min":1.2070000000000001},"last":0.84999999999999998}]},{"osd":5,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.52900000000000003,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":1.198,"5min":1.395,"15min":1.395},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.78300000000000003,"5min":0.56100000000000005,"15min":0.46100000000000002},"max":{"1min":1.121,"5min":1.1639999999999999,"15min":1.194},"last":1.0329999999999999}]},{"osd":6,"last update":"Tue Jul 12 09:51:36 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.91100000000000003,"15min":0.91400000000000003},"min":{"1min":0.65200000000000002,"5min":0.55500000000000005,"15min":0.434},"max":{"1min":1.077,"5min":1.1279999999999999,"15min":1.2410000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93200000000000005,"15min":0.93600000000000005},"min":{"1min":0.67000000000000004,"5min":0.52000000000000002,"15min":0.39600000000000002},"max":{"1min":1.1830000000000001,"5min":1.3759999999999999,"15min":1.3759999999999999},"last":0.92800000000000005}]}]},{"osd":6,"up_from":13,"seq":55834575708,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":4021,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4117574,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.78900000000000003,"15min":0.78300000000000003},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.52700000000000002},"max":{"1min":1.113,"5min":1.153,"15min":1.153},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.80100000000000005,"15min":0.81999999999999995},"min":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.55700000000000005},"max":{"1min":1.02,"5min":1.0580000000000001,"15min":1.2130000000000001},"last":0.73599999999999999}]},{"osd":1,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84099999999999997,"15min":0.87},"min":{"1min":0.69099999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.032,"5min":1.177,"15min":1.4570000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.89000000000000001,"15min":0.88900000000000001},"min":{"1min":0.75600000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.161,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.97199999999999998}]},{"osd":2,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.87,"15min":0.85099999999999998},"min":{"1min":0.61199999999999999,"5min":0.58799999999999997,"15min":0.55900000000000005},"max":{"1min":0.97199999999999998,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.82599999999999996,"15min":0.82999999999999996},"min":{"1min":0.70999999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0369999999999999,"5min":1.113,"15min":1.6870000000000001},"last":0.76200000000000001}]},{"osd":3,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.85899999999999999,"15min":0.84699999999999998},"min":{"1min":0.68999999999999995,"5min":0.65400000000000003,"15min":0.621},"max":{"1min":1.056,"5min":1.091,"15min":1.179},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.90600000000000003,"15min":0.91600000000000004},"min":{"1min":0.69499999999999995,"5min":0.63700000000000001,"15min":0.61599999999999999},"max":{"1min":0.95899999999999996,"5min":1.125,"15min":1.2150000000000001},"last":0.91200000000000003}]},{"osd":4,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.81799999999999995,"15min":0.79000000000000004},"min":{"1min":0.59699999999999998,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.141,"5min":1.1950000000000001,"15min":1.2010000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.84799999999999998,"15min":0.85399999999999998},"min":{"1min":0.63900000000000001,"5min":0.52400000000000002,"15min":0.47899999999999998},"max":{"1min":1.1319999999999999,"5min":1.175,"15min":1.413},"last":0.99099999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.88400000000000001,"15min":0.85299999999999998},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.54600000000000004},"max":{"1min":1.1850000000000001,"5min":1.3140000000000001,"15min":1.591},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.997,"5min":1.0049999999999999,"15min":0.997},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.626},"max":{"1min":1.2390000000000001,"5min":1.246,"15min":1.2589999999999999},"last":1.024}]},{"osd":7,"last update":"Tue Jul 12 09:52:27 2022","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.96999999999999997,"15min":0.95299999999999996},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.57999999999999996},"max":{"1min":1.073,"5min":1.365,"15min":1.391},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96399999999999997,"15min":0.95599999999999996},"min":{"1min":0.745,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1519999999999999,"5min":1.363,"15min":1.77},"last":0.90000000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798028,"kb_used_data":92901352,"kb_used_omap":3571,"kb_used_meta":0,"kb_avail":92901352,"statfs":{"total":95948165120,"available":95130984448,"internally_reserved":0,"allocated":95130984448,"data_stored":95130984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3657716,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.83499999999999996,"15min":0.82999999999999996},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.499},"max":{"1min":1.0700000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.82599999999999996,"15min":0.81399999999999995},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.51400000000000001},"max":{"1min":1.181,"5min":1.181,"15min":1.1830000000000001},"last":1}]},{"osd":1,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84699999999999998,"15min":0.83199999999999996},"min":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0349999999999999,"5min":1.149,"15min":1.149},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.85999999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.55400000000000005},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84899999999999998,"15min":0.85099999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0900000000000001,"5min":1.272,"15min":1.272},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.89600000000000002,"15min":0.88100000000000001},"min":{"1min":0.70099999999999996,"5min":0.61799999999999999,"15min":0.59599999999999997},"max":{"1min":1.135,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":1.0369999999999999}]},{"osd":3,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.82599999999999996,"15min":0.83199999999999996},"min":{"1min":0.63800000000000001,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":0.97299999999999998,"5min":1.0469999999999999,"15min":1.45},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.91000000000000003,"15min":0.90300000000000002},"min":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"max":{"1min":1.0880000000000001,"5min":1.3009999999999999,"15min":1.4079999999999999},"last":0.79500000000000004}]},{"osd":5,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.81200000000000006,"15min":0.80200000000000005},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.51800000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.258},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.86299999999999999},"min":{"1min":0.63800000000000001,"5min":0.54800000000000004,"15min":0.51200000000000001},"max":{"1min":1.0940000000000001,"5min":1.163,"15min":1.367},"last":0.69199999999999995}]},{"osd":6,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.86299999999999999,"15min":0.86099999999999999},"min":{"1min":0.63700000000000001,"5min":0.57499999999999996,"15min":0.56000000000000005},"max":{"1min":1.034,"5min":1.258,"15min":1.349},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.76200000000000001,"5min":0.623,"15min":0.59399999999999997},"max":{"1min":1.278,"5min":1.282,"15min":1.3109999999999999},"last":0.82699999999999996}]},{"osd":7,"last update":"Tue Jul 12 09:52:15 2022","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.93200000000000005,"15min":0.93999999999999995},"min":{"1min":0.64700000000000002,"5min":0.61399999999999999,"15min":0.60099999999999998},"max":{"1min":1.0669999999999999,"5min":1.3779999999999999,"15min":1.4199999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.056,"5min":1.0489999999999999,"15min":1.0449999999999999},"min":{"1min":0.93899999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1990000000000001,"5min":1.3740000000000001,"15min":1.4390000000000001},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834575708,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3341,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3421351,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.65500000000000003,"15min":0.64200000000000002},"min":{"1min":0.439,"5min":0.439,"15min":0.432},"max":{"1min":1.4670000000000001,"5min":1.4670000000000001,"15min":1.4670000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.76500000000000001,"15min":0.76200000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.877,"5min":1.0589999999999999,"15min":1.1180000000000001},"last":0.73399999999999999}]},{"osd":1,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.66100000000000003,"15min":0.66200000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.8169999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.74099999999999999,"15min":0.75800000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.5369999999999999},"last":0.96299999999999997}]},{"osd":2,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69299999999999995,"15min":0.68500000000000005},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1,"5min":1.248,"15min":1.248},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.79500000000000004,"15min":0.78500000000000003},"min":{"1min":0.56599999999999995,"5min":0.55600000000000005,"15min":0.43099999999999999},"max":{"1min":1.034,"5min":1.0700000000000001,"15min":1.5720000000000001},"last":1.0469999999999999}]},{"osd":4,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.92200000000000004,"15min":0.92100000000000004},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.64200000000000002},"max":{"1min":1.1379999999999999,"5min":1.236,"15min":1.405},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.93200000000000005,"15min":0.93300000000000005},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.2989999999999999,"5min":1.3360000000000001,"15min":1.708},"last":0.879}]},{"osd":5,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.94099999999999995,"15min":0.93799999999999994},"min":{"1min":0.78400000000000003,"5min":0.70099999999999996,"15min":0.65000000000000002},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.4319999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.998,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.68700000000000006,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":1.151}]},{"osd":6,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":1.022,"15min":1.004},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.274,"5min":1.966,"15min":1.966},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.0740000000000001,"15min":1.0660000000000001},"min":{"1min":0.873,"5min":0.79700000000000004,"15min":0.74299999999999999},"max":{"1min":1.411,"5min":1.4319999999999999,"15min":1.6519999999999999},"last":1.204}]},{"osd":7,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.78600000000000003,"5min":0.78400000000000003,"15min":0.73099999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.468},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.091,"15min":1.081},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.222,"5min":1.5920000000000001,"15min":1.706},"last":1.0900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575708,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2803,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2870758,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.621,"15min":0.62},"min":{"1min":0.496,"5min":0.443,"15min":0.438},"max":{"1min":1.153,"5min":1.153,"15min":1.155},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.56399999999999995,"5min":0.48399999999999999,"15min":0.47999999999999998},"max":{"1min":1.0189999999999999,"5min":1.026,"15min":1.1339999999999999},"last":0.73899999999999999}]},{"osd":1,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.47399999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81100000000000005,"5min":1.1379999999999999,"15min":1.2529999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76600000000000001,"15min":0.76000000000000001},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53000000000000003},"max":{"1min":0.92000000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.73199999999999998,"15min":0.74199999999999999},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.48499999999999999},"max":{"1min":1.042,"5min":1.069,"15min":1.1839999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.83599999999999997,"15min":0.83299999999999996},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":1.121}]},{"osd":4,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.94199999999999995,"15min":0.95799999999999996},"min":{"1min":0.74199999999999999,"5min":0.70199999999999996,"15min":0.69299999999999995},"max":{"1min":1.1599999999999999,"5min":1.256,"15min":1.3680000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.93600000000000005,"15min":0.93500000000000005},"min":{"1min":0.76000000000000001,"5min":0.69399999999999995,"15min":0.67200000000000004},"max":{"1min":1.135,"5min":1.262,"15min":1.3109999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.98599999999999999,"15min":0.996},"min":{"1min":0.68300000000000005,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.341},"last":1.0109999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.95999999999999996,"15min":0.98099999999999998},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.66100000000000003},"max":{"1min":1.2310000000000001,"5min":1.3360000000000001,"15min":1.476},"last":0.77800000000000002}]},{"osd":6,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.98499999999999999,"15min":0.98299999999999998},"min":{"1min":0.80600000000000005,"5min":0.74299999999999999,"15min":0.67600000000000005},"max":{"1min":1.204,"5min":1.425,"15min":1.425},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.135,"5min":1.0700000000000001,"15min":1.0640000000000001},"min":{"1min":0.84499999999999997,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.198}]},{"osd":7,"last update":"Tue Jul 12 09:52:10 2022","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0489999999999999,"15min":1.0720000000000001},"min":{"1min":0.83699999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.2829999999999999,"5min":1.591,"15min":1.591},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0740000000000001,"15min":1.0720000000000001},"min":{"1min":0.88700000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.27,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":1.4390000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575708,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3971,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4066332,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.63800000000000001,"15min":0.63500000000000001},"min":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.46899999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.216},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.70199999999999996,"15min":0.70399999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.98899999999999999,"5min":1.1779999999999999,"15min":1.226},"last":0.59799999999999998}]},{"osd":2,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.52700000000000002,"5min":0.46999999999999997,"15min":0.44},"max":{"1min":0.94999999999999996,"5min":1.143,"15min":1.1879999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77300000000000002,"15min":0.76800000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.52200000000000002},"max":{"1min":1.056,"5min":1.149,"15min":1.3620000000000001},"last":0.754}]},{"osd":3,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.81999999999999995,"15min":0.83999999999999997},"min":{"1min":0.65400000000000003,"5min":0.61499999999999999,"15min":0.56599999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.8779999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76400000000000001,"15min":0.78000000000000003},"min":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.80500000000000005}]},{"osd":4,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.89600000000000002,"15min":0.90000000000000002},"min":{"1min":0.752,"5min":0.64100000000000001,"15min":0.63700000000000001},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.016,"15min":1.016},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.68999999999999995},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.99399999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.008,"15min":0.99299999999999999},"min":{"1min":0.76600000000000001,"5min":0.73399999999999999,"15min":0.70799999999999996},"max":{"1min":1.5800000000000001,"5min":1.5800000000000001,"15min":1.5800000000000001},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.995,"15min":0.99199999999999999},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.63300000000000001},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":1.105}]},{"osd":6,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.004,"15min":1.0029999999999999},"min":{"1min":0.79900000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.0269999999999999,"15min":1.0169999999999999},"min":{"1min":0.83199999999999996,"5min":0.76900000000000002,"15min":0.68600000000000005},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.161}]},{"osd":7,"last update":"Tue Jul 12 09:52:19 2022","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.107,"15min":1.103},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.73099999999999998},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.0860000000000001,"15min":1.089},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.72399999999999998},"max":{"1min":1.6319999999999999,"5min":1.6319999999999999,"15min":1.6319999999999999},"last":1.214}]}]},{"osd":0,"up_from":13,"seq":55834575708,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":3794,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3886016,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46800000000000003,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88300000000000001,"5min":0.89200000000000002,"15min":0.91500000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.52100000000000002,"5min":0.497,"15min":0.497},"max":{"1min":0.82099999999999995,"5min":1.095,"15min":1.1819999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74299999999999999,"15min":0.72199999999999998},"min":{"1min":0.56000000000000005,"5min":0.53800000000000003,"15min":0.51900000000000002},"max":{"1min":0.96899999999999997,"5min":1.2210000000000001,"15min":1.2270000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.64200000000000002,"15min":0.66700000000000004},"min":{"1min":0.502,"5min":0.48399999999999999,"15min":0.42699999999999999},"max":{"1min":0.76000000000000001,"5min":1.129,"15min":1.224},"last":0.58899999999999997}]},{"osd":3,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.75,"15min":0.76300000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.131,"5min":1.131,"15min":1.1799999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.88300000000000001,"15min":0.88100000000000001},"min":{"1min":0.75800000000000001,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.141,"5min":1.2130000000000001,"15min":1.246},"last":0.84399999999999997}]},{"osd":4,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.84599999999999997,"15min":0.86399999999999999},"min":{"1min":0.66600000000000004,"5min":0.627,"15min":0.627},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.391},"last":0.747},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.94499999999999995,"15min":0.94399999999999995},"min":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.4219999999999999,"5min":1.4330000000000001,"15min":1.4330000000000001},"last":0.85999999999999999}]},{"osd":5,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.94799999999999995,"15min":0.96299999999999997},"min":{"1min":0.70299999999999996,"5min":0.67200000000000004,"15min":0.66500000000000004},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.81299999999999994,"5min":0.70299999999999996,"15min":0.69999999999999996},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.498},"last":0.91200000000000003}]},{"osd":6,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.014,"15min":1.024},"min":{"1min":0.78600000000000003,"5min":0.77400000000000002,"15min":0.71099999999999997},"max":{"1min":1.2370000000000001,"5min":1.3049999999999999,"15min":1.522},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.96399999999999997,"15min":0.99199999999999999},"min":{"1min":0.80400000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.1639999999999999,"5min":1.24,"15min":1.4650000000000001},"last":0.81499999999999995}]},{"osd":7,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.01,"15min":1.0329999999999999},"min":{"1min":0.753,"5min":0.74399999999999999,"15min":0.68799999999999994},"max":{"1min":1.242,"5min":1.3029999999999999,"15min":1.5489999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.0840000000000001,"15min":1.089},"min":{"1min":0.93400000000000005,"5min":0.79300000000000004,"15min":0.77200000000000002},"max":{"1min":1.546,"5min":1.546,"15min":1.569},"last":1.056}]}]},{"osd":5,"up_from":13,"seq":55834575708,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798156,"kb_used_data":92901224,"kb_used_omap":2742,"kb_used_meta":0,"kb_avail":92901224,"statfs":{"total":95948165120,"available":95130853376,"internally_reserved":0,"allocated":95130853376,"data_stored":95130853376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2807886,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.98499999999999999,"5min":1.194,"15min":1.224},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90500000000000003,"15min":0.91000000000000003},"min":{"1min":0.77000000000000002,"5min":0.65100000000000002,"15min":0.59999999999999998},"max":{"1min":1.054,"5min":1.161,"15min":1.161},"last":0.91800000000000004}]},{"osd":1,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81299999999999994,"15min":0.81599999999999995},"min":{"1min":0.57799999999999996,"5min":0.55400000000000005,"15min":0.5},"max":{"1min":1.042,"5min":1.198,"15min":1.22},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.79600000000000004,"15min":0.81000000000000005},"min":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.51600000000000001},"max":{"1min":1.0449999999999999,"5min":1.109,"15min":1.109},"last":0.88800000000000001}]},{"osd":2,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.85499999999999998,"15min":0.85899999999999999},"min":{"1min":0.63,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.075,"5min":1.127,"15min":1.1819999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.82599999999999996},"min":{"1min":0.61799999999999999,"5min":0.61599999999999999,"15min":0.55800000000000005},"max":{"1min":0.94399999999999995,"5min":1.1759999999999999,"15min":1.1859999999999999},"last":0.76700000000000002}]},{"osd":3,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80800000000000005,"15min":0.82499999999999996},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.57799999999999996},"max":{"1min":1.008,"5min":1.151,"15min":1.3360000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86499999999999999,"15min":0.879},"min":{"1min":0.68799999999999994,"5min":0.61199999999999999,"15min":0.54100000000000004},"max":{"1min":1.036,"5min":1.1639999999999999,"15min":1.1919999999999999},"last":0.94799999999999995}]},{"osd":4,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.69999999999999996,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.39000000000000001},"max":{"1min":1.0329999999999999,"5min":1.137,"15min":1.1719999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.92100000000000004,"15min":0.93799999999999994},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.58599999999999997},"max":{"1min":1.0720000000000001,"5min":1.212,"15min":1.4930000000000001},"last":1.01}]},{"osd":6,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86399999999999999,"15min":0.86899999999999999},"min":{"1min":0.64300000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.03,"5min":1.423,"15min":1.423},"last":1.095},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.88600000000000001,"15min":0.90700000000000003},"min":{"1min":0.60099999999999998,"5min":0.57599999999999996,"15min":0.48199999999999998},"max":{"1min":1.024,"5min":1.232,"15min":1.276},"last":0.96499999999999997}]},{"osd":7,"last update":"Tue Jul 12 09:52:21 2022","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.86799999999999999,"15min":0.89300000000000002},"min":{"1min":0.59199999999999997,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.042,"5min":1.2809999999999999,"15min":2.1200000000000001},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.998,"15min":1.0129999999999999},"min":{"1min":0.65400000000000003,"5min":0.623,"15min":0.61399999999999999},"max":{"1min":1.147,"5min":1.482,"15min":2.1019999999999999},"last":1.2490000000000001}]}]}]}} 2022-07-12T09:52:37.748 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2022-07-12T09:52:37.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1e2b9700 1 Processor -- start 2022-07-12T09:52:37.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1e2b9700 1 -- start start 2022-07-12T09:52:37.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1e2b9700 1 --2- >> v2:172.21.15.111:3300/0 conn(0x7f1a180b2b20 0x7f1a18060da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1e2b9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a180624d0 con 0x7f1a180620f0 2022-07-12T09:52:37.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1e2b9700 1 -- --> v1:172.21.15.111:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a180b31c0 con 0x7f1a180b1a70 2022-07-12T09:52:37.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1e2b9700 1 -- --> v2:172.21.15.111:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a180b3460 con 0x7f1a180b2b20 2022-07-12T09:52:37.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1d2b7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1a180620f0 0x7f1a181464d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:51354/0 (socket says 172.21.15.61:51354) 2022-07-12T09:52:37.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1d2b7700 1 -- 172.21.15.61:0/3850764392 learned_addr learned my addr 172.21.15.61:0/3850764392 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-12T09:52:37.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1dab8700 1 --2- 172.21.15.61:0/3850764392 >> v2:172.21.15.111:3300/0 conn(0x7f1a180b2b20 0x7f1a18060da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a0ffff700 1 -- 172.21.15.61:0/3850764392 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751815969 0 0) 0x7f1a180624d0 con 0x7f1a180620f0 2022-07-12T09:52:37.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a0ffff700 1 -- 172.21.15.61:0/3850764392 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19fc003260 con 0x7f1a180620f0 2022-07-12T09:52:37.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1dab8700 1 -- 172.21.15.61:0/3850764392 >> v1:172.21.15.111:6789/0 conn(0x7f1a180b1a70 legacy=0x7f1a18141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1dab8700 1 -- 172.21.15.61:0/3850764392 >> v1:172.21.15.61:6789/0 conn(0x7f1a180620f0 legacy=0x7f1a181464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1dab8700 1 -- 172.21.15.61:0/3850764392 --> v2:172.21.15.111:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180b3700 con 0x7f1a180b2b20 2022-07-12T09:52:37.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.858+0000 7f1a1dab8700 1 --2- 172.21.15.61:0/3850764392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a18060da0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f1a100032f0 tx=0x7f1a100091d0).ready entity=mon.1 client_cookie=e0109afbc41f372b server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:37.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a0ffff700 1 -- 172.21.15.61:0/3850764392 <== mon.1 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a1000b6c0 con 0x7f1a180b2b20 2022-07-12T09:52:37.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a0ffff700 1 -- 172.21.15.61:0/3850764392 <== mon.1 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1000b820 con 0x7f1a180b2b20 2022-07-12T09:52:37.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a0ffff700 1 -- 172.21.15.61:0/3850764392 <== mon.1 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a1000b990 con 0x7f1a180b2b20 2022-07-12T09:52:37.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/3850764392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 msgr2=0x7f1a18060da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/3850764392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a18060da0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7f1a100032f0 tx=0x7f1a100091d0).stop 2022-07-12T09:52:37.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/3850764392 shutdown_connections 2022-07-12T09:52:37.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/3850764392 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a18060da0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/3850764392 >> 172.21.15.61:0/3850764392 conn(0x7f1a180b70d0 msgr2=0x7f1a180b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:52:37.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/3850764392 shutdown_connections 2022-07-12T09:52:37.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/3850764392 wait complete. 2022-07-12T09:52:37.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 Processor -- start 2022-07-12T09:52:37.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- start start 2022-07-12T09:52:37.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 --2- >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180620f0 0x7f1a180af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 0x7f1a1809b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a1809a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180aefc0 con 0x7f1a180b1a70 2022-07-12T09:52:37.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180af260 con 0x7f1a180b2b20 2022-07-12T09:52:37.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1e2b9700 1 -- --> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180af500 con 0x7f1a180620f0 2022-07-12T09:52:37.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1cab6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 0x7f1a1809b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1cab6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 0x7f1a1809b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:34910/0 (socket says 172.21.15.61:34910) 2022-07-12T09:52:37.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1cab6700 1 -- 172.21.15.61:0/2200389227 learned_addr learned my addr 172.21.15.61:0/2200389227 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-12T09:52:37.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.862+0000 7f1a1dab8700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a1809a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1d2b7700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180620f0 0x7f1a180af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1cab6700 1 -- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180620f0 msgr2=0x7f1a180af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1cab6700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180620f0 0x7f1a180af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1cab6700 1 -- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 msgr2=0x7f1a1809a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:37.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1cab6700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a1809a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:37.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1cab6700 1 -- 172.21.15.61:0/2200389227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a10009470 con 0x7f1a180b1a70 2022-07-12T09:52:37.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1dab8700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a1809a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-12T09:52:37.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1cab6700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 0x7f1a1809b220 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7f1a180b8130 tx=0x7f1a14007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:37.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a14007030 con 0x7f1a180b1a70 2022-07-12T09:52:37.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1809b710 con 0x7f1a180b1a70 2022-07-12T09:52:37.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a14007190 con 0x7f1a180b1a70 2022-07-12T09:52:37.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a14006510 con 0x7f1a180b1a70 2022-07-12T09:52:37.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a180aff10 con 0x7f1a180b1a70 2022-07-12T09:52:37.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a18057460 con 0x7f1a180b1a70 2022-07-12T09:52:37.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.866+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1a14010030 con 0x7f1a180b1a70 2022-07-12T09:52:37.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.870+0000 7f1a0e7fc700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a00053ad0 0x7f1a00055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-12T09:52:37.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.870+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1a1406b810 con 0x7f1a180b1a70 2022-07-12T09:52:37.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.870+0000 7f1a1d2b7700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a00053ad0 0x7f1a00055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-12T09:52:37.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.870+0000 7f1a1d2b7700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a00053ad0 0x7f1a00055f90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f1a0800d000 tx=0x7f1a08009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-12T09:52:37.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:37.874+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a140429b0 con 0x7f1a180b1a70 2022-07-12T09:52:38.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.002+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f1a180af6c0 con 0x7f1a180b1a70 2022-07-12T09:52:38.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.010+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1a140429b0 con 0x7f1a180b1a70 2022-07-12T09:52:38.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.010+0000 7f1a0e7fc700 1 -- 172.21.15.61:0/2200389227 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v25)=0 v25) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1a14045fd0 con 0x7f1a180b1a70 2022-07-12T09:52:38.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a00053ad0 msgr2=0x7f1a00055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:38.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a00053ad0 0x7f1a00055f90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7f1a0800d000 tx=0x7f1a08009040).stop 2022-07-12T09:52:38.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 msgr2=0x7f1a1809b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-12T09:52:38.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 0x7f1a1809b220 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 rx=0x7f1a180b8130 tx=0x7f1a14007c10).stop 2022-07-12T09:52:38.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 shutdown_connections 2022-07-12T09:52:38.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:6800/21499,v1:172.21.15.111:6801/21499] conn(0x7f1a00053ad0 0x7f1a00055f90 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:38.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1a180b2b20 0x7f1a1809a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:38.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a180b1a70 0x7f1a1809b220 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:38.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 --2- 172.21.15.61:0/2200389227 >> [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] conn(0x7f1a180620f0 0x7f1a180af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-12T09:52:38.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 >> 172.21.15.61:0/2200389227 conn(0x7f1a180b70d0 msgr2=0x7f1a18065a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-12T09:52:38.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 shutdown_connections 2022-07-12T09:52:38.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-12T09:52:38.014+0000 7f1a1e2b9700 1 -- 172.21.15.61:0/2200389227 wait complete. 2022-07-12T09:52:38.029 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/ceph.py", line 1858, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T09:52:38.033 INFO:teuthology.misc:Shutting down mds daemons... 2022-07-12T09:52:38.034 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-12T09:52:38.035 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.050 INFO:tasks.ceph.mds.a:Stopped 2022-07-12T09:52:38.051 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-12T09:52:38.052 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.067 INFO:tasks.ceph.mds.b:Stopped 2022-07-12T09:52:38.069 INFO:teuthology.misc:Shutting down osd daemons... 2022-07-12T09:52:38.070 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2022-07-12T09:52:38.071 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.092 INFO:tasks.ceph.osd.0:Stopped 2022-07-12T09:52:38.093 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2022-07-12T09:52:38.094 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.155 INFO:tasks.ceph.osd.1:Stopped 2022-07-12T09:52:38.157 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2022-07-12T09:52:38.158 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.174 INFO:tasks.ceph.osd.2:Stopped 2022-07-12T09:52:38.176 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2022-07-12T09:52:38.177 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.196 INFO:tasks.ceph.osd.3:Stopped 2022-07-12T09:52:38.198 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2022-07-12T09:52:38.199 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.218 INFO:tasks.ceph.osd.4:Stopped 2022-07-12T09:52:38.219 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-07-12T09:52:38.219 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.236 INFO:tasks.ceph.osd.5:Stopped 2022-07-12T09:52:38.237 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2022-07-12T09:52:38.237 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.255 INFO:tasks.ceph.osd.6:Stopped 2022-07-12T09:52:38.256 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2022-07-12T09:52:38.257 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.317 INFO:tasks.ceph.osd.7:Stopped 2022-07-12T09:52:38.318 INFO:teuthology.misc:Shutting down mgr daemons... 2022-07-12T09:52:38.319 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2022-07-12T09:52:38.320 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.354 INFO:tasks.ceph.mgr.y:Stopped 2022-07-12T09:52:38.356 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2022-07-12T09:52:38.357 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.382 INFO:tasks.ceph.mgr.x:Stopped 2022-07-12T09:52:38.383 INFO:teuthology.misc:Shutting down mon daemons... 2022-07-12T09:52:38.384 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2022-07-12T09:52:38.384 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.424 INFO:tasks.ceph.mon.a:Stopped 2022-07-12T09:52:38.426 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2022-07-12T09:52:38.427 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.447 INFO:tasks.ceph.mon.b:Stopped 2022-07-12T09:52:38.449 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2022-07-12T09:52:38.450 INFO:teuthology.orchestra.run:waiting for 300 2022-07-12T09:52:38.510 INFO:tasks.ceph.mon.c:Stopped 2022-07-12T09:52:38.511 INFO:tasks.ceph:Checking cluster log for badness... 2022-07-12T09:52:38.512 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'evicting unresponsive client' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'has not responded to cap revoke by MDS for over' | egrep -v MDS_CLIENT_LATE_RELEASE | egrep -v 'responding to mclientcaps' | head -n 1 2022-07-12T09:52:38.537 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T09:52:38.538 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2022-07-12T09:52:38.806 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T09:52:38.807 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2022-07-12T09:52:39.081 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T09:52:39.083 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2022-07-12T09:52:39.287 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T09:52:39.288 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2022-07-12T09:52:39.488 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T09:52:39.490 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2022-07-12T09:52:40.007 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T09:52:40.008 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2022-07-12T09:52:40.245 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T09:52:40.247 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2022-07-12T09:52:40.412 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T09:52:40.413 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2022-07-12T09:52:40.570 INFO:tasks.ceph:Archiving mon data... 2022-07-12T09:52:40.573 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/data/mon.a.tgz 2022-07-12T09:52:40.575 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-12T09:52:40.589 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.owLb59Hrpk 2022-07-12T09:52:40.591 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.owLb59Hrpk 2022-07-12T09:52:40.813 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.owLb59Hrpk 2022-07-12T09:52:40.842 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.owLb59Hrpk is 495KB 2022-07-12T09:52:40.920 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.owLb59Hrpk 2022-07-12T09:52:40.937 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/data/mon.b.tgz 2022-07-12T09:52:40.938 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T09:52:40.947 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.AjvN9LtMPN 2022-07-12T09:52:40.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.AjvN9LtMPN 2022-07-12T09:52:41.224 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.AjvN9LtMPN 2022-07-12T09:52:41.251 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.AjvN9LtMPN is 654KB 2022-07-12T09:52:41.336 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.AjvN9LtMPN 2022-07-12T09:52:41.345 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/data/mon.c.tgz 2022-07-12T09:52:41.346 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2022-07-12T09:52:41.398 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.7yI9t4Hqoa 2022-07-12T09:52:41.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.7yI9t4Hqoa 2022-07-12T09:52:41.637 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.7yI9t4Hqoa 2022-07-12T09:52:41.704 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.7yI9t4Hqoa is 654KB 2022-07-12T09:52:41.786 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.7yI9t4Hqoa 2022-07-12T09:52:41.797 INFO:tasks.ceph:Cleaning ceph cluster... 2022-07-12T09:52:41.798 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-12T09:52:41.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-12T09:52:41.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-12T09:52:41.811 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-12T09:52:41.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-12T09:52:41.838 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-07-12T09:52:41.839 DEBUG:teuthology.orchestra.run.smithi061:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-12T09:52:41.843 DEBUG:teuthology.orchestra.run.smithi111:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-12T09:52:41.865 DEBUG:teuthology.orchestra.run.smithi121:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-12T09:52:41.872 DEBUG:teuthology.orchestra.run.smithi130:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-12T09:52:41.875 INFO:teuthology.orchestra.run.smithi061.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-12T09:52:41.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-12T09:52:41.902 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-12T09:52:41.916 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-12T09:52:41.982 INFO:teuthology.orchestra.run.smithi130.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-12T09:52:42.073 INFO:teuthology.orchestra.run.smithi121.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-12T09:52:42.080 INFO:tasks.ceph:Archiving crash dumps... 2022-07-12T09:52:42.084 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi061/crash 2022-07-12T09:52:42.085 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-12T09:52:42.107 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi111/crash 2022-07-12T09:52:42.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-12T09:52:42.130 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi121/crash 2022-07-12T09:52:42.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-12T09:52:42.158 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi130/crash 2022-07-12T09:52:42.159 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-12T09:52:42.181 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi137/crash 2022-07-12T09:52:42.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-12T09:52:42.203 INFO:tasks.ceph:Shutting down logrotate 2022-07-12T09:52:42.204 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.208 DEBUG:teuthology.orchestra.run.smithi111:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.213 DEBUG:teuthology.orchestra.run.smithi130:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.319 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-12T09:52:42.353 INFO:tasks.ceph:Compressing logs... 2022-07-12T09:52:42.354 DEBUG:teuthology.orchestra.run.smithi061:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:42.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:42.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:42.367 DEBUG:teuthology.orchestra.run.smithi130:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:42.371 DEBUG:teuthology.orchestra.run.smithi137:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:54.168 INFO:tasks.ceph:Archiving logs... 2022-07-12T09:52:54.170 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi061/log 2022-07-12T09:52:54.172 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-12T09:52:57.468 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi111/log 2022-07-12T09:52:57.470 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-12T09:52:58.895 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi121/log 2022-07-12T09:52:58.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-12T09:52:59.337 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi130/log 2022-07-12T09:52:59.340 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-12T09:52:59.389 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi137/log 2022-07-12T09:52:59.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-12T09:52:59.420 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-07-12T09:52:59.439 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph_297f944c54c8d8b03d4faee671f8bb06163c8262/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-12T09:52:59.447 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-07-12T09:52:59.451 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-12T09:52:59.456 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-12T09:52:59.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-12T09:52:59.463 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-12T09:52:59.468 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-12T09:52:59.646 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-07-12T09:52:59.664 INFO:teuthology.task.clock:Checking final clock skew... 2022-07-12T09:52:59.667 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T09:52:59.671 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T09:52:59.676 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T09:52:59.681 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T09:52:59.686 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-12T09:52:59.692 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T09:52:59.693 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2022-07-12T09:52:59.693 INFO:teuthology.orchestra.run.smithi061.stdout:+hv01.front.sepi 216.240.36.24 3 u 144 256 377 0.162 2.618 0.296 2022-07-12T09:52:59.694 INFO:teuthology.orchestra.run.smithi061.stdout:*hv02.front.sepi 163.237.218.18 2 u 159 256 377 0.169 -2.717 0.221 2022-07-12T09:52:59.694 INFO:teuthology.orchestra.run.smithi061.stdout:+hv03.front.sepi 216.229.4.69 3 u 72 256 377 0.166 -1.246 0.449 2022-07-12T09:52:59.694 INFO:teuthology.orchestra.run.smithi061.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-12T09:52:59.697 INFO:teuthology.orchestra.run.smithi111.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T09:52:59.697 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================== 2022-07-12T09:52:59.697 INFO:teuthology.orchestra.run.smithi111.stdout:+hv01.front.sepi 216.240.36.24 3 u 7 128 377 0.112 4.787 0.081 2022-07-12T09:52:59.698 INFO:teuthology.orchestra.run.smithi111.stdout:*hv02.front.sepi 163.237.218.18 2 u 30 128 377 0.191 -0.515 0.043 2022-07-12T09:52:59.698 INFO:teuthology.orchestra.run.smithi111.stdout:+hv03.front.sepi 216.229.4.69 3 u 21 128 377 0.174 1.711 0.592 2022-07-12T09:52:59.699 INFO:teuthology.orchestra.run.smithi111.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-12T09:52:59.701 INFO:teuthology.orchestra.run.smithi121.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T09:52:59.702 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================== 2022-07-12T09:52:59.702 INFO:teuthology.orchestra.run.smithi121.stdout:+hv01.front.sepi 216.240.36.24 3 u 80 256 377 0.142 6.277 1.010 2022-07-12T09:52:59.703 INFO:teuthology.orchestra.run.smithi121.stdout:*hv02.front.sepi 163.237.218.18 2 u 220 256 377 0.140 0.911 1.205 2022-07-12T09:52:59.703 INFO:teuthology.orchestra.run.smithi121.stdout:+hv03.front.sepi 216.229.4.69 3 u 213 256 377 0.159 1.889 1.078 2022-07-12T09:52:59.704 INFO:teuthology.orchestra.run.smithi121.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-12T09:52:59.709 INFO:teuthology.orchestra.run.smithi130.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T09:52:59.713 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================== 2022-07-12T09:52:59.714 INFO:teuthology.orchestra.run.smithi130.stdout:+hv01.front.sepi 216.240.36.24 3 u 126 128 377 0.141 1.336 0.359 2022-07-12T09:52:59.715 INFO:teuthology.orchestra.run.smithi130.stdout:*hv02.front.sepi 163.237.218.18 2 u 17 128 377 0.152 -4.219 0.521 2022-07-12T09:52:59.715 INFO:teuthology.orchestra.run.smithi130.stdout:+hv03.front.sepi 216.229.4.69 3 u 79 128 377 0.178 -2.272 0.426 2022-07-12T09:52:59.716 INFO:teuthology.orchestra.run.smithi130.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-12T09:52:59.717 INFO:teuthology.orchestra.run.smithi137.stdout: remote refid st t when poll reach delay offset jitter 2022-07-12T09:52:59.718 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================== 2022-07-12T09:52:59.718 INFO:teuthology.orchestra.run.smithi137.stdout:+hv01.front.sepi 216.240.36.24 3 u 14 128 377 0.142 1.800 0.211 2022-07-12T09:52:59.718 INFO:teuthology.orchestra.run.smithi137.stdout:*hv02.front.sepi 163.237.218.18 2 u 85 128 377 0.171 -3.308 0.132 2022-07-12T09:52:59.719 INFO:teuthology.orchestra.run.smithi137.stdout:+hv03.front.sepi 216.229.4.69 3 u 25 128 377 0.213 -1.671 0.338 2022-07-12T09:52:59.719 INFO:teuthology.orchestra.run.smithi137.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-12T09:52:59.720 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-07-12T09:52:59.730 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-07-12T09:52:59.731 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-07-12T09:52:59.769 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-07-12T09:52:59.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-07-12T09:52:59.795 INFO:teuthology.task.internal:Duration was 4928.206667 seconds 2022-07-12T09:52:59.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-07-12T09:52:59.811 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-07-12T09:52:59.812 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-12T09:52:59.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-12T09:52:59.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-12T09:52:59.823 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-12T09:52:59.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-12T09:52:59.877 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-07-12T09:52:59.878 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2022-07-12T09:52:59.880 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-12T09:52:59.898 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2022-07-12T09:52:59.899 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-12T09:52:59.916 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2022-07-12T09:52:59.917 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-12T09:52:59.939 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2022-07-12T09:52:59.941 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-12T09:52:59.959 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2022-07-12T09:52:59.960 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-12T09:52:59.979 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-07-12T09:52:59.981 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:59.985 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:59.993 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:52:59.997 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:53:00.000 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-12T09:53:00.052 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-07-12T09:53:00.053 DEBUG:teuthology.orchestra.run.smithi061:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-12T09:53:00.068 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-12T09:53:00.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-12T09:53:00.077 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-12T09:53:00.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-12T09:53:00.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-07-12T09:53:00.262 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-07-12T09:53:00.263 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-12T09:53:00.282 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-12T09:53:00.298 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-12T09:53:00.317 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-12T09:53:00.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-12T09:53:00.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-07-12T09:53:00.373 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.378 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.384 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.392 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2022-07-12T09:53:00.394 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2022-07-12T09:53:00.397 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2022-07-12T09:53:00.401 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2022-07-12T09:53:00.415 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2022-07-12T09:53:00.488 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:53:00.542 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:53:00.552 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:53:00.562 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:53:00.571 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-12T09:53:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-12T09:53:00.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-07-12T09:53:00.593 INFO:teuthology.task.internal:Transferring archived files... 2022-07-12T09:53:00.596 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi061 2022-07-12T09:53:00.597 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-12T09:53:00.642 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi111 2022-07-12T09:53:00.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-12T09:53:00.697 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi121 2022-07-12T09:53:00.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-12T09:53:00.736 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi130 2022-07-12T09:53:00.738 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-12T09:53:00.784 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-11_23:41:40-fs-octopus-release-distro-default-smithi/6925138/remote/smithi137 2022-07-12T09:53:00.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-12T09:53:00.833 INFO:teuthology.task.internal:Removing archive directory... 2022-07-12T09:53:00.835 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-12T09:53:00.839 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-12T09:53:00.845 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-12T09:53:00.850 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-12T09:53:00.855 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-12T09:53:00.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-07-12T09:53:00.899 INFO:teuthology.task.internal:Not uploading archives. 2022-07-12T09:53:00.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-07-12T09:53:00.912 INFO:teuthology.task.internal:Tidying up after the test... 2022-07-12T09:53:00.913 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-12T09:53:00.916 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-12T09:53:00.919 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-12T09:53:00.922 INFO:teuthology.orchestra.run.smithi061.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 12 09:53 /home/ubuntu/cephtest 2022-07-12T09:53:00.924 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-12T09:53:00.929 INFO:teuthology.orchestra.run.smithi111.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 12 09:53 /home/ubuntu/cephtest 2022-07-12T09:53:00.933 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-12T09:53:00.936 INFO:teuthology.orchestra.run.smithi130.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 12 09:53 /home/ubuntu/cephtest 2022-07-12T09:53:00.946 INFO:teuthology.orchestra.run.smithi137.stdout: 524315 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 12 09:53 /home/ubuntu/cephtest 2022-07-12T20:28:24.912 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-12T20:28:24.983 DEBUG:teuthology.task.console_log:Killing console logger for smithi061 2022-07-12T20:28:24.985 DEBUG:teuthology.task.console_log:Killing console logger for smithi111 2022-07-12T20:28:24.987 DEBUG:teuthology.task.console_log:Killing console logger for smithi121 2022-07-12T20:28:24.989 DEBUG:teuthology.task.console_log:Killing console logger for smithi130 2022-07-12T20:28:24.990 DEBUG:teuthology.task.console_log:Killing console logger for smithi137 2022-07-12T20:28:24.990 DEBUG:teuthology.exit:Finished running handlers